WorldMesh Navigable Multi-Room Scene Generation

A text description first drives construction of an explicit mesh scaffold (walls, floors) for an arbitrarily large, multi-room environment; that scaffold is rendered to condition a separate image-synthesis model per room, a promptable segmentation model masks individual objects room by room, a single-image-to-3D model reconstructs each masked object, and a depth model validates geometry before the scene is exported as a navigable, geometry-regularized 3D Gaussian Splat.

Properties

family: 3d
modality: text -> mesh scaffold -> mesh-conditioned room renders -> segmented objects -> reconstructed 3D scene
whyMultiModel: No single diffusion model can hold arbitrarily large, multi-room 3D consistency: WorldMesh decouples structural geometry (an explicit mesh scaffold) from appearance (a mesh-conditioned image model), then depends on a distinct promptable segmentation model to identify individual objects, a separate single-image 3D reconstruction model to lift each object into 3D, and a monocular metric-depth model to validate the resulting geometry, before a Gaussian-splat training stage produces the final navigable scene. Each stage solves a problem the others cannot: the scaffold has no appearance, the image model has no persistent 3D structure or object boundaries, the segmentation model has no 3D capability, and the reconstruction model needs an already-isolated object mask. This is distinct from the tracked text-to-image-depth-inpaint-room-mesh pattern (Text2Room: iterative per-view depth-lift + SD inpainting fused into one mesh, single room, no explicit floor-plan scaffold or per-object 3D reconstruction) and from segment-then-reconstruct-scene-kitbash (SAM 3 + SAM 3D Objects applied to a single existing cluttered photo, not to a text-generated, mesh-scaffolded, navigable multi-room environment).
steps: [object Object], [object Object], [object Object] +3 more
controls: Text prompt describing the floor plan and rooms; manual per-room object-mask clicks via the Gradio UI (Phase 1); choice of image model (cloud Nano Banana Pro vs. local FLUX.2-klein-9b distilled vs. undistilled base, the base being about 5x slower but higher quality); SAM 3 text/interactive segmentation prompts; depth-validation tolerance for geometry regularization.
exampleStack: Text prompt -> WorldMesh mesh scaffold (floor plan geometry) -> Nano Banana Pro or FLUX.2-klein-9b (mesh-conditioned room renders) -> SAM 3 (per-room object masks via Gradio UI) -> SAM 3D Objects (per-object 3D reconstruction) -> Depth Pro (depth validation) -> Nerfstudio Splatfacto/COLMAP (geometry-regularized 3DGS export).
useCases: [Navigable multi-room virtual-production or game-level generation from a text brief][Embodied-AI training environments needing consistent, arbitrarily-sized multi-room geometry][Architectural walkthrough prototyping generated purely from a text description][VR/AR scene prototyping with per-object 3D editability]
pitfalls: The published pipeline is two-phase and requires a manual masking step (a Gradio UI where the user clicks objects room by room) rather than being a single automated pass; the undistilled local FLUX.2-klein-9b-base variant is about 5x slower than the distilled variant for higher quality; reconstruction quality is bounded by SAM 3 segmentation accuracy in cluttered or occluded rooms, and errors in the mesh scaffold or depth validation propagate into the final Gaussian-splat geometry.
source: WorldMesh Navigable Multi-Room Scene Generation

Provenance

Source: https://github.com/mschneider456/worldmesh

WorldMesh Navigable Multi-Room Scene Generation

Properties

family

modality

text -> mesh scaffold -> mesh-conditioned room renders -> segmented objects -> reconstructed 3D scene

whyMultiModel

No single diffusion model can hold arbitrarily large, multi-room 3D consistency: WorldMesh decouples structural geometry (an explicit mesh scaffold) from appearance (a mesh-conditioned image model), then depends on a distinct promptable segmentation model to identify individual objects, a separate single-image 3D reconstruction model to lift each object into 3D, and a monocular metric-depth model to validate the resulting geometry, before a Gaussian-splat training stage produces the final navigable scene. Each stage solves a problem the others cannot: the scaffold has no appearance, the image model has no persistent 3D structure or object boundaries, the segmentation model has no 3D capability, and the reconstruction model needs an already-isolated object mask. This is distinct from the tracked text-to-image-depth-inpaint-room-mesh pattern (Text2Room: iterative per-view depth-lift + SD inpainting fused into one mesh, single room, no explicit floor-plan scaffold or per-object 3D reconstruction) and from segment-then-reconstruct-scene-kitbash (SAM 3 + SAM 3D Objects applied to a single existing cluttered photo, not to a text-generated, mesh-scaffolded, navigable multi-room environment).

steps

[object Object], [object Object], [object Object] +3 more

controls

Text prompt describing the floor plan and rooms; manual per-room object-mask clicks via the Gradio UI (Phase 1); choice of image model (cloud Nano Banana Pro vs. local FLUX.2-klein-9b distilled vs. undistilled base, the base being about 5x slower but higher quality); SAM 3 text/interactive segmentation prompts; depth-validation tolerance for geometry regularization.

exampleStack

Text prompt -> WorldMesh mesh scaffold (floor plan geometry) -> Nano Banana Pro or FLUX.2-klein-9b (mesh-conditioned room renders) -> SAM 3 (per-room object masks via Gradio UI) -> SAM 3D Objects (per-object 3D reconstruction) -> Depth Pro (depth validation) -> Nerfstudio Splatfacto/COLMAP (geometry-regularized 3DGS export).

useCases

[Navigable multi-room virtual-production or game-level generation from a text brief][Embodied-AI training environments needing consistent, arbitrarily-sized multi-room geometry][Architectural walkthrough prototyping generated purely from a text description][VR/AR scene prototyping with per-object 3D editability]

pitfalls

The published pipeline is two-phase and requires a manual masking step (a Gradio UI where the user clicks objects room by room) rather than being a single automated pass; the undistilled local FLUX.2-klein-9b-base variant is about 5x slower than the distilled variant for higher quality; reconstruction quality is bounded by SAM 3 segmentation accuracy in cluttered or occluded rooms, and errors in the mesh scaffold or depth validation propagate into the final Gaussian-splat geometry.

source

WorldMesh Navigable Multi-Room Scene Generation