Text-to-Image-Depth Room Mesh Fusion

Generates a room-scale 3D environment purely from a text prompt by iteratively rendering images at chosen camera poses, lifting each into 3D with monocular depth, and fusing/inpainting the growing mesh so each new view integrates seamlessly with previously generated geometry.

Properties

family: 3d
modality: text -> per-view 2D renders -> per-view depth -> fused, inpainted textured room mesh
whyMultiModel: A text-to-image model produces flat 2D pixels with no camera-consistent geometry; a monocular depth model lifts a single image to a partial point cloud but cannot generate novel content or fill disocclusions; a text-conditioned inpainting model fills the gaps exposed as the camera moves, and a mesh-fusion algorithm stitches each frame into one consistent mesh.
steps: [object Object], [object Object], [object Object] +1 more
controls: text prompt per region/object, camera viewpoint trajectory/selection strategy, depth alignment tolerance, inpainting mask region, mesh simplification thresholds
exampleStack: Stable Diffusion -> monocular depth -> Stable Diffusion inpainting -> Text2Room fusion -> textured room mesh (OBJ)
useCases: [Rapid environment blockouts for game levels from text briefs][Virtual production background generation][Synthetic training environments for embodied AI][Architectural concept walkthroughs]
pitfalls: Depth misalignment across frames causes seams, ghosting, or duplicated geometry; viewpoint selection can miss occluded regions leaving holes; style can drift between frames; scales to room interiors but not large open scenes.
source: Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (ICCV 2023, arXiv:2303.11989)

Provenance

Source: https://arxiv.org/abs/2303.11989

Properties

family

modality

text -> per-view 2D renders -> per-view depth -> fused, inpainted textured room mesh

whyMultiModel

A text-to-image model produces flat 2D pixels with no camera-consistent geometry; a monocular depth model lifts a single image to a partial point cloud but cannot generate novel content or fill disocclusions; a text-conditioned inpainting model fills the gaps exposed as the camera moves, and a mesh-fusion algorithm stitches each frame into one consistent mesh.

steps

[object Object], [object Object], [object Object] +1 more

controls

text prompt per region/object, camera viewpoint trajectory/selection strategy, depth alignment tolerance, inpainting mask region, mesh simplification thresholds

exampleStack

Stable Diffusion -> monocular depth -> Stable Diffusion inpainting -> Text2Room fusion -> textured room mesh (OBJ)

useCases

[Rapid environment blockouts for game levels from text briefs][Virtual production background generation][Synthetic training environments for embodied AI][Architectural concept walkthroughs]

pitfalls

Depth misalignment across frames causes seams, ghosting, or duplicated geometry; viewpoint selection can miss occluded regions leaving holes; style can drift between frames; scales to room interiors but not large open scenes.

source

Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (ICCV 2023, arXiv:2303.11989)