Geometry-Aware PBR Material Distillation

Takes an existing bare mesh and generates physically-based material maps (albedo, roughness, metallic) matched to its geometry, using a diffusion model conditioned on the mesh's own geometry and a chosen lighting environment, then distilling the shaded outputs into decomposed BRDF parameters via inverse rendering.

Properties

family: 3d
modality: untextured 3D mesh + text -> lit multi-view renders -> decomposed PBR material maps
whyMultiModel: An image-to-3D model only produces baked-in RGB texture or bare geometry; it cannot separate real material properties from shading. DreamMat needs a geometry- and light-aware diffusion model to render plausible shaded views under controlled illumination, plus a separate inverse-rendering distillation to decompose those views into albedo/roughness/metallic maps free of baked-in lighting.
steps: [object Object], [object Object], [object Object]
controls: environment light choice/rotation during conditioning, number of viewpoints rendered, per-material text prompt, distillation iteration count, mesh UV parameterization
exampleStack: Hunyuan3D 2.1 (bare mesh) -> DreamMat diffusion -> DreamMat PBR distillation -> Blender/Unreal import
useCases: [Relightable game/film asset texturing][Converting scanned or generated bare meshes into engine-ready PBR assets][Batch material re-skinning of an existing mesh library]
pitfalls: Without light-aware conditioning, shading bakes into albedo and looks wrong under new lighting; multi-view inconsistency causes seams at UV boundaries; distillation is compute-heavy per asset (minutes, not seconds).
source: DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models (SIGGRAPH 2024, arXiv:2405.17176)

Provenance

Source: https://arxiv.org/abs/2405.17176

Geometry-Aware PBR Material Distillation

Properties

family

modality

untextured 3D mesh + text -> lit multi-view renders -> decomposed PBR material maps

whyMultiModel

An image-to-3D model only produces baked-in RGB texture or bare geometry; it cannot separate real material properties from shading. DreamMat needs a geometry- and light-aware diffusion model to render plausible shaded views under controlled illumination, plus a separate inverse-rendering distillation to decompose those views into albedo/roughness/metallic maps free of baked-in lighting.

steps

[object Object], [object Object], [object Object]

controls

environment light choice/rotation during conditioning, number of viewpoints rendered, per-material text prompt, distillation iteration count, mesh UV parameterization

exampleStack

Hunyuan3D 2.1 (bare mesh) -> DreamMat diffusion -> DreamMat PBR distillation -> Blender/Unreal import

useCases

[Relightable game/film asset texturing][Converting scanned or generated bare meshes into engine-ready PBR assets][Batch material re-skinning of an existing mesh library]

pitfalls

Without light-aware conditioning, shading bakes into albedo and looks wrong under new lighting; multi-view inconsistency causes seams at UV boundaries; distillation is compute-heavy per asset (minutes, not seconds).

source

DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models (SIGGRAPH 2024, arXiv:2405.17176)