ControlNet Structure-Lock Chain

A preprocessor (canny, depth, or pose annotator) extracts a structural control map from a reference image; that map is then injected into a conditioned diffusion model to generate a new image that preserves the source structure.

Properties

family: style-control
modality: image + text -> image
whyMultiModel: Two genuinely distinct model types run in sequence: an annotator/extractor model (e.g. HED edge detector, MiDaS depth, OpenPose keypoint) produces a dense control signal, and a separate ControlNet-augmented diffusion model consumes that signal during generation. Neither step can substitute for the other.
steps: [object Object], [object Object]
controls: Annotator type (canny vs depth vs pose); ControlNet conditioning scale; text prompt; optional second ControlNet for combined conditions (e.g. depth + canny simultaneously).
exampleStack: MiDaS depth -> ControlNet-v1.1-depth + SDXL; or OpenPose -> ControlNet-OpenPose + SD 1.5; or Canny -> FLUX.1 ControlNet (Jasper canny).
useCases: [Pose-controlled character generation][Preserving scene layout across style variations][Sketch / lineart to rendered image]
pitfalls: Over-conditioning (scale too high) makes the output look traced with no creative latitude; mismatched annotator and ControlNet model type produces incoherent outputs; depth annotator struggles with textureless or reflective surfaces.
source: Adding Conditional Control to Text-to-Image Diffusion Models (arXiv:2302.05543)

Provenance

Source: https://arxiv.org/abs/2302.05543

ControlNet Structure-Lock Chain

Properties

family: style-control
modality: image + text -> image
whyMultiModel: Two genuinely distinct model types run in sequence: an annotator/extractor model (e.g. HED edge detector, MiDaS depth, OpenPose keypoint) produces a dense control signal, and a separate ControlNet-augmented diffusion model consumes that signal during generation. Neither step can substitute for the other.
steps: [object Object], [object Object]
controls: Annotator type (canny vs depth vs pose); ControlNet conditioning scale; text prompt; optional second ControlNet for combined conditions (e.g. depth + canny simultaneously).
exampleStack: MiDaS depth -> ControlNet-v1.1-depth + SDXL; or OpenPose -> ControlNet-OpenPose + SD 1.5; or Canny -> FLUX.1 ControlNet (Jasper canny).
useCases: [Pose-controlled character generation][Preserving scene layout across style variations][Sketch / lineart to rendered image]
pitfalls: Over-conditioning (scale too high) makes the output look traced with no creative latitude; mismatched annotator and ControlNet model type produces incoherent outputs; depth annotator struggles with textureless or reflective surfaces.
source: Adding Conditional Control to Text-to-Image Diffusion Models (arXiv:2302.05543)

Provenance

Source: https://arxiv.org/abs/2302.05543