Depth/Pose-Guided Video-to-Video Restyle

A source video is passed through a preprocessor that extracts a structural control signal (depth map, Canny edges, or OpenPose skeleton). That signal is fed as conditioning to a video diffusion model which generates new video that preserves the original motion and geometry while applying a new appearance, style, or character. The result is a restyled video with motion inherited from the reference footage.

Properties

family: style-control
modality: video -> control signal (depth/canny/pose) -> video
whyMultiModel: Video diffusion models cannot natively decompose appearance from structure; a separate ControlNet preprocessor model is required to extract the structural signal, which then constrains the generator. Neither the preprocessor nor the generator can do the other's job alone.
steps: [object Object], [object Object]
controls: Control signal type (depth vs. canny vs. pose); control strength; style and content prompts; sampler schedule (LTXVScheduler); frame dimensions.
exampleStack: Lotus Depth preprocessor -> LTX-2 ControlNet (guided video generation). OR OpenPose extractor -> Wan 2.1 Fun ControlNet.
useCases: [Motion retargeting (human to character)][Scene relighting and restyle][Product placement in existing footage][Previz from rough reference video]
pitfalls: Strong depth control preserves layout but can suppress texture creativity; OpenPose fails on non-standard human shapes; motion blur in source degrades extracted control quality.
source: RunComfy: LTX-2 ControlNet in ComfyUI - Depth-Controlled Video Workflow

Provenance

Source: https://www.runcomfy.com/comfyui-workflows/ltx-2-controlnet-in-comfyui-depth-controlled-video-workflow

Depth/Pose-Guided Video-to-Video Restyle

Properties

family

style-control

modality

video -> control signal (depth/canny/pose) -> video

whyMultiModel

Video diffusion models cannot natively decompose appearance from structure; a separate ControlNet preprocessor model is required to extract the structural signal, which then constrains the generator. Neither the preprocessor nor the generator can do the other's job alone.

steps

[object Object], [object Object]

controls

Control signal type (depth vs. canny vs. pose); control strength; style and content prompts; sampler schedule (LTXVScheduler); frame dimensions.

exampleStack

Lotus Depth preprocessor -> LTX-2 ControlNet (guided video generation). OR OpenPose extractor -> Wan 2.1 Fun ControlNet.

useCases

[Motion retargeting (human to character)][Scene relighting and restyle][Product placement in existing footage][Previz from rough reference video]

pitfalls

Strong depth control preserves layout but can suppress texture creativity; OpenPose fails on non-standard human shapes; motion blur in source degrades extracted control quality.

source

RunComfy: LTX-2 ControlNet in ComfyUI - Depth-Controlled Video Workflow