StreamDiffusionTD live multi-ControlNet + StreamV2V temporal-consistency chain

TouchDesigner tox StreamDiffusionTD runs SDXL-Turbo in real time driven by any TouchDesigner-generated control signal (camera feed, procedural TOP, depth/canny/normal maps built in TD) through multiple simultaneous TensorRT-accelerated ControlNets, then applies StreamV2V cached-attention video-to-video processing so consecutive live frames stay temporally coherent instead of flickering independently.

Properties

family: style-control
modality: live video/control signal -> image (real-time loop) -> temporally-consistent video
whyMultiModel: Distinct from baseline depth-pose-guided-v2v-restyle (offline video-to-video restyle via LTX/Wan Fun) because this runs live/interactively at TouchDesigner frame rates with a different model pair, SDXL-Turbo for per-frame generation plus StreamV2V for cross-frame temporal consistency, driven by arbitrary TD-authored control signals rather than a pre-extracted depth/pose track on a fixed video file.
steps: [object Object], [object Object]
controls: ControlNet on/off toggle mid-stream and per-unit weight, multiple simultaneous ControlNets, TensorRT engine build per model for real-time speed, any TD-authored conditioning image (audio-reactive, camera, procedural) wired into the second input
exampleStack: TouchDesigner TOP control -> SDXL-Turbo + multi-ControlNet (StreamDiffusionTD) -> StreamV2V temporal pass
useCases: [live VJ/installation real-time AI restyle of camera feed][interactive projection-mapped generative visuals reacting to audio or sensors][real-time previz restyle driven by procedural TD signals]
pitfalls: requires per-model TensorRT engine builds (CUDA/TRT setup) before streaming; ControlNet must be enabled at stream start to be usable mid-session; real-time frame rate depends heavily on GPU (documented benchmarks are for high-end cards)
source: DotSimulate StreamDiffusionTD docs

Provenance

Source: https://dotsimulate.com/docs/streamdiffusiontd

StreamDiffusionTD live multi-ControlNet + StreamV2V temporal-consistency chain

Properties

family

style-control

modality

live video/control signal -> image (real-time loop) -> temporally-consistent video

whyMultiModel

Distinct from baseline depth-pose-guided-v2v-restyle (offline video-to-video restyle via LTX/Wan Fun) because this runs live/interactively at TouchDesigner frame rates with a different model pair, SDXL-Turbo for per-frame generation plus StreamV2V for cross-frame temporal consistency, driven by arbitrary TD-authored control signals rather than a pre-extracted depth/pose track on a fixed video file.

steps

[object Object], [object Object]

controls

ControlNet on/off toggle mid-stream and per-unit weight, multiple simultaneous ControlNets, TensorRT engine build per model for real-time speed, any TD-authored conditioning image (audio-reactive, camera, procedural) wired into the second input

exampleStack

TouchDesigner TOP control -> SDXL-Turbo + multi-ControlNet (StreamDiffusionTD) -> StreamV2V temporal pass

useCases

[live VJ/installation real-time AI restyle of camera feed][interactive projection-mapped generative visuals reacting to audio or sensors][real-time previz restyle driven by procedural TD signals]

pitfalls

requires per-model TensorRT engine builds (CUDA/TRT setup) before streaming; ControlNet must be enabled at stream start to be usable mid-session; real-time frame rate depends heavily on GPU (documented benchmarks are for high-end cards)

source

DotSimulate StreamDiffusionTD docs