- family
- audio-driven
- modality
- text/image/video/audio references -> refined context -> synchronized audio+video (768p) -> upscaled audio+video (2K)
- whyMultiModel
- MiniMax's own H3 model card documents three named, separately versioned modules rather than one model run at different settings: H3-Context-IR (hosted-only instruction-refinement layer, not open-sourced), H3-Base (an open-weight Omni Transformer with distinct FL2VA and Ref2VA checkpoints that ships locally), and H3-Regenerate-2K (a hosted-only super-resolution regeneration model, also not open-sourced). The local H3-Base checkpoint cannot reach the documented 2K quality on its own; the card explicitly instructs users to 'combine a locally deployed SGLang service with the official H3-Context-IR and H3-Regenerate-2K APIs' to reproduce the full pipeline. This is distinct from the tracked ltx2-audio-conditioned-two-stage-upscale pattern (a single LTX-2.3 base checkpoint plus one spatial-upscaler checkpoint, no instruction-refinement stage) and from flux-krea-ovi-t2i2v-joint-audio-video (a separate open text-to-image model seeding an open joint audio-video diffuser, no context-refinement or regeneration stage at all).
- steps
- [object Object], [object Object], [object Object]
- controls
- choice of FL2VA (text/first-last-frame) vs Ref2VA (multi-image/video/audio reference) checkpoint; number and type of reference assets fed to Context-IR; whether to run the hybrid local+hosted path for full 2K output or stop at the local 768p H3-Base result; native canvas capped at 768px short edge (max 768x1344) for the local stage.
- exampleStack
- H3-Context-IR (hosted instruction refinement) -> MiniMax-H3 Base Ref2VA (local, SGLang-deployed, 768p synchronized audio+video) or MiniMax H3 Max (fal) (hosted, post-trained, faster-than-real-time alternative at the same 768p ceiling) -> H3-Regenerate-2K (hosted, context-conditioned 2K regeneration).
- useCases
[Reference-driven brand/product video where multiple images, a driving clip, and reference audio must be reconciled into one coherent shot][Localized or dubbed video that needs an audio reference plus a face/body reference reconciled before generation][High-resolution short-form video production where the free-tier 768p local model is generated first and only promising takes are sent to the paid 2K regeneration step]
- pitfalls
- The full 2K path depends on MiniMax's hosted Context-IR and Regenerate-2K APIs, which are 'not yet open-sourced,' so a fully local deployment is capped at 768p; the local H3-Base checkpoint 'uses full attention' rather than native sparse attention, making it computationally heavy at longer durations or higher resolutions even before the hosted regeneration step is invoked. MiniMax H3 Max (fal) swaps in a faster fal-hosted inference engine for the 768p stage but does not itself reach 2K, so the Regenerate-2K hand-off is still needed for the final resolution bump; fal's own announcement does not confirm lip-sync or identity-preservation behavior for MiniMax H3 Max, only text-to-video and image-to-video generation.
