DiffSensei MLLM Identity-Adapter Manga Generation

A multimodal LLM (SEED-X), fine-tuned as a text-compatible identity adapter, reads one or more character reference images and adjusts their expression, pose, and action to match each panel's text cues; those adapted features feed an SDXL diffusion backbone through masked cross-attention alongside per-panel character and dialogue bounding boxes, producing layout-aware, identity-consistent manga pages.

Properties

family: multi-shot
modality: character reference image(s) + per-panel text + character/dialogue bounding boxes -> customized manga panels
whyMultiModel: SDXL alone cannot hold multiple named characters' identities across panels while flexibly re-posing and re-expressing them per prompt; DiffSensei chains a separate SEED-X-initialized MLLM (updated via LoRA and a resampler) that performs the semantic identity-to-panel adaptation, an IP-Adapter-Plus-SDXL-initialized resampler that extracts the character features, and the SDXL diffusion generator that renders the panel. This differs from the tracked character-sheet-to-lettered-comic-panels pattern (IPAdapter FaceID + a static OpenPose ControlNet, with panel layout and lettering handled by separate non-generative ComfyUI tools): here the MLLM itself performs per-panel semantic pose/expression adaptation of the character before diffusion, rather than injecting a fixed identity embedding under a hand-picked pose skeleton.
steps: [object Object], [object Object], [object Object]
controls: per-panel text prompt; number and choice of character reference images; character bounding box per panel; dialogue bounding box per panel; LoRA/resampler fine-tuning stage.
exampleStack: IP-Adapter-Plus-SDXL (character feature extractor) + SEED-X MLLM (LoRA + resampler, identity/pose adaptation) -> SDXL (masked cross-attention panel diffusion with layout + dialogue boxes).
useCases: [Multi-character manga/webtoon page generation with per-panel pose and expression control from reference art][Customized manga generation from a small set of character reference images][Layout-aware panel generation with dialogue-bubble placement built into the conditioning rather than added as a post-process]
pitfalls: Panel generation depends on correctly assigned per-panel character and dialogue bounding boxes; because the identity adapter's cross-attention is masked by these boxes, an incorrect box placement misaligns a character's features onto the wrong panel region.
source: DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation (arXiv:2412.07589, CVPR 2025)

Provenance

Source: https://arxiv.org/abs/2412.07589

DiffSensei MLLM Identity-Adapter Manga Generation

Properties

family

multi-shot

modality

character reference image(s) + per-panel text + character/dialogue bounding boxes -> customized manga panels

whyMultiModel

SDXL alone cannot hold multiple named characters' identities across panels while flexibly re-posing and re-expressing them per prompt; DiffSensei chains a separate SEED-X-initialized MLLM (updated via LoRA and a resampler) that performs the semantic identity-to-panel adaptation, an IP-Adapter-Plus-SDXL-initialized resampler that extracts the character features, and the SDXL diffusion generator that renders the panel. This differs from the tracked character-sheet-to-lettered-comic-panels pattern (IPAdapter FaceID + a static OpenPose ControlNet, with panel layout and lettering handled by separate non-generative ComfyUI tools): here the MLLM itself performs per-panel semantic pose/expression adaptation of the character before diffusion, rather than injecting a fixed identity embedding under a hand-picked pose skeleton.

steps

[object Object], [object Object], [object Object]

controls

per-panel text prompt; number and choice of character reference images; character bounding box per panel; dialogue bounding box per panel; LoRA/resampler fine-tuning stage.

exampleStack

IP-Adapter-Plus-SDXL (character feature extractor) + SEED-X MLLM (LoRA + resampler, identity/pose adaptation) -> SDXL (masked cross-attention panel diffusion with layout + dialogue boxes).

useCases

[Multi-character manga/webtoon page generation with per-panel pose and expression control from reference art][Customized manga generation from a small set of character reference images][Layout-aware panel generation with dialogue-bubble placement built into the conditioning rather than added as a post-process]

pitfalls

Panel generation depends on correctly assigned per-panel character and dialogue bounding boxes; because the identity adapter's cross-attention is masked by these boxes, an incorrect box placement misaligns a character's features onto the wrong panel region.

source

DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation (arXiv:2412.07589, CVPR 2025)