[Model] Add native MAGI-2 Preview diffusion support (#5918)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu authored and GitHub committed
1 parent
13c617a140
commit
d4096e01fe
78 files changed
+9896
-459
No files matched your search
@@ -29,7 +29,7 @@ Each stage is implemented as a separate model class that can be configured indep
|
||||
|
||||
When adding a new model, you'll need to create the following structure:
|
||||
|
||||
```
|
||||
```text
|
||||
vllm_omni/model_executor/models/
|
||||
└── your_model_name/ # Model directory (e.g., qwen3_omni)
|
||||
├── __init__.py # Exports main model class
|
||||
@@ -422,22 +422,25 @@ The stage transition process follows these steps:
|
||||
Understanding the data structures is crucial for implementing stage transitions:
|
||||
|
||||
**Input to your function:**
|
||||
|
||||
- `stage_list[source_stage_id].engine_outputs`: List of `EngineCoreOutput` objects
|
||||
- - Each contains `outputs`: List of `RequestOutput` objects
|
||||
- Each `RequestOutput` has:
|
||||
- - - `token_ids`: Generated token IDs
|
||||
- `multimodal_output`: Dict with keys like `"code_predictor_codes"`, etc.These are the hidden states or intermediate outputs from the model's forward pass
|
||||
- `prompt_token_ids`: Original prompt token IDs
|
||||
- Each contains `outputs`: List of `RequestOutput` objects
|
||||
- Each `RequestOutput` has:
|
||||
- `token_ids`: Generated token IDs
|
||||
- `multimodal_output`: Dict with keys like `"code_predictor_codes"`, etc. These are the hidden states or intermediate outputs from the model's forward pass
|
||||
- `prompt_token_ids`: Original prompt token IDs
|
||||
|
||||
**Output from your function:**
|
||||
|
||||
- Must return `list[OmniTokensPrompt]` where each `OmniTokensPrompt` contains:
|
||||
- - `prompt_token_ids`: List[int] - Token IDs for the next stage
|
||||
- `prompt_token_ids`: List[int] - Token IDs for the next stage
|
||||
- `additional_information`: Dict[str, Any] - Optional metadata (e.g., embeddings, hidden states)
|
||||
- `multi_modal_data`: Optional multimodal data if needed
|
||||
|
||||
### How Model Outputs Are Stored
|
||||
|
||||
The model's `forward()` method returns an `OmniOutput` object that contains:
|
||||
|
||||
- `text_hidden_states`: Final hidden states for text generation
|
||||
- `multimodal_outputs`: Dict containing intermediate outputs
|
||||
|
||||
@@ -464,6 +467,7 @@ def forward(self, ...):
|
||||
```
|
||||
|
||||
These keys are then accessible in your stage transition function:
|
||||
|
||||
```python
|
||||
# In stage_input_processors/qwen3_omni.py
|
||||
thinker_prefill_embeddings = output.multimodal_output["0"] # Access by key
|
||||
@@ -493,7 +497,7 @@ thinker_hidden_states = output.multimodal_output["24"]
|
||||
Create stage transition processors in `vllm_omni/model_executor/stage_input_processors/your_model_name.py`. Each inter-stage edge should provide a **coherent processor set** rather than a single monolithic function:
|
||||
|
||||
| Suffix | Role | Runs when |
|
||||
|--------|------|-----------|
|
||||
| -------- | ------ | ----------- |
|
||||
| `*_full_payload` | Worker-side payload producer | `async_chunk=false`; accumulates tensors and ships via connector |
|
||||
| `*_async_chunk` | Scheduler-side streaming producer | `async_chunk=true`; emits per-chunk payloads |
|
||||
| `*_token_only` | Orchestrator placeholder builder | `async_chunk=false`; allocates downstream prompt slots only |
|
||||
@@ -554,7 +558,16 @@ For comprehensive testing guidelines, please refer to the [Test Writing Guide](.
|
||||
|
||||
## Adding a Model Recipe
|
||||
|
||||
After implementing and testing your model, please add a model recipe to the [vllm-project/recipes](https://github.com/vllm-project/recipes) repository. This helps other users understand how to use your model with vLLM-Omni.
|
||||
After implementing and testing your model, add a recipe under this repository's
|
||||
`recipes/` directory. Follow the
|
||||
[`add-recipe` skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/add-recipe/SKILL.md)
|
||||
and [`recipes/TEMPLATE.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/TEMPLATE.md),
|
||||
then update the model row in
|
||||
[`recipes/README.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/README.md).
|
||||
The external
|
||||
[vllm-project/recipes](https://github.com/vllm-project/recipes) repository can
|
||||
be used as a structural reference, but it is not the destination for a
|
||||
vLLM-Omni model recipe.
|
||||
|
||||
### What to Include
|
||||
|
||||
@@ -568,15 +581,10 @@ Your recipe should include:
|
||||
3. **Usage Examples**: Command-line examples demonstrating how to run the model
|
||||
4. **Configuration Details**: Important configuration parameters and their meanings
|
||||
|
||||
### Example
|
||||
|
||||
For reference, see the [LongCat recipe example](https://github.com/vllm-project/recipes/pull/179) which demonstrates the expected format and structure.
|
||||
|
||||
### Recipe Location
|
||||
|
||||
Create your recipe file in the appropriate directory structure:
|
||||
- For organization-specific models: `OrganizationName/ModelName.md`
|
||||
- For general models: `ModelName.md`
|
||||
Create `recipes/<vendor>/<model-family>.md` and keep commands, feature status,
|
||||
hardware evidence, and limitations specific to the model family.
|
||||
|
||||
The recipe should be a Markdown file that provides clear, reproducible instructions for users to get started with your model.
|
||||
|
||||
@@ -590,7 +598,7 @@ Adding a new model to vLLM-Omni involves:
|
||||
4. **Define pipeline topology and deploy defaults** in `pipeline.py` and `vllm_omni/deploy/`
|
||||
5. **Implement stage input processors** for stage transitions
|
||||
6. **Write tests** to verify functionality
|
||||
7. **Add model recipe** to the [vllm-project/recipes](https://github.com/vllm-project/recipes) repository (see [Adding a Model Recipe](#adding-a-model-recipe) section)
|
||||
7. **Add an in-repository model recipe** under `recipes/` and update its index (see [Adding a Model Recipe](#adding-a-model-recipe))
|
||||
|
||||
### Qwen3-Omni Reference Files
|
||||
|
||||
@@ -607,6 +615,7 @@ For a complete reference implementation, see:
|
||||
- **Testing**: `vllm_omni/tests/e2e/offline_inference/test_qwen3_omni.py`
|
||||
|
||||
For more information, see:
|
||||
|
||||
- [Architecture Overview](../../design/architecture_overview.md)
|
||||
- [Supported Models](../../models/supported_models.md)
|
||||
- [Pipeline and Deploy Configuration Guide](../../configuration/stage_configs.md)
|
||||
@@ -991,10 +991,13 @@ git push origin your-branch --force-with-lease
|
||||
|
||||
## Adding a Model Recipe
|
||||
|
||||
After implementing and testing your model, add a model recipe to the
|
||||
[vllm-project/recipes](https://github.com/vllm-project/recipes) repository so users can
|
||||
get started quickly. See [Adding an Omni-Modality Model](./adding_omni_model.md#adding-a-model-recipe)
|
||||
for the expected format.
|
||||
After implementing and testing your model, add a recipe under this repository's
|
||||
`recipes/` directory. Follow the
|
||||
[`add-recipe` skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/add-recipe/SKILL.md)
|
||||
and [`recipes/TEMPLATE.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/TEMPLATE.md),
|
||||
update the model row in
|
||||
[`recipes/README.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/README.md),
|
||||
and synchronize the relevant TTS example and speech API documentation.
|
||||
|
||||
## Summary
|
||||
|
||||
@@ -1011,7 +1014,7 @@ Adding a TTS model to vLLM-Omni involves:
|
||||
8. **Online serving** - add all 5 integration points to `serving_speech.py` in one commit
|
||||
9. **Tests** - cover single request, batching, and streaming
|
||||
10. **Pre-commit + DCO** - run `pre-commit` before pushing; sign every commit with `git commit -s`
|
||||
11. **Model recipe** - add to [vllm-project/recipes](https://github.com/vllm-project/recipes)
|
||||
11. **Model recipe** - add under the in-repository `recipes/` directory and update its index
|
||||
12. **Invariants** - re-check I1–I5 (streaming contract, consumer hygiene, hot-loop discipline, validation pyramid, per-request state) at the end of every phase
|
||||
|
||||
### Qwen3-TTS Reference Files
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="460" viewBox="0 0 760 460" role="img" aria-labelledby="title desc">
|
||||
<title id="title">MAGI-2 DLO one-step Pareto frontier</title>
|
||||
<desc id="desc">Warm user latency in seconds versus throughput per device. DP1SP4 is 5.735 seconds and 0.0436 outputs per second per device, DP2SP2 is 6.785 and 0.0737, and DP4 is 9.997 and 0.1000.</desc>
|
||||
<rect width="760" height="460" fill="#ffffff"/>
|
||||
<style>
|
||||
text { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #111827; }
|
||||
.title { font-size: 20px; font-weight: 600; }
|
||||
.axis-label { font-size: 14px; font-weight: 500; }
|
||||
.tick { font-size: 12px; fill: #4b5563; }
|
||||
.point-label { font-size: 13px; font-weight: 600; }
|
||||
.value-label { font-size: 11px; fill: #4b5563; }
|
||||
.grid { stroke: #e5e7eb; stroke-width: 1; }
|
||||
.axis { stroke: #6b7280; stroke-width: 1.5; }
|
||||
.frontier { fill: none; stroke: #2563eb; stroke-width: 3; }
|
||||
.point { fill: #2563eb; stroke: #ffffff; stroke-width: 2; }
|
||||
</style>
|
||||
|
||||
<text x="380" y="28" text-anchor="middle" class="title">One diffusion step</text>
|
||||
|
||||
<g aria-hidden="true">
|
||||
<line x1="85" y1="309" x2="700" y2="309" class="grid"/>
|
||||
<line x1="85" y1="226" x2="700" y2="226" class="grid"/>
|
||||
<line x1="85" y1="143" x2="700" y2="143" class="grid"/>
|
||||
<line x1="85" y1="61" x2="700" y2="61" class="grid"/>
|
||||
<line x1="85" y1="40" x2="85" y2="350" class="axis"/>
|
||||
<line x1="85" y1="350" x2="700" y2="350" class="axis"/>
|
||||
|
||||
<text x="73" y="313" text-anchor="end" class="tick">0.04</text>
|
||||
<text x="73" y="230" text-anchor="end" class="tick">0.06</text>
|
||||
<text x="73" y="147" text-anchor="end" class="tick">0.08</text>
|
||||
<text x="73" y="65" text-anchor="end" class="tick">0.10</text>
|
||||
|
||||
<text x="85" y="370" text-anchor="middle" class="tick">5</text>
|
||||
<text x="197" y="370" text-anchor="middle" class="tick">6</text>
|
||||
<text x="309" y="370" text-anchor="middle" class="tick">7</text>
|
||||
<text x="421" y="370" text-anchor="middle" class="tick">8</text>
|
||||
<text x="532" y="370" text-anchor="middle" class="tick">9</text>
|
||||
<text x="644" y="370" text-anchor="middle" class="tick">10</text>
|
||||
</g>
|
||||
|
||||
<text x="392" y="414" text-anchor="middle" class="axis-label">Warm user latency (seconds) — lower is better</text>
|
||||
<text x="20" y="195" transform="rotate(-90 20 195)" text-anchor="middle" class="axis-label">Throughput/device (outputs/s) — higher is better</text>
|
||||
|
||||
<polyline points="167,294 285,169 644,61" class="frontier"/>
|
||||
|
||||
<circle cx="167" cy="294" r="7" class="point"/>
|
||||
<text x="179" y="288" class="point-label">DP1SP4</text>
|
||||
<text x="179" y="305" class="value-label">5.735 s, 0.0436</text>
|
||||
|
||||
<circle cx="285" cy="169" r="7" class="point"/>
|
||||
<text x="297" y="163" class="point-label">DP2SP2</text>
|
||||
<text x="297" y="180" class="value-label">6.785 s, 0.0737</text>
|
||||
|
||||
<circle cx="644" cy="61" r="7" class="point"/>
|
||||
<text x="632" y="84" text-anchor="end" class="point-label">DP4</text>
|
||||
<text x="632" y="101" text-anchor="end" class="value-label">9.997 s, 0.1000</text>
|
||||
|
||||
<text x="700" y="444" text-anchor="end" class="tick">4x NVIDIA L20X · warm qualification run</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.1 KiB |
@@ -0,0 +1,65 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="760" height="460" viewBox="0 0 760 460" role="img" aria-labelledby="title desc">
|
||||
<title id="title">MAGI-2 DLO four-step Pareto frontier</title>
|
||||
<desc id="desc">Warm user latency in seconds versus throughput per device. DP1SP4 is 8.647 seconds and 0.0289 outputs per second per device, DP2SP2 is 11.682 and 0.0428, and DP4 is 20.957 and 0.0477.</desc>
|
||||
<rect width="760" height="460" fill="#ffffff"/>
|
||||
<style>
|
||||
text { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; fill: #111827; }
|
||||
.title { font-size: 20px; font-weight: 600; }
|
||||
.axis-label { font-size: 14px; font-weight: 500; }
|
||||
.tick { font-size: 12px; fill: #4b5563; }
|
||||
.point-label { font-size: 13px; font-weight: 600; }
|
||||
.value-label { font-size: 11px; fill: #4b5563; }
|
||||
.grid { stroke: #e5e7eb; stroke-width: 1; }
|
||||
.axis { stroke: #6b7280; stroke-width: 1.5; }
|
||||
.frontier { fill: none; stroke: #2563eb; stroke-width: 3; }
|
||||
.point { fill: #2563eb; stroke: #ffffff; stroke-width: 2; }
|
||||
</style>
|
||||
|
||||
<text x="380" y="28" text-anchor="middle" class="title">Four diffusion steps</text>
|
||||
|
||||
<g aria-hidden="true">
|
||||
<line x1="85" y1="350" x2="700" y2="350" class="grid"/>
|
||||
<line x1="85" y1="288" x2="700" y2="288" class="grid"/>
|
||||
<line x1="85" y1="226" x2="700" y2="226" class="grid"/>
|
||||
<line x1="85" y1="164" x2="700" y2="164" class="grid"/>
|
||||
<line x1="85" y1="102" x2="700" y2="102" class="grid"/>
|
||||
<line x1="85" y1="40" x2="700" y2="40" class="grid"/>
|
||||
<line x1="85" y1="40" x2="85" y2="350" class="axis"/>
|
||||
<line x1="85" y1="350" x2="700" y2="350" class="axis"/>
|
||||
|
||||
<text x="73" y="354" text-anchor="end" class="tick">0.025</text>
|
||||
<text x="73" y="292" text-anchor="end" class="tick">0.030</text>
|
||||
<text x="73" y="230" text-anchor="end" class="tick">0.035</text>
|
||||
<text x="73" y="168" text-anchor="end" class="tick">0.040</text>
|
||||
<text x="73" y="106" text-anchor="end" class="tick">0.045</text>
|
||||
<text x="73" y="44" text-anchor="end" class="tick">0.050</text>
|
||||
|
||||
<text x="85" y="370" text-anchor="middle" class="tick">8</text>
|
||||
<text x="173" y="370" text-anchor="middle" class="tick">10</text>
|
||||
<text x="261" y="370" text-anchor="middle" class="tick">12</text>
|
||||
<text x="349" y="370" text-anchor="middle" class="tick">14</text>
|
||||
<text x="436" y="370" text-anchor="middle" class="tick">16</text>
|
||||
<text x="524" y="370" text-anchor="middle" class="tick">18</text>
|
||||
<text x="612" y="370" text-anchor="middle" class="tick">20</text>
|
||||
<text x="700" y="370" text-anchor="middle" class="tick">22</text>
|
||||
</g>
|
||||
|
||||
<text x="392" y="414" text-anchor="middle" class="axis-label">Warm user latency (seconds) — lower is better</text>
|
||||
<text x="20" y="195" transform="rotate(-90 20 195)" text-anchor="middle" class="axis-label">Throughput/device (outputs/s) — higher is better</text>
|
||||
|
||||
<polyline points="113,302 247,129 654,69" class="frontier"/>
|
||||
|
||||
<circle cx="113" cy="302" r="7" class="point"/>
|
||||
<text x="125" y="296" class="point-label">DP1SP4</text>
|
||||
<text x="125" y="313" class="value-label">8.647 s, 0.0289</text>
|
||||
|
||||
<circle cx="247" cy="129" r="7" class="point"/>
|
||||
<text x="259" y="123" class="point-label">DP2SP2</text>
|
||||
<text x="259" y="140" class="value-label">11.682 s, 0.0428</text>
|
||||
|
||||
<circle cx="654" cy="69" r="7" class="point"/>
|
||||
<text x="642" y="92" text-anchor="end" class="point-label">DP4</text>
|
||||
<text x="642" y="109" text-anchor="end" class="value-label">20.957 s, 0.0477</text>
|
||||
|
||||
<text x="700" y="444" text-anchor="end" class="tick">4x NVIDIA L20X · warm qualification run</text>
|
||||
</svg>
|
||||
|
After Width: | Height: | Size: 3.6 KiB |
@@ -18,7 +18,7 @@ th {
|
||||
</style>
|
||||
|
||||
| Architecture | Models | Example HF Models | NVIDIA GPU | AMD GPU | Ascend NPU | Intel GPU | Recipe |
|
||||
|--------------|--------|-------------------|------------|---------|-----|-----------|--------|
|
||||
| -------------- | -------- | ------------------- | ------------ | --------- | ----- | ----------- | -------- |
|
||||
| `Qwen3OmniMoeForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) |
|
||||
| `Qwen2_5OmniForConditionalGeneration` | Qwen2.5-Omni | `Qwen/Qwen2.5-Omni-7B`, `Qwen/Qwen2.5-Omni-3B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen<sup>1</sup>) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ | | | | — |
|
||||
@@ -42,7 +42,7 @@ th {
|
||||
| `SanaWmPipeline` | SANA-WM | `BBBBruce/SANA-WM_bidirectional-stage1-diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-WM.md) |
|
||||
| `Wan22S2VPipeline` | Wan2.2-S2V | `Wan-AI/Wan2.2-S2V-14B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `Wan22VACEPipeline` | Wan2.1-VACE | `Wan-AI/Wan2.1-VACE-1.3B-diffusers`, `Wan-AI/Wan2.1-VACE-14B-diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `Wan22VACEPipeline` | Wan2.2-VACE | `Pyros13/Wan2.2-VACE-Fun-A14B-Diffusers` | ✅︎ | | | ✅︎ | — |
|
||||
| `Wan22VACEPipeline` | Wan2.2-VACE | `Pyros13/Wan2.2-VACE-Fun-A14B-Diffusers` | ✅︎ | | | ✅︎ | — |
|
||||
| `LTX2Pipeline` | LTX-2 / LTX-2.3 one-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` | ✅︎ | ✅︎ | | | — |
|
||||
| `LTX2TwoStagePipeline` | LTX-2 / LTX-2.3 ordinary two-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` + matching Lightricks LoRA and upsampler | ✅︎ | ✅︎ | | | — |
|
||||
| `LTX2DistilledOneStagePipeline` | LTX-2 / LTX-2.3 merged-distilled one-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled`, `diffusers/LTX-2.3-Distilled-Diffusers` | ✅︎ | ✅︎ | | | — |
|
||||
@@ -55,6 +55,7 @@ th {
|
||||
| `SanaVideoPipeline` | SANA-Video-2B T2V (native 480p/720p; Diffusers adapter validated at both resolutions) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) |
|
||||
| `SanaImageToVideoPipeline` | SANA-Video-2B I2V (native and Diffusers adapter validated at 480p/720p) | `Efficient-Large-Model/SANA-Video_2B_480p_diffusers`, `Efficient-Large-Model/SANA-Video_2B_720p_diffusers` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/SANA-Video-2B.md) |
|
||||
| `MiniMaxH3Pipeline` | MiniMax H3 T2VA, first/last-keyframe FL2VA, and mixed-reference Ref2VA | `MiniMaxAI/MiniMax-H3` | ✅︎ | ✅︎ | | | [Published](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) |
|
||||
| `Magi2Pipeline` | MAGI-2 native Preview-stage T2VA and I2VA (272p/540p) | `sand-ai/MAGI-2-preview` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SandAI/MAGI-2-preview-L20X.md) |
|
||||
| `DreamZeroPipeline` | DreamZero-DROID | `GEAR-Dreams/DreamZero-DROID` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `Pi0Pipeline` | π0 (Pi-Zero) VLA | `lerobot/pi0_base` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/lerobot/Pi0.md) |
|
||||
| `HeliosPipeline`, `HeliosPyramidPipeline` | Helios | `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled` | ✅︎ | ✅︎ | ✅︎ | | — |
|
||||
@@ -101,11 +102,11 @@ th {
|
||||
| `HunyuanVideo15ImageToVideoPipeline` | HunyuanVideo-1.5-I2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_i2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_i2v` | ✅︎ | ✅︎ | | | — |
|
||||
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
|
||||
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
|
||||
|`DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | ✅︎ | — |
|
||||
| `DyninOmniForConditionalGeneration` | Dynin-Omni | `snu-aidas/Dynin-Omni` | ✅︎ | | | ✅︎ | — |
|
||||
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | — |
|
||||
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
|
||||
|`HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
|
||||
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
|
||||
| `HiDreamO1ImagePipeline` | HiDream-O1-Image | `HiDream-ai/HiDream-O1-Image` | ✅︎ | | | | — |
|
||||
|
||||
The `Recipe` column links the corresponding deployment recipe. `Published` is
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Diffusion Advanced Features
|
||||
|
||||
<!-- markdownlint-disable MD028 MD060 -->
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
@@ -30,7 +32,6 @@ Cache methods trade minimal quality for significant speedup. Quality loss is typ
|
||||
| **[TeaCache](diffusion/cache_acceleration/teacache.md)** | Adaptive caching using modulated inputs | Quick setup, balanced quality/speed on single GPU |
|
||||
| **[Cache-DiT](diffusion/cache_acceleration/cache_dit.md)** | Multiple caching techniques: DBCache, TaylorSeer, SCM | Fine-grained control, tunable quality-speed tradeoff |
|
||||
|
||||
|
||||
#### Lossless Acceleration
|
||||
|
||||
Parallelism methods distribute computation across GPUs without quality loss (mathematically equivalent to single-GPU).
|
||||
@@ -72,7 +73,6 @@ Extension methods add specialized capabilities to diffusion models beyond standa
|
||||
| **[LoRA Inference](diffusion/lora.md)** | Enables inference with Low-Rank Adaptation (LoRA) adapters weights | Reinforcement learning extensions |
|
||||
| **[Frame Interpolation](diffusion/frame_interpolation.md)** | Inserts intermediate video frames after generation for smoother motion | Video generation pipelines that need higher temporal smoothness |
|
||||
|
||||
|
||||
### Execution Modes
|
||||
|
||||
Execution modes control how the diffusion pipeline processes requests and
|
||||
@@ -147,6 +147,7 @@ The following tables show which models support each feature:
|
||||
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (decode) | ✅ | ❌ |
|
||||
|
||||
> Notes:
|
||||
>
|
||||
> 1. Nextstep_1(T2I) does not support cache acceleration methods such as TeaCache or Cache-DiT.
|
||||
> 2. `Tongyi-MAI/Z-Image-Turbo` is a distilled model with minimal NFEs; CFG-Parallel is not necessary.
|
||||
> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload swaps the nested UND reasoner and GEN generator pathways; layerwise offload remains available for blockwise GEN/UND offload.
|
||||
@@ -170,18 +171,18 @@ The following tables show which models support each feature:
|
||||
| **Cosmos3** | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
|
||||
| **LongCat-Video-Avatar-1.5** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **MiniMax-H3** | ✅ (FL2VA) | ✅ | ✅ | ❌ | ✅ (DiT/TE) | ❌ | ✅ | ✅ | ✅ (tile) | ✅ (DiT) | ❌ |
|
||||
| **MAGI-2 Preview** | ❌ | ✅ | ✅ (Ulysses) | ✅ (2-way) | ✅ | ❌ | ✅ | ✅ (1-GPU/LW; DLO DP-AG/SP no-AG) | ✅ (tile) | ❌ | ❌ |
|
||||
| **SANA-WM** | ❌ | ❌ | ❌<sup>5</sup> | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
|
||||
> Notes:
|
||||
> 5. SANA-WM cannot support sequence parallelism: its bidirectional gated delta
|
||||
> recurrence carries state across frames, so a rank cannot denoise a slice of
|
||||
> the token sequence in isolation. Doing so would need a distributed scan or
|
||||
> an all-gather before every GDN block. The remaining ❌ columns are simply
|
||||
> unvalidated on this model, not known-broken.
|
||||
> recurrence carries state across frames, so a rank cannot denoise a slice of
|
||||
> the token sequence in isolation. Doing so would need a distributed scan or
|
||||
> an all-gather before every GDN block. The remaining ❌ columns are simply
|
||||
> unvalidated on this model, not known-broken.
|
||||
|
||||
> **Step execution note:** Helios supports single-request step execution only;
|
||||
> use `max_num_seqs=1`.
|
||||
|
||||
**Frame Interpolation Support**
|
||||
|
||||
- **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines; SANA-Video-2B native text-to-video and image-to-video pipelines
|
||||
@@ -193,7 +194,6 @@ The following tables show which models support each feature:
|
||||
|-----------------------|:---------:|:----------:|:---------------------:|:--------------:|:-----------------:|:-------------------:|:------:|:-------------------------:|:--------------------:|:--------------:|:----------------:|
|
||||
| **Stable-Audio-Open** | ✅ | ❌ | ❓ | ❓ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ |
|
||||
|
||||
|
||||
## Feature Compatibility
|
||||
|
||||
**Legend:**
|
||||
@@ -232,7 +232,6 @@ The following tables show which models support each feature:
|
||||
5. The compatibility matrix uses FP8 as the representative quantization method.
|
||||
6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
|
||||
|
||||
|
||||
## Multi-Thread Weight Loading
|
||||
|
||||
The loading guide now lives at [Diffusion Startup and
|
||||
|
||||
Reference in new issue
Block a user