[Model] Add native MAGI-2 Preview diffusion support (#5918)

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu authored and GitHub committed 2026-09-05 15:32:30 +08:00
1 parent 13c617a140
commit d4096e01fe
78 files changed
+9896 -459

No files matched your search

+26 -17
View File
@@ -29,7 +29,7 @@ Each stage is implemented as a separate model class that can be configured indep
When adding a new model, you'll need to create the following structure:
```
```text
vllm_omni/model_executor/models/
└── your_model_name/ # Model directory (e.g., qwen3_omni)
├── __init__.py # Exports main model class
@@ -422,22 +422,25 @@ The stage transition process follows these steps:
Understanding the data structures is crucial for implementing stage transitions:
**Input to your function:**
- `stage_list[source_stage_id].engine_outputs`: List of `EngineCoreOutput` objects
- - Each contains `outputs`: List of `RequestOutput` objects
- Each `RequestOutput` has:
- - - `token_ids`: Generated token IDs
- `multimodal_output`: Dict with keys like `"code_predictor_codes"`, etc.These are the hidden states or intermediate outputs from the model's forward pass
- `prompt_token_ids`: Original prompt token IDs
- Each contains `outputs`: List of `RequestOutput` objects
- Each `RequestOutput` has:
- `token_ids`: Generated token IDs
- `multimodal_output`: Dict with keys like `"code_predictor_codes"`, etc. These are the hidden states or intermediate outputs from the model's forward pass
- `prompt_token_ids`: Original prompt token IDs
**Output from your function:**
- Must return `list[OmniTokensPrompt]` where each `OmniTokensPrompt` contains:
- - `prompt_token_ids`: List[int] - Token IDs for the next stage
- `prompt_token_ids`: List[int] - Token IDs for the next stage
- `additional_information`: Dict[str, Any] - Optional metadata (e.g., embeddings, hidden states)
- `multi_modal_data`: Optional multimodal data if needed
### How Model Outputs Are Stored
The model's `forward()` method returns an `OmniOutput` object that contains:
- `text_hidden_states`: Final hidden states for text generation
- `multimodal_outputs`: Dict containing intermediate outputs
@@ -464,6 +467,7 @@ def forward(self, ...):
```
These keys are then accessible in your stage transition function:
```python
# In stage_input_processors/qwen3_omni.py
thinker_prefill_embeddings = output.multimodal_output["0"] # Access by key
@@ -493,7 +497,7 @@ thinker_hidden_states = output.multimodal_output["24"]
Create stage transition processors in `vllm_omni/model_executor/stage_input_processors/your_model_name.py`. Each inter-stage edge should provide a **coherent processor set** rather than a single monolithic function:
| Suffix | Role | Runs when |
|--------|------|-----------|
| -------- | ------ | ----------- |
| `*_full_payload` | Worker-side payload producer | `async_chunk=false`; accumulates tensors and ships via connector |
| `*_async_chunk` | Scheduler-side streaming producer | `async_chunk=true`; emits per-chunk payloads |
| `*_token_only` | Orchestrator placeholder builder | `async_chunk=false`; allocates downstream prompt slots only |
@@ -554,7 +558,16 @@ For comprehensive testing guidelines, please refer to the [Test Writing Guide](.
## Adding a Model Recipe
After implementing and testing your model, please add a model recipe to the [vllm-project/recipes](https://github.com/vllm-project/recipes) repository. This helps other users understand how to use your model with vLLM-Omni.
After implementing and testing your model, add a recipe under this repository's
`recipes/` directory. Follow the
[`add-recipe` skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/add-recipe/SKILL.md)
and [`recipes/TEMPLATE.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/TEMPLATE.md),
then update the model row in
[`recipes/README.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/README.md).
The external
[vllm-project/recipes](https://github.com/vllm-project/recipes) repository can
be used as a structural reference, but it is not the destination for a
vLLM-Omni model recipe.
### What to Include
@@ -568,15 +581,10 @@ Your recipe should include:
3. **Usage Examples**: Command-line examples demonstrating how to run the model
4. **Configuration Details**: Important configuration parameters and their meanings
### Example
For reference, see the [LongCat recipe example](https://github.com/vllm-project/recipes/pull/179) which demonstrates the expected format and structure.
### Recipe Location
Create your recipe file in the appropriate directory structure:
- For organization-specific models: `OrganizationName/ModelName.md`
- For general models: `ModelName.md`
Create `recipes/<vendor>/<model-family>.md` and keep commands, feature status,
hardware evidence, and limitations specific to the model family.
The recipe should be a Markdown file that provides clear, reproducible instructions for users to get started with your model.
@@ -590,7 +598,7 @@ Adding a new model to vLLM-Omni involves:
4. **Define pipeline topology and deploy defaults** in `pipeline.py` and `vllm_omni/deploy/`
5. **Implement stage input processors** for stage transitions
6. **Write tests** to verify functionality
7. **Add model recipe** to the [vllm-project/recipes](https://github.com/vllm-project/recipes) repository (see [Adding a Model Recipe](#adding-a-model-recipe) section)
7. **Add an in-repository model recipe** under `recipes/` and update its index (see [Adding a Model Recipe](#adding-a-model-recipe))
### Qwen3-Omni Reference Files
@@ -607,6 +615,7 @@ For a complete reference implementation, see:
- **Testing**: `vllm_omni/tests/e2e/offline_inference/test_qwen3_omni.py`
For more information, see:
- [Architecture Overview](../../design/architecture_overview.md)
- [Supported Models](../../models/supported_models.md)
- [Pipeline and Deploy Configuration Guide](../../configuration/stage_configs.md)
+8 -5
View File
@@ -991,10 +991,13 @@ git push origin your-branch --force-with-lease
## Adding a Model Recipe
After implementing and testing your model, add a model recipe to the
[vllm-project/recipes](https://github.com/vllm-project/recipes) repository so users can
get started quickly. See [Adding an Omni-Modality Model](./adding_omni_model.md#adding-a-model-recipe)
for the expected format.
After implementing and testing your model, add a recipe under this repository's
`recipes/` directory. Follow the
[`add-recipe` skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/add-recipe/SKILL.md)
and [`recipes/TEMPLATE.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/TEMPLATE.md),
update the model row in
[`recipes/README.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/README.md),
and synchronize the relevant TTS example and speech API documentation.
## Summary
@@ -1011,7 +1014,7 @@ Adding a TTS model to vLLM-Omni involves:
8. **Online serving** - add all 5 integration points to `serving_speech.py` in one commit
9. **Tests** - cover single request, batching, and streaming
10. **Pre-commit + DCO** - run `pre-commit` before pushing; sign every commit with `git commit -s`
11. **Model recipe** - add to [vllm-project/recipes](https://github.com/vllm-project/recipes)
11. **Model recipe** - add under the in-repository `recipes/` directory and update its index
12. **Invariants** - re-check I1–I5 (streaming contract, consumer hygiene, hot-loop discipline, validation pyramid, per-request state) at the end of every phase
### Qwen3-TTS Reference Files