docs: update vLLM-Omni architecture overview (#5914)
Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
Hongsheng Liu authored and GitHub committed
1 parent
593b404539
commit
9159cedbd7
3 files changed
+408
-146
No files matched your search
@@ -9,7 +9,7 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
||||
</h3>
|
||||
|
||||
<p align="center">
|
||||
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://deepwiki.com/vllm-project/vllm-omni"><b>DeepWiki</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> | <a href="docs/assets/WeChat.jpg"><b>WeChat</b></a> | <a href="https://arxiv.org/abs/2602.02204"><b>Paper</b></a> | <a href="https://docs.google.com/presentation/d/111-L8zF7A1j_YI_cR8JsblofdScdRr2f/edit?usp=sharing&ouid=110473603432222024453&rtpof=true&sd=true"><b>Slides</b></a> |
|
||||
| <a href="https://vllm-omni.readthedocs.io/en/latest/"><b>Documentation</b></a> | <a href="https://deepwiki.com/vllm-project/vllm-omni"><b>DeepWiki</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> | <a href="docs/assets/WeChat.jpg"><b>WeChat</b></a> | <a href="https://arxiv.org/abs/2602.02204"><b>Paper</b></a> | <a href="https://docs.google.com/presentation/d/1aPj0OGl_-ZVoib-Qne5dGDAlrRFB-PdHl6E-EE99g8E/edit?usp=sharing"><b>Slides</b></a> |
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
+1
-1
@@ -63,5 +63,5 @@ vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, in
|
||||
|
||||
For more information, checkout the following:
|
||||
|
||||
- [vllm-omni architecture design and recent roadmaps](https://docs.google.com/presentation/d/1XJWgv79lORl8rbaVvp2d5Sqs6ZEBgAgj/edit?slide=id.p1#slide=id.p1)
|
||||
- [vllm-omni architecture design and recent roadmaps](https://docs.google.com/presentation/d/1aPj0OGl_-ZVoib-Qne5dGDAlrRFB-PdHl6E-EE99g8E/edit?usp=sharing)
|
||||
- [vllm-omni announcement blogpost](https://blog.vllm.ai/2025/11/30/vllm-omni.html)
|
||||
@@ -1,6 +1,6 @@
|
||||
# Architecture Overview
|
||||
|
||||
This document outlines the architectural design for vLLM-Omni.
|
||||
This document outlines the architecture of vLLM-Omni.
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
@@ -9,194 +9,456 @@ This document outlines the architectural design for vLLM-Omni.
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
# Goals
|
||||
## Design goals
|
||||
|
||||
The primary goal of the vLLM-Omni project is to build the fastest and easiest-to-use open-source Omni-Modality model inference & serving engine. vLLM-Omni extends the original vLLM, which was created to support large language models for text-based autoregressive (AR) generation tasks. vLLM-Omni is designed to support:
|
||||
The primary goal of vLLM-Omni is to provide a fast and easy-to-use inference
|
||||
and serving engine for omni-modality models. vLLM-Omni extends vLLM's
|
||||
text-oriented autoregressive (AR) runtime with stage-based execution for
|
||||
non-textual outputs and non-autoregressive model components.
|
||||
|
||||
* **Non-textual Output:** Enables the integration, efficient processing and output of various data types, including but not limited to, images, audio, video, and action trajectories, alongside text.
|
||||
* **Non-Autoregressive Structure:** Support model structure beyond autoregressive, especially Diffusion Transformer (DiT), which is widely used in visual and audio generation.
|
||||
* **Integration with vLLM Core:** Maintain compatibility and leverage existing vLLM key modules and optimizations where applicable.
|
||||
* **Extensibility:** Design a modular and flexible architecture that can easily accommodate new modalities, model architectures, and output formats.
|
||||
The architecture is designed to:
|
||||
|
||||
* support text, image, audio, video, and action inputs and outputs;
|
||||
* compose autoregressive, generation, and diffusion stages in one pipeline;
|
||||
* reuse vLLM scheduling, cache, distributed-execution, and serving
|
||||
primitives where they fit; and
|
||||
* keep model topology, deployment placement, runtime lifecycle, transport,
|
||||
and public API concerns in separate layers.
|
||||
|
||||
# Representative omni-modality models
|
||||
## Model execution
|
||||
|
||||
According to analysis for current popular open-source models, most of them have the combination of AR+DiT. Specifically, they can be further categorized into 3 types below:
|
||||
These four examples show how vLLM-Omni maps model components to stage-local
|
||||
execution policies. Each summary covers the model-owned path, batching,
|
||||
attention, parallelism, quantization, and primary serving targets. The profiles
|
||||
are representative; a model may support multiple `PipelineConfig` and
|
||||
`DeployConfig` combinations.
|
||||
|
||||
**DiT as a main structure, with AR as text encoder (e.g.: Qwen-Image)**
|
||||
A powerful image generation foundation model capable of complex text rendering and precise image editing.
|
||||
| Model | Model-owned path | vLLM-Omni stage view | Primary serving target |
|
||||
| --- | --- | --- | --- |
|
||||
| Qwen3-Omni | Thinker → Talker → Code2Wav | Three stages with asynchronous chunk hand-off | TTFT/TTFP, TPOT, E2EL, audio RTF, and concurrent throughput |
|
||||
| HunyuanImage-3.0 | Multimodal AR understanding/reasoning plus image-generation DiT | AR-only, DiT-only, or split AR → DiT deployment | DiT E2EL/denoising latency and image throughput; TTFT/TPOT for AR tasks |
|
||||
| MiniMax-H3 | Text encoder → task-specific FL2VA or Ref2VA DiT → video/audio VAE decoder | Three logical component/phase boundaries within one registered diffusion stage | Video/audio E2EL and media throughput; RTF when audio streaming is measured |
|
||||
| Cosmos3 | Unified MoT reasoner tower plus diffusion generator tower | One diffusion pipeline materializes both towers inside the stage | Image/video/action E2EL and throughput; RTF for synchronized audio |
|
||||
|
||||
### Serving metric vocabulary
|
||||
|
||||
The target metric follows the output modality, not just the model name.
|
||||
|
||||
| Metric | Meaning in this document | Most useful for |
|
||||
| --- | --- | --- |
|
||||
| **TTFT** | Time to the first text token | AR reasoning or text output |
|
||||
| **TPOT** | Time per output token after the first token | AR decode stages |
|
||||
| **TTFP** | Time to the first streamed media packet, such as audio | Interactive speech and multimodal chat |
|
||||
| **E2EL** | End-to-end latency from request admission to the final output | Image, video, audio, and action requests |
|
||||
| **RTF** | Wall-clock processing time divided by generated audio duration; lower is faster than real time | Audio generation |
|
||||
| **Throughput** | Requests, tokens, or media seconds produced per wall-clock second | Offline and concurrent serving |
|
||||
|
||||
### Qwen3-Omni: streaming AR pipeline
|
||||
|
||||
Qwen3-Omni illustrates a multi-stage streaming AR pipeline: Thinker → Talker →
|
||||
Code2Wav. The [technical report](https://arxiv.org/abs/2509.17765) describes
|
||||
the model components, and the figure shows the stage-level streaming schedule
|
||||
used by vLLM-Omni.
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/dit-main-architecture.png">
|
||||
<img alt="Qwen-Image" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/dit-main-architecture.png" width=30%>
|
||||
</picture>
|
||||
<a href="https://github.com/vllm-project/vllm-omni/blob/main/docs/source/architecture/qwen3-omni-async-chunk.png">
|
||||
<img alt="Qwen3-Omni Thinker, Talker, and Code2Wav streaming stages" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/qwen3-omni-async-chunk.png" width="80%">
|
||||
</a>
|
||||
</p>
|
||||
|
||||
**AR as a main structure, with DiT as multi-modal generator (e.g. BAGEL)**
|
||||
A unified multimodal comprehension and generation model, with cot text output and visual generation.
|
||||
#### Model architecture by stage
|
||||
|
||||
| Stage | Model component | Stage output |
|
||||
| --- | --- | --- |
|
||||
| **Thinker** | AuT audio encoder, SigLIP2 vision encoder, and the 30B-total/3B-active Thinker MoE Transformer | Text tokens and conditioning for the Talker |
|
||||
| **Talker** | 3B-total/0.3B-active Talker MoE Transformer with the MTP code-prediction path | Text tokens and audio codec tokens |
|
||||
| **Code2Wav** | Approximately 200M-parameter ConvNet audio decoder | Waveform/audio chunks |
|
||||
|
||||
#### Representative stage execution
|
||||
|
||||
| Stage | Batching | Attention and execution | Parallelism | Quantization |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Thinker** | Continuous batching with stage-local `max_num_seqs` and token budgets | vLLM KV-cached AR attention; CUDA Graph execution when the stage is not eager | Independent stage placement; use the AR runtime's configured tensor parallelism | Supported ModelOpt FP8/NVFP4 or AutoRound checkpoint paths target the Thinker; encoders remain BF16 in the documented paths |
|
||||
| **Talker** | Continuous batching over autoregressive decode requests | KV-cached token decode with async-chunk output to Code2Wav | Independent stage placement; typically a separate GPU or replica from Thinker | BF16 baseline; no generic Talker quantization path is assumed |
|
||||
| **Code2Wav** | Static/chunk batching for codec-to-waveform decode | Non-AR chunked audio decode; eager execution is commonly retained | Separate stage or colocated placement, depending on the deployment | BF16 baseline; keep codec/vocoder precision model-specific |
|
||||
|
||||
For this pipeline, the main serving target is interactive responsiveness:
|
||||
TTFT for text, TTFP for the first audio packet, TPOT for autoregressive decode,
|
||||
E2EL for the complete answer, and RTF/throughput for sustained audio serving.
|
||||
Async chunking and streaming primarily reduce TTFP and improve stage overlap;
|
||||
batching and CUDA Graphs primarily improve E2EL and throughput. See the
|
||||
[Qwen3-Omni recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md)
|
||||
and [async chunk design](feature/async_chunk.md) for the deployment details.
|
||||
|
||||
### HunyuanImage-3.0: shared multimodal model, split deployment choices
|
||||
|
||||
HunyuanImage-3.0 illustrates a shared multimodal model with flexible AR, DiT,
|
||||
and split AR → DiT deployment. The official [technical
|
||||
report](https://arxiv.org/abs/2509.23951) provides the model view; the tables
|
||||
below map it to vLLM-Omni stages.
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/ar-main-architecture.png">
|
||||
<img alt="Bagel" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/ar-main-architecture.png" width=30%>
|
||||
</picture>
|
||||
<a href="https://github.com/Tencent-Hunyuan/HunyuanImage-3.0/blob/main/assets/framework.png">
|
||||
<img alt="HunyuanImage-3.0 framework for understanding, language modeling, and image generation" src="https://raw.githubusercontent.com/Tencent-Hunyuan/HunyuanImage-3.0/main/assets/framework.png" width="95%">
|
||||
</a>
|
||||
</p>
|
||||
|
||||
**AR+DiT (e.g. Qwen-Omni)**
|
||||
A natively end-to-end omni-modal LLM for multimodal inputs (text/image/audio/video...) and outputs (text/audio...).
|
||||
#### Model architecture by stage
|
||||
|
||||
| Stage or component | Model component | Stage output |
|
||||
| --- | --- | --- |
|
||||
| **AR understanding/reasoning** | Understanding and generation encoders feeding the Hunyuan-A13B decoder-only Transformer and text detokenizer | Text responses or image-generation conditioning |
|
||||
| **Image generation** | Generation encoder, diffusion prediction path/Gen. Decoder, and the image VAE decode path | Image latents and final images |
|
||||
|
||||
The vLLM-Omni deployment may expose the AR path only, the DiT path only, or
|
||||
both as an AR → DiT pipeline with a connector between them. That is a runtime
|
||||
decomposition of the model's capabilities; it should not be read as a claim
|
||||
that the published framework figure contains two unrelated backbones.
|
||||
|
||||
#### Representative stage execution
|
||||
|
||||
| Stage | Batching | Attention and execution | Parallelism | Quantization |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **AR** | Standard vLLM continuous batching when serving understanding/text tasks | KV-cached causal attention | Tensor/expert parallelism follows the selected AR deployment | Checkpoint-specific; do not infer DiT FP8 settings for the AR checkpoint |
|
||||
| **DiT + VAE** | Request batching or step-wise batching; Hunyuan step batching with more than one request requires `TORCH_SDPA` | `TORCH_SDPA` is used because the model mixes causal and full attention | Validated profiles include TP4, TP2 + sequence parallelism, TP2 + CFG parallelism, and expert parallelism for MoE weights | Online FP8 and ModelOpt mixed FP8/NVFP4 are documented for the DiT; VAE precision stays separate |
|
||||
|
||||
Image generation is therefore measured mainly with per-image E2EL/denoising
|
||||
latency, peak memory, and images-per-second throughput. TTFT and TPOT remain
|
||||
meaningful for the optional AR understanding path, but they are not the right
|
||||
headline metrics for the DiT path. See the
|
||||
[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md)
|
||||
for the validated batching, attention, and quantization combinations.
|
||||
|
||||
### MiniMax-H3: conditioning, denoising, and decode
|
||||
|
||||
MiniMax-H3 illustrates logical component and phase boundaries between text
|
||||
encoding, task-specific denoising, and media decoding:
|
||||
`Text Encoder → DiT → VAE Decoder`. The current registered topology co-locates
|
||||
these components in one diffusion stage. These boundaries are useful for
|
||||
profiling and offload; independent placement would require a future
|
||||
multi-stage `PipelineConfig`. The stage mapping follows the implementation and
|
||||
serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3)
|
||||
notes that a detailed technical report is forthcoming.
|
||||
|
||||
#### Model architecture by component
|
||||
|
||||
| Logical component | Model component | Component output |
|
||||
| --- | --- | --- |
|
||||
| **Text Encoder** | Tokenizer/processor and Qwen3-VL text encoder; vision/audio references are prepared alongside the text condition | Packed multimodal conditioning for denoising |
|
||||
| **DiT** | Task-specific `FL2VA` DiT for text/first-frame generation or `Ref2VA` DiT for mixed references | Video and audio latent tokens |
|
||||
| **VAE Decoder** | Video VAE and audio VAE decode the generated latents and assemble the synchronized output | Decoded video frames and stereo waveform, with FPS and audio sample rate metadata |
|
||||
|
||||
Both DiT partitions remain behind one task-selection point; `task` chooses
|
||||
which partition runs. In the current implementation, the three logical
|
||||
components execute inside one diffusion stage. This is different from
|
||||
Qwen3-Omni's three independently scheduled stages, even though both models
|
||||
produce synchronized audio and video-like outputs.
|
||||
|
||||
#### Representative component execution
|
||||
|
||||
| Logical component | Batching | Attention and execution | Parallelism | Quantization |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Text Encoder** | The current co-located pipeline follows the diffusion request scheduler; independent prompt/reference batching would require a separate multi-stage topology | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks; independent placement is not part of the current topology | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder |
|
||||
| **DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and Ulysses/VAE parallel groups on multi-GPU profiles; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears and is incompatible with layerwise offload |
|
||||
| **VAE Decoder** | Decode follows each generation request; tile/patch work can be distributed even when denoising is not batched | VAE decode kernels rather than DiT attention | VAE patch parallelism and native tiled decode within the diffusion stage | BF16/FP32 baseline; the documented H3 FP8 path leaves both VAEs unchanged |
|
||||
|
||||
The primary target is video/audio E2EL and media throughput, with the logical
|
||||
component boundaries making prompt encoding, denoising, and VAE decode costs
|
||||
separately measurable. RTF is useful when the audio stream is evaluated
|
||||
independently. TTFT and TPOT do not describe the main H3 generation path
|
||||
because the user-visible output is diffusion media rather than a token stream.
|
||||
The
|
||||
[MiniMax-H3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md)
|
||||
contains the hardware-specific profiles and warmup requirements.
|
||||
|
||||
### Cosmos3: unified MoT reasoner and generator
|
||||
|
||||
Cosmos3 illustrates a unified Mixture-of-Transformers (MoT) reasoner and
|
||||
diffusion generator. The official [technical
|
||||
report](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf)
|
||||
and NVIDIA's [technical explanation](https://developer.nvidia.com/blog/develop-physical-ai-reasoning-world-and-action-models-with-nvidia-cosmos-3/)
|
||||
provide the two-tower model view used below.
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/ar-dit-main-architecture.png">
|
||||
<img alt="Qwen-Omni" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/ar-dit-main-architecture.png" width=30%>
|
||||
</picture>
|
||||
<a href="https://developer.nvidia.com/blog/develop-physical-ai-reasoning-world-and-action-models-with-nvidia-cosmos-3/">
|
||||
<img alt="Cosmos3 autoregressive reasoner and diffusion generator towers" src="https://developer-blogs.nvidia.com/wp-content/uploads/2026/05/image-11-e1780000686151.webp" width="95%">
|
||||
</a>
|
||||
</p>
|
||||
|
||||
# vLLM-Omni main architecture
|
||||
#### Model architecture by stage
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/vllm-omni-main-architecture.png">
|
||||
<img alt="vLLM-Omni Main Architecture" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/vllm-omni-main-architecture.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
| Stage or component | Model component | Stage output |
|
||||
| --- | --- | --- |
|
||||
| **Reasoner tower** | Autoregressive VLM path with causal attention over language and optional vision/action context; runs once per generation request | Discrete reasoning/context tokens |
|
||||
| **Generator tower** | Diffusion path with full attention over the generator tokens and the reasoner K/V context; runs for each denoising step | Image, video, audio, or action tokens |
|
||||
| **Shared pipeline** | `Cosmos3OmniDiffusersPipeline` plus modality encoders/decoders and VAE paths | Final image/video/audio or action response |
|
||||
|
||||
## Key Components
|
||||
The reasoner and generator are model-owned components, not two independent
|
||||
vLLM-Omni stages by default. The current Cosmos3 pipeline materializes both
|
||||
inside one diffusion stage; model-level CPU offload can swap the reasoner and
|
||||
generator components between their phases.
|
||||
|
||||
| Component | Description |
|
||||
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **OmniRouter** | provide an intelligent router for Omni-modality requests dispatch |
|
||||
| **EntryPoints** | define the APIs for offline/online serving (APIServer, Omni/AsyncOmni), while `AsyncOmniEngine` and `Orchestrator` coordinate multi-stage AR/DiT execution |
|
||||
| **AR** | adapted for omni-modality models while inheriting efficient features from vLLM, such as cache management |
|
||||
| **Diffusion** | natively implemented and optimized using acceleration components |
|
||||
| **OmniConnector** | supports fully disaggregation based on E/P/D/G (Encoding/Processing/Decoding/Generation) disaggregation across stages |
|
||||
#### Representative stage execution
|
||||
|
||||
Disaggregated stages are managed through stage configuration. In Qwen3-Omni, Thinker/Talker/Code2wav are declared as separate configured stages, and runtime routing is handled by `Orchestrator` over `StageEngineCoreClient` / `StageDiffusionClient`.
|
||||
| Stage | Batching | Attention and execution | Parallelism | Quantization |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Reasoner + generator pipeline** | Use request batching or step execution only when the selected pipeline advertises that capability; otherwise use `max_num_seqs=1` | Causal attention for reasoner tokens and full attention for diffusion tokens; the validated recipe uses a platform-selected aiter FlashAttention path with `TORCH_SDPA` fallback | Ulysses/context parallelism or tensor parallelism for the transformer; VAE tiling and layerwise/model-level offload address activation and weight memory | Online FP8 is supported for the diffusion pipeline; AutoRound/checkpoint-specific paths and quality validation remain separate from the BF16 baseline |
|
||||
|
||||
For Cosmos3, generation E2EL and output throughput are the primary targets.
|
||||
Action-policy workloads add a control-loop latency target, while synchronized
|
||||
audio workloads add RTF. TTFT/TPOT are appropriate for a reasoner-only service,
|
||||
but not for the full diffusion generation path. See the
|
||||
[Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md)
|
||||
and [diffusion execution modes](../user_guide/diffusion/execution_modes.md)
|
||||
for the current runtime choices.
|
||||
|
||||
## Why the stage-based architecture follows from these examples
|
||||
|
||||
The four representative pipelines show that omni-modality serving is not one
|
||||
execution problem. Autoregressive decoding, diffusion denoising, multimodal
|
||||
encoding, and media decoding have different scheduling, attention, parallelism,
|
||||
memory, and quantization requirements. The architecture therefore needs a
|
||||
unit that can be optimized and resourced independently while remaining part of
|
||||
one request pipeline.
|
||||
|
||||
| Observation | Design implication | vLLM-Omni response |
|
||||
| --- | --- | --- |
|
||||
| AR, diffusion, encoder, and decoder components have different execution loops | One scheduler and one execution policy cannot fit every component | Specialized AR, generation, and diffusion stage runtimes |
|
||||
| A model can be deployed as AR-only, DiT-only, or AR → DiT | Model structure must be separate from deployment topology | `PipelineConfig` describes logical stages and relationships; `DeployConfig` describes placement and resources |
|
||||
| Text encoders, DiTs, and VAE decoders have different batch and memory behavior | Batching, attention, parallelism, and quantization must be stage-local | Each stage carries its own runtime and engine configuration |
|
||||
| MiniMax-H3 can co-locate or logically split its text encoder and VAE decoder | A stage boundary should be configurable rather than forcing a process boundary | Logical stages can be co-located or assigned independent runtime placement |
|
||||
| Qwen3-Omni transfers hidden states and codec chunks across stages | Large intermediate data needs typed transport and synchronization | `OmniConnector` transports payloads and KV-cache data without owning model routing |
|
||||
| Requests span multiple stages and may be cancelled or produce ordered streaming output | Cross-stage request state needs a separate control plane | `Orchestrator` owns correlation, cancellation, routing state, and output ordering |
|
||||
| Qwen3-Omni optimizes first-packet latency while image/video models optimize E2EL and throughput | Performance objectives must be model- and stage-aware | TTFT/TPOT/TTFP, E2EL, RTF, and throughput are measured according to the output modality |
|
||||
|
||||
This leads to a precise definition of a stage: it is a logical execution unit
|
||||
with its own model component, execution policy, resource ownership, and
|
||||
performance objectives. A stage is not necessarily a separate process. Cosmos3
|
||||
keeps its reasoner and generator inside one diffusion pipeline, while
|
||||
MiniMax-H3 exposes useful logical boundaries between text encoding, denoising,
|
||||
and VAE decoding. Both are valid mappings of the same abstraction.
|
||||
|
||||
The resulting separation can be summarized as:
|
||||
|
||||
```text
|
||||
Model structure What is computed and how components are related
|
||||
↓
|
||||
PipelineConfig Which logical stages exist and how they are connected
|
||||
↓
|
||||
Stage runtime policy How each stage batches, attends, parallelizes, and quantizes
|
||||
↓
|
||||
DeployConfig Where stages run, with which devices, replicas, and connectors
|
||||
```
|
||||
|
||||
`AsyncOmniEngine` and `Orchestrator` coordinate the pipeline, `OmniConnector`
|
||||
transports stage outputs, and `StageRuntime` materializes the selected
|
||||
placement. This separation lets one logical model support multiple serving
|
||||
topologies without coupling model code to process layout or deployment
|
||||
resources.
|
||||
|
||||
## System architecture
|
||||
|
||||
The figure shows the main components of the stage-based runtime: entrypoints,
|
||||
the shared `AsyncOmniEngine`, request orchestration, stage lifecycle
|
||||
management, AR and diffusion execution modules, model/layer operations,
|
||||
connector transport, and multimodal outputs. `AsyncOmniEngine` is the
|
||||
composition root between the entrypoints and the orchestrator.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph omni["vLLM-Omni"]
|
||||
subgraph entry["Entry points"]
|
||||
api["API server"]
|
||||
client["Omni / AsyncOmni"]
|
||||
end
|
||||
engine["AsyncOmniEngine<br/>composition root and background loop"]
|
||||
orchestrator["Orchestrator<br/>request state and cross-stage routing"]
|
||||
runtime["StageRuntime<br/>placement, replicas, readiness, and lifecycle"]
|
||||
ar["AR module<br/>vLLM engine, scheduler, cache, executor, worker, and model runner"]
|
||||
diffusion["Diffusion module<br/>scheduler, worker, model runner, and pipeline"]
|
||||
ops["Model / layer / ops<br/>attention, parallelism, and quantization"]
|
||||
connector["OmniConnector<br/>payload and KV transport plus synchronization"]
|
||||
outputs["Multimodal outputs<br/>streaming chunks and final artifacts"]
|
||||
|
||||
api --> engine
|
||||
client --> engine
|
||||
engine --> orchestrator
|
||||
engine --> runtime
|
||||
orchestrator --> runtime
|
||||
runtime --> ar
|
||||
runtime --> diffusion
|
||||
ar -. uses .-> ops
|
||||
diffusion -. uses .-> ops
|
||||
ar -. transports through .-> connector
|
||||
diffusion -. transports through .-> connector
|
||||
ar --> outputs
|
||||
diffusion --> outputs
|
||||
end
|
||||
```
|
||||
|
||||
### Key components
|
||||
|
||||
| Component | Responsibility |
|
||||
| --- | --- |
|
||||
| **Entrypoints** | Translate offline, CLI, OpenAI-compatible, and duplex requests into engine operations and render outputs back to public protocols. |
|
||||
| **Configuration resolution** | Combines pipeline topology, deployment settings, model metadata, and user overrides into a validated control-plane configuration. |
|
||||
| **AsyncOmniEngine** | Owns engine composition, the background event loop, stage initialization, request submission, and output collection. |
|
||||
| **Orchestrator** | Owns cross-stage request state, stage-to-stage routing, request correlation, cancellation, and output ordering. It does not own model selection or deployment placement. |
|
||||
| **StageRuntime** | Expands logical stages into local or distributed replicas, starts stage clients and processes, and manages readiness, affinity, failure, and shutdown. |
|
||||
| **AR runtime** | Extends vLLM's scheduler, KV-cache, worker, and model-runner path for omni-modality inputs and inter-stage outputs. |
|
||||
| **Diffusion runtime** | Schedules and executes denoising workloads through diffusion executors, workers, pipelines, acceleration backends, and output materialization. |
|
||||
| **OmniConnector** | Transports stage payloads and KV-cache data and provides synchronization. Connectors transport data; they do not choose the next logical stage. |
|
||||
| **Multimodal outputs** | `MultimodalPayload` separates tensor content from metadata, while `OmniRequestOutput` carries pipeline and diffusion results through the common output path. |
|
||||
|
||||
## Configuration and runtime resolution
|
||||
|
||||
The control plane has five conceptual layers. Authoring inputs are resolved
|
||||
once into a complete, transport-safe configuration before runtime processes are
|
||||
started. This keeps stage topology and model capabilities distinct from
|
||||
deployment placement and from process-local engine objects.
|
||||
|
||||
The figure shows only the primary hand-off object for each layer; the detailed
|
||||
inputs, fields, and ownership rules are described below.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
layer1["Layer 1 · Authoring inputs<br/>PipelineConfig + DeployConfig"]
|
||||
layer2["Layer 2 · Resolve once<br/>StageConfigFactory.create_from_model()<br/>VllmOmniConfig.from_pipeline_config()"]
|
||||
layer3["Layer 3 · Transport-safe control plane<br/>VllmOmniConfig"]
|
||||
layer4["Layer 4 · Runtime launch planning<br/>StageRuntime"]
|
||||
layer5["Layer 5 · Engine materialization<br/>VllmConfig / OmniDiffusionConfig"]
|
||||
|
||||
layer1 --> layer2 --> layer3 --> layer4 --> layer5
|
||||
```
|
||||
|
||||
The single resolution boundary is implemented by
|
||||
`StageConfigFactory.create_from_model()` and
|
||||
`VllmOmniConfig.from_pipeline_config()` in
|
||||
[`vllm_omni/config`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/config).
|
||||
The legacy `stage_args` YAML path remains only for models that have not yet
|
||||
migrated to `PipelineConfig` and `DeployConfig`.
|
||||
|
||||
In the typed path, each stage configuration derives from
|
||||
`BaseVllmOmniStageConfig` and is specialized as
|
||||
`VllmOmniARStageConfig`, `VllmOmniGenerationStageConfig`, or
|
||||
`VllmOmniDiffusionStageConfig`. The request and engine-spec fields belong to the
|
||||
control-plane boundary; the current implementation stores their equivalent
|
||||
projections in the structured stage configuration and materializes
|
||||
backend-specific engine objects during stage initialization.
|
||||
|
||||
The important ownership rules are:
|
||||
|
||||
1. `PipelineConfig` is the source of truth for stage topology, execution type,
|
||||
model capabilities, and stage relationships.
|
||||
2. `DeployConfig` describes placement, replicas, devices, connectors, and
|
||||
deploy-time defaults; it does not redefine the model graph.
|
||||
3. CLI and Python overrides are applied at the resolution boundary, with
|
||||
per-stage overrides taking precedence over global values where supported.
|
||||
4. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan`
|
||||
is runtime-private state, not a user configuration object.
|
||||
5. `VllmConfig` and the enriched `OmniDiffusionConfig` are materialized in the
|
||||
process that owns the corresponding engine.
|
||||
|
||||
## Main features
|
||||
|
||||
vLLM-Omni aims to be fast, flexible, and easy to use with the following features:
|
||||
The feature surface is grouped to match the
|
||||
[feature design documents](index.md#feature-design-documents). This page
|
||||
summarizes each feature's architectural role; the linked design document is
|
||||
the source for configuration, compatibility, and implementation details.
|
||||
|
||||
### Performance and Acceleration
|
||||
### Runtime and stage execution
|
||||
|
||||
The framework achieves high performance through several optimization techniques:
|
||||
* **Disaggregated inference:** Logical stages can run in separate processes,
|
||||
devices, or nodes while the orchestrator preserves their declared
|
||||
relationships. `OmniConnector` implementations transfer stage data and
|
||||
control-plane metadata. See [Disaggregated Inference](feature/disaggregated_inference.md).
|
||||
* **Asynchronous stage and output execution:** [Async Chunk](feature/async_chunk.md)
|
||||
forwards partial stage outputs as they become available. [Async Diffusion
|
||||
Output](feature/async_diffusion_output.md) overlaps device-to-host output
|
||||
packing with the next diffusion request, while [Async Omni Output
|
||||
Materialization](feature/omni_async_output_materialization.md) moves
|
||||
CPU-side payload construction off the AR decode critical path.
|
||||
* **Automatic prefix caching:** [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
|
||||
reuses KV-cache-aligned stage outputs and multimodal tensors for requests
|
||||
with common prefixes.
|
||||
|
||||
* **Efficient AR Support:** Leverages efficient KV cache management inherited from vLLM.
|
||||
* **Pipelined Execution:** Uses pipelined stage execution overlapping to ensure high throughput.
|
||||
* **Full Disaggregation:** Relies on the OmniConnector and dynamic resource allocation across stages.
|
||||
* **Diffusion Acceleration:** Includes integrated support for diffusion acceleration. This is managed by the acceleration layer, which handles:
|
||||
* **Cache:** Includes DBCache, TeaCache and third-party integration(e.g., [cache-dit](https://github.com/vipshop/cache-dit)).
|
||||
* **Parallelism:** Supports TP, CP, USP, and CFG.
|
||||
* **Attention:** Provides an interface for third-party integration (e.g., FA3, SAGE, MindIE-SD).
|
||||
* **Quantization:** Supports various quantization implementations including FP8 and AWQ.
|
||||
* **FusedOps:** Allows for custom and third-party integration.
|
||||
### Communication
|
||||
|
||||
### Classifier-Free Guidance (CFG) Companion Flow
|
||||
* **OmniConnector transport:** The connector contract carries tensors, KV-cache
|
||||
data, and transport metadata across stage boundaries. The available
|
||||
implementations cover shared memory and multi-node Mooncake, Mori, and
|
||||
Yuanrong transports; see [Disaggregated Inference](feature/disaggregated_inference.md)
|
||||
for the connector choices and configuration model.
|
||||
|
||||
vLLM-Omni natively models Classifier-Free Guidance (CFG) across disaggregated multi-stage setups via a "companion request" paradigm, eliminating redundant textual/multimodal context computation boundaries:
|
||||
1. **Prompt Expansion:** In the initial autoregressive (AR) stage, a customized `prompt_expand_func` hook intercepts incoming generation prompts and pairs them directly with negative companion prompts (e.g., a default negative prompt) on the fly, tagging the secondary prompt with a specific internal role (`cfg_text`).
|
||||
2. **Synchronized KV Cache Transfer:** The AR stage evaluates both the primary and companion sequence batches concurrently. The `OmniConnector` captures these specific structural dependencies and reliably passes the positive and negative outcome KV caches seamlessly across stage boundaries via shared memory or network protocols.
|
||||
3. **KV Cache Collection & Injection:** Upon reaching the downstream Diffusion (DiT) Engine, an assigned `cfg_kv_collect_func` automatically intercepts the mapped companion caches (`cfg_text_past_key_values`). These auxiliary dependencies are natively gathered and seamlessly bound to the primary generation sequence variables, enabling the DiT Engine to cleanly implement cross-attention CFG guidance over accurate conditioning and unconditioning structures in parallel.
|
||||
### Diffusion acceleration
|
||||
|
||||
### Flexibility and Usability
|
||||
* **Request and step batching:** [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
|
||||
defines request-batch and step-batch execution, scheduler admission, and the
|
||||
common streaming output path.
|
||||
* **Composable parallelism:** Diffusion stages can combine [CFG-Parallel](feature/cfg_parallel.md),
|
||||
[Expert Parallel](feature/expert_parallel.md), [HSDP](feature/hsdp.md),
|
||||
[Pipeline Parallel](feature/pipeline_parallel.md), [Sequence Parallel](feature/sequence_parallel.md),
|
||||
[Tensor Parallel](feature/tensor_parallel.md), and [VAE Patch Parallelism](feature/vae_parallel.md)
|
||||
according to the pipeline and hardware topology.
|
||||
* **Attention and cache acceleration:** [Skip-Softmax](feature/skip_softmax.md),
|
||||
[Cache-DiT](feature/cache_dit.md), and [TeaCache](feature/teacache.md)
|
||||
provide backend and denoising-step optimizations without changing the
|
||||
stage contract.
|
||||
* **Quantization and memory efficiency:** [Quantization](feature/quantization.md)
|
||||
resolves per-pipeline or per-component quantization configurations, while
|
||||
[Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
|
||||
streams diffusion blocks from host memory within the existing parallel
|
||||
topology.
|
||||
|
||||
vLLM-Omni is designed to be flexible and straightforward for users:
|
||||
## Interfaces
|
||||
|
||||
* **Heterogeneous Pipeline Abstraction:** Manages complex model workflows effectively.
|
||||
* **Hugging Face Integration:** Offers seamless integration with popular Hugging Face models.
|
||||
* **Distributed Inference:** Supports tensor, pipeline, data, and expert parallelism.
|
||||
* **Streaming Outputs:** Supports streaming outputs.
|
||||
* **Unified API:** Provides a consistent and unified API interface compatible with vLLM.
|
||||
* **OpenAI-compatible API Server:** Includes a FastAPI-based server for online serving that is compatible with the OpenAI API.
|
||||
|
||||
# Interface design
|
||||
|
||||
If you use vLLM, then you know how to use vLLM-Omni from Day 0:
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/vllm-omni-user-interface.png">
|
||||
<img alt="vLLM-Omni interface design" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/docs/source/architecture/vllm-omni-user-interface.png" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
Taking **Qwen3-Omni** as an example:
|
||||
|
||||
## Offline Inference
|
||||
The **Omni** class provides a Python interface for offline batched inference. Users initialize the Omni class with a Hugging Face model name and use the generate method, passing inputs that include both text prompts and multi-modal data:
|
||||
The public interfaces map onto the same engine and stage boundaries:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
offline["Offline Python<br/>Omni.generate()"] --> engine["AsyncOmniEngine"]
|
||||
online["OpenAI-compatible serving<br/>vllm serve ... --omni"] --> engine
|
||||
duplex["Experimental duplex WebSocket<br/>/v1/duplex or realtime duplex"] --> engine
|
||||
engine --> stages["Configured AR and diffusion stages"]
|
||||
stages --> result["Streaming or final multimodal output"]
|
||||
```
|
||||
# Create an omni runtime with HF model name.
|
||||
|
||||
### Offline inference
|
||||
|
||||
The **Omni** class provides a Python interface for offline batched inference:
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
|
||||
omni = Omni(model="Qwen/Qwen3-Omni-30B-A3B-Instruct")
|
||||
|
||||
# Example prompts.
|
||||
om_inputs = {"prompt": prompt,
|
||||
"multi_modal_data": {
|
||||
"video": video_frames,
|
||||
"audio": audio_signal,
|
||||
}}
|
||||
om_inputs = {
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {
|
||||
"video": video_frames,
|
||||
"audio": audio_signal,
|
||||
},
|
||||
}
|
||||
|
||||
# Generate texts and audio from the multi-modality inputs.
|
||||
outputs = omni.generate(om_inputs, sampling_params_list)
|
||||
```
|
||||
|
||||
## Online Serving
|
||||
Similar to vLLM, vLLM-Omni also provides a FastAPI-based server for online serving. Users can launch the server using the vllm serve command with the `--omni` flag:
|
||||
### Online serving
|
||||
|
||||
```
|
||||
The OpenAI-compatible server uses the same stage configuration and engine
|
||||
boundaries:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
Users can send requests to the server using curl:
|
||||
For example, a Qwen3-Omni chat request can contain text, image, audio, or
|
||||
video content and a `sampling_params_list` for its configured stages. See the
|
||||
[Qwen3-Omni serving example](../user_guide/examples/online_serving/qwen3_omni.md)
|
||||
and the [examples](https://github.com/vllm-project/vllm-omni/tree/main/examples)
|
||||
for complete requests.
|
||||
|
||||
```
|
||||
# prepare user content
|
||||
user_content='[
|
||||
{
|
||||
"type": "video_url",
|
||||
"video_url": {
|
||||
"url": "'"$SAMPLE_VIDEO_URL"'"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Why is this video funny?"
|
||||
}
|
||||
]'
|
||||
sampling_params_list='[
|
||||
'"$thinker_sampling_params"',
|
||||
'"$talker_sampling_params"',
|
||||
'"$code2wav_sampling_params"'
|
||||
]'
|
||||
mm_processor_kwargs="{}"
|
||||
|
||||
# send the request
|
||||
curl -sS -X POST http://localhost:8091/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @- <<EOF
|
||||
{
|
||||
"model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
|
||||
"sampling_params_list": $sampling_params_list,
|
||||
"mm_processor_kwargs": $mm_processor_kwargs,
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": $user_content
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
For more usages, please refer to [examples](https://github.com/vllm-project/vllm-omni/tree/main/examples).
|
||||
Some pipelines expose additional OpenAI-compatible endpoints, such as joint
|
||||
video/audio generation. Endpoint support remains model-specific; consult the
|
||||
relevant model guide before assuming that every OpenAI route applies to every
|
||||
pipeline.
|
||||
Reference in new issue
Block a user