From 9159cedbd7ce71e8d29899867e68524ead6c03bc Mon Sep 17 00:00:00 2001 From: Hongsheng Liu Date: Sun, 9 Aug 2026 09:16:18 +0800 Subject: [PATCH] docs: update vLLM-Omni architecture overview (#5914) Signed-off-by: Hongsheng Liu --- README.md | 2 +- docs/README.md | 2 +- docs/design/architecture_overview.md | 550 ++++++++++++++++++++------- 3 files changed, 408 insertions(+), 146 deletions(-) diff --git a/README.md b/README.md index 9d63a4b1e..418d0b38d 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Easy, fast, and cheap omni-modality model serving for everyone

-| Documentation | DeepWiki | User Forum | Developer Slack | WeChat | Paper | Slides | +| Documentation | DeepWiki | User Forum | Developer Slack | WeChat | Paper | Slides |

diff --git a/docs/README.md b/docs/README.md index f3dfe70c2..39df4e6a0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -63,5 +63,5 @@ vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, in For more information, checkout the following: -- [vllm-omni architecture design and recent roadmaps](https://docs.google.com/presentation/d/1XJWgv79lORl8rbaVvp2d5Sqs6ZEBgAgj/edit?slide=id.p1#slide=id.p1) +- [vllm-omni architecture design and recent roadmaps](https://docs.google.com/presentation/d/1aPj0OGl_-ZVoib-Qne5dGDAlrRFB-PdHl6E-EE99g8E/edit?usp=sharing) - [vllm-omni announcement blogpost](https://blog.vllm.ai/2025/11/30/vllm-omni.html) diff --git a/docs/design/architecture_overview.md b/docs/design/architecture_overview.md index 20c99fd9c..aa6f72770 100644 --- a/docs/design/architecture_overview.md +++ b/docs/design/architecture_overview.md @@ -1,6 +1,6 @@ # Architecture Overview -This document outlines the architectural design for vLLM-Omni. +This document outlines the architecture of vLLM-Omni.

@@ -9,194 +9,456 @@ This document outlines the architectural design for vLLM-Omni.

-# Goals +## Design goals -The primary goal of the vLLM-Omni project is to build the fastest and easiest-to-use open-source Omni-Modality model inference & serving engine. vLLM-Omni extends the original vLLM, which was created to support large language models for text-based autoregressive (AR) generation tasks. vLLM-Omni is designed to support: +The primary goal of vLLM-Omni is to provide a fast and easy-to-use inference +and serving engine for omni-modality models. vLLM-Omni extends vLLM's +text-oriented autoregressive (AR) runtime with stage-based execution for +non-textual outputs and non-autoregressive model components. -* **Non-textual Output:** Enables the integration, efficient processing and output of various data types, including but not limited to, images, audio, video, and action trajectories, alongside text. -* **Non-Autoregressive Structure:** Support model structure beyond autoregressive, especially Diffusion Transformer (DiT), which is widely used in visual and audio generation. -* **Integration with vLLM Core:** Maintain compatibility and leverage existing vLLM key modules and optimizations where applicable. -* **Extensibility:** Design a modular and flexible architecture that can easily accommodate new modalities, model architectures, and output formats. +The architecture is designed to: +* support text, image, audio, video, and action inputs and outputs; +* compose autoregressive, generation, and diffusion stages in one pipeline; +* reuse vLLM scheduling, cache, distributed-execution, and serving + primitives where they fit; and +* keep model topology, deployment placement, runtime lifecycle, transport, + and public API concerns in separate layers. -# Representative omni-modality models +## Model execution -According to analysis for current popular open-source models, most of them have the combination of AR+DiT. Specifically, they can be further categorized into 3 types below: +These four examples show how vLLM-Omni maps model components to stage-local +execution policies. Each summary covers the model-owned path, batching, +attention, parallelism, quantization, and primary serving targets. The profiles +are representative; a model may support multiple `PipelineConfig` and +`DeployConfig` combinations. -**DiT as a main structure, with AR as text encoder (e.g.: Qwen-Image)** - A powerful image generation foundation model capable of complex text rendering and precise image editing. +| Model | Model-owned path | vLLM-Omni stage view | Primary serving target | +| --- | --- | --- | --- | +| Qwen3-Omni | Thinker → Talker → Code2Wav | Three stages with asynchronous chunk hand-off | TTFT/TTFP, TPOT, E2EL, audio RTF, and concurrent throughput | +| HunyuanImage-3.0 | Multimodal AR understanding/reasoning plus image-generation DiT | AR-only, DiT-only, or split AR → DiT deployment | DiT E2EL/denoising latency and image throughput; TTFT/TPOT for AR tasks | +| MiniMax-H3 | Text encoder → task-specific FL2VA or Ref2VA DiT → video/audio VAE decoder | Three logical component/phase boundaries within one registered diffusion stage | Video/audio E2EL and media throughput; RTF when audio streaming is measured | +| Cosmos3 | Unified MoT reasoner tower plus diffusion generator tower | One diffusion pipeline materializes both towers inside the stage | Image/video/action E2EL and throughput; RTF for synchronized audio | + +### Serving metric vocabulary + +The target metric follows the output modality, not just the model name. + +| Metric | Meaning in this document | Most useful for | +| --- | --- | --- | +| **TTFT** | Time to the first text token | AR reasoning or text output | +| **TPOT** | Time per output token after the first token | AR decode stages | +| **TTFP** | Time to the first streamed media packet, such as audio | Interactive speech and multimodal chat | +| **E2EL** | End-to-end latency from request admission to the final output | Image, video, audio, and action requests | +| **RTF** | Wall-clock processing time divided by generated audio duration; lower is faster than real time | Audio generation | +| **Throughput** | Requests, tokens, or media seconds produced per wall-clock second | Offline and concurrent serving | + +### Qwen3-Omni: streaming AR pipeline + +Qwen3-Omni illustrates a multi-stage streaming AR pipeline: Thinker → Talker → +Code2Wav. The [technical report](https://arxiv.org/abs/2509.17765) describes +the model components, and the figure shows the stage-level streaming schedule +used by vLLM-Omni.

- - - Qwen-Image - + + Qwen3-Omni Thinker, Talker, and Code2Wav streaming stages +

-**AR as a main structure, with DiT as multi-modal generator (e.g. BAGEL)** - A unified multimodal comprehension and generation model, with cot text output and visual generation. +#### Model architecture by stage + +| Stage | Model component | Stage output | +| --- | --- | --- | +| **Thinker** | AuT audio encoder, SigLIP2 vision encoder, and the 30B-total/3B-active Thinker MoE Transformer | Text tokens and conditioning for the Talker | +| **Talker** | 3B-total/0.3B-active Talker MoE Transformer with the MTP code-prediction path | Text tokens and audio codec tokens | +| **Code2Wav** | Approximately 200M-parameter ConvNet audio decoder | Waveform/audio chunks | + +#### Representative stage execution + +| Stage | Batching | Attention and execution | Parallelism | Quantization | +| --- | --- | --- | --- | --- | +| **Thinker** | Continuous batching with stage-local `max_num_seqs` and token budgets | vLLM KV-cached AR attention; CUDA Graph execution when the stage is not eager | Independent stage placement; use the AR runtime's configured tensor parallelism | Supported ModelOpt FP8/NVFP4 or AutoRound checkpoint paths target the Thinker; encoders remain BF16 in the documented paths | +| **Talker** | Continuous batching over autoregressive decode requests | KV-cached token decode with async-chunk output to Code2Wav | Independent stage placement; typically a separate GPU or replica from Thinker | BF16 baseline; no generic Talker quantization path is assumed | +| **Code2Wav** | Static/chunk batching for codec-to-waveform decode | Non-AR chunked audio decode; eager execution is commonly retained | Separate stage or colocated placement, depending on the deployment | BF16 baseline; keep codec/vocoder precision model-specific | + +For this pipeline, the main serving target is interactive responsiveness: +TTFT for text, TTFP for the first audio packet, TPOT for autoregressive decode, +E2EL for the complete answer, and RTF/throughput for sustained audio serving. +Async chunking and streaming primarily reduce TTFP and improve stage overlap; +batching and CUDA Graphs primarily improve E2EL and throughput. See the +[Qwen3-Omni recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) +and [async chunk design](feature/async_chunk.md) for the deployment details. + +### HunyuanImage-3.0: shared multimodal model, split deployment choices + +HunyuanImage-3.0 illustrates a shared multimodal model with flexible AR, DiT, +and split AR → DiT deployment. The official [technical +report](https://arxiv.org/abs/2509.23951) provides the model view; the tables +below map it to vLLM-Omni stages.

- - - Bagel - + + HunyuanImage-3.0 framework for understanding, language modeling, and image generation +

-**AR+DiT (e.g. Qwen-Omni)** - A natively end-to-end omni-modal LLM for multimodal inputs (text/image/audio/video...) and outputs (text/audio...). +#### Model architecture by stage + +| Stage or component | Model component | Stage output | +| --- | --- | --- | +| **AR understanding/reasoning** | Understanding and generation encoders feeding the Hunyuan-A13B decoder-only Transformer and text detokenizer | Text responses or image-generation conditioning | +| **Image generation** | Generation encoder, diffusion prediction path/Gen. Decoder, and the image VAE decode path | Image latents and final images | + +The vLLM-Omni deployment may expose the AR path only, the DiT path only, or +both as an AR → DiT pipeline with a connector between them. That is a runtime +decomposition of the model's capabilities; it should not be read as a claim +that the published framework figure contains two unrelated backbones. + +#### Representative stage execution + +| Stage | Batching | Attention and execution | Parallelism | Quantization | +| --- | --- | --- | --- | --- | +| **AR** | Standard vLLM continuous batching when serving understanding/text tasks | KV-cached causal attention | Tensor/expert parallelism follows the selected AR deployment | Checkpoint-specific; do not infer DiT FP8 settings for the AR checkpoint | +| **DiT + VAE** | Request batching or step-wise batching; Hunyuan step batching with more than one request requires `TORCH_SDPA` | `TORCH_SDPA` is used because the model mixes causal and full attention | Validated profiles include TP4, TP2 + sequence parallelism, TP2 + CFG parallelism, and expert parallelism for MoE weights | Online FP8 and ModelOpt mixed FP8/NVFP4 are documented for the DiT; VAE precision stays separate | + +Image generation is therefore measured mainly with per-image E2EL/denoising +latency, peak memory, and images-per-second throughput. TTFT and TPOT remain +meaningful for the optional AR understanding path, but they are not the right +headline metrics for the DiT path. See the +[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md) +for the validated batching, attention, and quantization combinations. + +### MiniMax-H3: conditioning, denoising, and decode + +MiniMax-H3 illustrates logical component and phase boundaries between text +encoding, task-specific denoising, and media decoding: +`Text Encoder → DiT → VAE Decoder`. The current registered topology co-locates +these components in one diffusion stage. These boundaries are useful for +profiling and offload; independent placement would require a future +multi-stage `PipelineConfig`. The stage mapping follows the implementation and +serving recipe; the [release overview](https://minimaxi.com/blog/minimax-h3) +notes that a detailed technical report is forthcoming. + +#### Model architecture by component + +| Logical component | Model component | Component output | +| --- | --- | --- | +| **Text Encoder** | Tokenizer/processor and Qwen3-VL text encoder; vision/audio references are prepared alongside the text condition | Packed multimodal conditioning for denoising | +| **DiT** | Task-specific `FL2VA` DiT for text/first-frame generation or `Ref2VA` DiT for mixed references | Video and audio latent tokens | +| **VAE Decoder** | Video VAE and audio VAE decode the generated latents and assemble the synchronized output | Decoded video frames and stereo waveform, with FPS and audio sample rate metadata | + +Both DiT partitions remain behind one task-selection point; `task` chooses +which partition runs. In the current implementation, the three logical +components execute inside one diffusion stage. This is different from +Qwen3-Omni's three independently scheduled stages, even though both models +produce synchronized audio and video-like outputs. + +#### Representative component execution + +| Logical component | Batching | Attention and execution | Parallelism | Quantization | +| --- | --- | --- | --- | --- | +| **Text Encoder** | The current co-located pipeline follows the diffusion request scheduler; independent prompt/reference batching would require a separate multi-stage topology | Qwen3-VL attention and multimodal preprocessing | `--text-encoder-tp-size N` shards the encoder across the first `N` DiT ranks; independent placement is not part of the current topology | BF16/FP32 baseline; the H3 DiT FP8 path does not quantize the text encoder | +| **DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and Ulysses/VAE parallel groups on multi-GPU profiles; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears and is incompatible with layerwise offload | +| **VAE Decoder** | Decode follows each generation request; tile/patch work can be distributed even when denoising is not batched | VAE decode kernels rather than DiT attention | VAE patch parallelism and native tiled decode within the diffusion stage | BF16/FP32 baseline; the documented H3 FP8 path leaves both VAEs unchanged | + +The primary target is video/audio E2EL and media throughput, with the logical +component boundaries making prompt encoding, denoising, and VAE decode costs +separately measurable. RTF is useful when the audio stream is evaluated +independently. TTFT and TPOT do not describe the main H3 generation path +because the user-visible output is diffusion media rather than a token stream. +The +[MiniMax-H3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md) +contains the hardware-specific profiles and warmup requirements. + +### Cosmos3: unified MoT reasoner and generator + +Cosmos3 illustrates a unified Mixture-of-Transformers (MoT) reasoner and +diffusion generator. The official [technical +report](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) +and NVIDIA's [technical explanation](https://developer.nvidia.com/blog/develop-physical-ai-reasoning-world-and-action-models-with-nvidia-cosmos-3/) +provide the two-tower model view used below.

- - - Qwen-Omni - + + Cosmos3 autoregressive reasoner and diffusion generator towers +

-# vLLM-Omni main architecture +#### Model architecture by stage -

- - - vLLM-Omni Main Architecture - -

+| Stage or component | Model component | Stage output | +| --- | --- | --- | +| **Reasoner tower** | Autoregressive VLM path with causal attention over language and optional vision/action context; runs once per generation request | Discrete reasoning/context tokens | +| **Generator tower** | Diffusion path with full attention over the generator tokens and the reasoner K/V context; runs for each denoising step | Image, video, audio, or action tokens | +| **Shared pipeline** | `Cosmos3OmniDiffusersPipeline` plus modality encoders/decoders and VAE paths | Final image/video/audio or action response | -## Key Components +The reasoner and generator are model-owned components, not two independent +vLLM-Omni stages by default. The current Cosmos3 pipeline materializes both +inside one diffusion stage; model-level CPU offload can swap the reasoner and +generator components between their phases. -| Component | Description | -| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -| **OmniRouter** | provide an intelligent router for Omni-modality requests dispatch | -| **EntryPoints** | define the APIs for offline/online serving (APIServer, Omni/AsyncOmni), while `AsyncOmniEngine` and `Orchestrator` coordinate multi-stage AR/DiT execution | -| **AR** | adapted for omni-modality models while inheriting efficient features from vLLM, such as cache management | -| **Diffusion** | natively implemented and optimized using acceleration components | -| **OmniConnector** | supports fully disaggregation based on E/P/D/G (Encoding/Processing/Decoding/Generation) disaggregation across stages | +#### Representative stage execution -Disaggregated stages are managed through stage configuration. In Qwen3-Omni, Thinker/Talker/Code2wav are declared as separate configured stages, and runtime routing is handled by `Orchestrator` over `StageEngineCoreClient` / `StageDiffusionClient`. +| Stage | Batching | Attention and execution | Parallelism | Quantization | +| --- | --- | --- | --- | --- | +| **Reasoner + generator pipeline** | Use request batching or step execution only when the selected pipeline advertises that capability; otherwise use `max_num_seqs=1` | Causal attention for reasoner tokens and full attention for diffusion tokens; the validated recipe uses a platform-selected aiter FlashAttention path with `TORCH_SDPA` fallback | Ulysses/context parallelism or tensor parallelism for the transformer; VAE tiling and layerwise/model-level offload address activation and weight memory | Online FP8 is supported for the diffusion pipeline; AutoRound/checkpoint-specific paths and quality validation remain separate from the BF16 baseline | + +For Cosmos3, generation E2EL and output throughput are the primary targets. +Action-policy workloads add a control-loop latency target, while synchronized +audio workloads add RTF. TTFT/TPOT are appropriate for a reasoner-only service, +but not for the full diffusion generation path. See the +[Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) +and [diffusion execution modes](../user_guide/diffusion/execution_modes.md) +for the current runtime choices. + +## Why the stage-based architecture follows from these examples + +The four representative pipelines show that omni-modality serving is not one +execution problem. Autoregressive decoding, diffusion denoising, multimodal +encoding, and media decoding have different scheduling, attention, parallelism, +memory, and quantization requirements. The architecture therefore needs a +unit that can be optimized and resourced independently while remaining part of +one request pipeline. + +| Observation | Design implication | vLLM-Omni response | +| --- | --- | --- | +| AR, diffusion, encoder, and decoder components have different execution loops | One scheduler and one execution policy cannot fit every component | Specialized AR, generation, and diffusion stage runtimes | +| A model can be deployed as AR-only, DiT-only, or AR → DiT | Model structure must be separate from deployment topology | `PipelineConfig` describes logical stages and relationships; `DeployConfig` describes placement and resources | +| Text encoders, DiTs, and VAE decoders have different batch and memory behavior | Batching, attention, parallelism, and quantization must be stage-local | Each stage carries its own runtime and engine configuration | +| MiniMax-H3 can co-locate or logically split its text encoder and VAE decoder | A stage boundary should be configurable rather than forcing a process boundary | Logical stages can be co-located or assigned independent runtime placement | +| Qwen3-Omni transfers hidden states and codec chunks across stages | Large intermediate data needs typed transport and synchronization | `OmniConnector` transports payloads and KV-cache data without owning model routing | +| Requests span multiple stages and may be cancelled or produce ordered streaming output | Cross-stage request state needs a separate control plane | `Orchestrator` owns correlation, cancellation, routing state, and output ordering | +| Qwen3-Omni optimizes first-packet latency while image/video models optimize E2EL and throughput | Performance objectives must be model- and stage-aware | TTFT/TPOT/TTFP, E2EL, RTF, and throughput are measured according to the output modality | + +This leads to a precise definition of a stage: it is a logical execution unit +with its own model component, execution policy, resource ownership, and +performance objectives. A stage is not necessarily a separate process. Cosmos3 +keeps its reasoner and generator inside one diffusion pipeline, while +MiniMax-H3 exposes useful logical boundaries between text encoding, denoising, +and VAE decoding. Both are valid mappings of the same abstraction. + +The resulting separation can be summarized as: + +```text +Model structure What is computed and how components are related + ↓ +PipelineConfig Which logical stages exist and how they are connected + ↓ +Stage runtime policy How each stage batches, attends, parallelizes, and quantizes + ↓ +DeployConfig Where stages run, with which devices, replicas, and connectors +``` + +`AsyncOmniEngine` and `Orchestrator` coordinate the pipeline, `OmniConnector` +transports stage outputs, and `StageRuntime` materializes the selected +placement. This separation lets one logical model support multiple serving +topologies without coupling model code to process layout or deployment +resources. + +## System architecture + +The figure shows the main components of the stage-based runtime: entrypoints, +the shared `AsyncOmniEngine`, request orchestration, stage lifecycle +management, AR and diffusion execution modules, model/layer operations, +connector transport, and multimodal outputs. `AsyncOmniEngine` is the +composition root between the entrypoints and the orchestrator. + +```mermaid +flowchart TB + subgraph omni["vLLM-Omni"] + subgraph entry["Entry points"] + api["API server"] + client["Omni / AsyncOmni"] + end + engine["AsyncOmniEngine
composition root and background loop"] + orchestrator["Orchestrator
request state and cross-stage routing"] + runtime["StageRuntime
placement, replicas, readiness, and lifecycle"] + ar["AR module
vLLM engine, scheduler, cache, executor, worker, and model runner"] + diffusion["Diffusion module
scheduler, worker, model runner, and pipeline"] + ops["Model / layer / ops
attention, parallelism, and quantization"] + connector["OmniConnector
payload and KV transport plus synchronization"] + outputs["Multimodal outputs
streaming chunks and final artifacts"] + + api --> engine + client --> engine + engine --> orchestrator + engine --> runtime + orchestrator --> runtime + runtime --> ar + runtime --> diffusion + ar -. uses .-> ops + diffusion -. uses .-> ops + ar -. transports through .-> connector + diffusion -. transports through .-> connector + ar --> outputs + diffusion --> outputs + end +``` + +### Key components + +| Component | Responsibility | +| --- | --- | +| **Entrypoints** | Translate offline, CLI, OpenAI-compatible, and duplex requests into engine operations and render outputs back to public protocols. | +| **Configuration resolution** | Combines pipeline topology, deployment settings, model metadata, and user overrides into a validated control-plane configuration. | +| **AsyncOmniEngine** | Owns engine composition, the background event loop, stage initialization, request submission, and output collection. | +| **Orchestrator** | Owns cross-stage request state, stage-to-stage routing, request correlation, cancellation, and output ordering. It does not own model selection or deployment placement. | +| **StageRuntime** | Expands logical stages into local or distributed replicas, starts stage clients and processes, and manages readiness, affinity, failure, and shutdown. | +| **AR runtime** | Extends vLLM's scheduler, KV-cache, worker, and model-runner path for omni-modality inputs and inter-stage outputs. | +| **Diffusion runtime** | Schedules and executes denoising workloads through diffusion executors, workers, pipelines, acceleration backends, and output materialization. | +| **OmniConnector** | Transports stage payloads and KV-cache data and provides synchronization. Connectors transport data; they do not choose the next logical stage. | +| **Multimodal outputs** | `MultimodalPayload` separates tensor content from metadata, while `OmniRequestOutput` carries pipeline and diffusion results through the common output path. | + +## Configuration and runtime resolution + +The control plane has five conceptual layers. Authoring inputs are resolved +once into a complete, transport-safe configuration before runtime processes are +started. This keeps stage topology and model capabilities distinct from +deployment placement and from process-local engine objects. + +The figure shows only the primary hand-off object for each layer; the detailed +inputs, fields, and ownership rules are described below. + +```mermaid +flowchart TB + layer1["Layer 1 · Authoring inputs
PipelineConfig + DeployConfig"] + layer2["Layer 2 · Resolve once
StageConfigFactory.create_from_model()
VllmOmniConfig.from_pipeline_config()"] + layer3["Layer 3 · Transport-safe control plane
VllmOmniConfig"] + layer4["Layer 4 · Runtime launch planning
StageRuntime"] + layer5["Layer 5 · Engine materialization
VllmConfig / OmniDiffusionConfig"] + + layer1 --> layer2 --> layer3 --> layer4 --> layer5 +``` + +The single resolution boundary is implemented by +`StageConfigFactory.create_from_model()` and +`VllmOmniConfig.from_pipeline_config()` in +[`vllm_omni/config`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/config). +The legacy `stage_args` YAML path remains only for models that have not yet +migrated to `PipelineConfig` and `DeployConfig`. + +In the typed path, each stage configuration derives from +`BaseVllmOmniStageConfig` and is specialized as +`VllmOmniARStageConfig`, `VllmOmniGenerationStageConfig`, or +`VllmOmniDiffusionStageConfig`. The request and engine-spec fields belong to the +control-plane boundary; the current implementation stores their equivalent +projections in the structured stage configuration and materializes +backend-specific engine objects during stage initialization. + +The important ownership rules are: + +1. `PipelineConfig` is the source of truth for stage topology, execution type, + model capabilities, and stage relationships. +2. `DeployConfig` describes placement, replicas, devices, connectors, and + deploy-time defaults; it does not redefine the model graph. +3. CLI and Python overrides are applied at the resolution boundary, with + per-stage overrides taking precedence over global values where supported. +4. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan` + is runtime-private state, not a user configuration object. +5. `VllmConfig` and the enriched `OmniDiffusionConfig` are materialized in the + process that owns the corresponding engine. ## Main features -vLLM-Omni aims to be fast, flexible, and easy to use with the following features: +The feature surface is grouped to match the +[feature design documents](index.md#feature-design-documents). This page +summarizes each feature's architectural role; the linked design document is +the source for configuration, compatibility, and implementation details. -### Performance and Acceleration +### Runtime and stage execution -The framework achieves high performance through several optimization techniques: +* **Disaggregated inference:** Logical stages can run in separate processes, + devices, or nodes while the orchestrator preserves their declared + relationships. `OmniConnector` implementations transfer stage data and + control-plane metadata. See [Disaggregated Inference](feature/disaggregated_inference.md). +* **Asynchronous stage and output execution:** [Async Chunk](feature/async_chunk.md) + forwards partial stage outputs as they become available. [Async Diffusion + Output](feature/async_diffusion_output.md) overlaps device-to-host output + packing with the next diffusion request, while [Async Omni Output + Materialization](feature/omni_async_output_materialization.md) moves + CPU-side payload construction off the AR decode critical path. +* **Automatic prefix caching:** [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md) + reuses KV-cache-aligned stage outputs and multimodal tensors for requests + with common prefixes. -* **Efficient AR Support:** Leverages efficient KV cache management inherited from vLLM. -* **Pipelined Execution:** Uses pipelined stage execution overlapping to ensure high throughput. -* **Full Disaggregation:** Relies on the OmniConnector and dynamic resource allocation across stages. -* **Diffusion Acceleration:** Includes integrated support for diffusion acceleration. This is managed by the acceleration layer, which handles: - * **Cache:** Includes DBCache, TeaCache and third-party integration(e.g., [cache-dit](https://github.com/vipshop/cache-dit)). - * **Parallelism:** Supports TP, CP, USP, and CFG. - * **Attention:** Provides an interface for third-party integration (e.g., FA3, SAGE, MindIE-SD). - * **Quantization:** Supports various quantization implementations including FP8 and AWQ. - * **FusedOps:** Allows for custom and third-party integration. +### Communication -### Classifier-Free Guidance (CFG) Companion Flow +* **OmniConnector transport:** The connector contract carries tensors, KV-cache + data, and transport metadata across stage boundaries. The available + implementations cover shared memory and multi-node Mooncake, Mori, and + Yuanrong transports; see [Disaggregated Inference](feature/disaggregated_inference.md) + for the connector choices and configuration model. -vLLM-Omni natively models Classifier-Free Guidance (CFG) across disaggregated multi-stage setups via a "companion request" paradigm, eliminating redundant textual/multimodal context computation boundaries: -1. **Prompt Expansion:** In the initial autoregressive (AR) stage, a customized `prompt_expand_func` hook intercepts incoming generation prompts and pairs them directly with negative companion prompts (e.g., a default negative prompt) on the fly, tagging the secondary prompt with a specific internal role (`cfg_text`). -2. **Synchronized KV Cache Transfer:** The AR stage evaluates both the primary and companion sequence batches concurrently. The `OmniConnector` captures these specific structural dependencies and reliably passes the positive and negative outcome KV caches seamlessly across stage boundaries via shared memory or network protocols. -3. **KV Cache Collection & Injection:** Upon reaching the downstream Diffusion (DiT) Engine, an assigned `cfg_kv_collect_func` automatically intercepts the mapped companion caches (`cfg_text_past_key_values`). These auxiliary dependencies are natively gathered and seamlessly bound to the primary generation sequence variables, enabling the DiT Engine to cleanly implement cross-attention CFG guidance over accurate conditioning and unconditioning structures in parallel. +### Diffusion acceleration -### Flexibility and Usability +* **Request and step batching:** [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md) + defines request-batch and step-batch execution, scheduler admission, and the + common streaming output path. +* **Composable parallelism:** Diffusion stages can combine [CFG-Parallel](feature/cfg_parallel.md), + [Expert Parallel](feature/expert_parallel.md), [HSDP](feature/hsdp.md), + [Pipeline Parallel](feature/pipeline_parallel.md), [Sequence Parallel](feature/sequence_parallel.md), + [Tensor Parallel](feature/tensor_parallel.md), and [VAE Patch Parallelism](feature/vae_parallel.md) + according to the pipeline and hardware topology. +* **Attention and cache acceleration:** [Skip-Softmax](feature/skip_softmax.md), + [Cache-DiT](feature/cache_dit.md), and [TeaCache](feature/teacache.md) + provide backend and denoising-step optimizations without changing the + stage contract. +* **Quantization and memory efficiency:** [Quantization](feature/quantization.md) + resolves per-pipeline or per-component quantization configurations, while + [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md) + streams diffusion blocks from host memory within the existing parallel + topology. -vLLM-Omni is designed to be flexible and straightforward for users: +## Interfaces -* **Heterogeneous Pipeline Abstraction:** Manages complex model workflows effectively. -* **Hugging Face Integration:** Offers seamless integration with popular Hugging Face models. -* **Distributed Inference:** Supports tensor, pipeline, data, and expert parallelism. -* **Streaming Outputs:** Supports streaming outputs. -* **Unified API:** Provides a consistent and unified API interface compatible with vLLM. -* **OpenAI-compatible API Server:** Includes a FastAPI-based server for online serving that is compatible with the OpenAI API. - -# Interface design - -If you use vLLM, then you know how to use vLLM-Omni from Day 0: - -

- - - vLLM-Omni interface design - -

- -Taking **Qwen3-Omni** as an example: - -## Offline Inference -The **Omni** class provides a Python interface for offline batched inference. Users initialize the Omni class with a Hugging Face model name and use the generate method, passing inputs that include both text prompts and multi-modal data: +The public interfaces map onto the same engine and stage boundaries: +```mermaid +flowchart LR + offline["Offline Python
Omni.generate()"] --> engine["AsyncOmniEngine"] + online["OpenAI-compatible serving
vllm serve ... --omni"] --> engine + duplex["Experimental duplex WebSocket
/v1/duplex or realtime duplex"] --> engine + engine --> stages["Configured AR and diffusion stages"] + stages --> result["Streaming or final multimodal output"] ``` -# Create an omni runtime with HF model name. + +### Offline inference + +The **Omni** class provides a Python interface for offline batched inference: + +```python from vllm_omni.entrypoints.omni import Omni omni = Omni(model="Qwen/Qwen3-Omni-30B-A3B-Instruct") -# Example prompts. -om_inputs = {"prompt": prompt, - "multi_modal_data": { - "video": video_frames, - "audio": audio_signal, - }} +om_inputs = { + "prompt": prompt, + "multi_modal_data": { + "video": video_frames, + "audio": audio_signal, + }, +} -# Generate texts and audio from the multi-modality inputs. outputs = omni.generate(om_inputs, sampling_params_list) ``` -## Online Serving -Similar to vLLM, vLLM-Omni also provides a FastAPI-based server for online serving. Users can launch the server using the vllm serve command with the `--omni` flag: +### Online serving -``` +The OpenAI-compatible server uses the same stage configuration and engine +boundaries: + +```bash vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 ``` -Users can send requests to the server using curl: +For example, a Qwen3-Omni chat request can contain text, image, audio, or +video content and a `sampling_params_list` for its configured stages. See the +[Qwen3-Omni serving example](../user_guide/examples/online_serving/qwen3_omni.md) +and the [examples](https://github.com/vllm-project/vllm-omni/tree/main/examples) +for complete requests. -``` -# prepare user content -user_content='[ - { - "type": "video_url", - "video_url": { - "url": "'"$SAMPLE_VIDEO_URL"'" - } - }, - { - "type": "text", - "text": "Why is this video funny?" - } - ]' - sampling_params_list='[ - '"$thinker_sampling_params"', - '"$talker_sampling_params"', - '"$code2wav_sampling_params"' - ]' - mm_processor_kwargs="{}" - -# send the request -curl -sS -X POST http://localhost:8091/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d @- <