# Architecture Overview This document outlines the architecture of vLLM-Omni.

Omni-Modality Model Architecture

## Design goals The primary goal of vLLM-Omni is to provide a fast and easy-to-use inference and serving engine for omni-modality models. vLLM-Omni extends vLLM's text-oriented autoregressive (AR) runtime with stage-based execution for non-textual outputs and non-autoregressive model components. The architecture is designed to: * support text, image, audio, video, and action inputs and outputs; * compose autoregressive, generation, and diffusion stages in one pipeline; * reuse vLLM scheduling, cache, distributed-execution, and serving primitives where they fit; and * keep model topology, deployment placement, runtime lifecycle, transport, and public API concerns in separate layers. ## Model execution Model execution is easiest to compare on two separate axes. First, identify the model-owned components that run at each stage. Then describe how the runtime executes those components: batching, attention, parallelism, and quantization. The four examples below use that order. Their runtime rows are representative deployment profiles; a model can have more than one valid `PipelineConfig` and `DeployConfig`. | Model | Model-owned path | vLLM-Omni stage view | Primary serving target | | --- | --- | --- | --- | | Qwen3-Omni | Thinker → Talker → Code2Wav | Three stages with asynchronous chunk hand-off | TTFT/TTFP, TPOT, E2EL, audio RTF, and concurrent throughput | | HunyuanImage-3.0 | Multimodal AR understanding/reasoning plus image-generation DiT | AR-only, DiT-only, or split AR → DiT deployment | DiT E2EL/denoising latency and image throughput; TTFT/TPOT for AR tasks | | MiniMax-H3 | Shared multimodal conditioning plus task-specific FL2VA or Ref2VA DiT and video/audio VAEs | One diffusion stage; the request selects the DiT partition | Video/audio E2EL and media throughput; RTF when audio streaming is measured | | Cosmos3 | Unified MoT reasoner tower plus diffusion generator tower | One diffusion pipeline materializes both towers inside the stage | Image/video/action E2EL and throughput; RTF for synchronized audio | ### Serving metric vocabulary The target metric follows the output modality, not just the model name. | Metric | Meaning in this document | Most useful for | | --- | --- | --- | | **TTFT** | Time to the first text token | AR reasoning or text output | | **TPOT** | Time per output token after the first token | AR decode stages | | **TTFP** | Time to the first streamed media packet, such as audio | Interactive speech and multimodal chat | | **E2EL** | End-to-end latency from request admission to the final output | Image, video, audio, and action requests | | **RTF** | Wall-clock processing time divided by generated audio duration; lower is faster than real time | Audio generation | | **Throughput** | Requests, tokens, or media seconds produced per wall-clock second | Offline and concurrent serving | ### Qwen3-Omni: streaming AR pipeline Qwen3-Omni is a useful example of a pipeline whose user-visible latency is determined by several sequential model components. The Thinker produces text and conditioning, the Talker produces audio codec tokens, and Code2Wav renders those tokens into audio. The [Qwen3-Omni technical report](https://arxiv.org/abs/2509.17765) describes the model components; the figure below shows the stage-level streaming schedule used by vLLM-Omni.

Qwen3-Omni Thinker, Talker, and Code2Wav streaming stages

#### Model architecture by stage | Stage | Model component | Stage output | | --- | --- | --- | | **Thinker** | AuT audio encoder, SigLIP2 vision encoder, and the 30B-total/3B-active Thinker MoE Transformer | Text tokens and conditioning for the Talker | | **Talker** | 3B-total/0.3B-active Talker MoE Transformer with the MTP code-prediction path | Text tokens and audio codec tokens | | **Code2Wav** | Approximately 200M-parameter ConvNet audio decoder | Waveform/audio chunks | #### Representative stage execution | Stage | Batching | Attention and execution | Parallelism | Quantization | | --- | --- | --- | --- | --- | | **Thinker** | Continuous batching with stage-local `max_num_seqs` and token budgets | vLLM KV-cached AR attention; CUDA Graph execution when the stage is not eager | Independent stage placement; use the AR runtime's configured tensor parallelism | Supported ModelOpt FP8/NVFP4 or AutoRound checkpoint paths target the Thinker; encoders remain BF16 in the documented paths | | **Talker** | Continuous batching over autoregressive decode requests | KV-cached token decode with async-chunk output to Code2Wav | Independent stage placement; typically a separate GPU or replica from Thinker | BF16 baseline; no generic Talker quantization path is assumed | | **Code2Wav** | Static/chunk batching for codec-to-waveform decode | Non-AR chunked audio decode; eager execution is commonly retained | Separate stage or colocated placement, depending on the deployment | BF16 baseline; keep codec/vocoder precision model-specific | For this pipeline, the main serving target is interactive responsiveness: TTFT for text, TTFP for the first audio packet, TPOT for autoregressive decode, E2EL for the complete answer, and RTF/throughput for sustained audio serving. Async chunking and streaming primarily reduce TTFP and improve stage overlap; batching and CUDA Graphs primarily improve E2EL and throughput. See the [Qwen3-Omni recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen3-Omni.md) and [async chunk design](feature/async_chunk.md) for the deployment details. ### HunyuanImage-3.0: shared multimodal model, split deployment choices The official [HunyuanImage-3.0 technical report](https://arxiv.org/abs/2509.23951) presents a shared decoder-only backbone serving image understanding, language modeling, and image generation. Its framework figure is useful for separating model components from the deployment decomposition:

HunyuanImage-3.0 framework for understanding, language modeling, and image generation

#### Model architecture by stage | Stage or component | Model component | Stage output | | --- | --- | --- | | **AR understanding/reasoning** | Understanding and generation encoders feeding the Hunyuan-A13B decoder-only Transformer and text detokenizer | Text responses or image-generation conditioning | | **Image generation** | Generation encoder, diffusion prediction path/Gen. Decoder, and the image VAE decode path | Image latents and final images | The vLLM-Omni deployment may expose the AR path only, the DiT path only, or both as an AR → DiT pipeline with a connector between them. That is a runtime decomposition of the model's capabilities; it should not be read as a claim that the published framework figure contains two unrelated backbones. #### Representative stage execution | Stage | Batching | Attention and execution | Parallelism | Quantization | | --- | --- | --- | --- | --- | | **AR** | Standard vLLM continuous batching when serving understanding/text tasks | KV-cached causal attention | Tensor/expert parallelism follows the selected AR deployment | Checkpoint-specific; do not infer DiT FP8 settings for the AR checkpoint | | **DiT + VAE** | Request batching or step-wise batching; Hunyuan step batching with more than one request requires `TORCH_SDPA` | `TORCH_SDPA` is used because the model mixes causal and full attention | Validated profiles include TP4, TP2 + sequence parallelism, TP2 + CFG parallelism, and expert parallelism for MoE weights | Online FP8 and ModelOpt mixed FP8/NVFP4 are documented for the DiT; VAE precision stays separate | Image generation is therefore measured mainly with per-image E2EL/denoising latency, peak memory, and images-per-second throughput. TTFT and TPOT remain meaningful for the optional AR understanding path, but they are not the right headline metrics for the DiT path. See the [HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md) for the validated batching, attention, and quantization combinations. ### MiniMax-H3: one diffusion stage with task-selected DiTs MiniMax-H3 illustrates a different boundary. The vLLM-Omni implementation loads shared conditioning and decode components once and selects one of two task-specific DiTs per request. The current [MiniMax H3 release overview](https://minimaxi.com/blog/minimax-h3) says that a detailed H3 technical report is forthcoming, so this section uses the implementation and serving recipe as the source for the stage mapping rather than implying an unavailable published architecture figure. #### Model architecture by stage | Stage | Model component | Stage output | | --- | --- | --- | | **Diffusion stage: shared path** | Tokenizer/processor, Qwen3-VL text encoder, video VAE, and audio VAE | Packed text, image, video, and audio conditioning plus decoded media | | **Diffusion stage: `FL2VA`** | Task-specific DiT for text-to-video+audio (`t2va`) and first-frame-to-video+audio (`fl2va`) | Synchronized video and stereo audio | | **Diffusion stage: `Ref2VA`** | Task-specific DiT for mixed image/video/audio reference generation (`ref2va`) | Reference-conditioned video and stereo audio | Both DiT partitions live behind one logical diffusion stage; `task` selects which partition runs. This is different from Qwen3-Omni's three stage pipeline, even though both models produce synchronized audio and video-like outputs. #### Representative stage execution | Stage | Batching | Attention and execution | Parallelism | Quantization | | --- | --- | --- | --- | --- | | **Shared encoder + selected DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and VAE patch parallelism on two GPUs; Ulysses and tiled VAE patch parallelism on four GPUs; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears; text encoder and VAEs remain BF16/FP32, and FP8 is incompatible with layerwise offload | The primary target is video/audio E2EL and media throughput, with RTF useful when the audio stream is evaluated independently. TTFT and TPOT do not describe the main H3 generation path because the user-visible output is diffusion media rather than a token stream. The [MiniMax-H3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md) contains the hardware-specific profiles and warmup requirements. ### Cosmos3: unified MoT reasoner and generator Cosmos3 uses a Mixture-of-Transformers (MoT) design. The official [Cosmos 3 technical report](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) describes autoregressive reasoner tokens and diffusion generator tokens in one model sequence. The figure below is the corresponding two-tower overview from NVIDIA's [technical explanation](https://developer.nvidia.com/blog/develop-physical-ai-reasoning-world-and-action-models-with-nvidia-cosmos-3/).

Cosmos3 autoregressive reasoner and diffusion generator towers

#### Model architecture by stage | Stage or component | Model component | Stage output | | --- | --- | --- | | **Reasoner tower** | Autoregressive VLM path with causal attention over language and optional vision/action context; runs once per generation request | Discrete reasoning/context tokens | | **Generator tower** | Diffusion path with full attention over the generator tokens and the reasoner K/V context; runs for each denoising step | Image, video, audio, or action tokens | | **Shared pipeline** | `Cosmos3OmniDiffusersPipeline` plus modality encoders/decoders and VAE paths | Final image/video/audio or action response | The reasoner and generator are model-owned components, not two independent vLLM-Omni stages by default. The current Cosmos3 pipeline materializes both inside one diffusion stage; model-level CPU offload can swap the reasoner and generator components between their phases. #### Representative stage execution | Stage | Batching | Attention and execution | Parallelism | Quantization | | --- | --- | --- | --- | --- | | **Reasoner + generator pipeline** | Use request batching or step execution only when the selected pipeline advertises that capability; otherwise use `max_num_seqs=1` | Causal attention for reasoner tokens and full attention for diffusion tokens; the validated recipe uses a platform-selected aiter FlashAttention path with `TORCH_SDPA` fallback | Ulysses/context parallelism or tensor parallelism for the transformer; VAE tiling and layerwise/model-level offload address activation and weight memory | Online FP8 is supported for the diffusion pipeline; AutoRound/checkpoint-specific paths and quality validation remain separate from the BF16 baseline | For Cosmos3, generation E2EL and output throughput are the primary targets. Action-policy workloads add a control-loop latency target, while synchronized audio workloads add RTF. TTFT/TPOT are appropriate for a reasoner-only service, but not for the full diffusion generation path. See the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-Nano.md) and [diffusion execution modes](../user_guide/diffusion/execution_modes.md) for the current runtime choices. ## System architecture The runtime is organized around an engine, an orchestrator, stage lifecycle management, and independent AR and diffusion stage implementations. ```mermaid flowchart TB entry["Entrypoints
Omni / AsyncOmni / CLI / OpenAI-compatible APIs / duplex WebSocket"] engine["AsyncOmniEngine
engine composition root and background loop"] orchestrator["Orchestrator
request state, cross-stage routing, correlation, and output ordering"] runtime["StageRuntime or DistStageRuntime
placement, replicas, readiness, and lifecycle"] clients["StagePool and stage clients
StageEngineCoreClient / StageDiffusionClient"] ar["AR stages
vLLM engine, scheduler, worker, and model runner"] diffusion["Diffusion stages
DiffusionEngine, scheduler, executor, worker, and pipeline"] connector["OmniConnector
payload and KV transport plus synchronization"] outputs["MultimodalPayload and OmniRequestOutput
tensors, metadata, streaming chunks, and final artifacts"] entry --> engine engine --> orchestrator engine --> runtime runtime --> clients clients --> ar clients --> diffusion ar <--> connector diffusion <--> connector ar --> outputs diffusion --> outputs outputs --> entry ``` ### Key components | Component | Responsibility | | --- | --- | | **Entrypoints** | Translate offline, CLI, OpenAI-compatible, and duplex requests into engine operations and render outputs back to public protocols. | | **Configuration resolution** | Combines pipeline topology, deployment settings, model metadata, and user overrides into a validated control-plane configuration. | | **AsyncOmniEngine** | Owns engine composition, the background event loop, stage initialization, request submission, and output collection. | | **Orchestrator** | Owns cross-stage request state, stage-to-stage routing, request correlation, cancellation, and output ordering. It does not own model selection or deployment placement. | | **StageRuntime** | Expands logical stages into local or distributed replicas, starts stage clients and processes, and manages readiness, affinity, failure, and shutdown. | | **AR runtime** | Extends vLLM's scheduler, KV-cache, worker, and model-runner path for omni-modality inputs and inter-stage outputs. | | **Diffusion runtime** | Schedules and executes denoising workloads through diffusion executors, workers, pipelines, acceleration backends, and output materialization. | | **OmniConnector** | Transports stage payloads and KV-cache data and provides synchronization. Connectors transport data; they do not choose the next logical stage. | | **Multimodal outputs** | `MultimodalPayload` separates tensor content from metadata, while `OmniRequestOutput` carries pipeline and diffusion results through the common output path. | ## Configuration and runtime resolution The control plane has five conceptual layers. Authoring inputs are resolved once into a complete, transport-safe configuration before runtime processes are started. This keeps stage topology and model capabilities distinct from deployment placement and from process-local engine objects. The figure shows only the primary hand-off object for each layer; the detailed inputs, fields, and ownership rules are described below. ```mermaid flowchart TB layer1["Layer 1 · Authoring inputs
PipelineConfig + DeployConfig"] layer2["Layer 2 · Resolve once
OmniConfigResolveRequest
resolve_omni_config()"] layer3["Layer 3 · Transport-safe control plane
VllmOmniConfig"] layer4["Layer 4 · Runtime launch planning
StageRuntime"] layer5["Layer 5 · Engine materialization
VllmConfig / OmniDiffusionConfig"] layer1 --> layer2 --> layer3 --> layer4 --> layer5 ``` The resolver names in this diagram describe the intended single resolution boundary. In the current implementation, the corresponding path is carried out by `StageConfigFactory.create_from_model()` and `VllmOmniConfig.from_pipeline_config()` in [`vllm_omni/config`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/config). The legacy `stage_args` YAML path remains only for models that have not yet migrated to `PipelineConfig` and `DeployConfig`. In the typed path, the common `VllmOmniStageConfig` slot in the diagram is realized by `VllmOmniARStageConfig`, `VllmOmniGenerationStageConfig`, or `VllmOmniDiffusionStageConfig`. The request and engine-spec fields belong to the control-plane boundary; the current implementation stores their equivalent projections in the structured stage configuration and materializes backend-specific engine objects during stage initialization. The important ownership rules are: 1. `PipelineConfig` is the source of truth for stage topology, execution type, model capabilities, and stage relationships. 2. `DeployConfig` describes placement, replicas, devices, connectors, and deploy-time defaults; it does not redefine the model graph. 3. CLI and Python overrides are applied at the resolution boundary, with per-stage overrides taking precedence over global values where supported. 4. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan` is runtime-private state, not a user configuration object. 5. `VllmConfig` and the enriched `OmniDiffusionConfig` are materialized in the process that owns the corresponding engine. ## Main features The feature surface is grouped to match the [feature design documents](index.md#feature-design-documents). This page summarizes each feature's architectural role; the linked design document is the source for configuration, compatibility, and implementation details. ### Runtime and stage execution * **Disaggregated inference:** Logical stages can run in separate processes, devices, or nodes while the orchestrator preserves their declared relationships. `OmniConnector` implementations transfer stage data and control-plane metadata. See [Disaggregated Inference](feature/disaggregated_inference.md). * **Asynchronous stage and output execution:** [Async Chunk](feature/async_chunk.md) forwards partial stage outputs as they become available. [Async Diffusion Output](feature/async_diffusion_output.md) overlaps device-to-host output packing with the next diffusion request, while [Async Omni Output Materialization](feature/omni_async_output_materialization.md) moves CPU-side payload construction off the AR decode critical path. * **Automatic prefix caching:** [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md) reuses KV-cache-aligned stage outputs and multimodal tensors for requests with common prefixes. ### Communication * **OmniConnector transport:** The connector contract carries tensors, KV-cache data, and transport metadata across stage boundaries. The available implementations cover shared memory and multi-node Mooncake, Mori, and Yuanrong transports; see [Disaggregated Inference](feature/disaggregated_inference.md) for the connector choices and configuration model. ### Diffusion acceleration * **Request and step batching:** [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md) defines request-batch and step-batch execution, scheduler admission, and the common streaming output path. * **Composable parallelism:** Diffusion stages can combine [CFG-Parallel](feature/cfg_parallel.md), [Expert Parallel](feature/expert_parallel.md), [HSDP](feature/hsdp.md), [Pipeline Parallel](feature/pipeline_parallel.md), [Sequence Parallel](feature/sequence_parallel.md), [Tensor Parallel](feature/tensor_parallel.md), and [VAE Patch Parallelism](feature/vae_parallel.md) according to the pipeline and hardware topology. * **Attention and cache acceleration:** [Skip-Softmax](feature/skip_softmax.md), [Cache-DiT](feature/cache_dit.md), and [TeaCache](feature/teacache.md) provide backend and denoising-step optimizations without changing the stage contract. * **Quantization and memory efficiency:** [Quantization](feature/quantization.md) resolves per-pipeline or per-component quantization configurations, while [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md) streams diffusion blocks from host memory within the existing parallel topology. ## Interfaces The public interfaces map onto the same engine and stage boundaries: ```mermaid flowchart LR offline["Offline Python
Omni.generate()"] --> engine["AsyncOmniEngine"] online["OpenAI-compatible serving
vllm serve ... --omni"] --> engine duplex["Experimental duplex WebSocket
/v1/duplex or realtime duplex"] --> engine engine --> stages["Configured AR and diffusion stages"] stages --> result["Streaming or final multimodal output"] ``` ### Offline inference The **Omni** class provides a Python interface for offline batched inference: ```python from vllm_omni.entrypoints.omni import Omni omni = Omni(model="Qwen/Qwen3-Omni-30B-A3B-Instruct") om_inputs = { "prompt": prompt, "multi_modal_data": { "video": video_frames, "audio": audio_signal, }, } outputs = omni.generate(om_inputs, sampling_params_list) ``` ### Online serving The OpenAI-compatible server uses the same stage configuration and engine boundaries: ```bash vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 ``` For example, a Qwen3-Omni chat request can contain text, image, audio, or video content and a `sampling_params_list` for its configured stages. See the [Qwen3-Omni serving example](../user_guide/examples/online_serving/qwen3_omni.md) and the [examples](https://github.com/vllm-project/vllm-omni/tree/main/examples) for complete requests. Some pipelines expose additional OpenAI-compatible endpoints, such as joint video/audio generation. Endpoint support remains model-specific; consult the relevant model guide before assuming that every OpenAI route applies to every pipeline.