From 7d0ecd082f0b20a5c253283cb94ce5767ec38f14 Mon Sep 17 00:00:00 2001 From: Hongsheng Liu Date: Sat, 8 Aug 2026 13:13:00 +0800 Subject: [PATCH] docs: update architecture overview Signed-off-by: Hongsheng Liu --- docs/design/architecture_overview.md | 347 +++++++++++++++++---------- 1 file changed, 217 insertions(+), 130 deletions(-) diff --git a/docs/design/architecture_overview.md b/docs/design/architecture_overview.md index 20c99fd9c..0efa45294 100644 --- a/docs/design/architecture_overview.md +++ b/docs/design/architecture_overview.md @@ -1,6 +1,6 @@ # Architecture Overview -This document outlines the architectural design for vLLM-Omni. +This document outlines the architecture of vLLM-Omni.

@@ -9,194 +9,281 @@ This document outlines the architectural design for vLLM-Omni.

-# Goals +## Design goals -The primary goal of the vLLM-Omni project is to build the fastest and easiest-to-use open-source Omni-Modality model inference & serving engine. vLLM-Omni extends the original vLLM, which was created to support large language models for text-based autoregressive (AR) generation tasks. vLLM-Omni is designed to support: +The primary goal of vLLM-Omni is to provide a fast and easy-to-use inference +and serving engine for omni-modality models. vLLM-Omni extends vLLM's +text-oriented autoregressive (AR) runtime with stage-based execution for +non-textual outputs and non-autoregressive model components. -* **Non-textual Output:** Enables the integration, efficient processing and output of various data types, including but not limited to, images, audio, video, and action trajectories, alongside text. -* **Non-Autoregressive Structure:** Support model structure beyond autoregressive, especially Diffusion Transformer (DiT), which is widely used in visual and audio generation. -* **Integration with vLLM Core:** Maintain compatibility and leverage existing vLLM key modules and optimizations where applicable. -* **Extensibility:** Design a modular and flexible architecture that can easily accommodate new modalities, model architectures, and output formats. +The architecture is designed to: +* support text, image, audio, video, and action inputs and outputs; +* compose autoregressive, generation, and diffusion stages in one pipeline; +* reuse vLLM scheduling, cache, distributed-execution, and serving + primitives where they fit; and +* keep model topology, deployment placement, runtime lifecycle, transport, + and public API concerns in separate layers. -# Representative omni-modality models +## Model execution topologies -According to analysis for current popular open-source models, most of them have the combination of AR+DiT. Specifically, they can be further categorized into 3 types below: +vLLM-Omni is not limited to one model graph. A registered `PipelineConfig` +defines the stages and their relationships, while `DeployConfig` supplies the +placement and runtime choices for a particular deployment. -**DiT as a main structure, with AR as text encoder (e.g.: Qwen-Image)** - A powerful image generation foundation model capable of complex text rendering and precise image editing. +| Topology | Representative examples | Runtime shape | +| --- | --- | --- | +| AR plus downstream generation or diffusion | Qwen3-Omni, Qwen3-TTS, MiniCPM-o 4.5 | An AR stage produces text, codes, or conditioning for one or more downstream stages. | +| Diffusion-first generation | Qwen-Image, FLUX, HunyuanImage3, Cosmos3, Krea 2 | A diffusion stage owns denoising and may use an auxiliary text or multimodal encoder. | +| Joint multimodal generation | MiniMax H3, LTX-2 | A pipeline combines multiple model components to produce synchronized or multimodal artifacts such as video and audio. | +| Stateful or full-duplex interaction | MiniCPM-o 4.5 and experimental world-model paths | The runtime keeps session-scoped state and supports streaming, cancellation, or incremental input. | + +The following illustrations show common model-level arrangements. They are +examples of model topology, not constraints on the runtime. + +### DiT as the main structure, with AR as an encoder + +For example, Qwen-Image uses an AR or text-encoding component to condition a +diffusion transformer.

- Qwen-Image + Diffusion transformer as the main model structure

-**AR as a main structure, with DiT as multi-modal generator (e.g. BAGEL)** - A unified multimodal comprehension and generation model, with cot text output and visual generation. +### AR as the main structure, with a diffusion generator + +For example, BAGEL uses an AR model for multimodal understanding and text +reasoning, with a diffusion component for visual generation.

- Bagel + Autoregressive model with a diffusion generator

-**AR+DiT (e.g. Qwen-Omni)** - A natively end-to-end omni-modal LLM for multimodal inputs (text/image/audio/video...) and outputs (text/audio...). +### AR and DiT in one multimodal pipeline + +For example, Qwen-Omni combines multimodal encoders, an AR language model, +and a modality generator.

- Qwen-Omni + Autoregressive and diffusion components in one pipeline

-# vLLM-Omni main architecture +## System architecture -

- - - vLLM-Omni Main Architecture - -

+The runtime is organized around an engine, an orchestrator, stage lifecycle +management, and independent AR and diffusion stage implementations. -## Key Components +```mermaid +flowchart TB + entry["Entrypoints
Omni / AsyncOmni / CLI / OpenAI-compatible APIs / duplex WebSocket"] + engine["AsyncOmniEngine
engine composition root and background loop"] + orchestrator["Orchestrator
request state, cross-stage routing, correlation, and output ordering"] + runtime["StageRuntime or DistStageRuntime
placement, replicas, readiness, and lifecycle"] + clients["StagePool and stage clients
StageEngineCoreClient / StageDiffusionClient"] + ar["AR stages
vLLM engine, scheduler, worker, and model runner"] + diffusion["Diffusion stages
DiffusionEngine, scheduler, executor, worker, and pipeline"] + connector["OmniConnector
payload and KV transport plus synchronization"] + outputs["MultimodalPayload and OmniRequestOutput
tensors, metadata, streaming chunks, and final artifacts"] -| Component | Description | -| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | -| **OmniRouter** | provide an intelligent router for Omni-modality requests dispatch | -| **EntryPoints** | define the APIs for offline/online serving (APIServer, Omni/AsyncOmni), while `AsyncOmniEngine` and `Orchestrator` coordinate multi-stage AR/DiT execution | -| **AR** | adapted for omni-modality models while inheriting efficient features from vLLM, such as cache management | -| **Diffusion** | natively implemented and optimized using acceleration components | -| **OmniConnector** | supports fully disaggregation based on E/P/D/G (Encoding/Processing/Decoding/Generation) disaggregation across stages | + entry --> engine + engine --> orchestrator + engine --> runtime + runtime --> clients + clients --> ar + clients --> diffusion + ar <--> connector + diffusion <--> connector + ar --> outputs + diffusion --> outputs + outputs --> entry +``` -Disaggregated stages are managed through stage configuration. In Qwen3-Omni, Thinker/Talker/Code2wav are declared as separate configured stages, and runtime routing is handled by `Orchestrator` over `StageEngineCoreClient` / `StageDiffusionClient`. +### Key components + +| Component | Responsibility | +| --- | --- | +| **Entrypoints** | Translate offline, CLI, OpenAI-compatible, and duplex requests into engine operations and render outputs back to public protocols. | +| **Configuration resolution** | Combines pipeline topology, deployment settings, model metadata, and user overrides into a validated control-plane configuration. | +| **AsyncOmniEngine** | Owns engine composition, the background event loop, stage initialization, request submission, and output collection. | +| **Orchestrator** | Owns cross-stage request state, stage-to-stage routing, companion tracking, correlation, cancellation, and output ordering. It does not own model selection or deployment placement. | +| **StageRuntime** | Expands logical stages into local or distributed replicas, starts stage clients and processes, and manages readiness, affinity, failure, and shutdown. | +| **AR runtime** | Extends vLLM's scheduler, KV-cache, worker, and model-runner path for omni-modality inputs and inter-stage outputs. | +| **Diffusion runtime** | Schedules and executes denoising workloads through diffusion executors, workers, pipelines, acceleration backends, and output materialization. | +| **OmniConnector** | Transports stage payloads and KV-cache data and provides synchronization. Connectors transport data; they do not choose the next logical stage. | +| **Multimodal outputs** | `MultimodalPayload` separates tensor content from metadata, while `OmniRequestOutput` carries pipeline and diffusion results through the common output path. | + +## Configuration and runtime resolution + +The control plane has five conceptual layers. Authoring inputs are resolved +once into a complete, transport-safe configuration before runtime processes are +started. This keeps stage topology and model capabilities distinct from +deployment placement and from process-local engine objects. + +```mermaid +flowchart TB + subgraph authoring["Layer 1: Authoring inputs"] + pipeline["PipelineConfig
frozen stage topology and execution types
model capabilities and stage relationships"] + deploy["DeployConfig
placement, replicas, devices, and environment
deploy-time defaults and connector definitions"] + overrides["CLI / Python overrides"] + metadata["HF or diffusion model metadata"] + legacy["Legacy YAML adapter
migration-only input; removed after model parity"] + end + + request["Layer 2: OmniConfigResolveRequest"] + resolve["resolve_omni_config()
selects exactly one source path
loads model metadata once
applies precedence and stage overrides once
resolves connectors, requests, runtime placement, and engine specs
validates the complete pipeline"] + resolved["Layer 3: Resolved and transport-safe control-plane config
VllmOmniConfig
pipeline_config: PipelineConfig
orchestrator_config: VllmOmniOrchestratorConfig
stage_configs: tuple[VllmOmniStageConfig, ...]
stage_pipeline_config · runtime_config · connector_config · request_config · engine_spec"] + launch["Layer 4: Runtime-only launch planning
StageRuntime
replica expansion · device allocation · local or remote launch
ReplicaInitPlan remains private runtime state"] + materialize["Layer 5: Owning-process materialization
materialize_engine_config(stage)"] + ar_config["VllmConfig
vLLM-backed engine"] + diffusion_config["enriched OmniDiffusionConfig
diffusion engine"] + + authoring --> request + request --> resolve + resolve --> resolved + resolved --> launch + launch --> materialize + materialize --> ar_config + materialize --> diffusion_config +``` + +The resolver names in this diagram describe the intended single resolution +boundary. In the current implementation, the corresponding path is carried +out by `StageConfigFactory.create_from_model()` and +`VllmOmniConfig.from_pipeline_config()` in +[`vllm_omni/config`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/config). +The legacy `stage_args` YAML path remains only for models that have not yet +migrated to `PipelineConfig` and `DeployConfig`. + +In the typed path, the common `VllmOmniStageConfig` slot in the +diagram is realized by `VllmOmniARStageConfig`, +`VllmOmniGenerationStageConfig`, or `VllmOmniDiffusionStageConfig`. The +diagram's request and engine-spec fields describe the control-plane boundary; +the current implementation stores their equivalent projections in the +structured stage configuration and materializes backend-specific engine +objects during stage initialization. + +The important ownership rules are: + +1. `PipelineConfig` is the source of truth for stage topology, execution type, + model capabilities, and stage relationships. +2. `DeployConfig` describes placement, replicas, devices, connectors, and + deploy-time defaults; it does not redefine the model graph. +3. CLI and Python overrides are applied at the resolution boundary, with + per-stage overrides taking precedence over global values where supported. +4. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan` + is runtime-private state, not a user configuration object. +5. `VllmConfig` and the enriched `OmniDiffusionConfig` are materialized in the + process that owns the corresponding engine. ## Main features -vLLM-Omni aims to be fast, flexible, and easy to use with the following features: +### Stage-based execution and disaggregation -### Performance and Acceleration +Each logical stage declares its inputs, outputs, execution type, and connector +edges. The orchestrator routes requests according to those declared +relationships, while `OmniConnector` implementations move data or KV-cache +payloads across processes, devices, or nodes. Qwen3-Omni, for example, can +represent Thinker, Talker, and Code2Wav as separate stages. -The framework achieves high performance through several optimization techniques: +### AR and diffusion acceleration -* **Efficient AR Support:** Leverages efficient KV cache management inherited from vLLM. -* **Pipelined Execution:** Uses pipelined stage execution overlapping to ensure high throughput. -* **Full Disaggregation:** Relies on the OmniConnector and dynamic resource allocation across stages. -* **Diffusion Acceleration:** Includes integrated support for diffusion acceleration. This is managed by the acceleration layer, which handles: - * **Cache:** Includes DBCache, TeaCache and third-party integration(e.g., [cache-dit](https://github.com/vipshop/cache-dit)). - * **Parallelism:** Supports TP, CP, USP, and CFG. - * **Attention:** Provides an interface for third-party integration (e.g., FA3, SAGE, MindIE-SD). - * **Quantization:** Supports various quantization implementations including FP8 and AWQ. - * **FusedOps:** Allows for custom and third-party integration. +The two runtime families share the stage and output contracts while retaining +their specialized execution paths: -### Classifier-Free Guidance (CFG) Companion Flow +* **AR stages:** vLLM scheduling, KV-cache management, batching, CUDA Graphs, + and model-runner optimizations. +* **Diffusion stages:** continuous batching, compile granularity, cache + acceleration, attention backends, quantization, CPU or layerwise offload, + tensor/sequence/CFG/VAE parallelism, and distributed layerwise offload. -vLLM-Omni natively models Classifier-Free Guidance (CFG) across disaggregated multi-stage setups via a "companion request" paradigm, eliminating redundant textual/multimodal context computation boundaries: -1. **Prompt Expansion:** In the initial autoregressive (AR) stage, a customized `prompt_expand_func` hook intercepts incoming generation prompts and pairs them directly with negative companion prompts (e.g., a default negative prompt) on the fly, tagging the secondary prompt with a specific internal role (`cfg_text`). -2. **Synchronized KV Cache Transfer:** The AR stage evaluates both the primary and companion sequence batches concurrently. The `OmniConnector` captures these specific structural dependencies and reliably passes the positive and negative outcome KV caches seamlessly across stage boundaries via shared memory or network protocols. -3. **KV Cache Collection & Injection:** Upon reaching the downstream Diffusion (DiT) Engine, an assigned `cfg_kv_collect_func` automatically intercepts the mapped companion caches (`cfg_text_past_key_values`). These auxiliary dependencies are natively gathered and seamlessly bound to the primary generation sequence variables, enabling the DiT Engine to cleanly implement cross-attention CFG guidance over accurate conditioning and unconditioning structures in parallel. +### Classifier-Free Guidance companion flow -### Flexibility and Usability +CFG can be represented as companion requests that share the parent request's +stage affinity and transport lifecycle: -vLLM-Omni is designed to be flexible and straightforward for users: +1. A model-provided `prompt_expand_func` expands the initial prompt into a + primary request and one or more companion requests. +2. The `Orchestrator` tracks parent/companion identities and waits for the + required stage outputs. The connector transfers the associated KV-cache + payloads across the stage boundary. +3. A downstream diffusion stage invokes its `cfg_kv_collect_func` to collect + companion KV data and apply model-specific guidance inputs. -* **Heterogeneous Pipeline Abstraction:** Manages complex model workflows effectively. -* **Hugging Face Integration:** Offers seamless integration with popular Hugging Face models. -* **Distributed Inference:** Supports tensor, pipeline, data, and expert parallelism. -* **Streaming Outputs:** Supports streaming outputs. -* **Unified API:** Provides a consistent and unified API interface compatible with vLLM. -* **OpenAI-compatible API Server:** Includes a FastAPI-based server for online serving that is compatible with the OpenAI API. +See [CFG parallelism](feature/cfg_parallel.md) for the diffusion-side +configuration and constraints. -# Interface design +### Distributed inference and memory efficiency -If you use vLLM, then you know how to use vLLM-Omni from Day 0: +The deployment layer can combine stage replicas with vLLM parallelism and +diffusion parallelism. Composable strategies make common layouts reusable, +while CPU offload, layerwise offload, and distributed layerwise offload allow +large diffusion pipelines to run under tighter device-memory budgets. -

- - - vLLM-Omni interface design - -

+### Extensibility -Taking **Qwen3-Omni** as an example: +New model families contribute a pipeline definition, stage implementations, +input/output adapters, and optional acceleration or connector integrations. +The central engine and orchestrator remain model-agnostic; model-specific +behavior is selected by resolved stage metadata and explicit extension hooks. -## Offline Inference -The **Omni** class provides a Python interface for offline batched inference. Users initialize the Omni class with a Hugging Face model name and use the generate method, passing inputs that include both text prompts and multi-modal data: +## Interfaces +The public interfaces map onto the same engine and stage boundaries: + +```mermaid +flowchart LR + offline["Offline Python
Omni.generate()"] --> engine["AsyncOmniEngine"] + online["OpenAI-compatible serving
vllm serve ... --omni"] --> engine + duplex["Experimental duplex WebSocket
/v1/duplex or realtime duplex"] --> engine + engine --> stages["Configured AR and diffusion stages"] + stages --> result["Streaming or final multimodal output"] ``` -# Create an omni runtime with HF model name. + +### Offline inference + +The **Omni** class provides a Python interface for offline batched inference: + +```python from vllm_omni.entrypoints.omni import Omni omni = Omni(model="Qwen/Qwen3-Omni-30B-A3B-Instruct") -# Example prompts. -om_inputs = {"prompt": prompt, - "multi_modal_data": { - "video": video_frames, - "audio": audio_signal, - }} +om_inputs = { + "prompt": prompt, + "multi_modal_data": { + "video": video_frames, + "audio": audio_signal, + }, +} -# Generate texts and audio from the multi-modality inputs. outputs = omni.generate(om_inputs, sampling_params_list) ``` -## Online Serving -Similar to vLLM, vLLM-Omni also provides a FastAPI-based server for online serving. Users can launch the server using the vllm serve command with the `--omni` flag: +### Online serving -``` +The OpenAI-compatible server uses the same stage configuration and engine +boundaries: + +```bash vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 ``` -Users can send requests to the server using curl: +For example, a Qwen3-Omni chat request can contain text, image, audio, or +video content and a `sampling_params_list` for its configured stages. See the +[Qwen3-Omni serving example](../user_guide/examples/online_serving/qwen3_omni.md) +and the [examples](https://github.com/vllm-project/vllm-omni/tree/main/examples) +for complete requests. -``` -# prepare user content -user_content='[ - { - "type": "video_url", - "video_url": { - "url": "'"$SAMPLE_VIDEO_URL"'" - } - }, - { - "type": "text", - "text": "Why is this video funny?" - } - ]' - sampling_params_list='[ - '"$thinker_sampling_params"', - '"$talker_sampling_params"', - '"$code2wav_sampling_params"' - ]' - mm_processor_kwargs="{}" - -# send the request -curl -sS -X POST http://localhost:8091/v1/chat/completions \ - -H "Content-Type: application/json" \ - -d @- <