vllm-omni/docs/design/index.md

Design Documents

This section contains design documents and architecture specifications for vLLM-Omni. The sidebar groups documents by the system concern they describe, while the source paths remain stable so existing links continue to work.

Architecture Documents

• Architecture Overview

Feature Design Documents

Runtime and stage execution

• Disaggregated Inference
• Ray-based Execution
• Async Chunk
• Async Diffusion Output
• Async Omni Output Materialization
• Automatic Prefix Caching in Omni Models

Communication

OmniConnector implementations

• Mooncake Store Connector
• Mooncake Transfer Engine Connector
• Mori Transfer Engine Connector
• Shared Memory Connector
• Yuanrong Connector
• Yuanrong Transfer Engine Connector

Diffusion acceleration

Parallelism

• CFG-Parallel
• Expert Parallel
• Hybrid Sharded Data Parallel (HSDP)
• Pipeline Parallel
• Sequence Parallel
• Tensor Parallel
• VAE Patch Parallelism

Attention Backends

• Skip-Softmax

Quantization

• Quantization Overview

• Cache-DiT

• TeaCache

• Diffusion Continuous Batching

Infrastructure and Performance

• Prometheus Metrics
• Speech Generation Performance Optimizations

Module Design Documents

Runtime Modules

• AR Module
• DIT Module

Orchestration

• Entrypoint Module
• AsyncOmni Architecture (Qwen3-Omni Example)

RFCs and Experimental Work

• AR-Diffusion Pipeline Capability
• Diffusion Operator Boundary (RFC)
• Unified Error Handling (RFC)
• Production-Grade NVIDIA Blackwell Support (RFC)