nav: - Home: README.md - User Guide: - Getting Started: - getting_started/quickstart.md - getting_started/installation/* - Serving: - OpenAI-Compatible API: - Chat Completions API: serving/chat_completions_api.md - Image Generation: serving/image_generation_api.md - Image Edit: serving/image_edit_api.md - Text to Speech: serving/speech_api.md - Videos: serving/videos_api.md - Audio Generation: serving/audio_generate_api.md - Streaming Video Input: serving/video_stream_api.md - Examples: - examples/README.md - Offline Inference: - Image-To-Image: user_guide/examples/offline_inference/image_to_image.md - Image-To-Video: user_guide/examples/offline_inference/image_to_video.md - Speech-To-Video: user_guide/examples/offline_inference/speech_to_video.md - Text-To-Audio: user_guide/examples/offline_inference/text_to_audio.md - Text-To-Image: user_guide/examples/offline_inference/text_to_image.md - Text-To-Speech: user_guide/examples/offline_inference/text_to_speech.md - Text-To-Video: user_guide/examples/offline_inference/text_to_video.md - X-To-Text: user_guide/examples/offline_inference/x_to_text.md - X-To-Video-Audio: user_guide/examples/offline_inference/x_to_video_audio.md - Online Serving: - Image-To-Image: user_guide/examples/online_serving/image_to_image.md - Image-To-Video: user_guide/examples/online_serving/image_to_video.md - Speech-To-Video: user_guide/examples/online_serving/speech_to_video.md - Text-To-Image: user_guide/examples/online_serving/text_to_image.md - Text-To-Speech: user_guide/examples/online_serving/text_to_speech.md - Text-To-Video: user_guide/examples/online_serving/text_to_video.md - General: - usage/* - Configuration: - configuration/README.md - configuration/* - Models: - models/supported_models.md - Features: - Overview: features/README.md - Runtime and Stage Execution: - Execution Modes and Streaming: user_guide/diffusion/execution_modes.md - Prefill-Decode Disaggregation (experimental): features/pd_disaggregation.md - Sleep Mode: features/sleep_mode.md - Quantization: - Overview: user_guide/quantization/overview.md - Online Quantization: user_guide/quantization/online.md - Quantized KV Cache: user_guide/quantization/quantized_kvcache.md - FP8 W8A8: user_guide/quantization/fp8.md - Int8 W8A8: user_guide/quantization/int8.md - BitsAndBytes W4: user_guide/quantization/bitsandbytes.md - MXFP8 W8A8: user_guide/quantization/mxfp8.md - MXFP4 W4A4: user_guide/quantization/mxfp4.md - ModelOpt: user_guide/quantization/modelopt.md - AutoRound: user_guide/quantization/autoround.md - msModelSlim: user_guide/quantization/msmodelslim.md - Diffusion Acceleration: - Overview and Compatibility: - Overview: user_guide/diffusion_features.md - Feature Compatibility: user_guide/feature_compatibility.md - CPU Offloading: - Overview: user_guide/diffusion/cpu_offload.md - Model-Level: user_guide/diffusion/offloader/cpu_offload.md - Layerwise: user_guide/diffusion/offloader/layerwise_offload.md - Distributed Layerwise: user_guide/diffusion/offloader/distributed_layerwise_offload.md - Cache Acceleration: - TeaCache: user_guide/diffusion/cache_acceleration/teacache.md - Cache-DiT: user_guide/diffusion/cache_acceleration/cache_dit.md - Parallelism: - Overview: user_guide/diffusion/parallelism/overview.md - CFG Parallel: user_guide/diffusion/parallelism/cfg_parallel.md - Expert Parallel: user_guide/diffusion/parallelism/expert_parallel.md - Hybrid Sharded Data Parallel: user_guide/diffusion/parallelism/hsdp.md - Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md - Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md - VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md - Attention Backends: - Overview: user_guide/diffusion/attention_backends.md - Dense Backends: user_guide/diffusion/attention_backends/dense_backends.md - TRTLLM: user_guide/diffusion/attention_backends/trtllm.md - SageAttention: user_guide/diffusion/attention_backends/sage.md - Hugging Face Hub: user_guide/diffusion/attention_backends/huggingface_hub.md - RainFusion: user_guide/diffusion/attention_backends/rainfusion.md - Regional Compilation: user_guide/diffusion/regional_compilation.md - Frame Interpolation: user_guide/diffusion/frame_interpolation.md - Startup and Loading: user_guide/diffusion/startup_and_loading.md - LoRA: user_guide/diffusion/lora.md - Experimental: - Session State Manager: features/session_state_manager.md - Integrations: - Custom Diffusion Pipeline: features/custom_pipeline.md - ComfyUI: features/comfyui.md - Fault Tolerance: - Failure Modes: user_guide/fault_injection_reliability_matrix.md - Developer Guide: - General: - contributing/README.md - glob: contributing/* flatten_single_child_sections: true - Model Implementation: - contributing/model/README.md - contributing/model/adding_omni_model.md - contributing/model/adding_tts_model.md - contributing/model/adding_diffusion_model.md - CI: contributing/ci - Design Documents: - design/index.md - Architecture Documents: - Architecture Overview: design/architecture_overview.md - Feature Design: - Runtime and Stage Execution: - Disaggregated Inference: design/feature/disaggregated_inference.md - Host Weight Runtime: design/feature/host_weight_runtime.md - Async Chunk: design/feature/async_chunk.md - Async Diffusion Output: design/feature/async_diffusion_output.md - Async Omni Output Materialization: design/feature/omni_async_output_materialization.md - Automatic Prefix Caching: design/feature/prefix_caching.md - Realtime AR-Diffusion: design/feature/realtime_ar_diffusion.md - Communication: - OmniConnector Implementations: - Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md - Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md - Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md - Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md - Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md - Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md - Quantization: design/feature/quantization.md - Diffusion Acceleration: - Parallelism: - CFG-Parallel: design/feature/cfg_parallel.md - Expert Parallel: design/feature/expert_parallel.md - Hybrid Sharded Data Parallel (HSDP): design/feature/hsdp.md - Pipeline Parallel: design/feature/pipeline_parallel.md - Sequence Parallel: design/feature/sequence_parallel.md - Tensor Parallel: design/feature/tensor_parallel.md - VAE Patch Parallelism: design/feature/vae_parallel.md - Attention Optimization: - Backend Selection: design/feature/attention_backend_selection.md - Skip-Softmax: design/feature/skip_softmax.md - CPU Offloading: - Overview: design/feature/offloader/README.md - Model-Level: design/feature/offloader/cpu_offload.md - Layerwise: design/feature/offloader/layerwise_offload.md - Distributed Layerwise: design/feature/offloader/distributed_layerwise_offload.md - Cache-DiT: design/feature/cache_dit.md - TeaCache: design/feature/teacache.md - Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md - Infrastructure and Performance: - Prometheus Metrics: design/metrics.md - Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md - Module Design: - Entrypoints: design/module/entrypoints.md - vLLM-Omni Configuration: design/module/vllm_omni_config.md - Input, Output, and Modality Contracts: design/module/input_output_modality_contracts.md - Error Contracts: design/module/error_contracts.md - Engine Orchestration: design/module/engine_orchestration.md - Stage Runtime: design/module/stage_runtime.md - OmniConnector: design/module/omni_connector.md - Model Integration: design/module/model_integration.md - Autoregressive Runtime: design/module/ar_runtime.md - Diffusion: - Overview: design/module/diffusion/index.md - Runtime: design/module/diffusion/diffusion_runtime.md - Model Integration: design/module/diffusion/diffusion_model_integration.md - Continuous Batching: design/module/diffusion/continuous_batching.md - Parallelism: design/module/diffusion/parallelism.md - Offloader: design/module/diffusion/offloader.md - Execution Platforms: design/module/execution_platforms.md - Cache Management: design/module/cache_management.md - Host Weight Runtime: design/module/host_weight_runtime.md - Quantization: design/module/quantization.md - Observability: design/module/observability.md - Profiling: design/module/profiling.md - Benchmarking: design/module/benchmarking.md - Docs Guide: contributing/DOCS_GUIDE.md - API Reference: - api/README.md - api/vllm_omni - CLI Reference: cli - Community: - Governance: community/governance.md - Contact Us: community/contact_us.md - Community Meetings: community/meetings.md - Slack: https://slack.vllm.ai - Blog: https://blog.vllm.ai - Forum: https://discuss.vllm.ai