anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
ab2f2a3d658db5be5ec96a950da55cb74b48e7aa
vllm-omni/docs/engines/vllm_omni.md
T
tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork 
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:01:12 +08:00

80 KiB
Raw Blame History

vLLM-Omni (vllm-project/vllm-omni) — Engine Analysis

Field Value
Local path /data/zhoutaichang/embedding_infer/vllm-omni (links below are relative to analysis/engines/, pointing into the sibling vllm-omni/ checkout)
Remote https://github.com/vllm-project/vllm-omni.git (origin)
Branch main
HEAD 7be014bce6374f06c95b703763bdbac4c6198f31 — 2026-09-07 "[Bugfix][NPU] Fix MiniMax-H3 INT8 quantization dispatch (#6876)"
Dirty status clean (git status --short empty)
Submodules none (git submodule status empty)
Analysis date 2026-09-08
Installed package caveat The Python environment has vllm-omni installed editable from a different checkout: /home/zhoutaichang/feature/vllm-omni-main @ be335a86f (2026-08-28), version string 0.28.0rc2.dev22+gbe335a86f, with vllm==0.28.0. This report analyses the newer 7be014bc tree only; nothing was executed. [Verified]
Size 1,382 .py files / 508,455 LOC under vllm_omni/; 912 test_*.py files; 266 files under docs/; 90 deploy YAMLs. [Verified]

Evidence labels: [Verified] read in code/docs; [Inferred] reasoned from structure; [Proposal] design suggestion; [Unknown] not determinable from the tree.

0. Summary

  • vLLM-Omni is a Python-only serving framework layered on top of vLLM (v1 engine); it does not ship its own kernels, allocator, or C++ runtime. Its setup.py builds a pure-Python wheel and only selects a platform-specific requirements/*.txt (cuda|rocm|npu|xpu|musa|cpu). [Verified] (../../vllm-omni/setup.py, ../../vllm-omni/requirements/common.txt)
  • vLLM is not pinned in packaging metadata. No vllm== appears in requirements/ or pyproject.toml; instead version.py warns at import if major.minor differ, and docs pin vllm==0.28.0 for the 0.28.x line. [Verified] (../../vllm-omni/vllm_omni/version.py, ../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md)
  • It imports 284 distinct vllm.* modules and 104 files import vllm.v1.* internals; patch.py monkeypatches vLLM classes at import (EngineCoreRequest/Output(s), Request, StreamingUpdate, TokensPrompt, MRotaryEmbedding, ModelConfig.is_mm_prefix_lm, CuMemAllocator, NVFP4/FP8 kernels, torch inductor). Backend substitution below vLLM is therefore not a plugin exercise; it is a re-implementation of the AR runtime. [Verified] (../../vllm-omni/vllm_omni/patch.py)
  • Core abstraction: a model is a frozen PipelineConfig of stages (LLM_AR, LLM_GENERATION, DIFFUSION), and a DeployConfig YAML places stages on devices, sizes them, and wires named connectors. 60 pipelines are registered in OMNI_PIPELINES. [Verified] (../../vllm-omni/vllm_omni/config/stage_config.py, ../../vllm-omni/vllm_omni/config/pipeline_registry.py)
  • Each AR/generation stage is a separate vLLM EngineCoreProc subprocess (spawned by StageEngineCoreProcManager) reached over ZMQ; diffusion stages run either in a separate StageDiffusionProc (ZMQ PUSH/PULL) or inline in the orchestrator process. An Orchestrator asyncio loop routes outputs stage→stage. [Verified]
  • Inter-stage tensors travel host-side by default ("D2H2D") through OmniConnectors: SharedMemoryConnector (POSIX shm + msgpack); Mooncake (TCP/RDMA) and Yuanrong (Ascend) for multi-node, Mori (RDMA, plus an intra-node AMD xgmi backend configured in deploy/qwen3_omni_moe_mori_intranode.yaml); MooncakeTransferEngineConnector.put and the Mori connector pass torch.Tensor/ManagedBuffer payloads without msgpack serialization and can use device memory pools (repo docs still describe all connectors as D2H2D); none of these device paths is validated for an edge target. [Verified] (../../vllm-omni/vllm_omni/distributed/omni_connectors/)
  • Streaming audio is produced by async_chunk: talker decode steps accumulate codec frames (codec_chunk_frames, default 25, with a smaller dynamic first chunk) and push them to the code2wav stage, which decodes chunk-by-chunk while the request sits in a patched RequestStatus.WAITING_FOR_CHUNK. [Verified] (../../vllm-omni/docs/design/feature/async_chunk.md)
  • Platforms actually in tree: CUDA, ROCm, Ascend NPU (via vllm_ascend), Intel XPU, Moore Threads MUSA (via vllm_musa). There is no CPU platform/worker; requirements/cpu.txt exists only for docs builds and the fallback UnspecifiedOmniPlatform reports device_count()==0. [Verified] (../../vllm-omni/vllm_omni/platforms/__init__.py)
  • No edge/mobile targets: no Jetson/ARM/Metal/CoreML/Vulkan/OpenCL/QNN/NNAPI code, no ONNX/TensorRT export path (onnxruntime is used only to load a few speaker-embedding / codec helper models). [Verified]
  • TTS is a first-class citizen: 15 TTS model families, an OpenAI /v1/audio/speech endpoint with a per-model TTSModelAdapter registry, WebSocket streaming /v1/audio/speech/stream, PCM/WAV/MP3/Opus encoding via soundfile, and a repo-published Qwen3-TTS TTFP of 64 ms (H200, conc. 1; repo-reported, not reproduced). [Verified]
  • VLA is present but server-side: π0, GR00T-N1.7, InternVLA-A1 (single diffusion stage returning actions), DreamZero-DROID and LingBot-World (AR-diffusion world models with engine-level paged KV), Cosmos3 action modes; served over WS /v1/realtime/robot/openpi (msgpack). Actions are not streamed; one observation → one action chunk. [Verified]
  • Diffusion has its own runtime (DiffusionEngine, RequestScheduler/StepScheduler, DiffusionWorker, 11 attention backends, cache-DiT/TeaCache/MagCache/StepCache, CPU/layerwise/distributed offload, HSDP/Ulysses/Ring/CFG/VAE parallel) that reuses vLLM layers/quant/KVCacheManager as a library. [Verified]
  • Compilation: AR stages inherit vLLM CUDA graphs/torch.compile; diffusion uses regionally_compile() (torch.compile per repeated block) and per-model CUDA-graph wrappers (e.g., CUDAGraphDecoderWrapper for Qwen3-TTS code2wav). [Verified]
  • Quantization is inherited from vLLM (quantization_config path) with omni additions (Int8, MXFP4/MXFP8 for NPU, SVDQuant, quack FP8, bitsandbytes); non-AR TTS stages stay BF16 per docs. [Verified] (../../vllm-omni/docs/user_guide/quantization/overview.md)
  • Bottom line for the edge study: vLLM-Omni is the reference serving stack (pipeline/stage/streaming semantics, model ports, API contracts) but its execution runtime is vLLM+CUDA-class accelerators; an edge port must replace the AR runtime (vLLM EngineCore/scheduler/worker) and the diffusion worker while keeping PipelineConfig, stage input processors, connector payload schemas, and the OpenAI/OpenPI protocol layers. [Inferred]

1. Architecture overview

graph TD
  subgraph Entry["Entrypoints (vllm_omni/entrypoints)"]
    CLI["vllm serve --omni<br/>cli/serve.py"]
    API["OpenAI-compatible FastAPI<br/>openai/api_server.py"]
    OFF["Omni / AsyncOmni<br/>omni.py / async_omni.py"]
    DUP["Duplex + Realtime WS<br/>duplex/*, openai/realtime_connection.py"]
    OPI["OpenPI robot WS<br/>openpi/serving.py"]
  end
  subgraph Ctrl["Control plane (vllm_omni/engine)"]
    ENG["AsyncOmniEngine<br/>async_omni_engine.py"]
    ORCH["Orchestrator<br/>orchestrator.py"]
    RT["StageRuntime / DistStageRuntime<br/>stage_runtime.py"]
    POOL["StagePool (per logical stage)<br/>stage_pool.py"]
    MS["OmniMasterServer (ZMQ ROUTER)<br/>stage_engine_startup.py"]
  end
  subgraph Cfg["Config (vllm_omni/config)"]
    PC["PipelineConfig (frozen topology)<br/>stage_config.py"]
    DC["DeployConfig YAML<br/>vllm_omni/deploy/*.yaml"]
    REG["OMNI_PIPELINES<br/>pipeline_registry.py"]
  end
  subgraph AR["AR / generation stage (subprocess)"]
    ECP["StageEngineCoreProc (vLLM EngineCoreProc)"]
    SCH["OmniARScheduler / OmniGenerationScheduler<br/>core/sched/"]
    WRK["GPUARWorker / GPUGenerationWorker<br/>worker/"]
    MR["GPUARModelRunner / GPUGenerationModelRunner"]
    MODEL["Omni model classes<br/>model_executor/models/*"]
  end
  subgraph DIFF["Diffusion stage (subprocess or inline)"]
    SDP["StageDiffusionProc (ZMQ)"]
    DE["DiffusionEngine + RequestScheduler/StepScheduler"]
    DW["DiffusionWorker / DiffusionModelRunner"]
    PIPE["Pipelines<br/>diffusion/models/*"]
  end
  CONN["OmniConnector (SHM / Mooncake / Mori / Yuanrong)<br/>distributed/omni_connectors"]
  PLAT["OmniPlatform (cuda/rocm/npu/xpu/musa)<br/>platforms/"]
  VLLM["vLLM 0.28.x (patched by vllm_omni/patch.py)"]

  CLI --> API --> ENG
  OFF --> ENG
  DUP --> ENG
  OPI --> ENG
  ENG --> ORCH
  ENG --> RT
  RT --> POOL
  RT --> MS
  REG --> PC --> ENG
  DC --> ENG
  POOL -->|ZMQ| ECP
  POOL -->|ZMQ / inline| SDP
  ECP --> SCH --> WRK --> MR --> MODEL
  SDP --> DE --> DW --> PIPE
  MR -->|put/get chunks, KV| CONN
  DW -->|put/get| CONN
  WRK --> PLAT
  DW --> PLAT
  ECP --> VLLM
  MR --> VLLM

Key modules:

Module Path Responsibility Key symbols
Package init / patches ../../vllm-omni/vllm_omni/__init__.py, ../../vllm-omni/vllm_omni/patch.py Version check, then monkeypatch vLLM internals at import warn_if_misaligned_vllm_version, _patched_is_mm_prefix_lm, _patch_cumem_free_callback_cuda
Engine args ../../vllm-omni/vllm_omni/engine/arg_utils.py OmniEngineArgs(EngineArgs) per-stage fields; OrchestratorArgs; registers omni models into vLLM via entry point OmniEngineArgs, OrchestratorArgs, register_omni_models_to_vllm
Composition root ../../vllm-omni/vllm_omni/engine/async_omni_engine.py Resolves stage configs, boots stages, hosts orchestrator loop, request submit/abort/RPC/sleep AsyncOmniEngine, _create_default_diffusion_stage_cfg, add_request_async, abort_async
Orchestrator ../../vllm-omni/vllm_omni/engine/orchestrator.py Cross-stage routing, output ordering, cancellation, CFG companions, PD pair Orchestrator, _route_output, _forward_to_next_stage_unguarded, _prewarm_async_chunk_stages, _handle_abort
Stage runtime ../../vllm-omni/vllm_omni/engine/stage_runtime.py, ../../vllm-omni/vllm_omni/engine/stage_pool.py Expand logical stages into local/remote replicas; replica selection/affinity; metrics StageRuntime, DistStageRuntime, StagePool.pick, submit_initial
Stage process launch ../../vllm-omni/vllm_omni/engine/stage_engine_startup.py, ../../vllm-omni/vllm_omni/engine/stage_engine_core_proc.py, ../../vllm-omni/vllm_omni/engine/stage_engine_core_proc_manager.py ZMQ address allocation, headless replica registration, spawn of EngineCoreProc subclasses OmniMasterServer, StageEngineCoreProc.run_stage_core, launch_stage_replica
Stage client ../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py vLLM AsyncMPClient subclass per replica; KV sender endpoint StageEngineCoreClient, DPLBStageEngineCoreClient
Pipeline/deploy config ../../vllm-omni/vllm_omni/config/stage_config.py, ../../vllm-omni/vllm_omni/config/pipeline_registry.py, ../../vllm-omni/vllm_omni/config/omni_config.py Frozen topology, deploy overlay, precedence, scheduler resolution PipelineConfig, StagePipelineConfig, StageDeployConfig, StageExecutionType, _resolve_scheduler, OMNI_PIPELINES, register_pipeline
AR scheduler ../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py, ../../vllm-omni/vllm_omni/core/sched/omni_scheduler_mixin.py vLLM Scheduler subclass adding chunk/KV-transfer hooks OmniARScheduler, OmniARAsyncScheduler, OmniSchedulerMixin
Generation scheduler ../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py One-shot non-AR stage scheduling (code2wav, DiT-in-vLLM) OmniGenerationScheduler.schedule
Workers/runners ../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py, ../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py, ../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py, ../../vllm-omni/vllm_omni/worker/omni_connector_model_runner_mixin.py vLLM GPUWorker/GPUModelRunner subclasses; multimodal outputs, talker MTP, connector I/O GPUARWorker, GPUARModelRunner.execute_model, GPUGenerationModelRunner, OmniConnectorModelRunnerMixin.send_chunk
Model registry ../../vllm-omni/vllm_omni/model_executor/models/registry.py, ../../vllm-omni/vllm_omni/diffusion/registry.py 89 omni AR/generation archs + all vLLM archs; 66 diffusion pipelines OmniModelRegistry, DiffusionModelRegistry, register_diffusion_model
Stage input processors ../../vllm-omni/vllm_omni/model_executor/stage_input_processors/ Model-owned functions converting stage-N output → stage-N+1 input (full payload or async chunk) talker2code2wav_async_chunk (qwen3_tts.py)
Diffusion engine ../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py, ../../vllm-omni/vllm_omni/diffusion/sched/, ../../vllm-omni/vllm_omni/diffusion/executor/, ../../vllm-omni/vllm_omni/diffusion/worker/ Request/step batching, uniproc/multiproc executors, worker & runner DiffusionEngine, RequestScheduler, StepScheduler, MultiprocDiffusionExecutor, DiffusionWorker, DiffusionModelRunner
Diffusion attention ../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py, ../../vllm-omni/vllm_omni/diffusion/attention/selector.py, ../../vllm-omni/vllm_omni/diffusion/attention/layer.py Enum registry, per-role selection, platform defaults DiffusionAttentionBackendEnum, register_diffusion_backend, get_attn_backend_for_role, Attention
Connectors ../../vllm-omni/vllm_omni/distributed/omni_connectors/ Transport-only put/get + chunk transfer adapter OmniConnectorBase, SharedMemoryConnector, OmniConnectorFactory, OmniChunkTransferAdapter
Platforms ../../vllm-omni/vllm_omni/platforms/interface.py, ../../vllm-omni/vllm_omni/platforms/__init__.py Backend detection, worker class selection, diffusion attention defaults OmniPlatform, resolve_current_omni_platform_cls_qualname, CudaOmniPlatform, NPUOmniPlatform
I/O types ../../vllm-omni/vllm_omni/inputs/data.py, ../../vllm-omni/vllm_omni/outputs/__init__.py, ../../vllm-omni/vllm_omni/data_entry_keys.py, ../../vllm-omni/vllm_omni/request.py Prompt types with embeds/payloads; OmniRequestOutput; typed inter-stage payload OmniTokensPrompt, OmniRequestOutput, OmniPayload, OmniRequest
Serving ../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py, ../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py, ../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py HTTP/WS routes; TTS request building & audio encoding; robot policy WS omni_run_server, create_speech, ServingRealtimeRobotOpenPI.infer
Experimental ../../vllm-omni/vllm_omni/experimental/ar_diffusion/engine.py, ../../vllm-omni/vllm_omni/experimental/world_models/ AR-diffusion (world model) engine with paged KV; session state ARDiffusionEngine, ARDiffusionModelRunner

2. Device layer

Backends and registration. vllm_omni/platforms/__init__.py defines five builtin probe functions and an entry-point group vllm_omni.platform_plugins for out-of-tree platforms; exactly one may activate. [Verified] (../../vllm-omni/vllm_omni/platforms/__init__.py — builtin_omni_platform_plugins, resolve_current_omni_platform_cls_qualname)

Platform Probe Class (inherits vLLM/vendor platform) AR/generation worker Notes
CUDA pynvml.nvmlDeviceGetCount()>0 CudaOmniPlatform(OmniPlatform, CudaPlatformBase) ../../vllm-omni/vllm_omni/platforms/cuda/platform.py GPUARWorker, GPUGenerationWorker Default deploy dir vllm_omni/deploy; diffusion attention default chain (TRTLLM→cuDNN→FlashInfer→FA→SDPA) keyed on SM version [Verified]
ROCm amdsmi handles>0 RocmOmniPlatform(OmniPlatform, RocmPlatform) ../../vllm-omni/vllm_omni/platforms/rocm/platform.py same GPU workers Forces TRITON_ATTN/ROCM_AITER_FA for AR; patch_groupnorm.py; several deploy YAMLs flip code2wav to eager (MIOpen not capture-safe) [Verified]
Ascend NPU torch.npu.is_available() NPUOmniPlatform(OmniPlatform, vllm_ascend.NPUPlatform) ../../vllm-omni/vllm_omni/platforms/npu/platform.py NPUARWorker, NPUGenerationWorker (../../vllm-omni/vllm_omni/platforms/npu/worker/) HCCL; applies model patches for code2wav conv ops, 310P patches; YuanrongTransferEngineConnector; ACL graph wrapper [Verified]
Intel XPU torch.xpu.is_available() XPUOmniPlatform(OmniPlatform, XPUPlatform) ../../vllm-omni/vllm_omni/platforms/xpu/platform.py XPUARWorker, XPUGenerationWorker Disables XPU sampler kernel via env [Verified]
MUSA torchada.is_musa_platform() MUSAOmniPlatform(OmniPlatform, vllm_musa.MUSAPlatformBase) ../../vllm-omni/vllm_omni/platforms/musa/platform.py GPU workers requires torchada, mate, flash_attn_3 (../../vllm-omni/requirements/musa.txt) [Verified]
CPU — UnspecifiedOmniPlatform (device_type="cpu", get_device_count()==0) none No CPU worker/runner exists; cli/serve.py falls back to vLLM CpuPlatform only for argument parsing [Verified]

Device discovery/selection. Placement is declarative: each stage's devices: "0,1" in the deploy YAML must equal TP×DP_local×PP (×replicas); StageRuntime._scoped_spawn_device_env sets the platform's device_control_env_var (e.g., CUDA_VISIBLE_DEVICES) around subprocess spawn, and GPUARWorker.init_device maps local rank → visible device via current_omni_platform.get_torch_device. [Verified] (../../vllm-omni/vllm_omni/engine/stage_runtime.py, ../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py, ../../vllm-omni/docs/configuration/stage_configs.md)

Memory allocation/pools.

  • AR/generation stages inherit vLLM's paged KV allocation; OmniGPUWorkerBase.determine_available_memory re-implements the budget as total*gpu_memory_utilization − (weights + peak activation + non-torch) measured by memory_profiling around profile_run(), or honors explicit kv_cache_memory_bytes. [Verified] (../../vllm-omni/vllm_omni/worker/base.py)
  • Multi-stage co-location on one GPU is guarded by stage_admission.py (DeviceLedger, check_admission) using per-stage gpu_memory_utilization plus a CUDA-graph reserve, and by SH/EX device locks for sequential profiling (parallel_stage_init). [Verified] (../../vllm-omni/vllm_omni/engine/stage_admission.py)
  • Sleep mode uses vLLM's CuMemAllocator tagged pools (_maybe_get_memory_pool_context), patched in patch.py to avoid a CUDA double-free on shutdown. [Verified]
  • Diffusion workers: DiffusionWorker.init_device calls init_distributed_environment/initialize_model_parallel from vllm_omni.diffusion.distributed, and only takes a MemorySnapshot/KV budget when diffusion_kv_mode=paged_scheduler. Host-side: pinned memory for offload (pin_cpu_memory), CUDA host registration for DLO, and the Host Weight Runtime (mmap'd safetensors shared across same-host workers). [Verified] (../../vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py, ../../vllm-omni/vllm_omni/diffusion/offloader/, ../../vllm-omni/docs/design/feature/host_weight_runtime.md)
  • NVML process-scoped accounting helpers remain for diffusion memory reporting. [Verified] (../../vllm-omni/vllm_omni/worker/gpu_memory_utils.py)

Host↔device transfers. Inter-stage data is always staged to host: connector put() serializes CPU tensors (msgpack, OmniMsgpackEncoder, "zero-copy not implemented yet") into a POSIX shm segment; the receiving stage deserializes and re-uploads. GPUARModelRunner snapshots multimodal outputs to CPU on a dedicated copy stream (_get_or_create_omni_payload_copy_stream, _snapshot_tensor_payload_to_cpu_async) so D2H does not block decode ("async omni output materialization"). [Verified] (../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py, ../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py, ../../vllm-omni/docs/design/feature/omni_async_output_materialization.md)

Synchronization/streams. OmniPlatform.record_device_event() marks tensor readiness for async diffusion output (NPU overrides to sync the default stream first, since HCCL uses hidden streams). Prefix cache writes record a CUDA event on the copy stream (_PendingAsyncWrite). Ruff bans torch.cuda.* in favor of torch.accelerator.* to keep code device-neutral. [Verified] (../../vllm-omni/vllm_omni/platforms/interface.py, ../../vllm-omni/vllm_omni/core/prefix_cache.py, ../../vllm-omni/pyproject.toml)

Portability. Linux only (docs/getting_started/quickstart.md: "OS: Linux"); Python 3.10–3.13; Dockerfiles for cuda/rocm/xpu/npu (A2/A3). No macOS/Windows/ARM CPU/mobile code paths were found. [Verified] (../../vllm-omni/docker/Dockerfile.cuda, ../../vllm-omni/docker/Dockerfile.npu)

3. Kernel layer

Operator set & registration. vLLM-Omni has no compiled extension of its own; operators come from (a) vLLM's layers (vllm.model_executor.layers.{linear,layernorm,activation,rotary_embedding,fused_moe,quantization} — 62/25/15/9/5/17 import sites respectively) and vLLM's custom-op/IR mechanism, (b) PyTorch/Triton code inside model files (e.g., attention/fish_kvcache_triton.py, diffusion/layers/mot/ops/mot_gemm.py, diffusion/layers/fused_qk_norm_rope.py), and (c) optional external kernel packages (flash-attn 2/3/4, FlashInfer, SageAttention 2/3, cuDNN SDPA, HF kernels hub, quack FP8, TRT-LLM gen). [Verified]

Dispatch mechanisms.

  • AR stages: attention/backend dispatch is vLLM's (attention_backend engine arg, forced to TRITON_ATTN/ROCM_AITER_FA on ROCm by stage_init_utils.py). Omni adds a Fish-Speech-specific KV-cache attention Triton path selected by model arch (is_fish_kvcache_attn_active_for_model) with fallback counters. [Verified] (../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py)
  • Diffusion attention: Attention layer → get_attn_backend_for_role(role, head_size, attention_config) with precedence per-role config → role category → global default → current_omni_platform.get_diffusion_attn_backend_cls(); classes resolved from DiffusionAttentionBackendEnum (11 members) with runtime overrides via register_diffusion_backend. Each backend declares supported_platforms, validate_available(), head-size sets, mask/paged-KV capabilities. [Verified] (../../vllm-omni/vllm_omni/diffusion/attention/selector.py, ../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py, ../../vllm-omni/vllm_omni/diffusion/attention/backends/abstract.py)
  • Diffusion generic ops: CustomOp.dispatch_forward() picks forward_cuda|hip|npu|xpu|musa|native from current_omni_platform; HIP/MUSA default to the CUDA path, XPU to native. [Verified] (../../vllm-omni/vllm_omni/diffusion/layers/custom_op.py)
  • IR op priority: CUDA platform prefers ["vllm_c","native"] (optionally oink for rms_norm); per-pipeline get_*_ir_op_priority_func (Cosmos3) can merge overrides. [Verified] (../../vllm-omni/vllm_omni/platforms/cuda/platform.py, ../../vllm-omni/vllm_omni/diffusion/registry.py)

Quantization formats & where dequant happens. Unified quantization_config path (../../vllm-omni/vllm_omni/quantization/factory.py); vLLM quant methods are reused (ModelOpt FP8/NVFP4, AutoRound W4A16, TorchAO, compressed-tensors) and omni adds int8_config.py, mxfp4_config.py, mxfp8_config.py (Ascend), svdquant_config.py, bitsandbytes_config.py, inc_config.py, quack_fp8.py (Blackwell fused-bias FP8 GEMM). Dequant/scaled-GEMM occurs inside vLLM's LinearMethod.apply/ScaledMMLinearKernel (patched in patch.py for non-contiguous batched diffusion activations and FlashInfer output shape). GGUF loading for diffusion via gguf_adapters (requirements/common.txt). Runtime FP8 attention quant for diffusion FA (AttnQuantSpec). Docs state omni/TTS quantization is scoped to the AR LM stage; non-AR stages stay BF16. [Verified] (../../vllm-omni/docs/user_guide/quantization/overview.md, ../../vllm-omni/vllm_omni/quantization/)

Compilation / JIT / graph capture.

  • AR stages: vLLM compilation_config (cudagraph_mode: PIECEWISE|FULL, custom_ops) is settable per stage in deploy YAML (see platforms.npu.stages[*].compilation_config in ../../vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml). GPUARModelRunner._capture_talker_mtp_graphs captures a dedicated CUDA graph for the talker MTP/code-predictor path; supports_talker_mtp_graph_capture() is a platform hook. NPU uses ACLGraphWrapper via get_graph_wrapper_cls(). [Verified]
  • Generation stages (code2wav): model-owned wrappers, e.g., CUDAGraphDecoderWrapper with capture sizes derived from chunk ramp/left context (decode_cudagraph_capture_sizes, decode_cudagraph_batch_sizes connector extras). [Verified] (../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py, ../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml)
  • Diffusion: regionally_compile() applies torch.compile to each repeated block (_repeated_blocks), keeping offload hook wrappers outside the graph; diffusion_compile_granularity="regional", diffusion_compile_dynamic=True defaults; enforce_eager disables. patch.py also patches torch inductor's statically_known_multiple_of for FLUX fp8 dynamic shapes. [Verified] (../../vllm-omni/vllm_omni/diffusion/compile.py, ../../vllm-omni/docs/user_guide/diffusion/regional_compilation.md)

Fallback when an op is unsupported. Diffusion attention: explicit selection of an unavailable backend raises at platform resolution (fail-fast); automatic selection degrades to TORCH_SDPA. CustomOp.forward_native is the last resort. Fish KV attention records fallback reasons and uses vLLM attention otherwise. Platform-specific model patches (NPU code2wav conv2d, ROCm groupnorm) replace individual ops. [Verified]

Extension points.

  • Out-of-tree platform: entry point vllm_omni.platform_plugins returning a class path; general plugins vllm_omni.general_plugins. [Verified] (../../vllm-omni/vllm_omni/plugins/__init__.py)
  • Attention backend: register_diffusion_backend(enum, "pkg.Class"). [Verified]
  • Model: register_diffusion_model(arch, module, cls, pre/post-process funcs); AR models via OmniModelRegistry (vLLM _ModelRegistry); pipelines via register_pipeline. [Verified]
  • Connector: OmniConnectorFactory.register_connector(name, ctor). [Verified] (../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py)
  • Custom diffusion pipeline without registry edits: diffusion_load_format="dummy" + custom_pipeline_args={"pipeline_class": ...} + CustomPipelineWorkerExtension. [Verified] (../../vllm-omni/docs/features/custom_pipeline.md)
  • TTS serving adapter: @register_tts_adapter subclasses of TTSModelAdapter (ARTTSAdapter, DiffusionTTSAdapter). [Verified] (../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/__init__.py, ../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/base.py)
  • Contributor guides: ../../vllm-omni/docs/contributing/model/adding_omni_model.md, ../../vllm-omni/docs/contributing/model/adding_tts_model.md, ../../vllm-omni/docs/contributing/model/adding_diffusion_model.md. [Verified]

4. Model runner

Model loading / formats.

  • AR/generation stages: standard vLLM loader path (HF config.json + safetensors; trust_remote_code); the omni arch is looked up in OmniModelRegistry (vLLM archs + _OMNI_MODELS), and register_omni_models_to_vllm is installed as a vllm.general_plugins entry point so vLLM worker subprocesses that only import vllm still see omni archs. Per-stage hf_config_name, model_subdir, tokenizer_subdir, model_path_resolver select sub-configs inside multi-component repos. [Verified] (../../vllm-omni/pyproject.toml, ../../vllm-omni/vllm_omni/config/stage_config.py)
  • Diffusion stages: DiffusersPipelineLoader reads Diffusers-style model_index.json repos (safetensors index or unindexed shards, GGUF via download_gguf, ModelOpt/direct-mmap checkpoint adapters, multithreaded weight load, HWR host-weight plan); diffusion_load_format ∈ {default, dummy, diffusers} where diffusers runs the stock diffusers pipeline through DiffusersAdapterPipeline. Some VLA pipelines self-load (π0 loads model.safetensors in __init__; GR00T uses AutoModel/AutoProcessor). [Verified] (../../vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py, ../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py, ../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py)
  • No export/conversion pipeline exists (no ONNX/TensorRT/GGUF-export tooling; GGUF is consumed, not produced). [Verified]
  • Pipeline auto-detection: StageConfigFactory.create_from_model() maps HF model_type/architectures/_class_name to OMNI_PIPELINES; single-stage diffusion models without a registered pipeline use _create_default_diffusion_stage_cfg. [Verified] (../../vllm-omni/vllm_omni/config/config_factory.py, ../../vllm-omni/vllm_omni/engine/async_omni_engine.py)

Execution lifecycle (init → warmup → step).

  1. AsyncOmniEngine.__init__ → _resolve_stage_configs → StageRuntime.initialize() builds LogicalStageInitPlan/ReplicaInitPlan, runs admission, spawns replicas (LLM via launch_stage_replica → StageEngineCoreProcManager → StageEngineCoreProc.run_stage_core; diffusion via StageDiffusionProcManager.launch_headless or InlineStageDiffusionClient). [Verified]
  2. Inside an LLM stage, vLLM's EngineCore init runs init_device → load_model → determine_available_memory (profile run) → initialize_kv_cache → compile_or_warm_up_model (CUDA graph capture); omni overrides load_model (_init_talker_mtp, _prewarm_attention_capture_workspaces) and capture_model (_capture_talker_mtp_graphs). [Verified] (../../vllm-omni/vllm_omni/worker/gpu_model_runner.py, ../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py)
  3. Diffusion: DiffusionEngine.make_engine() → executor init → load_model → run_startup_warmup() (_dummy_run, KV profile requests when paged) → _busy_loop thread performing schedule → execute_batch/execute_step → update_from_output → per-request asyncio queue. [Verified] (../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py, ../../vllm-omni/docs/design/feature/diffusion_continuous_batching.md)
  4. Orchestrator _prewarm_async_chunk_stages pre-submits downstream stage requests when async_chunk is on, so they can park in WAITING_FOR_CHUNK. [Verified]
  5. Step: AR stage GPUARModelRunner.execute_model → _preprocess (model-specific preprocess builds inputs_embeds; _build_model_kwargs_extra) → _model_forward → sample_tokens (may run _run_post_sample_talker_mtp) → _build_omni_step_outputs → OmniModelRunnerOutput with multimodal_outputs/pooler_output. [Verified]

Dynamic shapes. AR stages rely on vLLM's padded batch buckets (max_cudagraph_capture_size, cudagraph_mode); talker MTP graphs are captured per batch bucket. Code2wav graph shapes are derived from the chunk-size schedule (initial_codec_chunk_frames, codec_chunk_frames, codec_left_context_frames, optional codec_chunk_ramp/adaptive ramp) and batch buckets; eager fallback on unmatched shapes (_get_padded_size returns None). Diffusion uses diffusion_compile_dynamic=True and per-request DiffusionRequestBatch slicing. [Verified]

State & KV cache.

  • AR/generation: inherits vLLM paged KV (KVCacheManager, block tables, prefix caching, chunked prefill, preemption). Omni additions: OmniTensorPrefixCache stores hidden states and multimodal tensors aligned to vLLM block slot mappings so prefix hits reuse stage outputs (../../vllm-omni/vllm_omni/core/prefix_cache.py, ../../vllm-omni/docs/design/feature/prefix_caching.md); retains_state_across_chunks keeps a request resident while awaiting chunks; KV transfer between stages (omni_kv_config, send_kv_cache/recv_kv_cache, rank-aware keys) for PD disaggregation and thinker→talker reuse. [Verified]
  • Across calls (sessions): streaming input sessions (OmniStreamingUpdate, resumable requests, session_id) and duplex sessions retain KV between segments; session_mode: duplex in deploy YAML. [Verified] (../../vllm-omni/vllm_omni/request.py, ../../vllm-omni/docs/design/fullduplex.md)
  • Diffusion: DiffusionKVCacheMode ∈ {dense_legacy, paged_scheduler}; DiffusionKVCacheManager wraps vLLM KVCacheManager (caching disabled) for scheduler-owned paged KV in AR-DiT/world models ("unified AR/DiT paged KV cache runtime", 0.28.0). Experimental ARDiffusionEngine keeps runner-owned paged KV per session with sliding window + sink frames (ARDiffusionKVCacheSpec). Prompt-embedding cache (enable_prompt_embed_cache). Eviction: LRU-style session accounting in experimental/world_models/session_state. [Verified] (../../vllm-omni/vllm_omni/diffusion/diffusion_kv/manager.py, ../../vllm-omni/vllm_omni/experimental/ar_diffusion/capability.py)

Multimodal stages.

  • Encoders (audio/vision) live inside the thinker model class (vLLM multimodal processors, mm_features); requires_multimodal_data lets a downstream stage receive raw media. [Verified]
  • Projectors/talkers: LLM_AR stage with engine_output_type="latent" emitting codec codes via multimodal_outputs. [Verified]
  • Decoders/vocoders: LLM_GENERATION stage (GPUGenerationModelRunner: no logits/sampling, returns tensors via pooler_output) e.g. Qwen3TTSCode2Wav, Qwen3OmniMoeCode2Wav, MossTTSCodecDecoder, FishSpeechDACDecoder, IndexTTS2S2MelDecoder; or a DIFFUSION stage for flow-matching decoders (CosyVoice3 DiT, MiniMaxMusic3Acoustic, OmniVoicePipeline). [Verified] (../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py, ../../vllm-omni/vllm_omni/model_executor/models/registry.py)
  • Diffusion (image/video/audio/action) as DIFFUSION stage pipelines. [Verified]

5. Scheduler

(a) Request / token / pipeline-level scheduling

  • Per-stage continuous batching is inherited from vLLM: each LLM stage runs its own Scheduler subclass chosen by _resolve_scheduler(execution_type, async_scheduling): OmniARScheduler (sync), OmniARAsyncScheduler (vLLM AsyncScheduler), OmniGenerationScheduler; diffusion stages return None (own scheduler). Batching knobs are per stage (max_num_seqs, max_num_batched_tokens, enable_chunked_prefill, enable_prefix_caching, async_scheduling). [Verified] (../../vllm-omni/vllm_omni/config/stage_config.py)
  • AR schedule hooks: OmniARScheduler.schedule drops aborted requests, _process_pending_omni_inputs, optionally defers waiting admission, calls super().schedule(), restores omni wait queues, post-processes with cached payloads, and wraps output with finished_requests_needing_kv_transfer. update_from_output triggers chunk_transfer_adapter.save_async per decode step and handles KV-transfer readiness. [Verified] (../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py)
  • Generation schedule (OmniGenerationScheduler.schedule): one-shot fast path feeding all input tokens at once (placeholder token if 0), chunk-aware: skips placeholder tokens when no new chunk arrived, finishes when is_done_receiving_chunks. [Verified]
  • Chunk lifecycle (OmniChunkTransferAdapter): keys {req_id}_{stage_id}_{chunk_id}; background recv_loop/save_loop threads; process_pending_chunks parks requests in WAITING_FOR_CHUNK (enum value −1 added by patch.py), restore_queues re-admits when ready, _preempt_non_active_running with active_stream_window. Full-payload (non-async) handoff uses WAITING_FOR_INPUT (−2) and OmniSchedulingCoordinator. [Verified] (../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py, ../../vllm-omni/vllm_omni/core/sched/omni_scheduling_coordinator.py)
  • Pipeline-level: Orchestrator._orchestration_loop (poll) or _orchestration_loop_event_driven (VLLM_OMNI_EVENT_DRIVEN_ORCH=1) reads each replica's EngineCoreOutputs, runs MultimodalOutputProcessor, then _route_output: final-output stages → OutputMessage to client queue; non-final → _forward_to_next_stage (unless the next stage receives async chunks, in which case data flows via connectors and the stage was pre-submitted). Diffusion next stages get inputs via custom_process_input_func(source_outputs, prompt, requires_multimodal_data); CFG companions are bundled (CfgCompanionTracker); PD pairs forward original prompt + kv_transfer_params. [Verified] (../../vllm-omni/vllm_omni/engine/orchestrator.py, ../../vllm-omni/vllm_omni/engine/cfg_companion_tracker.py)
  • Replica selection / load balancing: StagePool.pick/select_replica_id with LoadBalancer policies (--omni-lb-policy, default random), request→replica binding for affinity; OmniCoordinator tracks heartbeats for distributed replicas. [Verified] (../../vllm-omni/vllm_omni/engine/stage_pool.py, ../../vllm-omni/vllm_omni/distributed/omni_coordinator/omni_coordinator.py)
  • Diffusion scheduling: REQUEST_BATCH (RequestScheduler: fuse compatible requests by RequestBatchSamplingParamsKey, optional request_batch_max_wait_ms admission window) or STEP_BATCH (StepScheduler: continuous batching at denoise-step granularity, max_num_seqs>1), plus streaming_output for chunked video and submit_interaction for mid-stream prompt events. [Verified] (../../vllm-omni/vllm_omni/diffusion/sched/request_scheduler.py, ../../vllm-omni/vllm_omni/diffusion/sched/step_scheduler.py, ../../vllm-omni/docs/user_guide/diffusion/execution_modes.md)
  • Cancellation/abort: AsyncOmni.abort → AsyncOmniEngine.abort_async → AbortRequestMessage → Orchestrator._handle_abort → StagePool.abort_requests on every stage holding the request; AR final-stage abort yields a terminal output with finish_reason="abort" and the generated prefix; diffusion abort is whole-sample (DiffusionEngine.abort → _process_aborts_queue); connector cleanup(request_id) unlinks shm segments; abort_requests_collecting_outputs in the output processor; FINISHED_ABORTED requests are dropped before vLLM sees them. [Verified] (../../vllm-omni/docs/design/module/engine_orchestration.md)
  • Backpressure: generation admission gate in AsyncOmni._submit_with_admission; duplex_max_sessions engine-owned session capacity; max_num_seqs per stage; collect_timed_out_request_ids for chunk/input timeouts (connector_get_max_wait*); active_stream_window limits concurrently fed downstream streams. [Verified]

(b) Graph / operator / thread-level scheduling

  • Within a stage step the model forward is a single torch call sequence (eager, piecewise or full CUDA graph) — there is no omni-level operator DAG scheduler; ordering is the model's Python. [Verified]
  • Thread pools: chunk transfer adapter recv_loop/save_loop threads; async omni output builder thread (OmniAsyncGPUModelRunnerOutput._build_output_in_background); DiffusionEngine._busy_loop thread; multiproc diffusion _result_pump thread; orchestrator_monitor. [Verified]
  • Stream assignment: default compute stream plus a dedicated payload copy stream for D2H snapshots and prefix-cache writes; NPU synchronizes default stream before events. Diffusion sequence-parallel comms use GroupCoordinator in vllm_omni.diffusion.distributed. [Verified]
  • Diffusion regional compilation is the only compile-graph scheduling; N/A otherwise. [Verified]

6. I/O layer

  • Preprocessing. Media parsing reuses vllm.multimodal (audio via vllm.multimodal.audio — librosa is banned by ruff in favor of vLLM helpers/torchaudio); utils/audio.py provides mel filterbanks; per-model processors under transformers_utils/processors; TTS reference audio encoding happens in serving adapters (_encode_moss_references, _build_qwen3_tts_request); diffusion image preprocessing via get_*_pre_process_func. [Verified] (../../vllm-omni/vllm_omni/utils/audio.py, ../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py)
  • Tokenization. HF transformers tokenizers through vLLM's tokenizer layer (vllm.tokenizers imported in 20 files); owns_tokenizer marks the stage whose tokenizer the pipeline exposes; custom MammothModa2Tokenizer; speech tokenizers/codecs per model (e.g., Qwen3TTSTokenizer 12 Hz/25 Hz, s3tokenizer, whisper_vq). [Verified] (../../vllm-omni/vllm_omni/tokenizers/mammoth_moda2_tokenizer.py, ../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py)
  • Streaming. Text deltas via vLLM RequestOutputKind.DELTA; audio via async chunk OmniEngineCoreOutput.multimodal_output per chunk, accumulated by OmniRequestState.add_multimodal_tensor / multimodal_accumulation.py; HTTP TTS streams PCM through _generate_pcm_chunks/_iter_pcm_audio_bytes or SSE (_generate_audio_sse_events); WebSocket OmniStreamingSpeechHandler (/v1/audio/speech/stream, optional word alignments via forced aligner); StreamingAudioResampler for integer-ratio downsampling (24 kHz→8 kHz). Video streaming output via /v1/video/chat/stream, /v1/realtime/video. [Verified] (../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py, ../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py, ../../vllm-omni/vllm_omni/outputs/output_processor.py)
  • Buffering. Talker→code2wav frame accumulation (code_prompt_token_ids deque per request; chunk = 25 frames, left context 25–72 frames, first chunk 1–4 or dynamic); shm segments per chunk; per-request asyncio queues for diffusion; duplex realtime_input.py audio buffers with Silero VAD (SileroStreamingVAD, 16 kHz). [Verified] (../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py, ../../vllm-omni/vllm_omni/entrypoints/duplex/vad.py)
  • Postprocessing. Detokenize via vLLM output processor (sampling_constraints={"detokenize": False} on talkers); vocoders as generation/diffusion stages; diffusion get_*_post_process_func converts latents → PIL/video/actions; audio encoding with soundfile (wav, pcm, mp3, opus), speed change via torchaudio phase-vocoder. [Verified]
  • Application boundaries.
    • Python: Omni.generate(prompts, sampling_params_list, py_generator=...), AsyncOmni (implements vLLM EngineClient). [Verified] (../../vllm-omni/vllm_omni/entrypoints/omni.py, ../../vllm-omni/vllm_omni/entrypoints/async_omni.py)
    • CLI: vllm serve <model> --omni (vLLM subcommand plugin) and vllm-omni console script; stage-based launch --stage-id N --headless --omni-master-address/-port; vllm-omni bench serve. [Verified] (../../vllm-omni/vllm_omni/entrypoints/cli/main.py, ../../vllm-omni/vllm_omni/entrypoints/cli/serve.py)
    • HTTP (FastAPI): /v1/chat/completions (+/batch), /v1/audio/speech (+/batch), /v1/audio/generate, /v1/audio/voices (GET/POST/DELETE), /v1/images/generations, /v1/images/edits, /v1/videos (+/sync, list/get/delete/content), /v1/models, /health, /v1/omni/sleep|wakeup, /start_profile|/stop_profile. WebSocket: /v1/audio/speech/stream, /v1/video/chat/stream, /v1/realtime/video, /v1/realtime (turn-based PCM), /v1/realtime/robot/openpi, /v1/duplex. [Verified] (../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py)
    • No C API, no mobile bindings, no gRPC. [Verified]

7. Execution flow

(a) Text LLM decode (thinker-only pipeline, e.g. qwen3_omni_moe_thinker_only)

sequenceDiagram
  participant C as Client (HTTP)
  participant S as api_server / OmniOpenAIServingChat
  participant A as AsyncOmni.generate
  participant E as AsyncOmniEngine
  participant O as Orchestrator (asyncio)
  participant P as StagePool[0] / StageEngineCoreClient (ZMQ)
  participant EC as StageEngineCoreProc (vLLM EngineCore)
  participant SC as OmniARAsyncScheduler
  participant W as GPUARWorker / GPUARModelRunner
  C->>S: POST /v1/chat/completions (stream)
  S->>A: generate(prompt, sampling_params_list)
  A->>E: add_request_async (StageSubmissionMessage)
  E->>O: request queue
  O->>P: submit_initial(req) -> OmniEngineCoreRequest
  P->>EC: ZMQ add_request
  loop each engine step
    EC->>SC: schedule()
    SC-->>EC: SchedulerOutput (prefill/decode batch)
    EC->>W: execute_model -> sample_tokens
    W-->>EC: OmniModelRunnerOutput (token ids, hidden states opt.)
    EC->>SC: update_from_output
    EC-->>P: OmniEngineCoreOutputs (ZMQ)
    P-->>O: poll_llm_raw_output -> MultimodalOutputProcessor
    O-->>E: OutputMessage(final_output=True, DELTA)
    E-->>A: _process_orchestrator_results
    A-->>S: OmniRequestOutput
    S-->>C: SSE chunk
  end
  C-->>S: disconnect / cancel
  S->>A: abort(request_id)
  A->>O: AbortRequestMessage
  O->>P: abort_requests -> EC finish_requests(FINISHED_ABORTED)

[Verified] for the participants and call names (../../vllm-omni/vllm_omni/entrypoints/async_omni.py, ../../vllm-omni/vllm_omni/engine/orchestrator.py, ../../vllm-omni/vllm_omni/engine/stage_pool.py); exact vLLM-internal ZMQ framing inherited from AsyncMPClient [Inferred].

(b) TTS with async chunk streaming (Qwen3-TTS: talker → code2wav)

sequenceDiagram
  participant C as Client
  participant SP as OmniOpenAIServingSpeech (+Qwen3TTS adapter)
  participant O as Orchestrator
  participant T as Stage0 talker (OmniARScheduler + GPUARModelRunner)
  participant AD as OmniChunkTransferAdapter (stage0 sched proc)
  participant SHM as SharedMemoryConnector (/dev/shm)
  participant G as Stage1 code2wav (OmniGenerationScheduler + GPUGenerationModelRunner)
  participant M as Qwen3TTSCode2Wav (+CUDAGraphDecoderWrapper)
  C->>SP: POST /v1/audio/speech {input, voice, stream}
  SP->>SP: build prompt (prompt_embeds/additional_information, ref audio codes)
  SP->>O: AsyncOmni.generate (2 sampling params)
  O->>T: submit stage0 request
  O->>G: _prewarm_async_chunk_stages: pre-submit stage1 request (WAITING_FOR_CHUNK)
  loop each talker decode step
    T->>T: execute_model -> codec frame in multimodal_outputs
    T->>AD: update_from_output -> save_async(pooler_output, request)
    AD->>AD: talker2code2wav_async_chunk: accumulate frames; emit when >= initial/steady chunk (1..25 frames + left ctx)
    AD->>SHM: put(key=req_stage_chunk, OmniPayloadStruct{codes,meta})
  end
  G->>SHM: recv_loop get(key) -> chunk ready
  G->>G: restore_queues -> schedule (append chunk tokens)
  G->>M: execute_model -> chunked_decode_with_cudagraph
  M-->>G: waveform chunk (per request via ubatch_slices)
  G-->>O: OmniEngineCoreOutput.multimodal_output {audio}
  O-->>SP: OutputMessage(stage1 final_output=True)
  SP->>SP: soundfile encode (pcm/wav/mp3/opus), optional resample
  SP-->>C: chunked HTTP / SSE / WS audio
  T-->>AD: finished -> meta.finished chunk
  G-->>O: last chunk, finished=True -> cleanup shm segments

[Verified] (../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py, ../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/shm_connector.py, ../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py, ../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml, ../../vllm-omni/docs/design/feature/async_chunk.md).

VLA path (π0 / GR00T): WS /v1/realtime/robot/openpi → ServingRealtimeRobotOpenPI.infer(obs, session_id, reset) → _build_request puts the raw observation into sampling_params.extra_args["robot_obs"] → AsyncOmni.generate → single DIFFUSION stage (DiffusionEngine.step_streaming) → Pi0Pipeline.forward(req) (preprocess + flow-matching denoise) → DiffusionOutput(output={"actions": ndarray}) → multimodal_output["actions"] → msgpack reply. [Verified] (../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py, ../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py)

8. Tests and examples

  • Layout: ../../vllm-omni/tests/ (912 test files) split by module: engine/ (48), core/ (21), worker/ (14), diffusion/ (311), entrypoints/ (66), e2e/ (180: offline_inference/, online_serving/, accuracy/, features/), model_executor/ (136), distributed/ (18), platforms/ (9), examples/ (17), plus dfx/ (perf/reliability/stability). Root conftest.py is thin; fixtures in tests/helpers/fixtures/. [Verified] (../../vllm-omni/tests/conftest.py)
  • Markers / levels (pyproject): core_model (L1/L2 per PR), advanced_model (L3 per merge), full_model (L4 nightly), local_model; module markers diffusion|omni|tts|cache|parallel|sp|example; platform markers cpu|gpu|cuda|rocm|xpu|npu|musa; SKU markers H100|H800|H200|L4|B200|MI325|B60|S5000|A2|A3; cards_1..8. [Verified] (../../vllm-omni/pyproject.toml)
  • How to run (../../vllm-omni/docs/contributing/ci/test_execution_guide.md): uv pip install ".[dev]"; apt-get install espeak-ng jq; L1: cd tests && pytest -s -v -m "core_model and cpu"; L2/L3/L4 via bash tools/run_ready_jobs.sh|run_merge_jobs.sh|tools/nightly/run_nightly_jobs.sh which read .buildkite/cuda/test-{ready,merge,nightly}.yml; ad hoc pytest -s -v test_x.py --run-level=core_model. [Verified]
  • CI: Buildkite pipelines for cuda (../../vllm-omni/.buildkite/cuda/test-ready.yml, test-merge.yml, test-nightly.yml, test-weekly.yml), amd, intel (XPU, VLLM_VERSION: v0.28.0), npu (A2/A3); GitHub Actions only pre-commit.yml and build_wheel.yml. Nightly docker vllm/vllm-omni:nightly. [Verified]
  • Relevant examples: TTS ../../vllm-omni/examples/offline_inference/text_to_speech/ (qwen3_tts, cosyvoice3, fish_speech, gepard, glm_tts, higgs_audio_v2/v3, indextts2, ming_tts, moss_tts(_nano), omnivoice, voxcpm2, voxtral_tts; qwen3_tts/word_timestamps.py) and ../../vllm-omni/examples/online_serving/text_to_speech/; omni ../../vllm-omni/examples/offline_inference/qwen3_omni/end2end_async_chunk.py, ../../vllm-omni/examples/online_serving/qwen3_omni/openai_realtime_client.py; VLA ../../vllm-omni/examples/online_serving/pi0/openpi_client.py, ../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py, ../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py, ../../vllm-omni/examples/offline_inference/dreamzero/client_schedule.py; custom pipeline ../../vllm-omni/examples/offline_inference/custom_pipeline/image_to_image/custom_pipeline.py; apps ../../vllm-omni/apps/ComfyUI-vLLM-Omni. No mobile examples. [Verified]
  • Relevant e2e tests: ../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py, test_qwen3_omni_colocate_async.py, test_lingbot_world_v2_stepwise.py; ../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py, test_gr00t_openpi_expansion.py, test_dreamzero_expansion.py, test_minicpmo_4_5_duplex.py. [Verified]
  • Benchmarks: ../../vllm-omni/benchmarks/tts/bench_tts.py (+model_configs.yaml), benchmarks/diffusion, benchmarks/kernels, vllm-omni bench serve (TTFP/RTF metrics in vllm_omni/metrics). [Verified]

9. TTS relevance

Pipeline-level facts.

  • TTS models are two-stage LLM_AR (talker → codec tokens) + LLM_GENERATION (code2wav) or + DIFFUSION (flow-matching acoustic decoder) pipelines; single-stage native-AR TTS also exists (Gepard). sampling_constraints={"detokenize": False, "stop_token_ids": [...]} keep vLLM from detokenizing codec ids. [Verified] (../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py)
  • Streaming audio is produced by async_chunk: true with connector extras (codec_streaming, codec_chunk_frames, codec_left_context_frames, initial_codec_chunk_frames, codec_chunk_ramp, codec_chunk_adaptive, decode_batch_max_size, decode_cudagraph_*). Dynamic initial chunk sizing (compute_dynamic_initial_chunk_size) trades TTFP vs load. [Verified] (../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml, ../../vllm-omni/vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py)
  • Code2wav batching across requests uses ubatch_slices to split batched waveforms; CUDA graphs captured for the stateful streaming decoder shapes. [Verified] (../../vllm-omni/docs/design/feature/async_chunk.md, ../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py)
  • Serving: /v1/audio/speech with per-model adapters (qwen3_tts, cosyvoice3, fish_speech, voxtral, moss_tts, higgs_audio_v2/v3, indextts2, ming_tts, ming_flash_omni_tts, glm_tts, dots_tts, voxcpm2, omnivoice, step_audio2, covo_audio, audex, audex_tta, minimax_music3); voices registry endpoints (/v1/audio/voices, custom_voice_dir), speaker cache (utils/speaker_cache.py), forced aligner for word timestamps (utils/forced_aligner.py, --forced-aligner), speech usage accounting (speech_usage.py). [Verified] (../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/)
  • Realtime speech-to-speech: /v1/realtime (turn-based) and full-duplex /v1/duplex / /v1/realtime?duplex=1 with server VAD (Silero), used by MiniCPM-o 4.5, PersonaPlex (Moshi finetune), Nemotron VoiceChat. [Verified] (../../vllm-omni/docs/serving/realtime_duplex_api.md, ../../vllm-omni/docs/serving/full_duplex_api.md)
  • Repo-reported performance (not reproduced): Qwen3-TTS on H200, concurrency 1: E2E 941 ms, TTFP 64 ms, RTF 0.16 vs HF 15,513 ms / 2.64; Qwen3-Omni on A100: TTFP 0.934 s, RTF 0.32. [Verified as repo claims] (../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md)

TTS/audio model families in tree (registry keys in ../../vllm-omni/vllm_omni/model_executor/models/registry.py, dirs under ../../vllm-omni/vllm_omni/model_executor/models/):

Family Dir Stages Vocoder/codec
Qwen3-TTS (CustomVoice/VoiceDesign/Base) qwen3_tts Talker (AR) → Qwen3TTSCode2Wav (gen) 12 Hz/25 Hz speech tokenizer, ConvNet code2wav, x-vector speaker enc
CosyVoice3 cosyvoice3 (+ diffusion/models/cosyvoice3_audio) Talker → flow-matching DiT code2wav s3tokenizer; ONNX speaker embedding
Fish Speech S2 Pro fish_speech Slow AR (+fast AR) → FishSpeechDACDecoder DAC; custom KV-cache Triton attention
Gepard-1.0 gepard single-stage native AR FSQ/NanoCodec
VoxCPM2 voxcpm2 Talker (+ diffusion head) —
dots.tts dots_tts Talker —
Voxtral TTS voxtral_tts AR → VoxtralTTSAudioGeneration (flow matching) + audio tokenizer —
MOSS-TTS family (Delay/Realtime/Local, TTSD, SoundEffect, VoiceGenerator) moss_tts, moss_tts_nano Talker → MossTTSCodecDecoder MOSS codec
IndexTTS-2 / 2.5 indextts2 Talker → IndexTTS2S2MelDecoder s2mel + vocoder
GLM-TTS glm_tts Talker (+ voice clone helper, ONNX) —
Higgs-Audio v2 / v3 higgs_audio_v2, higgs_audio_v3 Talker → Code2Wav —
OmniVoice omnivoice (+ diffusion/models/omnivoice) diffusion pipeline —
Ming-TTS (dense/MoE), Ming-flash-omni TTS ming_tts, ming_flash_omni LLM → MingAudioVAEModel audio VAE
Audex (Nemotron) TTS/TTA/S2S audex Thinker → AudexCode2Wav / AudexXCodec1 XCodec
Step-Audio2, MiMo-Audio, Covo-Audio, MiniCPM-o 4.5, Qwen2.5/3-Omni, Nemotron VoiceChat, PersonaPlex, Dynin-Omni, Aura-Omni respective dirs omni thinker → talker → token2wav model codecs (Token2Wav / CosyVoice2 flow)
MiniMax-Music3 (text-to-music) minimax_music3 Talker → acoustic flow decoder —
Stable-Audio-Open diffusion/models/stable_audio diffusion —

What's missing for edge TTS. No CPU or mobile execution path for talker or vocoder; vocoder stages assume CUDA graphs or Ascend/ROCm patches; audio chunks cross a POSIX-shm + msgpack boundary (fine on a server, wasteful on a single-process edge device); no low-latency in-process single-stage mode (a stage is always a vLLM EngineCore subprocess with its own scheduler); no quantization validated for talker/code2wav stages (docs: "not validated"); no INT8/INT4 vocoder path; Omni offline API is batch-oriented (py_generator=True streams finished requests, not audio chunks — chunk streaming is only via AsyncOmni/HTTP). [Verified/Inferred]

10. VLA relevance

Pipeline-level facts.

  • VLA policies are single DIFFUSION stages with final_output_type="action"|"actions": PI0_PIPELINE (../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py), GR00T_N1D7_PIPELINE (../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py), InternVLA-A1 (InternVLAA1Pipeline, default single-stage fallback). Observations arrive in sampling_params.extra_args["robot_obs"]; the pipeline owns all preprocessing (image resize, tokenization of the language instruction, state normalization) and returns DiffusionOutput(output={"actions": ...}). [Verified]
  • Vision encoders: π0 uses PaliGemma (SigLIP + Gemma-2B prefix, paligemma_variant, action_expert_variant: gemma_300m, 224×224, up to 3 cameras); GR00T N1.7 uses its own VLM backbone through Gr00tN1d7/Gr00tN1d7Processor with embodiment-conditioned MLP and DiT action head (modeling/modules/dit.py, embodiment_conditioned_mlp.py); InternVLA-A1 uses a Qwen3-VL adapter (adapter_qwen3_vl.py) with a Cosmos-based world/action model (model_cosmos.py). [Verified] (../../vllm-omni/vllm_omni/deploy/pi0.yaml, ../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py, ../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py)
  • Structured/proprioceptive inputs: π0 max_state_dim: 32, max_action_dim: 32, chunk_size: 50, num_inference_steps: 10; GR00T dataio/state_action/{state_action_processor,action_chunking,pose}.py and embodiment_tags.py (DROID, G1, R1 Pro, LIBERO, SimplerEnv …); DreamZero transform/droid.py, roboarena.py, action_encoder.py. [Verified]
  • Action heads / diffusion-flow policies: flow matching (π0 sample_actions loop, Euler; GR00T DiT; InternVLA Cosmos CI); Cosmos3 action modes policy|forward_dynamics|inverse_dynamics with embodiment→domain id map (../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py). [Verified]
  • World models: DreamZero-DROID (DreamZeroPipeline, engine_backend: ARDiffusionEngine, step_cache DiT cache, dreamzero_tp1_cfg2.yaml CFG-parallel) and LingBot-World 2.0 (LingBotWorldCausalDMDPipeline, stepwise deploy lingbot_world_v2_stepwise.yaml) run on the experimental AR-diffusion engine: per-session paged KV (ARDiffusionKVCacheSpec: tokens_per_frame, window/sink frames), tick protocol with session_id/event_id/chunk_index/request_id, one latent block per request. SANA-WM (sana_wm) and Cosmos3 world generation also exist. [Verified] (../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md, ../../vllm-omni/vllm_omni/deploy/dreamzero.yaml)
  • Low-latency loop: OpenPI WebSocket (/v1/realtime/robot/openpi, msgpack + numpy ext, 64 MiB max frame, 30 s idle close, session_id/reset semantics, first frame = policy_server_config); request→response per observation; max_num_seqs: 1, enforce_eager: true, float32 for π0 by default. Session state manager (enable_session_state_manager, experimental/world_models/session_state) keeps model state across observations. [Verified] (../../vllm-omni/docs/serving/openpi_api.md, ../../vllm-omni/docs/features/session_state_manager.md)
  • Deploy YAMLs: ../../vllm-omni/vllm_omni/deploy/pi0.yaml, ../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml, ../../vllm-omni/vllm_omni/deploy/dreamzero.yaml, ../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml, ../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml. [Verified]

What's missing for edge VLA. No action-token streaming (one reply per observation); no explicit control-loop latency budget/deadline scheduling; VLA pipelines are eager PyTorch modules (no CUDA graph / compile path exercised for π0/GR00T by default; GR00T loads via HF AutoModel); no CPU/Jetson-specific path (π0 picks cuda if available else falls to CPU only implicitly via torch.cuda.is_available() in GR00T — untested); KV/state reuse across observations is only implemented for world models (AR-diffusion), not for VLM-prefix caching in π0/GR00T; no quantized VLA recipes. [Verified/Inferred]

11. Edge deployment profile

  • Binary size / dependencies. Pure-Python wheel; runtime deps (../../vllm-omni/requirements/common.txt): transformers>=5.10.1,<5.15, diffusers==0.40.0, kernels==0.15.2, accelerate, cache-dit, torchsde, openai-whisper, imageio[ffmpeg], x-transformers, einops, pyzmq, janus, msgpack, pydantic, gguf, cosmos-guardrail, soundfile, av; platform files add onnxruntime (+fa3-fwd on CUDA, onnxruntime-cann+torchaudio on NPU, torchada/mate/flash_attn_3 on MUSA, auto-round-lib on XPU). Plus vLLM 0.28.x itself (compiled CUDA kernels) and PyTorch. Docker images derive from vllm/vllm-openai:v0.28.0. Footprint is multi-GB; no size-optimized build exists. [Verified]
  • Build system. setuptools + setuptools-scm; setup.py picks the requirements file from VLLM_OMNI_TARGET_DEVICE or torch introspection; version suffix +rocm|+npu|+xpu|+musa|+cpu. No CMake, no native code. [Verified] (../../vllm-omni/setup.py)
  • Supported platforms (docs ../../vllm-omni/docs/getting_started/installation/README.md): NVIDIA CUDA (CC ≥ 7.0 per docs; diffusion FA needs ≥ 8.0; Blackwell needs FA4/cuDNN ≥ 9.5), AMD ROCm, Intel XPU, MThreads MUSA, Ascend NPU (Atlas A2/A3, 310P patches). Recipes mention consumer GPUs (RTX 4090/5090, RTX PRO, DGX Spark GB10 for MiniMax-H3). [Verified] (../../vllm-omni/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md)
  • Embedded NVIDIA / Jetson: no Jetson/aarch64-specific code, docs, or CI; the DGX Spark (GB10, aarch64+Blackwell) recipe is the closest evidence that an aarch64 CUDA host can run it, contingent on a vLLM aarch64 wheel. [Verified: absence] [Inferred: GB10 feasibility]
  • ARM CPU, Apple Silicon/Metal/CoreML, Android GPU/NPU (Vulkan/OpenCL/QNN/NNAPI): none. grep for metal|coreml|vulkan|opencl|qnn|nnapi|jetson over vllm_omni/ and docs/ found no implementation. [Verified: absence]
  • Model conversion path & constraints. Models are consumed directly as HF/Diffusers checkpoints (safetensors; GGUF for some diffusion models; ModelOpt/AutoRound/TorchAO pre-quantized checkpoints). There is no exporter to ONNX/TensorRT/CoreML/TFLite; the only "conversion" is loader-side (weight renaming, TP slicing, online quantization, Host Weight Runtime artifacts). Constraints: vLLM's model-class contract (forward(input_ids, positions, intermediate_tensors, inputs_embeds)), vLLM paged attention layout, and OmniPayload schemas between stages. [Verified]
  • Process model on the edge. Minimum footprint for a 2-stage TTS is: API server process + orchestrator thread + 2 vLLM EngineCoreProc subprocesses (each with scheduler + worker) + shm connector; single-stage diffusion can run inline (inline_diffusion/uni executor). [Verified] (../../vllm-omni/docs/configuration/stage_configs.md)
  • [Proposal] For an edge port, retain: PipelineConfig/StagePipelineConfig topology descriptions, stage_input_processors (chunking math in chunk_size_utils.py), OmniPayload typed schemas, TTS adapters' request building, OpenPI/realtime protocol handlers; replace: EngineCoreProc stages with in-process executors, SharedMemoryConnector with in-memory queues, vLLM model classes with the target runtime's graphs.

12. Limitations and unknowns

  • [Verified] Hard dependency on vLLM 0.28.x internals (284 modules, v1 engine, monkeypatched classes); minor-version drift is expected to break (version.py warning, patch.py FRAGILITY notes).
  • [Verified] No CPU/edge/mobile execution backend; requirements/cpu.txt is for docs builds only.
  • [Verified] The default single-node transport is host-staged (D2H2D) even when stages share a GPU (docs/design/feature/disaggregated_inference.md); zero-copy is a TODO in OmniMsgpackEncoder. Mooncake TE and Mori have device-memory fast paths, including Mori's intra-node XGMI backend; their applicability to an edge target is unvalidated.
  • [Verified] Each LLM stage is a separate subprocess with its own memory budget; co-location relies on gpu_memory_utilization partitioning and admission checks.
  • [Verified] Quantization for TTS/omni non-AR stages is documented as not validated; MXFP4/MXFP8 are Ascend-only.
  • [Verified] Many design docs are status: draft with "candidate invariants" (engine orchestration, stage runtime, connectors, platforms); the stage-client/process refactor #5441 is in flight.
  • [Verified] Experimental subsystems: AR-diffusion engine/world models (vllm_omni/experimental/ar_diffusion), full-duplex JoyVL/Mage-VL (experimental/fullduplex), PAGED_WORKER_LOCAL KV mode reserved-but-rejected.
  • [Verified] VLA serving returns whole action chunks; no intermediate streaming; session_id/reset ignored by π0 (stateless).
  • [Inferred] Latency floor for TTS on a single device includes subprocess ZMQ hops, shm serialization, and scheduler polling (connector_get_sleep_s: 0.01), which matter more on edge SoCs than on H100.
  • [Unknown] Actual runtime behavior of the 7be014bc tree was not executed here (installed editable is be335a86f); no benchmark numbers were reproduced.
  • [Unknown] Whether vLLM's aarch64/Jetson wheels satisfy the kernel packages omni's CUDA path probes (fa3-fwd, FlashInfer, FA4) — not determinable from this tree.
  • [Unknown] Memory footprint of the smallest supported TTS deployment (e.g., Qwen3-TTS 1.7B two-stage) — deploy YAMLs use gpu_memory_utilization: 0.3 per stage on H100 but no absolute numbers are published in-tree beyond recipes.

13. Reference index

Path Role
../../vllm-omni/pyproject.toml Package metadata, extras, pytest markers, vllm.general_plugins entry point
../../vllm-omni/setup.py Platform-aware dependency routing, version suffix
../../vllm-omni/requirements/common.txt Common runtime deps (no vllm pin)
../../vllm-omni/requirements/cpu.txt CPU requirements (docs builds)
../../vllm-omni/requirements/musa.txt MUSA deps
../../vllm-omni/README.md Project overview, model classes
../../vllm-omni/docker/Dockerfile.cuda CUDA image from vllm/vllm-openai:v0.28.0
../../vllm-omni/docker/Dockerfile.npu Ascend image
../../vllm-omni/vllm_omni/__init__.py Import-time version check + patch
../../vllm-omni/vllm_omni/version.py vLLM major.minor alignment warning
../../vllm-omni/vllm_omni/patch.py Monkeypatches of vLLM/torch/transformers
../../vllm-omni/vllm_omni/plugins/__init__.py Omni plugin groups
../../vllm-omni/vllm_omni/platforms/__init__.py Platform probes and resolution
../../vllm-omni/vllm_omni/platforms/interface.py OmniPlatform abstract hooks, UnspecifiedOmniPlatform
../../vllm-omni/vllm_omni/platforms/cuda/platform.py CUDA platform, diffusion attention default chain
../../vllm-omni/vllm_omni/platforms/rocm/platform.py ROCm platform
../../vllm-omni/vllm_omni/platforms/npu/platform.py Ascend platform (vllm_ascend)
../../vllm-omni/vllm_omni/platforms/npu/worker/ NPU AR/generation workers and runners
../../vllm-omni/vllm_omni/platforms/xpu/platform.py Intel XPU platform
../../vllm-omni/vllm_omni/platforms/musa/platform.py MUSA platform
../../vllm-omni/vllm_omni/engine/arg_utils.py OmniEngineArgs, OrchestratorArgs, model registration
../../vllm-omni/vllm_omni/engine/async_omni_engine.py AsyncOmniEngine composition root
../../vllm-omni/vllm_omni/engine/orchestrator.py Cross-stage routing, abort, CFG, PD
../../vllm-omni/vllm_omni/engine/cfg_companion_tracker.py CFG companion bookkeeping
../../vllm-omni/vllm_omni/engine/stage_runtime.py Local/distributed replica lifecycle
../../vllm-omni/vllm_omni/engine/stage_pool.py Per-stage replica pool, LB, metrics
../../vllm-omni/vllm_omni/engine/stage_engine_startup.py ZMQ master server, replica launch
../../vllm-omni/vllm_omni/engine/stage_engine_core_proc.py StageEngineCoreProc
../../vllm-omni/vllm_omni/engine/stage_engine_core_proc_manager.py Subprocess spawning per replica
../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py ZMQ client per replica
../../vllm-omni/vllm_omni/engine/stage_admission.py Device memory admission
../../vllm-omni/vllm_omni/engine/__init__.py OmniEngineCoreRequest/Output(s)
../../vllm-omni/vllm_omni/config/stage_config.py PipelineConfig, StageDeployConfig, scheduler resolution
../../vllm-omni/vllm_omni/config/pipeline_registry.py OMNI_PIPELINES
../../vllm-omni/vllm_omni/config/config_factory.py Pipeline auto-detection from HF config
../../vllm-omni/vllm_omni/config/omni_config.py Typed control-plane config
../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml Qwen3-TTS deploy (chunk streaming knobs)
../../vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml Qwen3-Omni 3-stage deploy with platform overrides
../../vllm-omni/vllm_omni/deploy/pi0.yaml π0 deploy
../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml GR00T deploy
../../vllm-omni/vllm_omni/deploy/dreamzero.yaml DreamZero deploy (AR-diffusion engine)
../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml DreamZero CFG-parallel deploy
../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml LingBot-World stepwise deploy
../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py AR scheduler
../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py Generation scheduler
../../vllm-omni/vllm_omni/core/sched/omni_scheduler_mixin.py Shared omni scheduling state
../../vllm-omni/vllm_omni/core/sched/omni_scheduling_coordinator.py Full-payload input coordination
../../vllm-omni/vllm_omni/core/prefix_cache.py Hidden-state/multimodal prefix cache
../../vllm-omni/vllm_omni/worker/base.py Worker base, memory profiling, sleep
../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py AR worker
../../vllm-omni/vllm_omni/worker/gpu_model_runner.py Shared omni runner (MRoPE, talker MTP init, preprocess)
../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py AR runner execute/sample/async output
../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py Non-AR generation runner
../../vllm-omni/vllm_omni/worker/omni_connector_model_runner_mixin.py Runner-side connector I/O (chunks, payloads, KV)
../../vllm-omni/vllm_omni/worker/gpu_memory_utils.py NVML memory helpers
../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py Fish Speech Triton KV attention
../../vllm-omni/vllm_omni/model_executor/models/registry.py Omni AR/generation model registry (89 archs)
../../vllm-omni/vllm_omni/model_executor/models/output_templates.py OmniOutput
../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py Qwen3-TTS topology
../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py Code2wav CUDA graph capture
../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py Qwen3-TTS speech tokenizer
../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py GR00T topology
../../vllm-omni/vllm_omni/model_executor/models/dreamzero/pipeline.py DreamZero topology
../../vllm-omni/vllm_omni/model_executor/models/lingbot_world/pipeline.py LingBot-World topology
../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py Talker→code2wav chunk/full-payload processors
../../vllm-omni/vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py Chunk ramp / dynamic IC math
../../vllm-omni/vllm_omni/diffusion/registry.py Diffusion pipeline registry (66)
../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py DiffusionEngine
../../vllm-omni/vllm_omni/diffusion/sched/request_scheduler.py Request-level batching
../../vllm-omni/vllm_omni/diffusion/sched/step_scheduler.py Step-level continuous batching
../../vllm-omni/vllm_omni/diffusion/executor/ Uniproc/multiproc executors
../../vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py Diffusion worker, device init, sleep
../../vllm-omni/vllm_omni/diffusion/worker/request_batch.py DiffusionRequestBatch
../../vllm-omni/vllm_omni/diffusion/compile.py Regional torch.compile
../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py Attention backend enum/overrides
../../vllm-omni/vllm_omni/diffusion/attention/backends/abstract.py Backend capability contract
../../vllm-omni/vllm_omni/diffusion/attention/selector.py Per-role backend resolution
../../vllm-omni/vllm_omni/diffusion/attention/layer.py Diffusion Attention layer
../../vllm-omni/vllm_omni/diffusion/layers/custom_op.py Platform op dispatch
../../vllm-omni/vllm_omni/diffusion/diffusion_kv/manager.py Scheduler-owned paged KV for diffusion
../../vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py Diffusers-style checkpoint loader
../../vllm-omni/vllm_omni/diffusion/offloader/ CPU/layerwise/distributed offload
../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py π0 topology
../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py π0 pipeline
../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py GR00T pipeline
../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py GR00T policy wrapper
../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py GR00T model
../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/pipeline_internvla_a1.py InternVLA-A1 pipeline
../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py InternVLA-A1 policy
../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py Cosmos3 action modes
../../vllm-omni/vllm_omni/experimental/ar_diffusion/engine.py ARDiffusionEngine
../../vllm-omni/vllm_omni/experimental/ar_diffusion/capability.py AR-diffusion KV spec contract
../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/base.py OmniConnectorBase
../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/shm_connector.py Shared-memory connector
../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py Connector factory/registry
../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py Chunk lifecycle adapter
../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py msgpack encoder for tensors
../../vllm-omni/vllm_omni/distributed/omni_coordinator/omni_coordinator.py Replica membership/heartbeats
../../vllm-omni/vllm_omni/inputs/data.py Omni prompt types
../../vllm-omni/vllm_omni/outputs/__init__.py OmniRequestOutput
../../vllm-omni/vllm_omni/outputs/output_processor.py Multimodal output processor
../../vllm-omni/vllm_omni/data_entry_keys.py OmniPayload schema
../../vllm-omni/vllm_omni/request.py OmniRequest, OmniStreamingUpdate
../../vllm-omni/vllm_omni/entrypoints/omni.py Offline Omni
../../vllm-omni/vllm_omni/entrypoints/async_omni.py AsyncOmni EngineClient
../../vllm-omni/vllm_omni/entrypoints/cli/main.py CLI interception of --omni
../../vllm-omni/vllm_omni/entrypoints/cli/serve.py vllm serve --omni subcommand
../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py HTTP/WS routes
../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py TTS serving
../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py TTS WebSocket streaming
../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py Audio encode/resample
../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/__init__.py TTS adapter registry
../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/base.py TTSModelAdapter
../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py OpenPI robot serving
../../vllm-omni/vllm_omni/entrypoints/duplex/vad.py Silero VAD
../../vllm-omni/vllm_omni/quantization/factory.py Quantization config factory
../../vllm-omni/vllm_omni/quantization/ Omni quant configs
../../vllm-omni/vllm_omni/utils/audio.py Mel filterbank utilities
../../vllm-omni/vllm_omni/tokenizers/mammoth_moda2_tokenizer.py Custom tokenizer
../../vllm-omni/docs/design/architecture_overview.md Architecture overview (mermaid, layers)
../../vllm-omni/docs/design/module/engine_orchestration.md Orchestration contract
../../vllm-omni/docs/design/module/stage_runtime.md Stage runtime contract
../../vllm-omni/docs/design/module/omni_connector.md Connector contract
../../vllm-omni/docs/design/module/execution_platforms.md Platform contract
../../vllm-omni/docs/design/module/ar_runtime.md AR runtime contract
../../vllm-omni/docs/design/module/input_output_modality_contracts.md I/O contracts
../../vllm-omni/docs/design/feature/async_chunk.md Async chunk design + perf table
../../vllm-omni/docs/design/feature/disaggregated_inference.md Connectors/D2H2D
../../vllm-omni/docs/design/feature/diffusion_continuous_batching.md Diffusion batching modes
../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md AR-diffusion sessions
../../vllm-omni/docs/design/feature/host_weight_runtime.md Host weight runtime
../../vllm-omni/docs/design/feature/prefix_caching.md Omni prefix caching
../../vllm-omni/docs/design/feature/omni_async_output_materialization.md Async output materialization
../../vllm-omni/docs/design/fullduplex.md Full-duplex runtime
../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md Repo-reported TTS perf
../../vllm-omni/docs/configuration/stage_configs.md Pipeline/deploy schema, precedence, stage CLI
../../vllm-omni/docs/getting_started/installation/README.md Platform list, vLLM alignment
../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md vllm==0.28.0 pin, CC≥7.0
../../vllm-omni/docs/getting_started/quickstart.md Linux/Python 3.12, offline API
../../vllm-omni/docs/user_guide/quantization/overview.md Quant modes × hardware matrix
../../vllm-omni/docs/user_guide/diffusion/execution_modes.md Diffusion execution modes
../../vllm-omni/docs/user_guide/diffusion/cpu_offload.md Offload strategies
../../vllm-omni/docs/user_guide/diffusion/regional_compilation.md Regional compile
../../vllm-omni/docs/features/custom_pipeline.md Custom pipeline extension
../../vllm-omni/docs/features/session_state_manager.md Session state manager
../../vllm-omni/docs/serving/openpi_api.md OpenPI WS protocol
../../vllm-omni/docs/serving/speech_api.md TTS HTTP API
../../vllm-omni/docs/serving/realtime_api.md Realtime audio WS
../../vllm-omni/docs/serving/realtime_duplex_api.md Duplex realtime
../../vllm-omni/docs/serving/full_duplex_api.md Full-duplex API
../../vllm-omni/docs/models/supported_models.md Model × hardware table
../../vllm-omni/docs/contributing/ci/test_execution_guide.md How to run tests
../../vllm-omni/docs/contributing/model/adding_tts_model.md TTS model guide
../../vllm-omni/docs/contributing/model/adding_omni_model.md Omni model guide
../../vllm-omni/docs/contributing/model/adding_diffusion_model.md Diffusion model guide
../../vllm-omni/tests/conftest.py Test bootstrap
../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py TTS e2e
../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py π0 e2e
../../vllm-omni/tests/e2e/online_serving/test_gr00t_openpi_expansion.py GR00T e2e
../../vllm-omni/.buildkite/cuda/test-ready.yml L1/L2 CI
../../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py Offline TTS example
../../vllm-omni/examples/online_serving/pi0/openpi_client.py π0 client
../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py DreamZero client
../../vllm-omni/examples/offline_inference/qwen3_omni/end2end_async_chunk.py Async chunk example
../../vllm-omni/examples/online_serving/qwen3_omni/openai_realtime_client.py Realtime client
../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py InternVLA example
../../vllm-omni/examples/offline_inference/custom_pipeline/image_to_image/custom_pipeline.py Custom pipeline example
../../vllm-omni/benchmarks/tts/bench_tts.py TTS benchmark
../../vllm-omni/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md DGX Spark recipe (aarch64 hint)
../../vllm-omni/apps/ComfyUI-vLLM-Omni ComfyUI app
Reference in new issue View Git Blame Copy Permalink