| Qwen3-Omni | Thinker → Talker → Code2Wav | Three stages with asynchronous chunk hand-off | TTFT/TTFP, TPOT, E2EL, audio RTF, and concurrent throughput |
| HunyuanImage-3.0 | Multimodal AR understanding/reasoning plus image-generation DiT | AR-only, DiT-only, or split AR → DiT deployment | DiT E2EL/denoising latency and image throughput; TTFT/TPOT for AR tasks |
| MiniMax-H3 | Shared multimodal conditioning plus task-specific FL2VA or Ref2VA DiT and video/audio VAEs | One diffusion stage; the request selects the DiT partition | Video/audio E2EL and media throughput; RTF when audio streaming is measured |
| Cosmos3 | Unified MoT reasoner tower plus diffusion generator tower | One diffusion pipeline materializes both towers inside the stage | Image/video/action E2EL and throughput; RTF for synchronized audio |
### Serving metric vocabulary
The target metric follows the output modality, not just the model name.
| Metric | Meaning in this document | Most useful for |
| **Thinker** | AuT audio encoder, SigLIP2 vision encoder, and the 30B-total/3B-active Thinker MoE Transformer | Text tokens and conditioning for the Talker |
| **Talker** | 3B-total/0.3B-active Talker MoE Transformer with the MTP code-prediction path | Text tokens and audio codec tokens |
| **Thinker** | Continuous batching with stage-local `max_num_seqs` and token budgets | vLLM KV-cached AR attention; CUDA Graph execution when the stage is not eager | Independent stage placement; use the AR runtime's configured tensor parallelism | Supported ModelOpt FP8/NVFP4 or AutoRound checkpoint paths target the Thinker; encoders remain BF16 in the documented paths |
| **Talker** | Continuous batching over autoregressive decode requests | KV-cached token decode with async-chunk output to Code2Wav | Independent stage placement; typically a separate GPU or replica from Thinker | BF16 baseline; no generic Talker quantization path is assumed |
| **Code2Wav** | Static/chunk batching for codec-to-waveform decode | Non-AR chunked audio decode; eager execution is commonly retained | Separate stage or colocated placement, depending on the deployment | BF16 baseline; keep codec/vocoder precision model-specific |
For this pipeline, the main serving target is interactive responsiveness:
TTFT for text, TTFP for the first audio packet, TPOT for autoregressive decode,
E2EL for the complete answer, and RTF/throughput for sustained audio serving.
Async chunking and streaming primarily reduce TTFP and improve stage overlap;
batching and CUDA Graphs primarily improve E2EL and throughput. See the
<img alt="HunyuanImage-3.0 framework for understanding, language modeling, and image generation" src="https://raw.githubusercontent.com/Tencent-Hunyuan/HunyuanImage-3.0/main/assets/framework.png" width="95%">
| Stage or component | Model component | Stage output |
| --- | --- | --- |
| **AR understanding/reasoning** | Understanding and generation encoders feeding the Hunyuan-A13B decoder-only Transformer and text detokenizer | Text responses or image-generation conditioning |
| **Image generation** | Generation encoder, diffusion prediction path/Gen. Decoder, and the image VAE decode path | Image latents and final images |
The vLLM-Omni deployment may expose the AR path only, the DiT path only, or
both as an AR → DiT pipeline with a connector between them. That is a runtime
decomposition of the model's capabilities; it should not be read as a claim
that the published framework figure contains two unrelated backbones.
| **AR** | Standard vLLM continuous batching when serving understanding/text tasks | KV-cached causal attention | Tensor/expert parallelism follows the selected AR deployment | Checkpoint-specific; do not infer DiT FP8 settings for the AR checkpoint |
| **DiT + VAE** | Request batching or step-wise batching; Hunyuan step batching with more than one request requires `TORCH_SDPA` | `TORCH_SDPA` is used because the model mixes causal and full attention | Validated profiles include TP4, TP2 + sequence parallelism, TP2 + CFG parallelism, and expert parallelism for MoE weights | Online FP8 and ModelOpt mixed FP8/NVFP4 are documented for the DiT; VAE precision stays separate |
Image generation is therefore measured mainly with per-image E2EL/denoising
latency, peak memory, and images-per-second throughput. TTFT and TPOT remain
meaningful for the optional AR understanding path, but they are not the right
for the validated batching, attention, and quantization combinations.
### MiniMax-H3: one diffusion stage with task-selected DiTs
MiniMax-H3 illustrates a different boundary. The vLLM-Omni implementation
loads shared conditioning and decode components once and selects one of two
task-specific DiTs per request. The current
[MiniMax H3 release overview](https://minimaxi.com/blog/minimax-h3) says that a
detailed H3 technical report is forthcoming, so this section uses the
implementation and serving recipe as the source for the stage mapping rather
than implying an unavailable published architecture figure.
#### Model architecture by stage
| Stage | Model component | Stage output |
| --- | --- | --- |
| **Diffusion stage: shared path** | Tokenizer/processor, Qwen3-VL text encoder, video VAE, and audio VAE | Packed text, image, video, and audio conditioning plus decoded media |
| **Diffusion stage: `FL2VA`** | Task-specific DiT for text-to-video+audio (`t2va`) and first-frame-to-video+audio (`fl2va`) | Synchronized video and stereo audio |
| **Diffusion stage: `Ref2VA`** | Task-specific DiT for mixed image/video/audio reference generation (`ref2va`) | Reference-conditioned video and stereo audio |
Both DiT partitions live behind one logical diffusion stage; `task` selects
which partition runs. This is different from Qwen3-Omni's three stage
pipeline, even though both models produce synchronized audio and video-like
| **Shared encoder + selected DiT** | The current H3 implementation executes one generation request per diffusion batch; use concurrency for service-level throughput | cuDNN attention for the validated two-GPU consumer profile; TRTLLM attention or FlashAttention-4 on supported Blackwell profiles | TP2 plus text-encoder TP and VAE patch parallelism on two GPUs; Ulysses and tiled VAE patch parallelism on four GPUs; DLO/CPU offload trade memory for transfer time | Online FP8 applies to eligible DiT linears; text encoder and VAEs remain BF16/FP32, and FP8 is incompatible with layerwise offload |
The primary target is video/audio E2EL and media throughput, with RTF useful
when the audio stream is evaluated independently. TTFT and TPOT do not
describe the main H3 generation path because the user-visible output is
| Stage or component | Model component | Stage output |
| --- | --- | --- |
| **Reasoner tower** | Autoregressive VLM path with causal attention over language and optional vision/action context; runs once per generation request | Discrete reasoning/context tokens |
| **Generator tower** | Diffusion path with full attention over the generator tokens and the reasoner K/V context; runs for each denoising step | Image, video, audio, or action tokens |
| **Shared pipeline** | `Cosmos3OmniDiffusersPipeline` plus modality encoders/decoders and VAE paths | Final image/video/audio or action response |
The reasoner and generator are model-owned components, not two independent
vLLM-Omni stages by default. The current Cosmos3 pipeline materializes both
inside one diffusion stage; model-level CPU offload can swap the reasoner and
| **Reasoner + generator pipeline** | Use request batching or step execution only when the selected pipeline advertises that capability; otherwise use `max_num_seqs=1` | Causal attention for reasoner tokens and full attention for diffusion tokens; the validated recipe uses a platform-selected aiter FlashAttention path with `TORCH_SDPA` fallback | Ulysses/context parallelism or tensor parallelism for the transformer; VAE tiling and layerwise/model-level offload address activation and weight memory | Online FP8 is supported for the diffusion pipeline; AutoRound/checkpoint-specific paths and quality validation remain separate from the BF16 baseline |
For Cosmos3, generation E2EL and output throughput are the primary targets.
Action-policy workloads add a control-loop latency target, while synchronized
audio workloads add RTF. TTFT/TPOT are appropriate for a reasoner-only service,
but not for the full diffusion generation path. See the
| **Entrypoints** | Translate offline, CLI, OpenAI-compatible, and duplex requests into engine operations and render outputs back to public protocols. |
| **Configuration resolution** | Combines pipeline topology, deployment settings, model metadata, and user overrides into a validated control-plane configuration. |
| **AsyncOmniEngine** | Owns engine composition, the background event loop, stage initialization, request submission, and output collection. |
| **Orchestrator** | Owns cross-stage request state, stage-to-stage routing, request correlation, cancellation, and output ordering. It does not own model selection or deployment placement. |
| **StageRuntime** | Expands logical stages into local or distributed replicas, starts stage clients and processes, and manages readiness, affinity, failure, and shutdown. |
| **AR runtime** | Extends vLLM's scheduler, KV-cache, worker, and model-runner path for omni-modality inputs and inter-stage outputs. |
| **Diffusion runtime** | Schedules and executes denoising workloads through diffusion executors, workers, pipelines, acceleration backends, and output materialization. |
| **OmniConnector** | Transports stage payloads and KV-cache data and provides synchronization. Connectors transport data; they do not choose the next logical stage. |
| **Multimodal outputs** | `MultimodalPayload` separates tensor content from metadata, while `OmniRequestOutput` carries pipeline and diffusion results through the common output path. |
## Configuration and runtime resolution
The control plane has five conceptual layers. Authoring inputs are resolved
once into a complete, transport-safe configuration before runtime processes are
started. This keeps stage topology and model capabilities distinct from
deployment placement and from process-local engine objects.