[Docs] Redirect the former PD guide URL

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
[Docs] Document current PD design
2026-08-13 12:48:43 +08:00 · 2026-08-13 12:44:42 +08:00 · 2026-08-13 12:23:56 +08:00 · 2026-08-13 06:54:34 +08:00 · 2026-08-13 06:50:41 +08:00
27 changed files with 360 additions and 849 deletions
@@ -43,6 +43,7 @@ nav:
    - Overview: features/README.md
    - Runtime and Stage Execution:
      - Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
      - Prefill-Decode Disaggregation (experimental): features/pd_disaggregation.md
      - Sleep Mode: features/sleep_mode.md
    - Quantization:
      - Overview: user_guide/quantization/overview.md
@@ -14,10 +14,6 @@ For introduction, please check [Introduction for stage config](./stage_configs.m
 - **[GPU Memory Calculation and Configuration](./gpu_memory_utilization.md)** - Guide on how to calculate memory requirements and set up `gpu_memory_utilization` for optimal performance
 ## Multi-Stage Recipes
 - **[Prefill-Decode Disaggregation](./pd_disaggregation.md)** - How to derive a PD-aware Qwen3-Omni stage config from the default config without introducing another bundled YAML
 ## Optimization Features
 - **[Diffusion Features Overview](../user_guide/diffusion_features.md)** - Complete overview of all diffusion model features and supported models
@@ -1,171 +0,0 @@
 # Prefill-Decode (PD) Disaggregation
 PD disaggregation splits the Qwen3-Omni thinker into separate prefill and decode
 stages so prompt processing and token generation can run on different workers.
 This is documented as a stage-config recipe instead of a bundled YAML because the
 deployment-specific values usually change per environment:
 - GPU placement
 - `tensor_parallel_size`
 - connector backend and connector ports
 - connector IPs or bootstrap addresses
 Start from the [default Qwen3-Omni stage config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
 and copy it to your own file, for example `qwen3_omni_pd.yaml`. Then apply the
 changes below.
 ## Requirements
 - 3+ GPUs for a basic layout: prefill, decode, and talker+code2wav
 - A KV connector supported by vLLM, such as `MooncakeConnector`
 - Matching `tensor_parallel_size` on the prefill and decode thinker stages
 ## 1. Split the thinker into prefill and decode stages
 Replace the original thinker stage with two stages:
 ```yaml
 stage_args:
  - stage_id: 0
    stage_type: llm
    is_prefill_only: true
    runtime:
      devices: "0"
    engine_args:
      max_num_seqs: 16
      model_stage: thinker
      model_arch: Qwen3OmniMoeForConditionalGeneration
      worker_type: ar
      scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
      gpu_memory_utilization: 0.9
      enforce_eager: true
      trust_remote_code: true
      engine_output_type: latent
      distributed_executor_backend: "mp"
      enable_prefix_caching: false
      max_num_batched_tokens: 32768
      hf_config_name: thinker_config
      tensor_parallel_size: 1
      kv_transfer_config:
        kv_connector: "MooncakeConnector"
        kv_role: "kv_producer"
        kv_rank: 0
        kv_parallel_size: 2
        kv_connector_extra_config:
          mooncake_bootstrap_port: 25201
    final_output: false
    is_comprehension: true
    default_sampling_params:
      temperature: 0.4
      top_p: 0.9
      top_k: 1
      max_tokens: 2048
      seed: 42
      detokenize: True
      repetition_penalty: 1.05
  - stage_id: 1
    stage_type: llm
    is_decode_only: true
    runtime:
      devices: "1"
    engine_args:
      max_num_seqs: 64
      model_stage: thinker
      model_arch: Qwen3OmniMoeForConditionalGeneration
      worker_type: ar
      scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
      gpu_memory_utilization: 0.9
      enforce_eager: true
      trust_remote_code: true
      engine_output_type: latent
      distributed_executor_backend: "mp"
      enable_prefix_caching: false
      max_num_batched_tokens: 32768
      hf_config_name: thinker_config
      tensor_parallel_size: 1
      kv_transfer_config:
        kv_connector: "MooncakeConnector"
        kv_role: "kv_consumer"
        kv_rank: 1
        kv_parallel_size: 2
        kv_connector_extra_config:
          mooncake_bootstrap_port: 25202
    engine_input_source: [0]
    final_output: true
    final_output_type: text
    is_comprehension: true
    default_sampling_params:
      temperature: 0.4
      top_p: 0.9
      top_k: 1
      max_tokens: 2048
      seed: 42
      detokenize: True
      repetition_penalty: 1.05
 ```
 Notes:
 - `is_prefill_only: true` marks the thinker stage that only saves KV.
 - `is_decode_only: true` marks the thinker stage that resumes from remote KV.
 - `kv_transfer_config` is required on both stages.
 - The orchestrator forces the prefill stage to run with `max_tokens=1`, so the
  prefill side only processes the prompt and exports KV.
 ## 2. Shift the downstream stages by one index
 After inserting the extra thinker stage, renumber the remaining stages:
 ```yaml
  - stage_id: 2
    runtime:
      devices: "2"
    engine_input_source: [1]
    sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_token_only
  - stage_id: 3
    runtime:
      devices: "2"
    engine_args:
      max_num_seqs: 1
    engine_input_source: [2]
    sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_token_only
 ```
 Compared with the default Qwen3-Omni config:
 - the talker becomes stage `2` instead of stage `1`
 - the code2wav stage becomes stage `3` instead of stage `2`
 - the talker now reads from decode stage `1`
 ## 3. Add runtime edges for the four-stage pipeline
 ```yaml
 runtime:
  enabled: true
  edges:
    - from: 0
      to: 1
    - from: 1
      to: 2
    - from: 2
      to: 3
 ```
 ## 4. Launch with your custom config
 ```bash
 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
    --deploy-config /path/to/qwen3_omni_pd.yaml
 ```
 ## Operational Notes
 - `MooncakeConnector` does not support heterogeneous TP sizes across the PD
  pair. Keep prefill and decode at the same `tensor_parallel_size`.
 - If the thinker requires TP=2, both thinker stages must use TP=2 and be given
  separate GPU sets, for example `"0,1"` for prefill and `"2,3"` for decode.
 - Choose connector ports and addresses that match your deployment. The values
  shown above are examples only.
@@ -1,6 +1,6 @@
 # Prometheus Metrics Design
-This document describes how vLLM-Omni exposes Prometheus metrics for multi-stage pipelines, the constraints that shaped the design, and how the pipeline-level metrics coexist with upstream vLLM per-engine metrics.
+This document describes how vLLM-Omni exposes Prometheus metrics for multi-stage pipelines, the constraints that shaped the design, and how the pipeline-level metrics coexist with upstream vLLM per-engine metrics. This iteration covers the text and audio surface; diffusion / image / video families land in a follow-up.
 ## Objectives
@@ -20,13 +20,19 @@ an implementation contract has a user-facing workflow.
 | Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
 | Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
 The [Prefill-Decode Disaggregation design reference](pd_disaggregation.md)
 describes the experimental Qwen3-Omni runtime topology and its current
 integration boundaries. It is not yet a supported launch recipe.
 Some runtime designs are deliberately not promoted as standalone User Guide
 features yet:
- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and
+- Generic, OmniConnector-based
  [Disaggregated Inference](../design/feature/disaggregated_inference.md) is not
  yet a standalone User Guide feature. Its
  [OmniConnector implementations](../design/index.md#communication) describe
-  topology and transport contracts. Practical deployment configuration remains
+  transport contracts, while practical connector configuration remains under
-  under [Pipeline and deploy configurations](../configuration/stage_configs.md).
+  [Pipeline and deploy configurations](../configuration/stage_configs.md).
 - [Async Chunk](../design/feature/async_chunk.md) and
  [Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
  are model- and pipeline-dependent. Use the selected model's deploy
@@ -0,0 +1,140 @@
 # Prefill-Decode (PD) Disaggregation (experimental)
 !!! warning "Experimental design reference"
    PD disaggregation is only partially integrated on the current `main` branch.
    The runtime contains PD detection and routing scaffolding, but there is no
    supported deploy overlay or end-to-end validated launch recipe yet. Treat this
    page as a description of the current design, not as production guidance.
 Prefill-decode disaggregation splits the Qwen3-Omni Thinker into two logical
 stages: a prefill worker that processes the prompt and produces KV cache, and a
 decode worker that imports that cache and generates tokens. The Talker and
 Code2Wav stages remain downstream; PD does not disaggregate the entire
 Qwen3-Omni pipeline.
 ## Current design
 ### Runtime topology
 ```mermaid
 flowchart LR
    client["Client request"] --> orchestrator["Omni Orchestrator"]
    orchestrator -->|"original prompt<br/>max_tokens = 1"| prefill["Thinker prefill<br/>KV producer"]
    prefill -->|"KV blocks"| connector[("vLLM KV connector")]
    prefill -->|"KV routing metadata"| orchestrator
    orchestrator -->|"original prompt<br/>remote KV parameters"| decode["Thinker decode<br/>KV consumer"]
    connector -->|"remote KV load"| decode
    decode -->|"generated text"| client
 ```
 The orchestrator owns the request lifecycle, while the vLLM KV connector owns
 the bulk KV transfer. The original prompt is submitted to both Thinker stages:
 the decode worker still needs the request tokens and multimodal features to
 construct its request, while the imported KV cache prevents it from repeating
 the completed prefill computation.
 For the full Qwen3-Omni pipeline, splitting the Thinker adds one logical stage:
 | Stage | Role | Output or handoff |
 | --- | --- | --- |
 | 0 | Thinker prefill | Saves prompt KV and prompt-side multimodal output |
 | 1 | Thinker decode | Loads remote KV, generates text, and produces decode-side conditioning |
 | 2 | Talker | Converts combined Thinker conditioning into codec codes |
 | 3 | Code2Wav | Converts codec codes into audio |
 Text remains a final output of the decode stage, and audio remains a final
 output of Code2Wav.
 ### Request lifecycle
 1. At startup, PD detection looks for one stage marked `is_prefill_only` and
   one downstream stage marked `is_decode_only`. The decode stage must list the
   prefill stage as an input source.
 2. The orchestrator clones the Thinker sampling parameters for prefill, forces
   `max_tokens=1`, clears stop conditions, and sets producer-side
   `kv_transfer_params`. This makes prefill finish by length after exporting KV.
 3. When prefill finishes, the orchestrator captures its KV routing metadata and
   Qwen multimodal output under the original request ID.
 4. The orchestrator submits the original prompt to decode with consumer-side
   `kv_transfer_params`. Those parameters identify the transfer, prefill engine,
   bootstrap endpoint, and, when required by the connector, remote request.
 5. The decode engine imports the KV cache and continues generation. The normal
   Omni stage path then carries Thinker conditioning through Talker and
   Code2Wav.
 ### Control and data paths
 PD uses two different connector layers. They solve different handoffs and are
 not interchangeable.
 | Path | Payload | Owner |
 | --- | --- | --- |
 | PD control path | Request ID, original prompt, `transfer_id`, bootstrap address, engine ID, and connector-specific request metadata | Omni Orchestrator |
 | PD data path | Thinker attention KV blocks | vLLM KV connector |
 | Downstream model path | Thinker hidden states and embeddings, Talker codec codes, and audio tensors | vLLM-Omni stage connectors and processors |
 ### Detection and validation
 The current runtime accepts one logical prefill/decode pair per pipeline. Each
 stage may use tensor parallelism, but both sides of the pair must satisfy these
 invariants:
 - both stages define `kv_transfer_config`
 - prefill uses `kv_producer` or `kv_both`
 - decode uses `kv_consumer` or `kv_both`
 - both stages use the same KV connector
 - `kv_buffer_device` and `kv_buffer_size` agree when set on both stages
 - both stages use the same `tensor_parallel_size`
 These checks happen before request processing so an inconsistent PD topology
 fails during engine initialization.
 ## What is implemented today
 | Area | Current state |
 | --- | --- |
 | Pair detection and topology validation | Implemented in the shared PD helper |
 | Prefill sampling preparation | Implemented for sync and async entrypoints |
 | Prefill-to-decode routing | Implemented in the current Orchestrator |
 | Deploy-based configuration | Not yet migrated; the supported `stages:` deploy schema cannot currently expand Qwen3-Omni into the four-stage PD topology |
 | Qwen3 Thinker-to-Talker handoff | Bridge state and embedding-merge helpers exist, but the merge is not wired into the live stage processor |
 | Connector compatibility | Incomplete; the Orchestrator currently requires `remote_request_id`, while the pinned Mooncake connector completes prefill without returning that metadata |
 | Validation coverage | PD entrypoint tests and the PD deploy overlay in online end-to-end tests are temporarily disabled during migration |
 | Performance | No non-regression or throughput/latency result is established for the current path |
 Because these pieces are still converging, the previous legacy `stage_args` YAML
 and `vllm serve` command have been removed. The serve CLI now accepts the
 deploy-based `stages:` schema, so retaining the legacy recipe would imply a
 launch path that current `main` rejects.
 ## Configuration contract under development
 The intended deploy integration starts from the
 [default Qwen3-Omni deploy config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
 and expands its frozen three-stage pipeline into the four stages shown above.
 A complete integration must:
 - split the Thinker definition without duplicating the Talker or Code2Wav model
  contracts
 - remap downstream stage IDs, input sources, and stage connectors
 - inject producer and consumer `kv_transfer_config` values into the two Thinker
  stages
 - allocate separate device sets for prefill and decode
 - preserve both final outputs: decode text and Code2Wav audio
 - provide connector-specific routing metadata without hard-coding one connector
  contract
 The required number of GPUs depends on tensor-parallel sizes and whether Talker
 and Code2Wav share a device. A basic TP=1 layout uses one device for prefill, one
 for decode, and one for the two downstream audio stages.
 ## Implementation map
 - PD detection, validation, and sampling-parameter preparation:
  [`PDDisaggregationMixin`](gh-file:vllm_omni/entrypoints/pd_utils.py)
 - startup detection and connector metadata:
  [`AsyncOmniEngine`](gh-file:vllm_omni/engine/async_omni_engine.py)
 - request-scoped prefill-to-decode routing:
  [`Orchestrator`](gh-file:vllm_omni/engine/orchestrator.py)
 - Qwen3 downstream conditioning helpers:
  [Qwen3-Omni stage input processor](gh-file:vllm_omni/model_executor/stage_input_processors/qwen3_omni.py)
@@ -14,10 +14,7 @@ A: At first, you can check current [issues](https://github.com/vllm-project/vllm
 > Q: Does vLLM-Omni support AWQ or any other quantization?
-A: AWQ is not currently listed as a validated vLLM-Omni path. Quantization
+A: We plan to introduce GGUF FP8 prequantized models and online FP8 quantization in version 0.16.0. Support for other quantization types will follow in future releases. For details, please see our [Q1 quantization roadmap](https://github.com/vllm-project/vllm-omni/issues/1057).
 support depends on the model, pipeline stage, checkpoint format, and hardware
 backend. See the current [quantization support matrix](../user_guide/quantization/overview.md)
 for online methods, pre-quantized checkpoints, and model-level validation.
 > Q: Does vLLM-Omni support multimodal streaming input and output?
@@ -1,23 +1,23 @@
 # Production Metrics
-vLLM-Omni exposes Prometheus metrics via the `/metrics` endpoint on the OpenAI-compatible API server.
+vLLM-Omni exposes Prometheus metrics via the `/metrics` endpoint on the OpenAI-compatible API server. This page covers the text and audio surface; diffusion / image / video metrics are tracked in a follow-up PR.
 ```bash
 vllm-omni serve Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8000 --log-stats
 curl http://localhost:8000/metrics
 ```
-**`--log-stats` is required to populate metric data.** Without the flag, the endpoint still returns `200 OK` but the upstream `vllm:*` wrap is not registered at all, and the 15 `vllm_omni:*` families are registered as placeholders with no data written to them. This keeps the runtime cost essentially zero for deployments that don't need monitoring. With the flag, all ~80 families populate.
+**`--log-stats` is required to populate metric data.** Without the flag, the endpoint still returns `200 OK` but the upstream `vllm:*` wrap is not registered at all, and the 15 `vllm:omni_*` families are registered as placeholders with no data written to them. This keeps the runtime cost essentially zero for deployments that don't need monitoring. With the flag, all ~80 families populate.
 ## Metric Namespaces
 | Prefix | Source | Present when |
 |--------|--------|--------------|
-| `vllm_omni:` | vLLM-Omni orchestrator / audio modality / cross-stage transfer | Pipeline-dependent |
+| `vllm:omni_` | vLLM-Omni orchestrator / audio modality / cross-stage transfer | Pipeline-dependent |
 | `vllm:` | Upstream vLLM engine, wrapped by `OmniPrometheusStatLogger` to expose `{stage, replica}` | Pipeline includes an LLM (AR) stage |
 | `http_` / `process_` | Uvicorn / Python runtime | Always |
-## Pipeline-Level Metrics (`vllm_omni:`)
+## Pipeline-Level Metrics (`vllm:omni_`)
 Defined in `vllm_omni/metrics/prometheus.py`. Track request lifecycle across the full multi-stage pipeline.
@@ -25,66 +25,42 @@ Defined in `vllm_omni/metrics/prometheus.py`. Track request lifecycle across the
 | Metric | Type | Labels | Description |
 |--------|------|--------|-------------|
-| `vllm_omni:num_requests_running` | Gauge | `model_name` | Pipeline-global in-flight requests (dispatched to engine, not yet finalized) |
+| `vllm:omni_num_requests_running` | Gauge | `model_name` | Pipeline-global in-flight requests (dispatched to engine, not yet finalized) |
-| `vllm_omni:num_requests_waiting` | Gauge | `model_name` | Requests waiting in the Orchestrator queue |
+| `vllm:omni_num_requests_waiting` | Gauge | `model_name` | Requests waiting in the Orchestrator queue |
-| `vllm_omni:requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason. `finished_reason` ∈ {`stop`, `length`, `abort`, ...} mirroring upstream `vllm:request_success_total`; aborts cover client disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
+| `vllm:omni_requests_success_total` | Counter | `model_name`, `finished_reason` | Total requests by completion reason. `finished_reason` ∈ {`stop`, `length`, `abort`, ...} mirroring upstream `vllm:request_success_total`; aborts cover client disconnect / cancellation paths in addition to upstream `FinishReason.ABORT` |
 ### Latency
 | Metric | Type | Labels | Description |
 |--------|------|--------|-------------|
-| `vllm_omni:e2e_request_latency_s` | Histogram | `model_name` | Pipeline-global end-to-end request latency in seconds |
+| `vllm:omni_e2e_request_latency_s` | Histogram | `model_name` | Pipeline-global end-to-end request latency in seconds |
-## Audio Modality Metrics (`vllm_omni:`)
+## Audio Modality Metrics (`vllm:omni_`)
 Emitted at request finalize, except for `audio_ttfp_s` (streaming-hook at the first audio packet) and `audio_underrun_s` / `audio_continuity_ok_total` (streaming finalize, after the chunk stream is exhausted). All carry `{model_name, stage, replica}` plus the listed extra label.
 | Metric | Type | Extra label | Description |
 |--------|------|-------------|-------------|
-| `vllm_omni:audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
+| `vllm:omni_audio_ttfp_s` | Histogram | — | Time from request arrival to first audio packet/frame |
-| `vllm_omni:audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
+| `vllm:omni_audio_duration_s` | Histogram | — | Audio content duration (`audio_frames / sample_rate`) |
-| `vllm_omni:audio_rtf` | Histogram | — | Real-time factor (`stage_gen_time_s / audio_duration_s`); streaming TTS SLO red line `< 1`; uses `RTF_BUCKETS` |
+| `vllm:omni_audio_rtf` | Histogram | — | Real-time factor (`stage_gen_time_s / audio_duration_s`); streaming TTS SLO red line `< 1`; uses `RTF_BUCKETS` |
-| `vllm_omni:audio_frames_total` | Counter | — | Cumulative audio frame count; throughput via `rate()` |
+| `vllm:omni_audio_frames_total` | Counter | — | Cumulative audio frame count; throughput via `rate()` |
-| `vllm_omni:audio_underrun_s` | Histogram | — | Per-request worst-case player deficit; `> 0` indicates listener heard silent gaps |
+| `vllm:omni_audio_underrun_s` | Histogram | — | Per-request worst-case player deficit; `> 0` indicates listener heard silent gaps |
-| `vllm_omni:audio_continuity_ok_total` | Counter | `threshold_ms` | Incremented when the request's worst underrun stayed below `threshold_ms` |
+| `vllm:omni_audio_continuity_ok_total` | Counter | `threshold_ms` | Incremented when the request's worst underrun stayed below `threshold_ms` |
-| `vllm_omni:audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
+| `vllm:omni_audio_skipped_requests_total` | Counter | `reason` | Silent-loss counter — code2wav rejected malformed codec input and returned `200 OK` with empty audio |
 The continuity math comes from `vllm_omni/benchmarks/audio_continuity.py::compute_continuity_stats` so the server-side observation aligns with the bench-side definition.
-## Diffusion Metrics (`vllm_omni:`)
+## Cross-Stage Transfer Metrics (`vllm:omni_`)
 Per-request timing breakdowns for diffusion (image/video) stages. Emitted at request finalize when `stage_metrics.diffusion_metrics` is present. All carry `{model_name, stage, replica}`.
 | Metric | Type | Description |
 |--------|------|-------------|
 | `vllm_omni:diffusion_exec_s` | Histogram | DiT forward pass execution time per request in seconds |
 | `vllm_omni:diffusion_exec_per_step_s` | Histogram | DiT forward pass execution time per denoising step in seconds (`exec_s / num_inference_steps`) |
 | `vllm_omni:diffusion_preprocess_s` | Histogram | Diffusion input preprocessing time per request in seconds |
 | `vllm_omni:diffusion_postprocess_s` | Histogram | Diffusion output postprocessing (VAE decode) time per request in seconds |
 `diffusion_exec_s` uses `SECONDS_BUCKETS` (50 ms → 300 s); the other three use `SECONDS_FAST_BUCKETS` (1 ms → 60 s). `diffusion_exec_per_step_s` is skipped when `num_inference_steps` is not set.
 ### Example PromQL
 ```promql
 # P99 DiT exec time by stage
 histogram_quantile(0.99, rate(vllm_omni:diffusion_exec_s_bucket[5m]))
 # Ratio of preprocess to total exec wall time
 rate(vllm_omni:diffusion_preprocess_s_sum[5m])
  / rate(vllm_omni:diffusion_exec_s_sum[5m])
 ```
 ## Cross-Stage Transfer Metrics (`vllm_omni:`)
 Per-physical-transfer histograms tracking the data hop between adjacent stages. Labels `{model_name, from_stage, from_replica, to_stage, to_replica}` let dashboards attribute latency to specific replica edges. `from_replica` / `to_replica` are resolved from the orchestrator's sticky-routing binding (`stage_pool.get_bound_replica_id(request_id)`), so no extra plumbing through `TransferEdgeStats` is needed.
 | Metric | Type | Description |
 |--------|------|-------------|
-| `vllm_omni:transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
+| `vllm:omni_transfer_size_bytes` | Histogram | Per-transfer payload size in bytes |
-| `vllm_omni:transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
+| `vllm:omni_transfer_tx_s` | Histogram | Sender-side time (serialize + submit to connector) |
-| `vllm_omni:transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
+| `vllm:omni_transfer_rx_s` | Histogram | Receiver-side time (recv + deserialize) |
-| `vllm_omni:transfer_in_flight_s` | Histogram | Network in-flight time (TX done → RX recv start) |
+| `vllm:omni_transfer_in_flight_s` | Histogram | Network in-flight time (TX done → RX recv start) |
 ## vLLM Engine Metrics (`vllm:`)
@@ -107,9 +83,9 @@ For the full list of upstream metrics, see [the vLLM docs](https://github.com/vl
 | Metric group | Multi-stage LLM (Qwen3-Omni) |
 |---|---|
-| `vllm_omni:` request tracking + latency | With `--log-stats` |
+| `vllm:omni_` request tracking + latency | With `--log-stats` |
-| `vllm_omni:` audio modality | With `--log-stats`, if pipeline has a talker stage |
+| `vllm:omni_` audio modality | With `--log-stats`, if pipeline has a talker stage |
-| `vllm_omni:` transfer | With `--log-stats`, if pipeline has ≥ 2 stages |
+| `vllm:omni_` transfer | With `--log-stats`, if pipeline has ≥ 2 stages |
 | `vllm:` engine metrics (per `(stage, replica)`) | With `--log-stats` |
 | `vllm:` MFU metrics | With `--log-stats --enable-mfu-metrics` |
@@ -118,5 +94,5 @@ For the full list of upstream metrics, see [the vLLM docs](https://github.com/vl
 - All time-bearing metrics use the `_s` suffix (values in seconds). Buckets are `SECONDS_BUCKETS` for e2e / generation-style values and `SECONDS_FAST_BUCKETS` (1 ms → 60 s) for the fine-grained transfer and audio-underrun values.
 - Counters use the `_total` suffix (auto-appended by `prometheus_client`).
 - Sizes use the `_bytes` suffix.
- All omni-specific families are prefixed `vllm_omni:`. The upstream `unregister_vllm_metrics()` function is monkey-patched (see `vllm_omni/patch.py`) to a scoped version that still strips upstream `vllm:*` collectors so multi-engine init within one process does not crash on duplicate registration, but preserves anything prefixed `vllm_omni:`.
+- All omni-specific families are prefixed `vllm:omni_`. The upstream `unregister_vllm_metrics()` function is monkey-patched (see `vllm_omni/patch.py`) to a scoped version that still strips upstream `vllm:*` collectors so multi-engine init within one process does not crash on duplicate registration, but preserves anything prefixed `vllm:omni_` / `vllm_omni`.
- Text and audio first-output use distinct families (`vllm:time_to_first_token_seconds` reused from upstream for text; `vllm_omni:audio_ttfp_s` for audio) rather than a single metric with a `modality` label.
+- Text and audio first-output use distinct families (`vllm:time_to_first_token_seconds` reused from upstream for text; `vllm:omni_audio_ttfp_s` for audio) rather than a single metric with a `modality` label.
@@ -92,7 +92,7 @@ current support.
 |--------|--------------|-------------|----------|
 | **[FP8](quantization/fp8.md)** | `quantization="fp8"` | FP8 W8A8 on validated transformer stages | Memory reduction, inference speedup |
 | **[INT8](quantization/int8.md)** | `quantization="int8"` | INT8 W8A8 on validated transformer stages | Memory reduction, broad GPU compatibility |
-| **[BitsAndBytes W4](quantization/bitsandbytes.md)** | `quantization="bitsandbytes"` | Online NF4/FP4 weight-only quantization on validated CUDA transformer stages | Reducing transformer weight memory without preparing a separate checkpoint |
+| **[GGUF](quantization/gguf.md)** | `quantization="gguf"` | Native GGUF transformer-only weights (Q4, Q8, etc.) | Memory reduction on consumer GPUs |
 ## Supported Models
@@ -143,8 +143,7 @@ python text_to_video.py \
 - `--enable-layerwise-offload`: enable layerwise offloading on DiT modules.
 - `--frame-rate`: generation FPS for pipelines that require it (e.g., LTX2).
 - `--audio-sample-rate`: fallback audio sample rate when the pipeline returns audio.
- `--quantization`: online quantization method, for example `fp8`; supported
+- `--quantization`: quantization method (`fp8` for FP8, `gguf` for GGUF).
  pre-quantized checkpoints are normally auto-detected from checkpoint metadata.
 - `--flow-shift`: scheduler flow_shift parameter.
 - `--lora-path`: path to PEFT LoRA adapter folder or checkpoint file.
 - `--lora-scale`: scale factor for LoRA weights.
@@ -90,8 +90,8 @@ vllm serve "Qwen/Qwen-Image" \
    --quantization-config '{"method": "fp8"}'
 ```
-Other vLLM-Omni quantization methods, such as `modelopt`, `mxfp4`, `mxfp8`,
+Other vLLM-Omni quantization methods, such as `gguf`, `modelopt`, `mxfp4`,
-serialized checkpoints, static FP8 configs, and layer-name skip lists
+`mxfp8`, serialized checkpoints, static FP8 configs, and layer-name skip lists
 such as `ignored_layers`, are intentionally not translated by this compatibility
 shortcut. Use Diffusers-native configuration through `diffusers_load_kwargs` or
 a native vLLM-Omni pipeline for those cases.
@@ -10,10 +10,6 @@ checkpoint's `config.json` and auto-detects
 AutoRound is static quantization: no `--quantization` flag is needed at
 inference time when the checkpoint already contains the quantization config.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels) defined in the overview. A listed
 checkpoint is not enough by itself to establish end-to-end model support.
 ## Hardware Support
 | Device | Support |
@@ -25,57 +21,46 @@ checkpoint is not enough by itself to establish end-to-end model support.
 | Intel XPU | ✅ |
 | Ascend NPU | ❌ |
-Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
-guide. This table is method-level backend availability. AutoRound is an Intel
+guide. AutoRound is Intel-supported.
 project; vLLM selects a compatible CUDA or XPU compute backend at load time.
 ## Model Type Support
 ### Diffusion Model (Qwen-Image, Wan2.2)
-| Model | Checkpoint | Scope | Scheme | Validation |
+| Model | Checkpoint | Scope | Scheme | Backend |
-|-------|------------|-------|--------|------------|
+|-------|------------|-------|--------|---------|
-| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | Validated |
+| FLUX.1-dev | `vllm-project-org/FLUX.1-dev-AutoRound-w4a16` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
-| Qwen-Image | `INC4AI/Qwen-Image-AutoRound-W4A16` | Diffusion transformer | W4A16 | Validated with end-to-end generation and quality comparison |
+| Qwen-Image | Not listed | Diffusion transformer | W4A16 | Not validated |
-| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
+| Wan2.2-I2V | `Intel/Wan2.2-I2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
-| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Both diffusion transformers | W4A16 | CI-backed |
+| Wan2.2-T2V | `Intel/Wan2.2-T2V-A14B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
-| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | Validated; not in the scheduled Wan AutoRound job |
+| Wan2.2-TI2V | `Intel/Wan2.2-TI2V-5B-Diffusers-int4-AutoRound` | Diffusion transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
 CUDA execution uses GPTQ-Marlin or another compatible vLLM AutoRound backend;
 Intel XPU uses the Intel-supported backend. The text encoder and VAE stay in
 BF16 for all entries above.
 ### World Model (Cosmos3)
-| Model | Checkpoint | Scope | Scheme | Validation |
+| Model | Checkpoint | Scope | Scheme | Backend |
-|-------|------------|-------|--------|------------|
+|-------|------------|-------|--------|---------|
-| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | Integrated; no model-specific end-to-end test is maintained in-tree |
+| Cosmos3-Nano | `Intel/Cosmos3-Nano-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
-| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | Validated with a manual serving run |
+| Cosmos3-Super | `Intel/Cosmos3-Super-int4-AutoRound` | World-model transformer | W4A16 | GPTQ-Marlin or Intel-supported AutoRound backend |
 The Cosmos VAE and guardrail components stay in BF16.
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
-| Model | Checkpoint | Scope | Scheme | Validation |
+| Model | Checkpoint | Scope | Scheme | Backend |
-|-------|------------|-------|--------|------------|
+|-------|------------|-------|--------|---------|
-| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
+| Qwen2.5-Omni-7B | `Intel/Qwen2.5-Omni-7B-int4-AutoRound` | Language-model stage | W4A16 | AutoRound |
-| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | Validated outside scheduled CI |
+| Qwen3-Omni-30B-A3B-Instruct | `Intel/Qwen3-Omni-30B-A3B-Instruct-int4-AutoRound` | Thinker language-model stage | W4A16 | AutoRound |
 | Qwen3-TTS | Not listed | TTS language-model stage | W4A16 | Not validated |
 AutoRound support is checkpoint-driven. A model is supported when its
 checkpoint uses a compatible INC/AutoRound config and the target stage maps to
 vLLM-Omni's runtime module names.
 For Qwen-Omni, only the thinker language model uses the AutoRound checkpoint.
 Audio and vision encoders, the talker, and waveform-decoder stages stay in
 BF16.
 ### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
-| Model | Checkpoint | Scope | Validation |
+| Model | Scope | Status | Notes |
-|-------|------------|-------|------------|
+|-------|-------|--------|-------|
-| GLM-Image | `Intel/GLM-Image-int4-AutoRound` | Diffusion transformer | CI-backed |
+| GLM-Image | Diffusion transformer  | ✅ | `Intel/GLM-Image-int4-AutoRound` |
-| BAGEL | Not listed | Checkpoint-defined diffusion or transformer stage | Not validated |
+| BAGEL | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint |
 ## Configuration
@@ -134,14 +119,6 @@ The checkpoint should contain a config like:
 At load time, vLLM-Omni builds an `OmniINCConfig`, remaps checkpoint block names
 to runtime module names, and selects the matching vLLM compute backend.
 Checkpoint auto-detection and successful layer construction establish the
 **Integrated** level. Promote a model to **Validated** only after the named
 checkpoint completes an end-to-end generation or multimodal request and its
 output passes a model-appropriate sanity or quality check. Compare against the
 BF16 baseline for quality-sensitive image and video paths. Promote it to
 **CI-backed** only when that named checkpoint is selected by scheduled hardware
 CI.
 Example checkpoint creation:
 ```bash
@@ -9,10 +9,6 @@ stay in BF16/FP16. No pre-quantized checkpoint is required.
 This is an online (dynamic) path: load the normal HuggingFace checkpoint and
 quantize during model loading via the `bitsandbytes` CUDA kernels.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). BitsAndBytes kernel support does
 not imply that every diffusion transformer has been quality-validated.
 ## Hardware Support
 | Device | Support |
@@ -25,7 +21,7 @@ not imply that every diffusion transformer has been quality-validated.
 | Intel XPU | ❌ |
 | Ascend NPU | ❌ |
-Legend: `✅` backend available, `❌` unsupported. Non-CUDA platforms raise in
+Legend: `✅` supported, `❌` unsupported. Non-CUDA platforms raise in
 `get_quant_method()`; CUDA requires compute capability 7.5+ (`SM 75+`).
 Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
@@ -34,11 +30,11 @@ Requires the optional `bitsandbytes` package (`pip install bitsandbytes`).
 ### Diffusion Model (Qwen-Image, Wan2.2)
-| Model | HF models | Scope | Level | Recommendation |
+| Model | HF models | CUDA | Mode | Recommendation |
-|-------|-----------|-------|-------|----------------|
+|-------|-----------|:----:|------|----------------|
-| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Diffusion transformer | Validated | All heavy linear layers; sensitive embedders stay BF16 |
+| Z-Image | `Tongyi-MAI/Z-Image-Turbo` | Yes | Online W4 weight-only | All heavy linear layers; sensitive embedders stay BF16 |
-| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Diffusion transformer | Not validated | Compare with BF16 before enabling |
+| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Not validated | Online W4 weight-only | Compare vs BF16 before enabling |
-| Wan2.2 | Wan2.2 diffusion pipelines | One or both diffusion transformers | Not validated | Validate quality and cascade behavior before enabling |
+| Wan2.2 | Wan2.2 diffusion pipelines | Not validated | Online W4 weight-only | Validate before enabling in docs |
 Other diffusion models may work if their transformer uses supported linear
 layers, but they are not validated in this guide.
@@ -108,10 +104,6 @@ Multi-GPU tensor parallelism (`tensor_parallel_size` > 1) is not validated for
 BitsAndBytes in diffusion models; each rank quantizes its own weight shard
 independently.
 Z-Image-Turbo is validated outside scheduled full-model BitsAndBytes CI. The
 real CUDA kernel smoke tests cover quantization and matrix multiplication, but
 do not establish model-level quality for unlisted architectures.
 If quality regresses, use `ignored_layers` to keep sensitive projections in BF16
 (for example `to_out` or `w2`).
@@ -13,11 +13,6 @@ components, such as the tokenizer, scheduler, text encoder, vision/audio
 encoder, and VAE, on the base checkpoint unless a model-specific guide says
 otherwise.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels) defined in the overview. ModelOpt
 format detection and checkpoint adaptation are shared runtime integrations;
 each model and stage still requires its own end-to-end validation.
 !!! note
    ModelOpt checkpoints are pre-quantized checkpoints. Do not pass
    `--quantization fp8` for these checkpoints. The checkpoint
@@ -63,11 +58,11 @@ stay unquantized unless the model-specific recipe validates otherwise.
 | Intel XPU | ⭕ |
 | Ascend NPU | ❌ |
-Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
-guide. The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt
+The optional CUTLASS FP8 runtime override requires CUDA SM89+. ModelOpt NVFP4
-NVFP4 and mixed FP8/NVFP4 diffusion checkpoints are currently validated on
+and mixed FP8/NVFP4 diffusion checkpoints are currently validated on Blackwell
-Blackwell CUDA systems in the recipes below; other CUDA generations require
+CUDA systems in the recipes below; other CUDA generations require separate
-separate backend and quality validation.
+backend and quality validation.
 ## Model Type Support
@@ -84,9 +79,6 @@ separate backend and quality validation.
 | HunyuanImage-3.0 | `feizhai123/hunyuan-image3-modelopt-mixed-experts-nvfp4-dense-fp8` | MoE diffusion transformer | Validated for ModelOpt mixed FP8/NVFP4 checkpoints |
 | Wan2.2 | Not available | Diffusion transformer | Not validated |
 The validated diffusion entries above have named checkpoint and recipe
 evidence. They are not all exercised by scheduled full-model CI.
 For full serving commands and benchmark context, see
 [`recipes/Qwen/Qwen-Image.md`](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Qwen/Qwen-Image.md)
 and
@@ -96,8 +88,8 @@ and
 | Model | Scope | Status |
 |-------|-------|--------|
-| Qwen3-Omni | Thinker language-model stage (FP8) | Validated outside scheduled CI |
+| Qwen3-Omni | Thinker language-model stage | ModelOpt FP8 checkpoint path |
-| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | CI-backed for the H100 Marlin fallback; native Blackwell path is recipe-validated |
+| Qwen3-Omni | Thinker language-model stage (W4A4 NVFP4) | Validated; see [Qwen3-Omni NVFP4 W4A4](#qwen3-omni-nvfp4-w4a4-thinker) below |
 | Qwen3-TTS | TTS language-model stage | Not validated |
 Audio encoder, vision encoder, talker, and code2wav stages stay in BF16 unless
@@ -110,9 +102,9 @@ Qwen3-Omni-30B-A3B-Instruct thinker language model. The thinker text body
 (attention + MoE experts) is quantized to NVFP4 with FP8 per-tensor input
 scales; the audio encoder, vision encoder, talker, and code2wav stay in BF16.
-| Variant | HF checkpoint | Hardware and validation |
+| Variant | HF checkpoint | Hardware |
-|---------|---------------|-------------------------|
+|---------|---------------|----------|
-| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ native FlashInfer FP4 GEMM; H100 Marlin fallback is CI-backed |
+| W4A4 NVFP4 (full thinker) | `YihongJin/Qwen3-Omni-30B-A3B-Instruct-NVFP4-W4A4-full-thinker-awqclip` | sm_100+ (Blackwell, FlashInfer FP4 GEMM) |
 Calibration uses ModelOpt `mtq.NVFP4_DEFAULT_CFG` with the `awq_clip` algorithm
 on 1024 ultrachat samples chat-templated through the Qwen3-Omni tokenizer.
@@ -240,6 +232,3 @@ omni = Omni(
 5. Keep `--quantization fp8` for online FP8 from BF16 checkpoints; use this
   ModelOpt path only when the checkpoint already contains ModelOpt quantized
   weights and scales.
 6. The Qwen3-Omni full-model CI smoke exercises the H100 Marlin fallback. A
   scheduled Blackwell test is still required before treating the native
   FlashInfer FP4 kernel path as CI-backed.
@@ -3,17 +3,12 @@
 ## Overview
 [msModelSlim](https://github.com/Ascend/msmodelslim) is an Ascend compression
-toolkit for producing pre-quantized model checkpoints. vLLM-Omni consumes its
+toolkit for producing pre-quantized model checkpoints. In vLLM-Omni, these
-output either through a compatible generic Ascend checkpoint path or through
+checkpoints run through the Ascend/NPU path with `--quantization ascend`.
 the native MXFP8 and MXFP4 merge workflows documented in this guide.
 msModelSlim is static quantization: quantized weights are generated offline
 before vLLM-Omni inference starts.
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). A producer recipe or hardware
 backend alone is an integration point, not model-level validation.
 ## Hardware Support
 | Device | Support |
@@ -25,28 +20,23 @@ backend alone is an integration point, not model-level validation.
 | Intel XPU | ❌ |
 | Ascend NPU | ✅ |
-Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
 guide.
 ## Model Type Support
 ### Diffusion Model (Qwen-Image, Wan2.2)
-| Model | Base model | Scope | Level | Notes |
+| Model | Base model | Scope | Hardware | Notes |
-|-------|------------|-------|-------|-------|
+|-------|------------|-------|----------|-------|
-| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Integrated | Upstream msModelSlim provides a producer recipe; use the [MXFP8](mxfp8.md) or [MXFP4](mxfp4.md) workflow for vLLM-Omni's native merged formats |
+| Wan2.2 | Wan2.2 diffusion weights | DiT or diffusion stage | Ascend NPU | Upstream msModelSlim provides a Wan2.2 quantization recipe; vLLM-Omni inference validation is not listed |
-| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Not validated | No supported checkpoint workflow is documented |
+| Qwen-Image | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | DiT or diffusion stage | Ascend NPU | Not validated in this guide |
-| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Integrated | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation and validate against BF16 |
+| HunyuanImage-3.0 | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | DiT or diffusion stage | Ascend A2/A3 NPU | Generate quantized weights with the HunyuanImage-3.0 msModelSlim adaptation |
 Public Hugging Face quantized weights are not available yet. Use the
 [HunyuanImage-3.0 msModelSlim adaptation](https://gitcode.com/betta18/msmodelslim/tree/hyimage3_mxfp8)
 to generate the checkpoint manually.
 msModelSlim is also the producer used by the native offline MXFP8 and MXFP4
 DualScale workflows. Those workflows preprocess and annotate each transformer
 checkpoint and are auto-detected at inference; follow their method pages rather
 than passing `--quantization ascend`.
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
 | Model | Scope | Status | Notes |
@@ -63,32 +53,25 @@ than passing `--quantization ascend`.
 ## Configuration
 For a compatible generic Ascend checkpoint, select the Ascend quantization
 method explicitly:
 Offline inference:
 ```bash
-python text_to_image.py --model <compatible-ascend-checkpoint> --quantization ascend
+python text_to_image.py --model <quantized-model-path> --quantization ascend
 ```
 Online serving:
 ```bash
-vllm serve <compatible-ascend-checkpoint> --omni --quantization ascend
+vllm serve <quantized-model-path> --omni --quantization ascend
 ```
 ## Parameters
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
-| `--quantization` | str | - | Use `ascend` only for a compatible generic Ascend-format checkpoint; omit it for native merged MXFP8/MXFP4 checkpoints |
+| `--quantization` | str | - | Use `ascend` for msModelSlim-produced checkpoints |
 | `model` | str | - | Path to the quantized checkpoint generated by Ascend tooling |
 Do not pass `--quantization ascend` for a Wan checkpoint processed by the native
 MXFP8 or MXFP4 merge tools. Their transformer metadata selects `mxfp8` or
 `mxfp4_dualscale` automatically.
 Example msModelSlim command for a Wan2.2 W8A8 checkpoint:
 ```bash
@@ -110,6 +93,3 @@ For HunyuanImage-3.0, use the Hunyuan-specific adaptation linked above.
   tooling; it is not a load-time CUDA quantizer.
 3. Keep the quantized checkpoint aligned with the same model architecture and
   stage config used for BF16 inference.
 4. No named full-model msModelSlim checkpoint is currently scheduled in NPU
   CI. Treat new producer outputs as **Integrated** until an end-to-end BF16
   comparison establishes **Validated** support.
@@ -13,10 +13,6 @@ vLLM-Omni provides two quantization methods with different scale structures:
 | `mxfp4` | Single-scale (per-32 fine only) | Online only | Quick accuracy baseline; no checkpoint prep needed |
 | `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | Production; better accuracy; offline recommended |
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). Current model guidance covers
 Wan diffusion transformers only; the text encoder and VAE stay in BF16.
 !!! tip "Recommended: `mxfp4_dualscale` offline"
    For production deployments, use the `mxfp4_dualscale` offline mode with a
    pre-quantized checkpoint produced by msModelSlim. Offline checkpoints load
@@ -38,45 +34,26 @@ Wan diffusion transformers only; the text encoder and VAE stay in BF16.
 ## Hardware Support
-| Device | Online `mxfp4` | Online `mxfp4_dualscale` | Offline `mxfp4_dualscale` |
+| Device | Support |
-|--------|-----------------|---------------------------|----------------------------|
+|--------|---------|
-| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ❌ |
+| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
-| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ❌ |
+| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ |
-| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ❌ |
+| NVIDIA Ampere GPU (SM 80+) | ⭕ |
-| AMD ROCm (gfx950 / MI355X) | ✅ | ❌ | ❌ |
+| AMD ROCm (gfx950 / MI355X) | ✅ |
-| Intel XPU | ❌ | ❌ | ❌ |
+| Intel XPU | ⭕ |
-| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ✅ |
+| Ascend NPU (Atlas 950 A5) | ✅ |
-Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
 guide. Backend availability does not imply full-model checkpoint validation.
 !!! note "Why NVIDIA is marked unsupported"
    This table describes the methods currently wired into vLLM-Omni, not every
    low-precision format NVIDIA hardware could execute. `DiffusionMXFP4Config`
    has runtime implementations only for Ascend and ROCm gfx950; on CUDA it
    raises `NotImplementedError` because no CUDA MXFP4 linear method or
    checkpoint loader is registered.
    On NVIDIA, use [ModelOpt](modelopt.md) NVFP4 for the documented 4-bit
    checkpoint path. NVFP4 and OCP MXFP4 have different checkpoint metadata,
    scale layouts, and kernel contracts, so they are not interchangeable. A
    native CUDA MXFP4 path would require a CUDA linear implementation, loader
    contract, and model-level output validation.
 The ROCm path dispatches online single-scale `mxfp4` to AITER on gfx950. It has
 kernel and dispatch coverage, but this guide does not yet claim end-to-end
 quality validation for a named Wan checkpoint on ROCm. DualScale and
 pre-quantized MXFP4 checkpoints are not supported on ROCm.
 ## Model Type Support
 ### Diffusion Model (Wan2.2)
-| Model | Online (Ascend) | Offline (Ascend) | Validation and notes |
+| Model | Online | Offline | Notes |
-|-------|-----------------|------------------|----------------------|
+|-------|--------|---------|-------|
-| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; both cascade transformers are quantized |
+| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade (`transformer` + `transformer_2`); both transformers quantized with the same config |
-| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | Validated outside scheduled full-model offline NPU CI; same cascade scheme as T2V-A14B |
+| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade; same scheme as T2V-A14B |
-| Wan2.2-TI2V-5B | ❌ | ❌ | Unsupported because W4A4 causes unacceptable accuracy loss |
+| Wan2.2-TI2V-5B | ❌ | ❌ | Parameter count too small; W4A4 causes unacceptable accuracy loss |
 The choice between `mxfp4` and `mxfp4_dualscale` in **online mode** is about
 quantization quality, not model compatibility — both work on cascade (A14B) and
@@ -106,13 +83,12 @@ by the merge script); there is no offline `mxfp4` single-scale format.
 ### `mxfp4` — Single-Scale Online Mode
-Online mode requires no pre-processing. On Ascend, vLLM-Omni quantizes BF16
+Online mode requires no pre-processing. vLLM-Omni quantizes BF16 weights to
-weights at load time using `npu_dynamic_mx_quant`. On ROCm gfx950, the same
+MXFP4 at load time using `npu_dynamic_mx_quant`. A single block scale
 `mxfp4` config dispatches to the AITER online kernel. A single block scale
 (`float8_e8m0fnu`, one per 32 K elements) is computed on the fly; no
-calibration `mul_scale` is available. The config applies equally to
+calibration `mul_scale` is available. Applies equally to single-transformer
-single-transformer and cascade (A14B) models, but named Wan model validation is
+and cascade (A14B) models — both transformers in a cascade receive the same
-currently limited to the documented Ascend path.
+quantization config automatically.
 ```python
 from vllm_omni import Omni
@@ -135,10 +111,10 @@ vllm serve <your-model> --omni --quantization mxfp4
 ### `mxfp4_dualscale` — DualScale Online Mode
-Ascend-only online DualScale mode computes both fine and coarse scales on the
+Online DualScale mode computes both fine and coarse scales on the fly from BF16
-fly from BF16 weights using `npu_dynamic_dual_level_mx_quant`. It applies
+weights using `npu_dynamic_dual_level_mx_quant`. Applies equally to
-equally to single-transformer and cascade (A14B) models. Compared to `mxfp4`
+single-transformer and cascade (A14B) models. Compared to `mxfp4` online,
-online, DualScale provides better quantization accuracy at higher compute cost.
+DualScale provides better quantization accuracy at higher compute cost.
 The default configuration keeps the leading 5 transformer blocks in BF16
 (`num_bf16_fallback_layers=5`). Accuracy evaluation on Wan2.2-A14B shows this
@@ -365,11 +341,6 @@ unless they appear in `ignored_layers`.
   quantization. Use `ignored_layers` and `num_bf16_fallback_layers` to trade
   off compression vs. accuracy for precision-sensitive layers.
 8. The native methods, merge mappings, and per-transformer cascade metadata
   have unit coverage. The current Buildkite configuration does not schedule a
   named full-model offline MXFP4 checkpoint on NPU, so this path is
   **Validated**, not **CI-backed**.
 ## Adapting MXFP4 for a New Model
 This section is aimed at developers who want to add MXFP4 support to a model
@@ -15,45 +15,30 @@ This method supports three modes:
 | **Offline (Native)** | msModelSlim-exported MXFP8 weights converted to diffusers format via `merge_mxfp8_checkpoint.py` — weights and scales are loaded directly from the preprocessed checkpoint |
 | **Offline (AutoRound)** | AutoRound MXFP8 checkpoints with `data_type="mx_fp"` — auto-detected from `config.json` |
 The validation terms on this page follow the
 [support levels](overview.md#support-levels). Current model guidance covers
 Wan diffusion transformers only; the text encoder and VAE stay in BF16.
 ## Hardware Support
 | Device | Online | Offline (Native) | Offline (AutoRound) |
 |--------|--------|------------------|---------------------|
-| NVIDIA Blackwell GPU (SM 100+) | ❌ | ❌ | ⭕ |
+| NVIDIA Blackwell GPU (SM 100+) | ⭕ | ⭕ | ⭕ |
-| NVIDIA Ada/Hopper GPU (SM 89+) | ❌ | ❌ | ⭕ |
+| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ | ⭕ | ⭕ |
-| NVIDIA Ampere GPU (SM 80+) | ❌ | ❌ | ⭕ |
+| NVIDIA Ampere GPU (SM 80+) | ⭕ | ⭕ | ⭕ |
-| AMD ROCm | ❌ | ❌ | ⭕ |
+| AMD ROCm | ⭕ | ⭕ | ⭕ |
 | Intel XPU | ✅ | ❌ | ✅ |
 | Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ⭕ |
-Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
 guide. Backend availability does not imply full-model checkpoint validation.
-The native `DiffusionMXFP8Config` currently dispatches only to Intel XPU and
+**Note**: Intel XPU only supports AutoRound MXFP8 for offline mode. Use AutoRound quantized checkpoints or online mode for XPU.
 Ascend NPU implementations. CUDA reaches `NotImplementedError`, so the NVIDIA
 rows are marked unsupported even though this is an implementation gap rather
 than a general claim about NVIDIA hardware capabilities. On NVIDIA, use
 [FP8](fp8.md) for load-time quantization or [ModelOpt](modelopt.md) FP8 for
 pre-quantized checkpoints. Adding native CUDA MXFP8 requires a registered CUDA
 linear method, a compatible checkpoint contract, and model-level validation.
 Native `mxfp8` is available online on Intel XPU and online or offline on
 Ascend. Intel XPU only supports AutoRound MXFP8 for offline mode. The AutoRound
 column is a separate checkpoint format and compute path.
 ## Model Type Support
 ### Diffusion Model (Wan2.2)
-| Model | Mode | Scope | Validation |
+| Model | Mode | Notes |
-|-------|------|-------|------------|
+|-------|------|-------|
-| Wan2.2-T2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI |
+| Wan2.2-T2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) |
-| Wan2.2-I2V-A14B | Online + Offline | MoE cascade: `transformer` + `transformer_2` | Validated outside scheduled full-model offline NPU CI |
+| Wan2.2-I2V-A14B | Online + Offline | MoE cascade; quantizes two transformers (`transformer` + `transformer_2`) |
-| Wan2.2-TI2V-5B | Online + Offline | Single diffusion transformer | Validated outside scheduled full-model offline NPU CI |
+| Wan2.2-TI2V-5B | Online + Offline | Single transformer |
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -233,7 +218,3 @@ omni = Omni(model="<autoround-mxfp8-model>")
   `--output-dir`), regenerate it with the current script. The old script
   wrote a separate `quantization_config.json` that is not read by vLLM-Omni;
   the current script injects the config directly into `transformer/config.json`.
 4. The native loader, merge mappings, and Wan cascade config propagation have
   unit coverage. The current Buildkite configuration does not schedule a
   named full-model offline MXFP8 checkpoint on NPU, so this path is
   **Validated**, not **CI-backed**.
@@ -6,7 +6,7 @@ Online quantization means vLLM-Omni computes quantized weights and scales while
 loading the model. Use it when you want memory savings without preparing a
 separate quantized checkpoint.
-This mode is different from pre-quantized checkpoint formats such as ModelOpt,
+This mode is different from pre-quantized checkpoint formats such as GGUF,
 AutoRound, msModelSlim, or serialized Int8 checkpoints. Those formats are
 prepared before serving and are documented in their method-specific guides.
 For MXFP8 and MXFP4, use this page for load-time quantization from BF16
@@ -15,22 +15,18 @@ by msModelSlim and the merge tools.
 ## Hardware Support
-| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | MXFP8 W8A8 | MXFP4 W4A4 |
+| Device | FP8 W8A8 | Int8 W8A8 | MXFP8 W8A8 | MXFP4 W4A4 |
-|--------|----------|-----------|------------------|------------|------------|
+|--------|----------|-----------|------------|------------|
-| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ❌ | ❌ |
+| NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ⭕ | ⭕ |
-| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ❌ | ❌ |
+| NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ |
-| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ❌ | ❌ |
+| NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ |
-| AMD ROCm (gfx950) | ⭕ | ⭕ | ❌ | ❌ | ✅ |
+| AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ |
-| Intel XPU | ⭕ | ⭕ | ❌ | ✅ | ❌ |
+| Intel XPU | ⭕ | ⭕ | ✅ | ⭕ |
-| Ascend NPU | ❌ | ✅ | ❌ | ✅ | ✅ |
+| Ascend NPU | ❌ | ✅ | ✅ | ✅ |
-Legend: `✅` backend available, `❌` unsupported, `⭕` not verified in this
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
-guide. These entries describe method-level backend availability, not validation
+guide. FP8 on Ampere may use a weight-only path where available. MXFP8 and
-for every model. FP8 on Ampere may use a weight-only path where available.
+MXFP4 are documented for the Ascend NPU path.
 On ROCm gfx950, MXFP4 means online single-scale `mxfp4`; online and offline
 `mxfp4_dualscale` are Ascend-only. The native MXFP8/MXFP4 configs do not
 currently register CUDA implementations; use FP8 or ModelOpt FP8/NVFP4 on
 NVIDIA instead.
 ## Model Type Support
@@ -40,9 +36,8 @@ NVIDIA instead.
 |--------|-------|----------------|--------|
 | FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
 | Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
 | BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Z-Image-Turbo; Qwen-Image and Wan2.2 are not validated | Validated for Z-Image-Turbo |
 | MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Validated on Ascend NPU and Intel XPU |
-| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Validated on Ascend NPU; the ROCm gfx950 backend is integrated but not model-validated; TI2V-5B is unsupported |
+| MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Ascend NPU only; TI2V-5B is not supported |
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -64,7 +59,6 @@ from vllm_omni import Omni
 omni_fp8 = Omni(model="<your-model>", quantization="fp8")
 omni_int8 = Omni(model="<your-model>", quantization="int8")
 omni_bnb = Omni(model="Tongyi-MAI/Z-Image-Turbo", quantization="bitsandbytes")
 omni_mxfp8 = Omni(model="<your-model>", quantization="mxfp8")
 omni_mxfp4 = Omni(model="<your-model>", quantization="mxfp4")
 omni_mxfp4_dualscale = Omni(model="<your-model>", quantization="mxfp4_dualscale")
@@ -75,7 +69,6 @@ CLI:
 ```bash
 vllm serve <your-model> --omni --quantization fp8
 vllm serve <your-model> --omni --quantization int8
 vllm serve Tongyi-MAI/Z-Image-Turbo --omni --quantization bitsandbytes
 vllm serve <your-model> --omni --quantization mxfp8
 vllm serve <your-model> --omni --quantization mxfp4
 vllm serve <your-model> --omni --quantization mxfp4_dualscale
@@ -92,19 +85,14 @@ config = build_quant_config({
 })
 ```
 Component routing is an extension mechanism, not automatic support for every
 pipeline component. See [Quantization Scope](overview.md#quantization-scope)
 before applying it to an encoder, VAE, or decoder.
 ## Parameters
 | Parameter | Methods | Description |
 |-----------|---------|-------------|
-| `method` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"bitsandbytes"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
+| `method` | FP8, Int8, MXFP8, MXFP4 | Quantization method: `"fp8"`, `"int8"`, `"mxfp8"`, `"mxfp4"`, or `"mxfp4_dualscale"` |
-| `ignored_layers` | FP8, Int8, BitsAndBytes, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
+| `ignored_layers` | FP8, Int8, MXFP8, MXFP4 | Layer name patterns to keep in BF16/FP16 |
 | `activation_scheme` | FP8, Int8 | The runtime value `"dynamic"` selects online activation scaling |
 | `weight_block_size` | FP8 | Optional block-wise FP8 weight quantization size |
 | `quant_type` | BitsAndBytes | Weight format: `"nf4"` (default) or `"fp4"` |
 | `num_bf16_fallback_layers` | MXFP4 DualScale | Leading transformer blocks to keep in BF16 for online `mxfp4_dualscale`; defaults to `5` |
 ## Validation and Notes
@@ -16,123 +16,72 @@ For the internal architecture and backend extension points, see the
 | Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
 | Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
-Online quantization starts from a normal BF16/FP16 checkpoint and repeats the
+## Hardware Support
 conversion on each fresh model load. It is the simplest path for experiments.
 Pre-quantized checkpoints store the packed weights and scales produced by a
 separate tool; they are easier to reuse across deployments and can carry
 calibrated or mixed-precision policies that load-time conversion cannot
 reproduce.
-## Support Levels
+| Device | FP8 W8A8 | Int8 W8A8 | BitsAndBytes W4 | ModelOpt | MXFP8 W8A8 | MXFP4 W4A4 | AutoRound | msModelSlim |
 |--------|----------|-----------|------------------|----------|------------|------------|-----------|-------------|
 | NVIDIA Blackwell GPU (SM 100+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
 | NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ |
 | NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
 | AMD ROCm | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ⭕ | ❌ |
 | Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ✅ | ❌ |
 | Ascend NPU | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
-Quantization support is specific to a model, component, checkpoint format, and
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
-hardware backend. The tables in this guide use the following levels:
+guide. FP8 on Ampere may use a weight-only path where available.
-| Level | Meaning |
+## Model Type Support
 |-------|---------|
 | **CI-backed** | A named model and quantized checkpoint are exercised by scheduled full-model hardware CI. |
 | **Validated** | A named model and checkpoint have end-to-end or quality-validation evidence, but the check is not necessarily scheduled in CI. |
 | **Integrated** | The config, loader, quantized layers, or model wiring exists, but this guide has no completed end-to-end validation for the named model and checkpoint. |
 | **Not validated** | There is no current support recommendation for this model/component combination. |
 | **Unsupported** | The combination is intentionally rejected or known not to work. |
-A method being available on a device does not make every model on that device
+### Diffusion Model (Qwen-Image, Wan2.2)
 supported. Likewise, successfully detecting a checkpoint's
 `quantization_config` establishes loader compatibility, not output quality.
-## Hardware × Quantization Method
+These models run a diffusion transformer as the primary inference module. The
 default quantization target is the transformer; tokenizer, scheduler, text
 encoder, and VAE stay on the base checkpoint unless a method guide says
 otherwise.
-This matrix uses concrete inference paths as columns. Producer toolkits are not
+| Method | Guide | Mode | Example models | Status |
-separate methods: NVIDIA ModelOpt produces the ModelOpt FP8/NVFP4 formats,
+|--------|-------|------|----------------|--------|
-Intel AutoRound produces AutoRound checkpoints, and Ascend msModelSlim produces
+| FP8 W8A8 | [FP8](fp8.md) | Online W8A8 or checkpoint FP8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
-checkpoints consumed by the native MXFP or compatible Ascend paths.
+| Int8 W8A8 | [Int8](int8.md) | Online or serialized W8A8 | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
 | BitsAndBytes W4 | [BitsAndBytes](bitsandbytes.md) | Online W4 weight-only | Z-Image-Turbo; Qwen-Image/Wan2.2 not validated | Validated for Z-Image-Turbo |
 | ModelOpt | [ModelOpt](modelopt.md) | Pre-quantized FP8 checkpoints | Qwen-Image, Z-Image, FLUX.2, HunyuanImage-3.0 | Validated for ModelOpt FP8 diffusion checkpoints |
 | MXFP8 W8A8 | [MXFP8](mxfp8.md) | Online W8A8 or offline pre-quantized | Wan2.2-T2V-A14B, I2V-A14B, TI2V-5B | Ascend NPU only; validated for Wan2.2 |
 | MXFP4 W4A4 | [MXFP4](mxfp4.md) | `mxfp4`: online single-scale only; `mxfp4_dualscale`: online or offline dual-scale (offline recommended) | Wan2.2-T2V-A14B, I2V-A14B | Ascend NPU only; validated for Wan2.2 A14B cascade models; TI2V-5B not supported; offline `mxfp4_dualscale` uses calibrated `mul_scale` for best accuracy |
 | AutoRound | [AutoRound](autoround.md) | Pre-quantized W4A16 checkpoints | FLUX.1-dev; Qwen-Image/Wan2.2 not validated | Checkpoint-driven |
 | msModelSlim | [msModelSlim](msmodelslim.md) | Pre-quantized Ascend checkpoints | Wan2.2 recipe; HunyuanImage-3.0 inference target | Ascend/NPU path |
-| Hardware | [Online FP8](fp8.md) | [Int8](int8.md) | [BitsAndBytes W4](bitsandbytes.md) | [ModelOpt FP8](modelopt.md) | [ModelOpt NVFP4](modelopt.md#supported-modelopt-checkpoint-formats) | [AutoRound W4A16](autoround.md) | [OCP MXFP8](mxfp8.md) | [OCP MXFP4](mxfp4.md) |
+### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
 |----------|-----------------------|------------------|------------------------------------------|--------------------------------|-------------------------------------------------------------------------|--------------------------------------|--------------------------|--------------------------|
 | NVIDIA Blackwell (SM 100+) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
 | NVIDIA Hopper (SM 90) | ✅ | ✅ | ✅ | ✅ | ✅ (Marlin fallback) | ✅ | ❌ | ❌ |
 | NVIDIA Ada (SM 89) | ✅ | ✅ | ✅ | ✅ | ⭕ | ✅ | ❌ | ❌ |
 | NVIDIA Ampere (SM 80+) | ✅ | ✅ | ✅ | ⭕ | ⭕ | ✅ | ❌ | ❌ |
 | AMD ROCm (gfx950 for MXFP4) | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ⭕ | ❌ | ✅ (online only) |
 | Intel XPU | ⭕ | ⭕ | ❌ | ⭕ | ⭕ | ✅ | ✅ | ❌ |
 | Ascend NPU | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ |
-Legend: `✅` backend available, `❌` rejected or no implementation, `⭕` not
+These models combine an AR language model with audio, vision, talker, or TTS
-verified in this guide. This matrix does not establish model-level support; use
+stages. Quantization is scoped to the AR language-model stage when the
-the model-and-stage tables below for that.
+checkpoint contains a supported `quantization_config`; the non-AR stages stay
 in BF16 unless the model guide explicitly adds support.
-The Int8 column covers online and serialized Int8 paths. OCP MXFP8 is online on
+| Method | Guide | Scope | Example models | Status |
-Intel XPU and online/offline on Ascend; Intel's separate AutoRound MXFP8
+|--------|-------|-------|----------------|--------|
-checkpoint path is described in the [MXFP8 guide](mxfp8.md). OCP MXFP4 on AMD
+| ModelOpt | [ModelOpt](modelopt.md) | Thinker or language-model checkpoint config | Qwen3-Omni thinker | ModelOpt checkpoint path |
-means online single-scale `mxfp4` on gfx950, while MXFP4 DualScale is
+| Int8 | [Int8](int8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
-Ascend-only. NVIDIA's NVFP4 and OCP MXFP4 have different checkpoint metadata,
+| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
-scale layouts, and kernel contracts. Mixed ModelOpt FP8/NVFP4 checkpoints also
+| MXFP8 | [MXFP8](mxfp8.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
-require a model-specific mixed-precision policy and validation. FP8 on Ampere
+| MXFP4 | [MXFP4](mxfp4.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
-may use a weight-only path where available.
+| AutoRound | [AutoRound](autoround.md) | Thinker or language-model checkpoint config | Qwen2.5-Omni, Qwen3-Omni | Supported through AutoRound checkpoints |
 | msModelSlim | [msModelSlim](msmodelslim.md) | Not currently validated for omni/TTS stages | Qwen3-Omni, Qwen3-TTS | Not validated |
-## Current Model and Stage Support
+### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
-The current offline checkpoint guidance is transformer-first. A listed
+These models split generation across multiple stages. Quantization must be
-checkpoint quantizes the stage named in the **Scope** column; it does not imply
+attached to the intended stage rather than applied globally.
 that the model's encoders, VAE, decoder, or other stages are also quantized.
-### Vendor and Runtime Ecosystems
+| Method | Guide | Scope | Example models | Status |
-
+|--------|-------|-------|----------------|--------|
-The producer and runtime backend are related, but they are not the same support
+| FP8 | [FP8](fp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
-claim. A producer can emit several checkpoint formats, and each format still
+| Int8 | [Int8](int8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Requires model-specific validation |
-needs a compatible vLLM-Omni loader, model-stage wiring, and validation. A
+| BitsAndBytes | [BitsAndBytes](bitsandbytes.md) | Stage-specific transformer or DiT module | BAGEL, GLM-Image | Not validated |
-runtime may also provide online quantization without an offline producer path.
+| ModelOpt | [ModelOpt](modelopt.md) | Checkpoint-defined diffusion stage | BAGEL, GLM-Image | Requires model-specific validation |
-
+| MXFP8 | [MXFP8](mxfp8.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
-| Ecosystem | Current paths | Current vLLM-Omni scope | Model evidence |
+| MXFP4 | [MXFP4](mxfp4.md) | Stage-specific DiT or transformer module | BAGEL, GLM-Image | Not validated |
-|-----------|---------------|-------------------------|----------------|
+| AutoRound | [AutoRound](autoround.md) | Checkpoint-defined stage | BAGEL, GLM-Image | No validated checkpoint listed |
-| NVIDIA | ModelOpt FP8, NVFP4, and mixed FP8/NVFP4 | Diffusion transformer or Qwen-Omni thinker | Qwen-Image and other named diffusion checkpoints are validated; Qwen3-Omni NVFP4 has H100 CI for the Marlin fallback |
+| msModelSlim | [msModelSlim](msmodelslim.md) | Ascend-generated stage weights | GLM-Image | Requires model-specific adaptation |
 | Intel | AutoRound W4A16 and AutoRound MXFP8 | Diffusion/world-model transformer or Qwen-Omni thinker | Wan A14B and GLM-Image W4A16 are CI-backed; Qwen-Image, Qwen-Omni, and Cosmos3-Super have validation outside scheduled CI |
 | AMD ROCm | Online OCP MXFP4 through AITER on gfx950; no pre-quantized MXFP4 or AMD Quark workflow is integrated | Diffusion transformer linear layers | Integrated backend with dispatch and weight-allocation unit coverage; no named-model quality validation or scheduled gfx950 full-model CI |
 | Ascend | msModelSlim output through native MXFP8/MXFP4 merge workflows or a compatible generic Ascend format | Wan diffusion transformers | Native Wan MXFP8/MXFP4 paths are validated outside scheduled full-model NPU CI; other model families remain integrated or unvalidated |
 ### Pre-Quantized Model Matrix
 | Model family | Method | Scope | Level | Notes |
 |--------------|--------|-------|-------|-------|
 | Qwen-Image | [ModelOpt](modelopt.md) FP8 or mixed FP8/NVFP4 | Diffusion transformer | Validated | Named Qwen-Image 2512 checkpoints and recipes are available. Text encoder and VAE stay BF16. |
 | Qwen-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Uses `INC4AI/Qwen-Image-AutoRound-W4A16`; text encoder and VAE stay BF16. |
 | Wan2.2 I2V/T2V A14B | [AutoRound](autoround.md) W4A16 | Both diffusion transformers | CI-backed | The high-noise and low-noise DiTs are quantized; text encoder and VAE stay BF16. |
 | Wan2.2 TI2V 5B | [AutoRound](autoround.md) W4A16 | Diffusion transformer | Validated | Named checkpoint available; not covered by the scheduled Wan AutoRound job. |
 | Wan2.2 | [MXFP8](mxfp8.md) | One or both diffusion transformers | Validated | Native Ascend and AutoRound checkpoint formats are integrated; no scheduled full-model offline NPU job. |
 | Wan2.2 I2V/T2V A14B | [MXFP4 DualScale](mxfp4.md) | Both diffusion transformers | Validated | Offline calibrated checkpoints are recommended; no scheduled full-model offline NPU job. |
 | Qwen2.5-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Encoders and output/audio stages stay BF16. |
 | Qwen3-Omni | [ModelOpt](modelopt.md) FP8 | Thinker language model | Validated | Tested outside scheduled CI. |
 | Qwen3-Omni | [ModelOpt](modelopt.md) NVFP4 | Thinker language model | CI-backed | H100 CI exercises the Marlin fallback; the native Blackwell FP4 path has separate recipe validation. |
 | Qwen3-Omni | [AutoRound](autoround.md) W4A16 | Thinker language model | Validated | Audio encoder, vision encoder, talker, and Code2Wav stay BF16. |
 | GLM-Image | [AutoRound](autoround.md) W4A16 | Diffusion transformer stage | CI-backed | Text-to-image and image-to-image full-model smoke coverage. |
 | Cosmos3-Super | [AutoRound](autoround.md) W4A16 | World-model transformer | Validated | Manually validated serving path; VAE and other components stay BF16. |
 | Cosmos3-Nano | [AutoRound](autoround.md) W4A16 | World-model transformer | Integrated | Named checkpoint is listed, but no model-specific end-to-end test is maintained in-tree. |
 | MiniMax H3 | None | None | Not validated | No offline vendor checkpoint path is documented; online FP8 targets the DiT only. |
 The method pages contain the complete checkpoint lists, including FLUX,
 Z-Image, and HunyuanImage-3.0 variants.
 ### Online Quantization
 | Model family | Methods | Scope | Level |
 |--------------|---------|-------|-------|
 | Qwen-Image | FP8, Int8 | Diffusion transformer | Validated |
 | Z-Image | Int8, BitsAndBytes W4 | Diffusion transformer | Validated |
 | Wan2.2 | MXFP8 | One or both diffusion transformers | Validated on the documented XPU/NPU paths |
 | Wan2.2 A14B | MXFP4 | Both diffusion transformers | Validated on Ascend; the ROCm gfx950 online backend is integrated but not model-validated |
 | MiniMax H3 | FP8 | DiT and optional reference DiT | Validated |
 | Qwen3-Omni | Generic online methods, such as FP8 | Thinker language model only | Integrated; not model-validated as an online recommendation |
 See [Online Quantization](online.md) for configuration and hardware details.
 ### Unlisted and Long-Tail Models
 For models not listed above, generic quantized linear layers or compatible
 checkpoint metadata are only an integration starting point. The default support
 target is the model's primary transformer, DiT, or AR language-model stage.
 Text/audio/vision encoders, VAEs, vocoders, decoders, and output stages remain
 BF16 until a model-specific integration supplies quant-aware layers, checkpoint
 mapping, end-to-end output validation, and backend coverage.
 !!! note
    "Online quantization" means vLLM-Omni computes the quantization data while
@@ -143,8 +92,8 @@ mapping, end-to-end output validation, and backend coverage.
 ### Diffusion Model (Qwen-Image, Wan2.2)
-The default target is the diffusion transformer. `build_quant_config()` can
+The default target is the diffusion transformer. Component routing is available
-construct a component router:
+through `build_quant_config()`:
 ```python
 from vllm_omni.quantization import build_quant_config
@@ -155,19 +104,11 @@ config = build_quant_config({
 })
 ```
 !!! warning "Routing syntax is not a support claim"
    A component dictionary only selects a quantization config by module prefix.
    It does not make an encoder or VAE quantizable. The model must construct
    that component from quantization-aware layers, and an offline checkpoint
    must contain matching packed weights and scales. Current user guidance
    keeps diffusion text encoders and VAEs in BF16.
 | Component | Default quantized? | Notes |
 |-----------|--------------------|-------|
-| Diffusion transformer | Yes, for listed model/method pairs | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
+| Diffusion transformer | Yes | Primary target for FP8, Int8, BitsAndBytes, ModelOpt, MXFP8, MXFP4, AutoRound, and msModelSlim |
 | Second diffusion transformer | Yes, for listed Wan2.2 A14B methods | `transformer` and `transformer_2` read their own offline checkpoint metadata; arbitrary mixed methods are not currently validated |
 | Text encoder | No | Keep BF16 unless a method-specific guide documents support |
-| VAE | No | Keep BF16 |
+| VAE | No | Keep BF16; storage-only paths are method-specific |
 | Scheduler/tokenizer | No | Loaded from the base model repository |
 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -180,18 +121,6 @@ config = build_quant_config({
 | Talker or TTS stage | No | BF16 unless model-specific support is documented |
 | Code2Wav | No | BF16 |
 An explicit `ComponentQuantizationConfig` can be used by model developers to
 route different configs to `audio_tower`, `visual`, and `language_model`, but
 the supported Qwen-Omni checkpoints currently quantize only the thinker
 language model.
 ### World Model (Cosmos3)
 | Component | Default quantized? | Notes |
 |-----------|--------------------|-------|
 | World-model transformer | AutoRound checkpoints only | Cosmos3-Super is validated; Cosmos3-Nano is integrated but not end-to-end tested in-tree |
 | VAE and guardrail components | No | BF16 |
 ### Multi-Stage Diffusion Model (BAGEL, GLM-Image)
 | Component | Default quantized? | Notes |
@@ -212,7 +141,6 @@ build_quant_config("fp8")
 build_quant_config({"method": "fp8", "activation_scheme": "static"})
 build_quant_config("bitsandbytes")
 build_quant_config("auto-round", bits=4, group_size=128)
 # Component routing syntax; model-specific support is still required.
 build_quant_config({"transformer": {"method": "fp8"}, "vae": None})
 build_quant_config(None)
 ```
@@ -77,6 +77,7 @@ plugins:
        user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
        user_guide/diffusion/cpu_offload_diffusion.md: user_guide/diffusion/cpu_offload.md
        user_guide/quantization/index.md: user_guide/quantization/overview.md
        configuration/pd_disaggregation.md: features/pd_disaggregation.md
        design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
        design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
  - glightbox
@@ -95,7 +95,6 @@ def test_formatter_preserves_single_video_audio_actions_and_metadata(
        "resolution": 512,
        "width": None,
        "height": None,
        "num_inference_steps": 1,
    }
@@ -42,10 +42,6 @@ _EXPECTED_FAMILIES = [
    defs.AUDIO_UNDERRUN_S,
    defs.AUDIO_CONTINUITY_OK_METRIC,
    defs.AUDIO_SKIPPED_REQUESTS_METRIC,
    defs.DIFFUSION_EXEC_S,
    defs.DIFFUSION_EXEC_PER_STEP_S,
    defs.DIFFUSION_PREPROCESS_S,
    defs.DIFFUSION_POSTPROCESS_S,
 ]
@@ -59,10 +55,6 @@ class TestRegistration:
        mod.observe_audio_underrun("s", "r", 0.01)
        mod.inc_audio_continuity_ok("s", "r", 100)
        mod.inc_audio_skipped("s", "r", "malformed_codec")
        mod.observe_diffusion_exec("s", "r", 0.5)
        mod.observe_diffusion_exec_per_step("s", "r", 0.01)
        mod.observe_diffusion_preprocess("s", "r", 0.01)
        mod.observe_diffusion_postprocess("s", "r", 0.2)
        out = generate_latest(REGISTRY).decode()
        for name in _EXPECTED_FAMILIES:
@@ -164,50 +156,6 @@ class TestAudio:
        assert unknown == 1.0
 # ---------------------------------------------------------------------------
 # Diffusion observe API
 # ---------------------------------------------------------------------------
 class TestDiffusion:
    def test_diffusion_exec_observed(self, mod: OmniModalityMetrics) -> None:
        stage, replica = "dit_exec", "0"
        mod.observe_diffusion_exec(stage, replica, 1.25)
        out = generate_latest(REGISTRY).decode()
        prefix = f'{defs.DIFFUSION_EXEC_S}_sum{{model_name="{_MODEL}",replica="{replica}",stage="{stage}"}}'
        assert _sample_value(out, prefix) == 1.25
    def test_diffusion_preprocess_observed(self, mod: OmniModalityMetrics) -> None:
        stage, replica = "dit_pre", "0"
        mod.observe_diffusion_preprocess(stage, replica, 0.03)
        out = generate_latest(REGISTRY).decode()
        prefix = f'{defs.DIFFUSION_PREPROCESS_S}_sum{{model_name="{_MODEL}",replica="{replica}",stage="{stage}"}}'
        assert _sample_value(out, prefix) == pytest.approx(0.03)
    def test_diffusion_postprocess_observed(self, mod: OmniModalityMetrics) -> None:
        stage, replica = "dit_post", "0"
        mod.observe_diffusion_postprocess(stage, replica, 0.8)
        out = generate_latest(REGISTRY).decode()
        prefix = f'{defs.DIFFUSION_POSTPROCESS_S}_sum{{model_name="{_MODEL}",replica="{replica}",stage="{stage}"}}'
        assert _sample_value(out, prefix) == 0.8
    def test_diffusion_exec_per_step_observed(self, mod: OmniModalityMetrics) -> None:
        stage, replica = "dit_per_step", "0"
        mod.observe_diffusion_exec_per_step(stage, replica, 0.078)
        out = generate_latest(REGISTRY).decode()
        prefix = f'{defs.DIFFUSION_EXEC_PER_STEP_S}_sum{{model_name="{_MODEL}",replica="{replica}",stage="{stage}"}}'
        assert _sample_value(out, prefix) == pytest.approx(0.078)
    def test_log_stats_false_skips(self) -> None:
        silent = OmniModalityMetrics(model_name="silent-model", log_stats=False)
        silent.observe_diffusion_exec("s", "r", 1.0)
        silent.observe_diffusion_exec_per_step("s", "r", 0.02)
        silent.observe_diffusion_preprocess("s", "r", 0.01)
        silent.observe_diffusion_postprocess("s", "r", 0.5)
        out = generate_latest(REGISTRY).decode()
        assert 'model_name="silent-model"' not in out
 # ---------------------------------------------------------------------------
 # Stub-driven routing tests for the finalize / streaming helpers.
 # ---------------------------------------------------------------------------
@@ -235,18 +183,6 @@ class _StubModMetrics:
    def inc_audio_skipped(self, s, r, reason):
        self.calls.append(("inc_audio_skipped", s, r, reason))
    def observe_diffusion_exec(self, s, r, v):
        self.calls.append(("observe_diffusion_exec", s, r, v))
    def observe_diffusion_exec_per_step(self, s, r, v):
        self.calls.append(("observe_diffusion_exec_per_step", s, r, v))
    def observe_diffusion_preprocess(self, s, r, v):
        self.calls.append(("observe_diffusion_preprocess", s, r, v))
    def observe_diffusion_postprocess(self, s, r, v):
        self.calls.append(("observe_diffusion_postprocess", s, r, v))
 class _Bag:
    def __init__(self, **kw):
@@ -303,88 +239,8 @@ class TestObserveModalityAtFinalize:
        )
        assert ("observe_audio_duration", "1", "0", 1.0) in stub.calls
-    def test_diffusion_path_full(self):
+    def test_non_audio_output_type_skipped(self):
-        stub = _StubModMetrics()
+        # Image / video / text output types are out of scope for this module.
        stage_metrics = _Bag(
            stage_gen_time_ms=1000.0,
            diffusion_metrics={
                "preprocess_time_s": 0.05,
                "diffusion_engine_exec_time_s": 1.2,
                "postprocess_time_s": 0.3,
                "image_num": 1,
                "resolution": 1024,
                "num_inference_steps": 50,
            },
        )
        observe_modality_at_finalize(
            stub,
            output_type="image",
            stage_id=2,
            replica_id=0,
            stage_metrics=stage_metrics,
            engine_outputs=_Bag(),
        )
        assert ("observe_diffusion_preprocess", "2", "0", 0.05) in stub.calls
        assert ("observe_diffusion_exec", "2", "0", 1.2) in stub.calls
        assert ("observe_diffusion_postprocess", "2", "0", 0.3) in stub.calls
        assert ("observe_diffusion_exec_per_step", "2", "0", pytest.approx(0.024)) in stub.calls
    def test_diffusion_per_step_skipped_without_num_inference_steps(self):
        stub = _StubModMetrics()
        stage_metrics = _Bag(
            stage_gen_time_ms=1000.0,
            diffusion_metrics={
                "diffusion_engine_exec_time_s": 1.2,
                "preprocess_time_s": 0.05,
                "postprocess_time_s": 0.3,
            },
        )
        observe_modality_at_finalize(
            stub,
            output_type="image",
            stage_id=2,
            replica_id=0,
            stage_metrics=stage_metrics,
            engine_outputs=_Bag(),
        )
        assert ("observe_diffusion_exec", "2", "0", 1.2) in stub.calls
        assert not any(c[0] == "observe_diffusion_exec_per_step" for c in stub.calls)
    def test_diffusion_metrics_none_skipped(self):
        stub = _StubModMetrics()
        observe_modality_at_finalize(
            stub,
            output_type="image",
            stage_id=2,
            replica_id=0,
            stage_metrics=_Bag(stage_gen_time_ms=500.0, diffusion_metrics=None),
            engine_outputs=_Bag(),
        )
        assert not any(c[0].startswith("observe_diffusion") for c in stub.calls)
    def test_diffusion_zero_values_observed(self):
        stub = _StubModMetrics()
        stage_metrics = _Bag(
            stage_gen_time_ms=500.0,
            diffusion_metrics={
                "preprocess_time_s": 0.0,
                "diffusion_engine_exec_time_s": 0.0,
                "postprocess_time_s": 0.0,
            },
        )
        observe_modality_at_finalize(
            stub,
            output_type="image",
            stage_id=2,
            replica_id=0,
            stage_metrics=stage_metrics,
            engine_outputs=_Bag(),
        )
        assert ("observe_diffusion_preprocess", "2", "0", 0.0) in stub.calls
        assert ("observe_diffusion_exec", "2", "0", 0.0) in stub.calls
        assert ("observe_diffusion_postprocess", "2", "0", 0.0) in stub.calls
    def test_non_audio_output_type_skips_audio_but_still_checks_diffusion(self):
        stub = _StubModMetrics()
        for output_type in ("text", "image", "video"):
            observe_modality_at_finalize(
@@ -395,7 +251,7 @@ class TestObserveModalityAtFinalize:
                stage_metrics=_Bag(stage_gen_time_ms=100.0),
                engine_outputs=_Bag(),
            )
-        assert not any(c[0].startswith("observe_audio") or c[0].startswith("inc_audio") for c in stub.calls)
+        assert stub.calls == []
    def test_replica_id_none_skipped(self):
        stub = _StubModMetrics()
@@ -138,7 +138,6 @@ def format_diffusion_outputs(
        ),
        "width": request.sampling_params.width,
        "height": request.sampling_params.height,
        "num_inference_steps": request.sampling_params.num_inference_steps,
    }
    # Only the primary payload determines the response type. Some image models
@@ -568,7 +568,6 @@ class OmniBase(PDDisaggregationMixin):
            msg_id = id(result)
            consumed = self._consumed_metric_message_ids(req_id)
            if msg_id not in consumed:
                metrics.accumulate_diffusion_metrics(stage_meta.stage_type, req_id, engine_outputs)
                metrics.on_stage_metrics(stage_id, req_id, _m, output_type)
                consumed.add(msg_id)
@@ -134,15 +134,6 @@ AUDIO_CONTINUITY_OK_METRIC = METRIC_PREFIX + AUDIO_CONTINUITY_OK
 AUDIO_SKIPPED_REQUESTS_METRIC = METRIC_PREFIX + AUDIO_SKIPPED_REQUESTS
 # ============================================================================
 # Diffusion family (per-stage + per-replica diffusion timing breakdowns)
 # ============================================================================
 DIFFUSION_EXEC_S = METRIC_PREFIX + "diffusion_exec_s"
 DIFFUSION_EXEC_PER_STEP_S = METRIC_PREFIX + "diffusion_exec_per_step_s"
 DIFFUSION_PREPROCESS_S = METRIC_PREFIX + "diffusion_preprocess_s"
 DIFFUSION_POSTPROCESS_S = METRIC_PREFIX + "diffusion_postprocess_s"
 # ============================================================================
 # Cross-stage Transfer family (per-physical-hop TX/RX/in-flight timings)
 # ============================================================================
@@ -78,35 +78,6 @@ _audio_skipped_family = Counter(
 )
 # ----------------------------------------------------------------------------
 # Diffusion family
 # ----------------------------------------------------------------------------
 _diffusion_exec_family = Histogram(
    defs.DIFFUSION_EXEC_S,
    "DiT forward pass execution time per request in seconds.",
    labelnames=_stage_labels,
    buckets=defs.SECONDS_BUCKETS,
 )
 _diffusion_exec_per_step_family = Histogram(
    defs.DIFFUSION_EXEC_PER_STEP_S,
    "DiT forward pass execution time per denoising step in seconds.",
    labelnames=_stage_labels,
    buckets=defs.SECONDS_FAST_BUCKETS,
 )
 _diffusion_preprocess_family = Histogram(
    defs.DIFFUSION_PREPROCESS_S,
    "Diffusion input preprocessing time per request in seconds.",
    labelnames=_stage_labels,
    buckets=defs.SECONDS_FAST_BUCKETS,
 )
 _diffusion_postprocess_family = Histogram(
    defs.DIFFUSION_POSTPROCESS_S,
    "Diffusion output postprocessing (VAE decode) time per request in seconds.",
    labelnames=_stage_labels,
    buckets=defs.SECONDS_FAST_BUCKETS,
 )
 class OmniModalityMetrics:
    """Per-modality observe API. Stage/replica are passed at observe time
    because a single OmniModalityMetrics instance per pipeline serves all
@@ -168,44 +139,6 @@ class OmniModalityMetrics:
            reason=reason or "unknown",
        ).inc()
    # ---- Diffusion --------------------------------------------------------
    def observe_diffusion_exec(self, stage: str, replica: str, seconds: float) -> None:
        if not self._log_stats:
            return
        _diffusion_exec_family.labels(
            model_name=self._model_name,
            stage=stage,
            replica=replica,
        ).observe(seconds)
    def observe_diffusion_exec_per_step(self, stage: str, replica: str, seconds: float) -> None:
        if not self._log_stats:
            return
        _diffusion_exec_per_step_family.labels(
            model_name=self._model_name,
            stage=stage,
            replica=replica,
        ).observe(seconds)
    def observe_diffusion_preprocess(self, stage: str, replica: str, seconds: float) -> None:
        if not self._log_stats:
            return
        _diffusion_preprocess_family.labels(
            model_name=self._model_name,
            stage=stage,
            replica=replica,
        ).observe(seconds)
    def observe_diffusion_postprocess(self, stage: str, replica: str, seconds: float) -> None:
        if not self._log_stats:
            return
        _diffusion_postprocess_family.labels(
            model_name=self._model_name,
            stage=stage,
            replica=replica,
        ).observe(seconds)
 def observe_modality_at_finalize(
    mod_metrics: OmniModalityMetrics,
@@ -228,14 +161,19 @@ def observe_modality_at_finalize(
    """
    if replica_id is None or stage_metrics is None or output_type is None:
        return
    if output_type != "audio":
        return
    stage_label = str(stage_id)
    replica_label = str(replica_id)
    gen_time_s = float(getattr(stage_metrics, "stage_gen_time_ms", 0.0)) / 1000.0
    mm_out = extract_mm_output(engine_outputs)
    if output_type == "audio":
    sample_rate = defs.resolve_audio_sample_rate(mm_out)
    # `stage_metrics.audio_generated_frames` is the legacy per-chunk
    # accumulator field on StageRequestStats. No production path currently
    # fills it, so the fallback below is the live source — but we leave the
    # field lookup in place in case the accumulator gets re-wired upstream.
    n_frames = int(getattr(stage_metrics, "audio_generated_frames", 0) or 0)
    if n_frames == 0:
        n_frames = count_audio_frames(mm_out)
@@ -249,28 +187,17 @@ def observe_modality_at_finalize(
            defs.compute_audio_rtf(gen_time_s, duration_s),
        )
    else:
        # Request completed (finish_reason ∈ {stop, length} — error paths
        # don't reach finalize) but no audio samples were produced. Covers
        # silent `return None` skips in the talker→code2wav stage
        # processors and the `parsed.append((0,0))` malformed-length path
        # in qwen3-tts code2wav. raise-paths surface via the upstream
        # vllm:request_success_total{finished_reason="error"} channel and
        # never reach this branch.
        mod_metrics.inc_audio_skipped(stage_label, replica_label, "no_audio_data")
-
+    # audio_underrun / continuity are emitted from the streaming path in
-    dm = getattr(stage_metrics, "diffusion_metrics", None)
+    # observe_audio_streaming_finalize; finalize is too late for the
-    if dm:
+    # per-chunk timeline they need.
        _key_map = {
            "diffusion_engine_exec_time_s": mod_metrics.observe_diffusion_exec,
            "preprocess_time_s": mod_metrics.observe_diffusion_preprocess,
            "postprocess_time_s": mod_metrics.observe_diffusion_postprocess,
        }
        for key, observe_fn in _key_map.items():
            val = dm.get(key)
            if val is not None:
                observe_fn(stage_label, replica_label, float(val))
        exec_time = dm.get("diffusion_engine_exec_time_s")
        num_steps = dm.get("num_inference_steps")
        if exec_time is not None and num_steps and num_steps > 0:
            mod_metrics.observe_diffusion_exec_per_step(
                stage_label,
                replica_label,
                float(exec_time) / int(num_steps),
            )
 def observe_audio_first_packet(
@@ -44,7 +44,7 @@ class StageRequestStats:
    final_output_type: str | None = None
    request_id: str | None = None
    postprocess_time_ms: float = 0.0
-    diffusion_metrics: dict[str, float] = None
+    diffusion_metrics: dict[str, int] = None
    audio_generated_frames: int = 0
    audio_sample_rate: int = 0
    audio_duration_s: float = 0.0
@@ -579,7 +579,7 @@ class OrchestratorAggregator:
        if final_output_type is not None:
            stats.final_output_type = final_output_type
        stats.diffusion_metrics = (
-            {k: float(v) for k, v in self.diffusion_metrics.pop(req_id, {}).items()}
+            {k: int(v) for k, v in self.diffusion_metrics.pop(req_id, {}).items()}
            if req_id in self.diffusion_metrics
            else None
        )
@@ -644,19 +644,8 @@ class OrchestratorAggregator:
        if isinstance(diffusion_metrics, list):
            diffusion_metrics = diffusion_metrics[0]
        if diffusion_metrics:
            _MS_TO_S = {
                "diffusion_engine_exec_time_ms": "diffusion_engine_exec_time_s",
                "preprocess_time_ms": "preprocess_time_s",
                "postprocess_time_ms": "postprocess_time_s",
                "diffusion_engine_total_time_ms": "diffusion_engine_total_time_s",
            }
            for key, value in diffusion_metrics.items():
-                if value is None:
+                self.diffusion_metrics[req_id][key] += value
                    continue
                if key in _MS_TO_S:
                    self.diffusion_metrics[req_id][_MS_TO_S[key]] = float(value) / 1000.0
                else:
                    self.diffusion_metrics[req_id][key] = value
    def on_forward(
        self,
Author	SHA1	Message	Date
Hongsheng Liu	fab29e2873	[Docs] Redirect the former PD guide URL Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-08-13 12:48:43 +08:00
Hongsheng Liu	1ade14d147	[Docs] Document current PD design Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-08-13 12:44:42 +08:00
Hongsheng Liu	b6313918d6	Merge branch 'main' into codex-pd-navigation	2026-08-13 12:23:56 +08:00
Hongsheng Liu	f27e70dce7	[Docs] Move PD guide into Features Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-08-13 06:54:34 +08:00
Hongsheng Liu	c46abfef59	[Docs] Mark PD disaggregation as experimental Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-08-13 06:50:41 +08:00