docs: define module ownership contracts

Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>
[Config] Read engine args from VllmOmniConfig (#5678 )
2026-08-07 11:45:59 +08:00 · 2026-08-07 11:17:33 +08:00 · 2026-08-07 10:25:04 +08:00 · 2026-08-07 00:12:07 +08:00 · 2026-08-06 23:21:43 +08:00 · 2026-08-06 21:20:04 +08:00
250 changed files with 23829 additions and 2752 deletions
@@ -460,6 +460,21 @@ steps:
            exit $$EXIT2
        mirror_hardwares: h100_2

+      - label: ":full_moon: Diffusion X2V · Perf Test · MiniMax-H3"
+        key: nightly-diffusion-x2v-performance-minimax-h3
+        timeout_in_minutes: 180
+        commands:
+          - export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
+          - export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
+          - |
+            set +e
+            pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json
+            EXIT=$$?
+            buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
+            buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
+            exit $$EXIT
+        mirror_hardwares: h100_4
+
  - group: ":card_index_dividers: Diffusion Test"
    key: nightly-diffusion-test-group
    depends_on: upload-nightly-pipeline
@@ -309,7 +309,7 @@ For Omni or custom models, create:

 Required updates:
 1. `docs/user_guide/diffusion/parallelism/overview.md` — parallelism support overview/table
-2. `docs/user_guide/diffusion/cpu_offload_diffusion.md` — if CPU offload supported (add to supported models table)
+2. `docs/user_guide/diffusion/cpu_offload.md` — if CPU offload supported (add to supported models table)
 3. `docs/user_guide/diffusion/cache_acceleration/teacache.md` — if TeaCache supported
 4. `docs/user_guide/diffusion/cache_acceleration/cache_dit.md` — if Cache-DiT supported
 5. Offline example docs under `examples/offline_inference/<name>/` (`README.md` or category-specific `.md`)
@@ -16,15 +16,15 @@
 /.claude/ @hsliuustc0106

 # Tests
-/tests/ @yenuo26
+/tests/ @yenuo26 @NickCao

 # CI
-/.buildkite/ @yenuo26 @congw729
+/.buildkite/ @yenuo26 @congw729 @NickCao

 # Engine and model_executor
 /vllm_omni/core/ @tzhouam @yinpeiqi @fake0fan @Sy0307
 /vllm_omni/engine/ @tzhouam @yinpeiqi @fake0fan @Sy0307
-/vllm_omni/entrypoints/ @tzhouam @yinpeiqi @fake0fan @alex-jw-brooks
+/vllm_omni/entrypoints/ @tzhouam @yinpeiqi @fake0fan @alex-jw-brooks @NickCao
 /vllm_omni/model_executor/ @tzhouam @gcanlin
 /vllm_omni/worker/ @tzhouam @gcanlin

@@ -37,7 +37,7 @@

 # Omni and TTS support
 /vllm_omni/transformers_utils/ @tzhouam @yuanheng-zhao
-/vllm_omni/model_executor/models/ @linyueqian @ZeldaHuang @princepride @yuanheng-zhao @amy-why-3459 @Bounty-hunter @Sy0307 @gcanlin
+/vllm_omni/model_executor/models/ @linyueqian @ZeldaHuang @princepride @yuanheng-zhao @amy-why-3459 @Bounty-hunter @Sy0307 @gcanlin @NickCao

 # Benchmarks
 /benchmarks/accuracy/ @david6666666
@@ -1,5 +1,6 @@
 import asyncio
 import base64
+import json
 import mimetypes
 import os
 import time
@@ -13,6 +14,17 @@ from tqdm import tqdm
 DEFAULT_EDITS_BOT_TASK = "think"


+def _is_minimax_h3_mixed_reference(model: str) -> bool:
+    """MiniMax-H3 Ref2VA consumes reference videos as file paths.
+
+    Its reference preparer ffprobes / ffmpegs the reference file, so single
+    ``video_reference`` uploads are sent as a base64 data URL: the server
+    persists them to disk and keeps ``source_path``, handing the pipeline a
+    real file path instead of a decoded frame list.
+    """
+    return "MiniMax-H3" in (model or "")
+
+
@dataclass
 class RequestFuncInput:
    prompt: str
@@ -355,13 +367,21 @@ async def async_request_v1_videos(
        )
    elif input.video_paths and len(input.video_paths) > 0:
        video_path = input.video_paths[0]
-        image_file = open(video_path, "rb")
-        form.add_field(
-            "input_reference",
-            image_file,
-            filename=os.path.basename(video_path),
-            content_type=_guess_mime_type(video_path),
-        )
+        if _is_minimax_h3_mixed_reference(input.model):
+            with open(video_path, "rb") as reference_file:
+                video_b64 = base64.b64encode(reference_file.read()).decode("ascii")
+            form.add_field(
+                "video_reference",
+                json.dumps([{"video_url": f"data:video/mp4;base64,{video_b64}"}]),
+            )
+        else:
+            image_file = open(video_path, "rb")
+            form.add_field(
+                "input_reference",
+                image_file,
+                filename=os.path.basename(video_path),
+                content_type=_guess_mime_type(video_path),
+            )

    job_id = None
    job_status = None
@@ -858,7 +858,12 @@ class RandomDataset(BaseDataset):
            height = int(self.args.height or profile.get("height") or 720)
            num_frames = int(self.args.num_frames or profile.get("num_frames") or 16)
            fps = float(self.args.fps or profile.get("fps") or 8)
-            writer = cv2.VideoWriter(video_path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (width, height))
+            # cv2 typically writes MPEG-4 Part 2 (``mp4v``), which pipelines such
+            # as MiniMax-H3 Ref2VA reject because they require H.264/H.265
+            # reference videos. Write the synthetic clip first, then re-encode
+            # it to H.264 when ffmpeg is available; otherwise keep ``mp4v``.
+            raw_path = video_path + ".raw.mp4"
+            writer = cv2.VideoWriter(raw_path, cv2.VideoWriter_fourcc(*"mp4v"), fps, (width, height))
            if not writer.isOpened():
                raise RuntimeError("cv2.VideoWriter failed to open")
            for frame_idx in range(num_frames):
@@ -871,6 +876,10 @@ class RandomDataset(BaseDataset):
                frame[max(y - 36, 0) : min(y + 36, height), x : min(x + 96, width), :] = (64, 128, 220)
                writer.write(frame)
            writer.release()
+            if _reencode_video_to_h264(raw_path, video_path):
+                os.unlink(raw_path)
+            else:
+                os.replace(raw_path, video_path)
            return [video_path]
        except Exception as e:
            raise RuntimeError(
@@ -878,6 +887,40 @@ class RandomDataset(BaseDataset):
            ) from e


+def _reencode_video_to_h264(source: str, destination: str) -> bool:
+    """Re-encode a video to H.264/MP4 using ffmpeg when available.
+
+    Returns False when ffmpeg is missing or the re-encode fails so callers can
+    fall back to the original container/codec.
+    """
+    import shutil
+    import subprocess
+
+    if shutil.which("ffmpeg") is None:
+        return False
+    try:
+        result = subprocess.run(
+            [
+                "ffmpeg",
+                "-y",
+                "-loglevel",
+                "error",
+                "-i",
+                source,
+                "-c:v",
+                "libx264",
+                "-pix_fmt",
+                "yuv420p",
+                destination,
+            ],
+            capture_output=True,
+            text=True,
+        )
+        return result.returncode == 0 and os.path.exists(destination)
+    except Exception:
+        return False
+
+
 def _compute_expected_latency_ms_from_base(req: RequestFuncInput, args, base_time_ms: float | None) -> float | None:
    """Compute expected execution time (ms) based on a base per-step-per-frame unit time.

@@ -68,13 +68,12 @@ nav:
        - Sequence Parallel: user_guide/diffusion/parallelism/sequence_parallel.md
        - Tensor Parallel: user_guide/diffusion/parallelism/tensor_parallel.md
        - VAE Parallelism: user_guide/diffusion/parallelism/vae_parallelism.md
-      - CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md
+      - CPU Offloading: user_guide/diffusion/cpu_offload.md
      - LoRA: user_guide/diffusion/lora.md
      - Custom Pipeline: features/custom_pipeline.md
-      - Request-Level Batching: user_guide/diffusion/request_batching.md
-      - Step Execution: user_guide/diffusion/step_execution.md
+      - Execution Modes: user_guide/diffusion/execution_modes.md
    - Quantization:
-      - Overview: user_guide/quantization/overview.md
+      - Quantization: user_guide/quantization/overview.md
      - Online Quantization: user_guide/quantization/online.md
      - Quantized KV Cache: user_guide/quantization/quantized_kvcache.md
      - FP8 W8A8: user_guide/quantization/fp8.md
@@ -102,32 +101,65 @@ nav:
  - CI: contributing/ci
  - Design Documents:
    - design/index.md
-    - design/architecture_overview.md
+    - Architecture Documents:
+      - Architecture Overview: design/architecture_overview.md
    - Feature Design:
-      - design/feature/disaggregated_inference.md
-      - design/feature/ray_based_execution.md
-      - design/feature/omni_connectors/
-      - design/feature/prefix_caching.md
-      - design/feature/cfg_parallel.md
-      - design/feature/expert_parallel.md
-      - design/feature/sequence_parallel.md
-      - design/feature/tensor_parallel.md
-      - design/feature/vae_parallel.md
-      - design/feature/hsdp.md
-      - design/feature/cache_dit.md
-      - design/feature/teacache.md
-      - design/feature/skip_softmax.md
-      - design/feature/async_chunk.md
-      - design/feature/omni_async_output_materialization.md
-      - design/feature/vae_parallel.md
-      - design/feature/diffusion_step_execution.md
-      - design/feature/diffusion_request_level_batching.md
-      - design/feature/diffusion_continuous_batching.md
+      - Runtime and Stage Execution:
+        - Disaggregated Inference: design/feature/disaggregated_inference.md
+        - Async Chunk: design/feature/async_chunk.md
+        - Async Diffusion Output: design/feature/async_diffusion_output.md
+        - Async Omni Output Materialization: design/feature/omni_async_output_materialization.md
+        - Automatic Prefix Caching: design/feature/prefix_caching.md
+      - Communication:
+        - OmniConnector Implementations:
+          - Mooncake Store Connector: design/feature/omni_connectors/mooncake_store_connector.md
+          - Mooncake Transfer Engine Connector: design/feature/omni_connectors/mooncake_transfer_engine_connector.md
+          - Mori Transfer Engine Connector: design/feature/omni_connectors/mori_transfer_engine_connector.md
+          - Shared Memory Connector: design/feature/omni_connectors/shared_memory_connector.md
+          - Yuanrong Store Connector: design/feature/omni_connectors/yuanrong_connector.md
+          - Yuanrong Transfer Engine Connector: design/feature/omni_connectors/yuanrong_transfer_engine_connector.md
+      - Diffusion Acceleration:
+        - Parallelism:
+          - CFG-Parallel: design/feature/cfg_parallel.md
+          - Expert Parallel: design/feature/expert_parallel.md
+          - Hybrid Sharded Data Parallel (HSDP): design/feature/hsdp.md
+          - Pipeline Parallel: design/feature/pipeline_parallel.md
+          - Sequence Parallel: design/feature/sequence_parallel.md
+          - Tensor Parallel: design/feature/tensor_parallel.md
+          - VAE Patch Parallelism: design/feature/vae_parallel.md
+        - Attention Backends:
+          - Skip-Softmax: design/feature/skip_softmax.md
+        - Quantization: design/feature/quantization.md
+        - Cache-DiT: design/feature/cache_dit.md
+        - TeaCache: design/feature/teacache.md
+        - Diffusion Continuous Batching: design/feature/diffusion_continuous_batching.md
+        - Distributed Layerwise Offload: design/feature/distributed_layerwise_offload.md
+    - Infrastructure and Performance:
+      - Prometheus Metrics: design/metrics.md
+      - Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
    - Module Design:
-      - design/module/ar_module.md
-      - design/module/dit_module.md
-      - design/module/entrypoint_module.md
-      - design/module/async_omni_architecture.md
+      - Entrypoints: design/module/entrypoints.md
+      - vLLM-Omni Configuration: design/module/vllm_omni_config.md
+      - Input, Output, and Modality Contracts: design/module/input_output_modality_contracts.md
+      - Error Contracts: design/module/error_contracts.md
+      - Engine Orchestration: design/module/engine_orchestration.md
+      - Stage Runtime: design/module/stage_runtime.md
+      - OmniConnector: design/module/omni_connector.md
+      - Model Integration: design/module/model_integration.md
+      - Autoregressive Runtime: design/module/ar_runtime.md
+      - Diffusion:
+        - Overview: design/module/diffusion/index.md
+        - Runtime: design/module/diffusion/diffusion_runtime.md
+        - Model Integration: design/module/diffusion/diffusion_model_integration.md
+        - Continuous Batching: design/module/diffusion/continuous_batching.md
+        - Parallelism: design/module/diffusion/parallelism.md
+        - Offloader: design/module/diffusion/offloader.md
+      - Execution Platforms: design/module/execution_platforms.md
+      - Cache Management: design/module/cache_management.md
+      - Quantization: design/module/quantization.md
+      - Observability: design/module/observability.md
+      - Profiling: design/module/profiling.md
+      - Benchmarking: design/module/benchmarking.md
  - Docs Guide: contributing/DOCS_GUIDE.md
 - API Reference:
  - api/README.md
@@ -40,6 +40,7 @@ Committers have write access and merge rights. They typically have deep expertis
 - [@Isotr0py](https://github.com/Isotr0py): Diffusion and Quantization
 - [@linyueqian](https://github.com/linyueqian): TTS and Omni Support
 - [@lishunyang12](https://github.com/lishunyang12): Quantization and Configuration
+- [@NickCao](https://github.com/NickCao): CI, Tests, Entrypoint, and TTS Model Support
 - [@princepride](https://github.com/princepride): Diffusion and Omni Support
 - [@RuixiangMa](https://github.com/RuixiangMa): Diffusion models, parallel, cache, and docs
 - [@SamitHuang](https://github.com/SamitHuang): RL, Diffusion, and cache
@@ -387,8 +387,8 @@ that support request-level batching with `step_execution` disabled.

 Use this together with `max_num_seqs > 1` for bursty serving traffic. `0`
 disables admission waiting and preserves the lowest first-request latency.
-For diffusion request-level batching tuning, see
-[Request-Level Batching](../user_guide/diffusion/request_batching.md).
+For diffusion execution and batching tuning, see
+[Diffusion Execution Modes](../user_guide/diffusion/execution_modes.md).

 Default: `0.0`

@@ -1,6 +1,6 @@
 **Scope**

-For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload_diffusion/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/).
+For diffusion models, the L4 functionality test covers all or common *diffusion features* that are supported by this model, including several [parallelism acceleration methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/parallelism_acceleration/), [CPU offloading](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cpu_offload/), [TeaCache](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/teacache/) and [Cache-DiT](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/diffusion/cache_dit_acceleration/) cache backends, [quantization methods](https://docs.vllm.ai/projects/vllm-omni/en/latest/user_guide/quantization/overview/).

 **Test Case Design**

@@ -44,9 +44,11 @@ Pass **`--test-config-file`** to run one JSON file as-is, or omit it for the bul
            "ignore_eos": true,
            "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
            "baseline": {
-                "mean_ttft_ms": [500, 800],
-                "mean_audio_ttfp_ms": [2000, 3500],
-                "mean_audio_rtf": [0.25, 0.35]
+                "H100": {
+                    "mean_ttft_ms": [500, 800],
+                    "mean_audio_ttfp_ms": [2000, 3500],
+                    "mean_audio_rtf": [0.25, 0.35]
+                }
            }
        }
    ]
@@ -142,10 +144,6 @@ pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py \
  --test-config-file tests/dfx/perf/tests/test_bagel_vllm_omni.json
 pytest -s -v tests/dfx/perf/scripts/run_benchmark.py \
  --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json
-
-# Optional baseline assertion (default off)
-pytest -sv tests/dfx/perf/scripts/run_diffusion_benchmark.py --assert-baseline \
-  --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
 ```

 See also [Markers for Tests](#markers-for-tests) for registered hardware markers (`H100`, `L4`, `cuda`, `distributed_cuda`, …).
@@ -202,4 +200,4 @@ You can add any benchmark running parameters you need here. For all optional par
 | num_prompts     | int / array | Yes      | 10,[10, 20, 30] | Number of requests. Supports single values or arrays. If a single value is used, it will be automatically expanded to match the number of qps or max_concurrency, e.g., [10,10,10]. If an array is used, its length must match the number of qps or max_concurrency. |
 | request_rate    | float / array | No  | 0.5, [0.5, 1, inf] | Queries per second. Supports single values or arrays. If a single value is used, it will be automatically expanded to match the number of num_prompts, e.g., [1,1,1]. If an array is used, its length must match the number of num_prompts.                          |
 | max_concurrency | int / array | No       | 1, [1, 2, 3]    | Maximum concurrent in-flight requests. Same array / expansion rules as `request_rate` (mutually exclusive with QPS mode).                                                                                                                                                                                                             |
-| baseline        | object      | No       | see above       | Optional per-metric thresholds; keys must match benchmark output fields. Scalar, list (per sweep step), or object (keyed by concurrency or QPS string).  
+| baseline        | object      | No       | see below       | Optional thresholds. **Hardware-nested only**: top-level keys must be `[hardware-resource]` pytest markers from `pyproject.toml` (`get_hardware_mark_list()`). Under each hardware key, metric names are free-form; values are scalars or sweep-aligned lists. Example: `{"H100": {"throughput_qps": […], "custom_stage_ms": 1.0}, "A3": {…}}`. Runtime filename aliases (`_RUNTIME_DEVICE_ALIASES`) are separate and do not expand this allowlist. |
@@ -755,7 +755,8 @@ omni = Omni(model="your-model", ulysses_degree=2, ring_degree=2)

 ### Step Execution

-See detailed design guide: [How to add step execution support](../../design/feature/diffusion_step_execution.md)
+See the detailed
+[Diffusion Continuous Batching design guide](../../design/feature/diffusion_continuous_batching.md).

 Use this only when your pipeline can be split into stable request-scoped and
 step-scoped phases. The reference implementation is
@@ -769,9 +770,8 @@ step-scoped phases. The reference implementation is
 Do not enable `step_execution=True` until those four methods are implemented
 and validated against the request-level path.

-If you want the pipeline to work with the experimental batched step-wise path
-(`max_num_seqs > 1`), also see:
-[Continuous Batching for Step-Wise Diffusion](../../design/feature/diffusion_continuous_batching.md).
+The same design guide covers the experimental batched step-wise path used when
+`max_num_seqs > 1`.

 If you expose this in example scripts or recipes, keep it opt-in. Surface
 runtime features like `step_execution` as optional flags instead of silently
@@ -824,7 +824,7 @@ omni = Omni(model="your-model",

 ### CPU Offload

-See detailed guide: [CPU Offloading for Diffusion Models](../../user_guide/diffusion/cpu_offload_diffusion.md)
+See detailed guide: [CPU Offloading for Diffusion Models](../../user_guide/diffusion/cpu_offload.md)

 vLLM-Omni provides two offloading strategies to reduce GPU memory usage:

@@ -1,165 +1,232 @@
-# Continuous Batching for Step-Wise Diffusion
+# Diffusion Continuous Batching

-!!! warning "Experimental Feature"
-    This feature is experimental. It currently applies only to native
-    diffusion pipelines running with `step_execution=True`.
-
-This document describes the batching extension built on top of
-[Diffusion Step Execution](diffusion_step_execution.md). The base
-step-execution contract is unchanged. The batching work is mainly in the
-scheduler and runner layers.
-
-## Why It Helps
-
-Step-wise execution breaks a long denoise loop into scheduler-visible units.
-That gives the runtime a place to admit other compatible requests between
-steps instead of waiting for an entire request to finish.
-
-This matters most in low-MFU or bursty serving scenarios:
-
- one request's denoise step may not fully saturate the GPU
- several compatible requests can share the same denoise forward pass
- throughput and device utilization can improve without changing request-local
-  scheduler state
-
-This is **not** a guaranteed single-request latency win. The main benefit is
-usually higher utilization and better throughput when the workload contains
-multiple in-flight compatible requests.
+This document describes the unified diffusion execution architecture, including
+request execution, request-level batching, step execution, continuous batching,
+and output delivery. For user-facing configuration and CLI examples, see
+[Diffusion Execution Modes](../../user_guide/diffusion/execution_modes.md).

 ## Overview

-With continuous batching enabled:
+`DiffusionEngine` selects an execution mode from `step_execution` and uses
+`max_num_seqs` as that mode's scheduler capacity:

- the scheduler may keep multiple compatible requests active at the same time
- the runner packs request-local step state into one `InputBatch`
- `denoise_step()` runs on that batch
- `step_scheduler()` and `post_decode()` still run per request
+| Configuration | Engine mode | Scheduler | Execution |
+|---|---|---|---|
+| `step_execution=False`, `max_num_seqs=1` | `REQUEST_BATCH` | `RequestScheduler` | One complete request-level `forward()` |
+| `step_execution=False`, `max_num_seqs>1` | `REQUEST_BATCH` | `RequestScheduler` | One fused `forward()` over compatible requests |
+| `step_execution=True`, `max_num_seqs=1` | `STEP_BATCH` | `StepScheduler` | One request advanced one denoise step per scheduler tick |
+| `step_execution=True`, `max_num_seqs>1` | `STEP_BATCH` | `StepScheduler` | Compatible requests advanced together in step waves |

-The current implementation is conservative:
+Serial request execution is handled inside `REQUEST_BATCH`; it is not a
+separate engine mode. Similarly, continuous batching is the multi-request
+configuration of `STEP_BATCH`, not an independent execution mode.

- only compatible requests are batched together
- per-request progress and completion remain independent
+The engine performs the following initialization:

-Here, "continuous batching" means the step-wise path enabled by
-`step_execution=True`. Request-mode `DiffusionRequestBatch` is static
-request-level batching for one full pipeline `forward()` call; it does not
-admit or remove requests between denoise steps.
+1. Resolve model-specific pre- and post-processing hooks.
+2. Select `REQUEST_BATCH` or `STEP_BATCH`.
+3. Construct the configured `DiffusionExecutor`.
+4. Construct `RequestScheduler` or `StepScheduler`, unless a scheduler was
+   explicitly injected.
+5. Bind `execute_batch` or `execute_step`.

-## Enablement
+`DiffusionEngine.make_engine()` owns startup warmup after engine construction.
+This keeps construction, backend selection, and warmup failure cleanup in one
+lifecycle.

-Use `--step-execution` as the feature gate, then increase `--max-num-seqs`
-above `1` if you want batching:
+## Unified Output Stream

-```bash
-vllm serve Qwen/Qwen-Image --omni \
-  --port 8091 \
-  --step-execution \
-  --max-num-seqs 8
+All asynchronous diffusion requests use the same per-request output stream,
+regardless of whether the caller exposes streaming results to the user.
+
+```python
+async for output in engine.step_streaming(request):
+    ...
 ```

-`--max-num-seqs 1` keeps the step-wise path without enabling batching.
+When a request is admitted, the engine creates an
+`asyncio.Queue[DiffusionOutput]` keyed by its request ID. The busy loop sends
+both intermediate chunks and final results through that queue:

-For a reproducible replay flow using the bundled serving benchmark, see the
-Qwen-Image replay commands in
-[`benchmarks/diffusion/README.md`](gh-file:benchmarks/diffusion/README.md)
-and
-[`benchmarks/diffusion/performance_dashboard/qwen_image_serving_performance.md`](gh-file:benchmarks/diffusion/performance_dashboard/qwen_image_serving_performance.md).
+```text
+request
+  -> scheduler.add_request()
+  -> scheduler.schedule()
+  -> execute_batch() or execute_step()
+  -> scheduler.update_from_output()
+  -> per-request output queue
+  -> step_streaming()
+```

-## Scheduler
+Streaming callers forward every yielded output. Non-streaming callers consume
+the same generator to completion and return only its final output. This avoids
+maintaining separate future-based and queue-based result paths.

-The scheduler derives its batch capacity from `max_num_seqs` through
-`max_num_running_reqs`.
+`DiffusionEngine.step()` and
+`async_add_req_and_wait_for_response()` remain deprecated compatibility
+wrappers. New integrations should consume `step_streaming()` or
+`async_add_req_and_stream_response()`.

-Batch admission is gated by
-[`StepBatchSamplingParamsKey`](gh-file:vllm_omni/diffusion/sched/interface.py),
-which is built from shape-sensitive and CFG-sensitive sampling fields. This is
-the core correctness rule for batching: requests are only co-batched when they
-share the same denoise tensor contract.
+### Completion and Cancellation

-There are three important details:
+Output delivery and scheduler lifecycle are related but separate:

- `num_inference_steps` is not part of the key, so requests with different
-  total step counts can still share a batch
- requests also do not need to be at the same current denoise progress; active
-  requests can continue batching even when their current step indices diverge
- admission is still FIFO, so an incompatible request at the head of the
-  waiting queue blocks later compatible requests
+- A terminal scheduler result always finalizes request state.
+- A disconnected or cancelled consumer removes only its delivery queue.
+- The scheduler may still finish and clean up the request after its queue has
+  been removed.
+- Engine shutdown sends an error output to all remaining streams.

-Today that compatibility rule is still shape-sensitive. `height`, `width`,
-`num_frames`, and CFG-related fields remain part of the key, so different
-resolutions or incompatible guidance settings do **not** co-batch yet. The
-key also covers LoRA identity (`lora_int_id`, `lora_scale`), so requests
-targeting different adapters or scales run in separate batches and the
-worker can activate exactly one adapter per step.
+This separation prevents cancelled consumers from leaving active request IDs in
+the scheduler.

-The scheduler batching unit is one logical `OmniDiffusionRequest`. In the
-step-wise path, runtime tensor batching is represented as `StepInputBatch`. For
-request-mode prompt semantics, see
-[Request-Level Batching](../../user_guide/diffusion/request_batching.md).
+## Request-Batch Execution

-## Runner
+Request-batch execution runs the complete diffusion pipeline for each scheduler
+wave. Each `OmniDiffusionRequest` remains an independent logical request with
+its own prompt, sampling parameters, seed, request ID, output, error, and abort
+state.

-The runner keeps persistent per-request execution state in
-[`StepRequestState`](gh-file:vllm_omni/diffusion/worker/utils.py),
-while the scheduler owns a separate lightweight
-[`SchedulerRequestState`](gh-file:vllm_omni/diffusion/sched/interface.py)
-for queueing and lifecycle tracking.
+With `max_num_seqs=1`, `execute_batch()` handles the serial request path. With a
+larger value, a pipeline can opt into fused execution by declaring:

-For each step, the runner builds an
-[`InputBatch`](gh-file:vllm_omni/diffusion/worker/input_batch.py) from the
-active request states:
+```python
+supports_request_batch = True
+```

- prompt embeddings and masks are normalized and padded
- dynamic tensors such as `latents` and `timesteps` are gathered each step
- buffers are reused when batch composition stays the same
+Its `forward()` method must accept `DiffusionRequestBatch` and return one
+`DiffusionOutput` per request. The runner validates the result count and maps
+outputs back to their original request IDs with `BatchRunnerOutput`.

-The step-wise batched path is:
+Current in-tree request-batch implementations include the Flux, LTX-2, SD3, and
+Qwen-Image pipelines. Treat the pipeline capability flag in the source as
+authoritative because support continues to expand.

-1. Run `prepare_encode()` for newly admitted requests.
+### Request-Batch Data Flow
+
+- `DiffusionSchedulerOutput` carries scheduled request IDs and payloads.
+- `DiffusionRequestBatch` provides the pipeline-facing static batch.
+- `BatchRunnerOutput` carries per-request results back to the engine.
+
+Request-local setup, such as seed handling and output/error mapping, remains
+per request. Cache refresh, homogeneous LoRA activation, and
+`pipeline.forward(req_batch)` can be shared by the batch.
+
+### Compatibility and Admission
+
+`RequestBatchSamplingParamsKey` contains fields that affect the tensor contract,
+including shape, guidance, output count, and LoRA identity. Only compatible
+requests can share a fused forward call.
+
+Admission is FIFO and conservative. An incompatible request at the front of the
+waiting queue can prevent later compatible requests from joining the current
+wave.
+
+`request_batch_max_wait_ms` optionally delays the first schedule operation of a
+new wave so bursty arrivals can accumulate. It applies only when:
+
+- the pipeline supports request batching;
+- step execution is disabled;
+- no requests are currently running; and
+- the configured wait is greater than zero.
+
+The wait ends when the queue reaches capacity, stabilizes briefly, reaches its
+deadline, or the engine stops.
+
+## Step-Batch Execution
+
+Step execution exposes denoising progress to the scheduler. A supporting
+pipeline implements four stateful operations:
+
+| Operation | Responsibility |
+|---|---|
+| `prepare_encode(state)` | Validate input, encode prompts, initialize latents and timesteps, and create request-local scheduler state |
+| `denoise_step(input_batch, *, states=...)` | Run one denoise forward for the scheduler-provided request states |
+| `step_scheduler(state, noise_pred)` | Update latents and advance request progress |
+| `post_decode(state)` | Decode and postprocess a completed request or output boundary |
+
+Persistent request state lives in `StepRequestState`. Pipeline-specific fields
+that do not belong in the shared contract should be stored in `state.extra`.
+Queueing and lifecycle metadata remain in the scheduler's request state.
+
+Current native pipelines that explicitly enable step execution include
+Qwen-Image, HunyuanImage3, and Helios. Step execution alone does not imply
+continuous-batching support: Qwen-Image accepts batched step states.
+HunyuanImage3 accepts batched step states only when its resolved self-attention
+backend is `TORCH_SDPA`; otherwise it rejects groups larger than one request.
+Configure `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or
+`diffusion_attention_config.default.backend=TORCH_SDPA` when
+`max_num_seqs>1`. See the
+[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md)
+for its validated configuration. Helios supports only a single active step
+request and must use `max_num_seqs=1`.
+
+### Continuous Batching
+
+When `max_num_seqs>1`, `StepScheduler` can keep multiple compatible requests
+active. The runner gathers their state into `InputBatch`, performs one batched
+denoise forward, then applies scheduler updates per request:
+
+1. Receive transferred KV payloads, then run `prepare_encode()` for newly
+   admitted requests.
 2. Build or refresh `InputBatch`.
-3. Run one batched `denoise_step(input_batch)`.
-4. Slice the batched `noise_pred` back per request.
-5. Run per-request `step_scheduler()`.
-6. Run `post_decode()` only for requests that finished denoising.
-7. Scatter updated latents back into persistent request state with
-   [`scatter_latents()`](gh-file:vllm_omni/diffusion/worker/input_batch.py).
+3. Run one batched `denoise_step(input_batch, states=states)`.
+4. Slice noise predictions back to each request.
+5. Run each request's `step_scheduler()`.
+6. Run `post_decode()` at a chunk boundary or request completion.
+7. Scatter updated latents back into persistent request state.

-This keeps the shared work limited to the denoise forward pass while preserving
-request-local scheduler state and outputs.
+`StepBatchSamplingParamsKey` protects the batched tensor contract. Requests can
+have different total step counts and current step indices, but shape-sensitive,
+CFG-sensitive, and LoRA fields must be compatible. FIFO head-of-line blocking
+also applies here.

-## Engine
+Chunk-capable pipelines may emit intermediate results. Final-only step
+pipelines emit when the request finishes even if `streaming_output=True`.

-[`DiffusionEngine`](gh-file:vllm_omni/diffusion/diffusion_engine.py) provides
-the background loop and async add-request path needed for multiple requests to
-accumulate in the scheduler.
+## Model Author Guidelines

-When `step_execution=True`, the engine routes work through the step-wise
-executor path. The continuous batching behavior is defined by scheduler-side
-compatibility gating and runner-side `StepInputBatch` packing.
+When converting a request-level pipeline:

-## Current Limitations
+- Reuse the same helpers as `forward()` to avoid behavior drift.
+- Copy request-scoped scheduler state into `state.scheduler`.
+- Advance `state.step_index` only in `step_scheduler()`.
+- Keep model forward work in `denoise_step()`.
+- Keep latent mutation in `step_scheduler()`.
+- Keep final decoding equivalent to the tail of `forward()`.
+- Store masks, condition latents, and other request-local tensors in the
+  request state rather than on the shared pipeline.

- Experimental feature; use `max_num_seqs=1` for the conservative single-request
-  step path.
- Only native pipelines that already support `step_execution=True`.
- Only homogeneous batches keyed by `StepBatchSamplingParamsKey` are supported.
- `cache_backend`, KV transfer, and other request-mode extras are not wired
-  into the batched step-wise path yet.
- Future work can relax the current same-shape restriction with richer
-  heterogeneous batching policies such as bucketing or padded execution for
-  different resolutions.
+Before setting `supports_step_execution = True`, validate output parity for the
+same seed and sampling parameters, scheduler isolation across concurrent
+requests, abort cleanup, progress reporting, and all supported CFG paths.

-## Related Files
+Before setting `supports_request_batch = True`, validate the batch input
+contract, one-output-per-request result shape, request identity and error
+mapping, seeded concurrency, LoRA compatibility, and tensor IPC.

- Scheduler base:
-  [`vllm_omni/diffusion/sched/base_scheduler.py`](gh-file:vllm_omni/diffusion/sched/base_scheduler.py)
- Scheduler interface:
-  [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
- Step scheduler:
-  [`vllm_omni/diffusion/sched/step_scheduler.py`](gh-file:vllm_omni/diffusion/sched/step_scheduler.py)
- Runner:
-  [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
- Input batch:
-  [`vllm_omni/diffusion/worker/input_batch.py`](gh-file:vllm_omni/diffusion/worker/input_batch.py)
- Tests:
-  [`tests/diffusion/test_diffusion_scheduler.py`](gh-file:tests/diffusion/test_diffusion_scheduler.py)
+## Limitations
+
+- Request batching requires an explicit pipeline capability when
+  `max_num_seqs>1`.
+- Step execution requires the four-method stateful pipeline contract.
+- Both batching paths currently require homogeneous compatibility keys.
+- FIFO scheduling can reduce batching opportunities.
+- `request_batch_max_wait_ms` trades first-request latency for burst
+  coalescing.
+- All diffusion cache backends are currently unsupported in step mode.
+- KV transfer is supported for newly admitted step requests; some other
+  request-mode extras remain unsupported.
+- Step continuous batching remains experimental; use `max_num_seqs=1` for the
+  conservative step path.
+
+## Related Implementation
+
+- Engine: [`vllm_omni/diffusion/diffusion_engine.py`](gh-file:vllm_omni/diffusion/diffusion_engine.py)
+- Model contract: [`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py)
+- Scheduler interface: [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
+- Request scheduler: [`vllm_omni/diffusion/sched/request_scheduler.py`](gh-file:vllm_omni/diffusion/sched/request_scheduler.py)
+- Step scheduler: [`vllm_omni/diffusion/sched/step_scheduler.py`](gh-file:vllm_omni/diffusion/sched/step_scheduler.py)
+- Runner: [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
+- Request batch: [`vllm_omni/diffusion/worker/request_batch.py`](gh-file:vllm_omni/diffusion/worker/request_batch.py)
+- Step input batch: [`vllm_omni/diffusion/worker/input_batch.py`](gh-file:vllm_omni/diffusion/worker/input_batch.py)
@@ -1,175 +0,0 @@
-# Request-Level Batching for Diffusion
-
-This document describes the request-mode batching path for diffusion pipelines.
-For end-user enablement and tuning, see
-[Request-Level Batching](../../user_guide/diffusion/request_batching.md).
-
-This is separate from
-[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
-Request-level batching runs one full pipeline `forward()` over a static batch of
-compatible requests. Step-wise continuous batching admits work between denoise
-steps when `step_execution=True`.
-
-## Why It Helps
-
-The request-level design avoids coupling several logical requests to one request
-object. This keeps request identity, abort/error handling, and per-request
-metadata unambiguous while still allowing one fused pipeline forward pass for
-bursty or concurrent traffic.
-
-## Overview
-
-With request-level batching enabled:
-
- each `OmniDiffusionRequest` contains one `prompt` and one `request_id`
- the scheduler groups compatible waiting requests into one scheduler wave
- `DiffusionRequestBatch` wraps the scheduled requests for pipeline `forward()`
- batch-capable pipelines return `list[DiffusionOutput]`, one output per
-  request
- `BatchRunnerOutput` maps each result back to its original `request_id`
-
-Pipelines opt in with `supports_request_batch = True` and a `forward()` method
-that accepts `DiffusionRequestBatch` and returns `list[DiffusionOutput]`.
-Pipelines that do not opt in keep the existing per-request execution path.
-
-## Enablement
-
-Request-level batching is the request-mode path, so `step_execution` must remain
-disabled. Increase `max_num_seqs` above `1` to let the scheduler keep multiple
-compatible requests active:
-
-```bash
-vllm serve Qwen/Qwen-Image --omni \
-  --port 8091 \
-  --max-num-seqs 4
-```
-
-For bursty online ingress, `request_batch_max_wait_ms` can add a bounded
-admission wait before the first `schedule()` of a scheduler wave:
-
-```bash
-vllm serve Qwen/Qwen-Image --omni \
-  --port 8091 \
-  --max-num-seqs 4 \
-  --request-batch-max-wait-ms 20
-```
-
-`request_batch_max_wait_ms=0` disables this wait and is the default.
-
-## Request Contract
-
-`OmniDiffusionRequest` represents one logical request. It owns one prompt,
-sampling parameters, request id, and request-local metadata. Runtime batches are
-formed by the scheduler and represented separately from the request payload.
-
-Runtime batching is represented by:
-
- [`DiffusionSchedulerOutput`](gh-file:vllm_omni/diffusion/sched/interface.py)
-  for scheduled request ids and request payloads
- [`DiffusionRequestBatch`](gh-file:vllm_omni/diffusion/worker/request_batch.py)
-  for the pipeline-facing request batch
- [`BatchRunnerOutput`](gh-file:vllm_omni/diffusion/worker/utils.py) for
-  per-request results
-
-`DiffusionRequestBatch` intentionally exposes compatibility properties such as
-`prompts`, `sampling_params`, `request_id`, and `kv_sender_info` so migrated
-pipelines can stay close to upstream code while using a batch-aware contract.
-
-## Scheduler
-
-The scheduler derives its capacity from `max_num_seqs` through
-`max_num_running_reqs`. It exposes waiting/running queue counters so the engine
-can decide whether admission wait is useful before scheduling a new wave.
-
-Batch compatibility is controlled by
-[`RequestBatchSamplingParamsKey`](gh-file:vllm_omni/diffusion/sched/interface.py). The key
-contains shape-sensitive and guidance-sensitive fields, including output count
-and LoRA identity. Requests with incompatible shapes, CFG settings, output
-counts, LoRA adapters, or LoRA scales are kept in separate batches.
-
-Admission is conservative:
-
- the scheduler only batches compatible requests
- FIFO ordering is preserved
- an incompatible request at the head of the waiting queue blocks later
-  compatible requests
-
-## Engine
-
-[`DiffusionEngine`](gh-file:vllm_omni/diffusion/diffusion_engine.py) resolves
-request-batch capability during initialization from the configured pipeline
-class, including custom pipeline classes.
-
-The capability check uses the pipeline class attribute
-`supports_request_batch = True`. Pipelines that set this attribute must implement
-a request-batch-compatible `forward()` contract and return one
-`DiffusionOutput` per request; the runner validates that return shape at runtime.
-
-When the selected pipeline is batch-capable and `step_execution=False`, request
-mode routes scheduler waves through the batch executor path. Otherwise it keeps
-the per-request executor path.
-
-The optional admission wait runs only when:
-
- request batching is supported
- `step_execution=False`
- `request_batch_max_wait_ms > 0`
- no requests are currently running
-
-The wait exits early when the waiting queue reaches capacity, when the queue is
-stable for a short window, when the deadline expires, or when the engine stops.
-
-## Executor And Runner
-
-The executor exposes two request-mode entries:
-
- `execute_request`: one worker call per scheduled request
- `execute_batch`: one worker call for the whole `DiffusionSchedulerOutput`
-
-On the batch path, the worker builds a `DiffusionRequestBatch` and runs the
-pipeline once. Request-local setup remains per request:
-
- KV transfer metadata
- random generator and seed handling
- request output/error/abort mapping
-
-Shared batch setup happens once per batch when possible:
-
- cache refresh
- LoRA activation for the homogeneous adapter key
- pipeline `forward(req_batch)`
-
-Large tensor IPC still uses the shared-memory packing path. The packer traverses
-both normal `RunnerOutput.result` wrappers and nested batch results so batched
-outputs do not fall back to pickle IPC for tensor payloads.
-
-## Current Limitations
-
- Only pipelines that declare the request-batch contract use fused batch
-  execution.
- Batches are homogeneous under `RequestBatchSamplingParamsKey`; heterogeneous resolution or
-  incompatible guidance settings do not co-batch yet.
- FIFO scheduling can reduce batching opportunities when an incompatible
-  request is at the front of the queue.
- `request_batch_max_wait_ms` improves burst coalescing but can add latency to
-  the first request in a scheduler wave. Keep it small for latency-sensitive
-  serving.
- Step-wise continuous batching is documented separately and only applies when
-  `step_execution=True`.
-
-## Related Files
-
- Request object and request batch:
-  [`vllm_omni/diffusion/request.py`](gh-file:vllm_omni/diffusion/request.py)
- Scheduler interface:
-  [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
- Scheduler base:
-  [`vllm_omni/diffusion/sched/base_scheduler.py`](gh-file:vllm_omni/diffusion/sched/base_scheduler.py)
- Engine:
-  [`vllm_omni/diffusion/diffusion_engine.py`](gh-file:vllm_omni/diffusion/diffusion_engine.py)
- Worker runner:
-  [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
- Executor interface:
-  [`vllm_omni/diffusion/executor/abstract.py`](gh-file:vllm_omni/diffusion/executor/abstract.py)
- Tests:
-  [`tests/diffusion/test_diffusion_engine.py`](gh-file:tests/diffusion/test_diffusion_engine.py)
@@ -1,131 +0,0 @@
-# Adding Step Execution Support for Diffusion Pipelines
-
-This guide documents vLLM-Omni's stepwise diffusion contract for model authors
-and contributors implementing `step_execution=True` support for a diffusion
-pipeline.
-
-For end-user enablement, supported models, and current limitations, see
-[Step Execution](../../user_guide/diffusion/step_execution.md).
-
-This document describes the base step-execution contract only. For the
-experimental batching policy layered on top of the step-wise path, see
-[Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
-
-## Current Support Scope
-
-`step_execution` is **not** a generic diffusion toggle. It only works for
-pipelines that implement the segmented stateful contract in
-[`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py).
-
-This page is intentionally author-facing. Treat runtime enablement
-(`step_execution=True` in Python or `--step-execution` in serving) as an
-opt-in user knob layered on top of the implementation contract below.
-
-Current in-tree support:
-
-| Pipeline | Example models | Step execution |
-|----------|----------------|----------------|
-| `QwenImagePipeline` | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes |
-| All other diffusion pipelines | `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, `QwenImageLayeredPipeline`, GLM-Image, Wan, Flux, etc. | No |
-
-Current engine/runtime limitations:
-
- Continuous batching with `max_num_seqs > 1` is experimental and documented in
-  [Continuous Batching for Step-Wise Diffusion](diffusion_continuous_batching.md).
-  Keep `max_num_seqs=1` for the conservative single-request step path.
- `cache_backend` is not supported in step mode.
- Request-mode extras such as KV transfer are not wired into step mode yet.
- Unsupported pipelines now fail early during model loading instead of failing on the first request.
-
-## Execution Contract
-
-Step mode is driven by four pipeline methods plus the shared mutable request
-state object:
-
- `prepare_encode(state)`: one-time request preparation.
- `denoise_step(state)`: compute the noise prediction for the current step.
- `step_scheduler(state, noise_pred)`: mutate latents and advance step state.
- `post_decode(state)`: decode the final output after denoising is complete.
-
-The state lives in
-[`vllm_omni/diffusion/worker/utils.py`](gh-file:vllm_omni/diffusion/worker/utils.py)
-as `StepRequestState`. Store request-scoped tensors there, or use
-`state.extra` for model-specific fields that do not justify extending the core
-dataclass.
-
-The worker-side step loop lives in
-[`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py):
-
-1. `prepare_encode()` runs once for a new request.
-2. `denoise_step()` runs every scheduler tick.
-3. `step_scheduler()` mutates `state.latents` and advances `state.step_index`.
-4. `post_decode()` runs exactly once after `state.denoise_completed` becomes true.
-
-## Recommended Split
-
-When converting an existing request-level `forward()` pipeline, keep the split
-strict and mechanical:
-
-| Request-level phase | Stepwise method | What belongs there |
-|---------------------|-----------------|--------------------|
-| Input validation, prompt encoding, latent init, timestep prep, per-request scheduler creation | `prepare_encode()` | Anything that should happen once per request |
-| Transformer forward / noise prediction | `denoise_step()` | Pure denoise computation for the current timestep |
-| `scheduler.step(...)` and `step_index += 1` | `step_scheduler()` | Only latent/state mutation for one step |
-| VAE decode / postprocess | `post_decode()` | Final decode only |
-
-Keep the stepwise path reusing the same helpers as the request-level path
-whenever possible. Reimplementing the denoise loop from scratch is the easiest
-way to introduce behavioral drift.
-
-## Qwen-Image Reference
-
-[`pipeline_qwen_image.py`](gh-file:vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py)
-is the reference implementation and is split correctly for the current
-contract:
-
- `prepare_encode()` reuses `_prepare_generation_context()` so prompt encoding,
-  latent init, timestep creation, CFG setup, and shape bookkeeping stay aligned
-  with `forward()`.
- `prepare_encode()` deep-copies `self.scheduler` **after**
-  `prepare_timesteps()` so request-specific scheduler state is isolated.
- `denoise_step()` reuses `_build_denoise_kwargs()` plus
-  `predict_noise_maybe_with_cfg()`, so sequential CFG, CFG-parallel, and
-  non-CFG behavior stay identical to the request-level path.
- `step_scheduler()` only calls
-  `scheduler_step_maybe_with_cfg(..., per_request_scheduler=state.scheduler)`
-  and increments `state.step_index`.
- `post_decode()` reuses `_decode_latents()`, so the final image decode matches
-  the normal `forward()` path.
-
-That decomposition is the target pattern for future models.
-
-## Rules For New Pipelines
-
- Do not keep request-scoped scheduler state on `self.scheduler`. Copy it into
-  `state.scheduler` during `prepare_encode()`.
- Do not mutate `state.step_index` inside `denoise_step()`. Only
-  `step_scheduler()` should advance the step.
- Do not decode partial outputs in `denoise_step()` or `step_scheduler()`.
- If the request-level pipeline has condition latents, masks, or edit-specific
-  tensors, store them in `state` or `state.extra`, not in global pipeline
-  attributes.
- Preserve CFG behavior by sharing the same helper path used by `forward()`.
- Keep `post_decode()` equivalent to the tail of `forward()`.
-
-## Validation Checklist
-
-Before marking a pipeline as `supports_step_execution = True`, verify:
-
- Stepwise output matches request-level output for the same seed and sampling params.
- Per-request scheduler state is isolated across concurrent requests.
- Abort during denoise does not leak cached state.
- `step_index` reported by `RunnerOutput` matches the scheduler progress.
- CFG-parallel and non-CFG paths both work if the request-level pipeline supports them.
-
-## Related Files
-
- Contract: [`vllm_omni/diffusion/models/interface.py`](gh-file:vllm_omni/diffusion/models/interface.py)
- State: [`vllm_omni/diffusion/worker/utils.py`](gh-file:vllm_omni/diffusion/worker/utils.py)
- Runner loop: [`vllm_omni/diffusion/worker/diffusion_model_runner.py`](gh-file:vllm_omni/diffusion/worker/diffusion_model_runner.py)
- Scheduler transport: [`vllm_omni/diffusion/sched/interface.py`](gh-file:vllm_omni/diffusion/sched/interface.py)
- Reference pipeline: [`vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py`](gh-file:vllm_omni/diffusion/models/qwen_image/pipeline_qwen_image.py)
@@ -0,0 +1,156 @@
+# Distributed Layerwise Offload
+
+This document describes distributed layerwise offload (DLO) for diffusion
+models. DLO keeps only a small number of DiT blocks on the accelerator and
+streams the remaining blocks from host memory. The distributed backend can
+either shard those host-side weights across an existing parallel group or keep
+the standard loader's rank-local weights and avoid an additional collective.
+
+For user-facing commands, see the
+[distributed layerwise offloading guide](../../user_guide/diffusion/cpu_offload.md)
+and the [Cosmos3 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md).
+
+## Status
+
+DLO is implemented for multi-device diffusion execution. The default
+AllGather path is the primary path for DP and SP deployments. The
+`--dlo-no-use-allgather` path is a rank-local compatibility mode: it is useful
+for standard-loader sharding, workstation bring-up, and systems where an
+additional DLO collective is undesirable, but it does not reduce host weight
+storage across ranks.
+
+The compatibility matrix below describes the current implementation. The
+unit-level guards are covered, but not every parallelism combination has a
+full model-and-hardware end-to-end test.
+
+## Design
+
+### DLO consumes the existing parallel topology
+
+DLO does not create a new DP, TP, or SP topology. It reads the configured
+`DiffusionParallelConfig` and attaches offload hooks to the DiT blocks after the
+standard distributed groups have been initialized.
+
+The DLO weight-sharding group is selected as follows:
+
+1. Use the existing DP group when `data_parallel_size > 1`.
+2. When DP is one and SP is greater than one, use the SP group.
+3. Otherwise, run rank-locally without a DLO process group.
+
+TP is deliberately not used as DLO's AllGather group. HSDP has its own
+parameter-sharding lifecycle and is not allowed to be sharded a second time by
+DLO's AllGather path.
+
+### AllGather path
+
+With the default `dlo_use_allgather=True`, each rank stores approximately
+`1 / group_size` of each streamable block in pinned host memory. The next
+block's shard is copied to a device buffer and reconstructed with
+`all_gather_into_tensor` on a communication stream while the current block is
+executing.
+
+```text
+Compute:    [Block N]             [Block N+1]          [Block N+2]
+H2D:                      [shard N+1]           [shard N+2]
+AllGather:                [full N+1]             [full N+2]
+Buffers:    [current slot]       [prefetch slot]       [current slot]
+```
+
+![DLO double-buffer prefetch pipeline](../figures/dlo/dlo_pipeline.gif)
+
+The backend uses two shared device buffers, so accelerator weight residency is
+bounded by the largest streamed blocks rather than the complete model.
+
+When DP is greater than one, the engine can process one request per DP rank in
+the same denoising wave. Because AllGather is a collective, all participating
+requests must take the same execution path at every denoising step.
+
+### Rank-local path without DLO AllGather
+
+With `--dlo-no-use-allgather`, DLO forces its internal offload shard size to
+one. The regular model loader remains responsible for preparing each rank's
+weights, including TP-local tensors or HSDP-managed parameters. DLO then
+streams those rank-local tensors block by block using H2D copies only.
+
+This mode means:
+
+- DP still provides independent replicas, but DLO does not shard weights
+  across DP ranks.
+- SP still performs its normal activation/attention collectives, but DLO does
+  not shard weights across SP ranks.
+- TP/HSDP/SP collectives, if configured, are not disabled by this flag; only
+  DLO's additional weight AllGather is disabled.
+- Pure DP deployments keep a full host-side model copy per rank, subject to
+  shared OS page-cache behavior.
+- The scheduler does not require a synchronized DP request wave for DLO.
+
+This path is intentionally not implemented by reusing the AllGather mmap
+loader. It relies on the standard loader so model-specific TP/HSDP transforms
+remain intact.
+
+## Parallelism compatibility
+
+| Parallelism | DLO + AllGather | DLO without AllGather |
+|---|---|---|
+| **DP** | Supported primary path. DLO shards host weights across the DP group and can run DP multi-concurrency. | Supported rank-local path. DP replicas remain independent; no DLO weight sharding or cross-DP DLO collective. |
+| **SP** | Supported in the implementation. With DP=1, DLO uses the SP group for host-weight sharding; SP still shards sequence/activation work. | SP remains active, but DLO keeps standard-loader rank-local weights and adds no SP weight collective. |
+| **TP > 1** | Unsupported in the DLO mmap path. TP-aware loader callbacks are bypassed, so the backend rejects this configuration when it enters that path. | Standard loading is retained and TP-local tensors can be streamed. This is the intended compatibility path, but it still needs broader model and hardware validation. |
+| **HSDP** | Rejected. HSDP has already sharded parameters, so DLO AllGather would double-shard them. | Accepted by configuration. HSDP owns parameter sharding and its own gathers; DLO only stages rank-local parameters. End-to-end coverage is limited. |
+
+### Combined dimensions
+
+- **DP + SP:** DLO uses the DP group for weight sharding when DP is greater
+  than one; SP continues to use its own sequence-parallel group. If DP is one,
+  the SP group becomes DLO's sharding group in AllGather mode.
+- **DP + TP/SP without AllGather:** standard model loading defines the
+  rank-local tensor layout. DLO adds no cross-DP, cross-TP, or cross-SP weight
+  collective.
+- **HSDP + SP:** the general parallel configuration permits HSDP over SP, but
+  DLO must use `--dlo-no-use-allgather`. HSDP remains responsible for weight
+  materialization and synchronization.
+- **HSDP + DP or TP:** rejected independently by the diffusion parallel
+  configuration.
+
+## Request and loading constraints
+
+AllGather DP multi-concurrency requires:
+
+- explicit `num_inference_steps`;
+- the same `num_inference_steps` for all requests in a wave; and
+- identical request arguments that affect the collective execution path.
+
+The no-AllGather path does not impose these DLO-specific synchronized-wave
+requirements.
+
+The mmap loader is used only by the supported DLO+AllGather path when the
+model has a compatible checkpoint layout. Online quantization is incompatible
+with that sharded mmap path; use `--dlo-no-use-allgather` or disable online
+quantization. Models that do not meet the mmap requirements use the regular
+loader path.
+
+## Validation coverage
+
+Current source-level validation includes:
+
+- HSDP + DLO + AllGather rejection;
+- HSDP + DLO without AllGather acceptance at configuration level;
+- TP rejection in the DLO+AllGather mmap path;
+- resident-layer requests requiring no-AllGather;
+- DP request-wave validation for denoising-step compatibility;
+- sharding, double-buffer, AllGather-size, and heterogeneous-block regression
+  tests.
+
+The highest-value missing coverage is end-to-end numerical comparison against
+ordinary layerwise offload for DP+SP, TP+no-AllGather, and HSDP+SP+no-AllGather
+on the target CUDA/NCCL or CANN/HCCL hardware.
+
+## Recommendations
+
+- Use **DP + DLO AllGather** for the supported throughput and host-memory
+  scaling path.
+- Use **SP + DLO AllGather** for long-sequence workloads when DP concurrency is
+  not the goal.
+- Use **no-AllGather** to bring up TP or workstation PCIe configurations, with
+  the expectation of higher host-memory use and lower validation confidence.
+- Prefer **HSDP alone** for production HSDP deployments until the combined
+  HSDP + DLO no-AllGather path has broader end-to-end coverage.
@@ -0,0 +1,249 @@
+# Quantization
+
+This document describes the quantization architecture in vLLM-Omni and the
+extension points for contributors. For user-facing configuration, supported
+hardware, and method-specific instructions, see the
+[quantization user guide](../../user_guide/quantization/overview.md).
+
+## Goals and scope
+
+The design has four goals:
+
+1. Provide one `quantization_config` entry point for diffusion-only,
+   multi-stage omni/TTS, and multi-stage diffusion models.
+2. Reuse vLLM's quantization registry and `QuantizationConfig` contract where
+   a vLLM backend already provides the required behavior.
+3. Allow different components of one pipeline to use different quantization
+   configurations.
+4. Preserve the distinction between online quantization of a BF16/FP16
+   checkpoint and loading a pre-quantized checkpoint with serialized scales.
+
+This document covers configuration resolution, component routing, model
+integration, backend extension, and validation. Kernel implementations and
+user-facing support matrices are outside its scope.
+
+## Architecture
+
+The quantization path is split across the following layers:
+
+| Layer | Responsibility | Source |
+| --- | --- | --- |
+| Public API | Exposes `build_quant_config`, component routing, and backend registration | `vllm_omni/quantization/__init__.py` |
+| Factory | Normalizes method names, selects Omni overrides or vLLM registry backends, detects ModelOpt configs, and reconciles checkpoint metadata | `vllm_omni/quantization/factory.py` |
+| Component router | Resolves a quantization config from a layer prefix using longest-prefix matching | `vllm_omni/quantization/component_config.py` |
+| Method configs | Implement `QuantizationConfig` and select the platform-specific linear method | `vllm_omni/quantization/*_config.py` |
+| Diffusion configuration | Stores the canonical runtime config and propagates checkpoint configuration | `vllm_omni/diffusion/data.py` |
+| Worker and model integration | Passes the resolved config into vLLM's model configuration and model layers | `vllm_omni/diffusion/worker/diffusion_worker.py` and `vllm_omni/diffusion/models/` |
+
+The runtime contract remains vLLM's `QuantizationConfig`:
+
+```text
+user/CLI or model config
+          |
+          v
+OmniDiffusionConfig.quantization_config
+          |
+          +--> build_quant_config()
+          |        |
+          |        +--> Omni override
+          |        +--> vLLM registry backend
+          |        +--> ComponentQuantizationConfig
+          |
+          +--> checkpoint metadata reconciliation
+          |
+          +--> diffusion worker / vLLM config
+          |
+          +--> model layers call get_quant_method(layer, prefix)
+```
+
+The model layer receives either a method implementation or `None`. Returning
+`None` leaves that layer unquantized.
+
+## Configuration resolution
+
+`build_quant_config()` accepts the following forms:
+
+| Input | Resolution |
+| --- | --- |
+| `None` or `"none"` | Disable quantization. |
+| Method string, such as `"fp8"` | Resolve an Omni override first, then the vLLM quantization registry. |
+| Flat dictionary with `method` or `quant_method` | Normalize the method and construct its `QuantizationConfig`. |
+| Per-component dictionary | Construct a `ComponentQuantizationConfig`. |
+| Existing `QuantizationConfig` | Pass through without rebuilding it. |
+
+Method aliases are normalized case-insensitively, with `-` and `_` treated as
+equivalent. `auto-round` and `auto_round` use the Omni INC/AutoRound adapter.
+ModelOpt checkpoint metadata is detected from its method, producer metadata, or
+`quant_algo`, then mapped to the corresponding vLLM ModelOpt configuration.
+
+The factory uses Omni-specific overrides for methods whose diffusion behavior
+or platform dispatch is not provided by the vLLM registry. The current override
+set includes INT8, BitsAndBytes, MXFP8, MXFP4, MXFP4 dual-scale, and INC/
+AutoRound. Other methods, such as FP8, GGUF, and ModelOpt, are resolved through
+vLLM's registry when their configuration is compatible.
+
+## Per-component routing
+
+Multi-stage models do not necessarily want one quantization policy for every
+module. A component map expresses that policy without requiring each pipeline
+to construct separate vLLM configurations:
+
+```python
+quantization_config = {
+    "transformer": {"method": "fp8"},
+    "vae": None,
+    "default": None,
+}
+```
+
+`ComponentQuantizationConfig` applies the following rules:
+
+1. Component keys are interpreted as runtime layer-prefixes.
+2. The longest matching prefix wins.
+3. `None` disables quantization for the matching component.
+4. `default`, when present, handles prefixes with no explicit match.
+5. A matching child config delegates to its normal
+   `get_quant_method(layer, prefix)` implementation.
+
+Prefix matching happens after the model's weight-name mapping. New model
+integrations must verify the actual runtime prefixes, especially when a
+`WeightsMapper` changes checkpoint names. A prefix mismatch can silently route
+a layer to the default configuration, so component-routing tests should cover
+both positive matches and fall-through behavior.
+
+## Online and pre-quantized checkpoints
+
+The same method name can represent two different loading paths:
+
+- **Online**: the source checkpoint contains BF16/FP16 weights; the method
+  creates quantized weights or scales while the model is loaded.
+- **Pre-quantized**: the checkpoint contains quantized weights and the metadata
+  required by the method; the loader consumes those serialized tensors.
+
+Method configuration owns the mode flag, for example
+`is_checkpoint_int8_serialized` or `is_checkpoint_mxfp8_serialized`. The
+configuration's `get_quant_method()` selects the corresponding linear method.
+
+For pipelines with transformer-specific `config.json` files, such as cascade
+models, `resolve_quant_config_from_disk()` reconciles the active configuration
+with each transformer's checkpoint metadata. Its invariants are:
+
+- If no active configuration exists, valid checkpoint metadata can be used for
+  auto-detection.
+- A method mismatch is an error rather than an implicit conversion.
+- Serialized-checkpoint flags rebuild an online configuration into the
+  matching offline configuration.
+- Checkpoint-specific `ignored_layers` are preserved by rebuilding the active
+  configuration when they differ.
+- AutoRound MXFP8 metadata (`data_type="mx_fp"`) selects its offline path.
+
+This prevents a model from interpreting serialized weights with an online
+loader or silently applying the wrong method to one transformer in a
+multi-transformer pipeline.
+
+## Model and layer integration
+
+Diffusion model constructors receive the resolved configuration and pass it to
+quantizable vLLM layers. Each method configuration is responsible for:
+
+- identifying supported layers in `get_quant_method()`;
+- creating the correct parameter shapes and loaders;
+- processing weights after loading when conversion is required; and
+- applying the platform-specific quantized operation at runtime.
+
+Model code must keep precision-sensitive or unsupported modules unquantized.
+The shared `safe_quant_config()` helper is used by model integrations for
+normalization and modulation layers where a generic FP8 configuration would be
+unsafe. Text encoders, audio/vision encoders, VAEs, schedulers, and tokenizers
+are not implicitly quantized just because the main transformer is quantized;
+they require explicit component routing and method support.
+
+Weight quantization and quantized KV cache are separate paths. The former is
+resolved through `quantization_config`; runtime attention/KV-cache quantization
+is configured by the diffusion attention configuration and must be validated
+independently.
+
+## Platform and parallelism boundaries
+
+Platform checks belong in the method implementation rather than in the common
+factory. This keeps configuration parsing portable while allowing CUDA, NPU,
+and XPU methods to select different kernels or reject unsupported execution at
+model construction time. The user guide is the source of truth for the current
+hardware support matrix.
+
+Quantization methods must also respect the weight layout produced by the
+configured parallelism strategy:
+
+- Tensor-parallel layers may receive partitioned weight shapes and loaders.
+- Sequence-parallel execution changes activation movement but does not by
+  itself change the quantization configuration.
+- Data-parallel replicas build their own method state; quantization metadata
+  must be available consistently on every rank.
+- `WeightsMapper` transformations must be applied before matching ignored-layer
+  lists or component prefixes.
+
+Distributed layerwise offload has an additional boundary: its sharded mmap
+AllGather path is not compatible with online quantization because the path
+expects already materialized checkpoint weights. Use the rank-local
+`--dlo-no-use-allgather` path or a compatible pre-quantized checkpoint when
+combining these features. See the [DLO design document](distributed_layerwise_offload.md)
+for the full compatibility matrix.
+
+## Adding a quantization backend
+
+Contributors should follow this sequence:
+
+1. Check whether an existing vLLM backend can provide the required weight
+   format and kernel behavior. Prefer reuse over a duplicate Omni backend.
+2. If Omni-specific behavior is required, implement a `QuantizationConfig` and
+   its `QuantizeMethodBase`/linear method in `vllm_omni/quantization/`.
+3. Implement `get_name()`, supported activation dtypes, minimum capability,
+   checkpoint parsing, ignored-layer handling, and `get_quant_method()`.
+4. Add a lazy factory override in `factory.py`. Keep optional platform
+   dependencies out of package import paths and normalize aliases to one
+   canonical method name.
+5. If the method is pre-quantized, define the checkpoint metadata and the
+   serialized loading path before adding an online path.
+6. Add model-specific component routing only when the model contains stages
+   that must use different policies.
+7. Document user-visible configuration in the quantization user guide and keep
+   implementation rationale here.
+
+The public factory registration hook,
+`register_quantization_override(method, builder)`, is intended for integrations
+that need to register an Omni-compatible builder without changing the vLLM
+registry.
+
+## Validation
+
+Validation should cover the complete path, not only whether a config object can
+be instantiated:
+
+| Area | Required checks |
+| --- | --- |
+| Factory | Strings, aliases, flat dictionaries, invalid inputs, passthrough, and component maps |
+| Method config | Online/offline selection, ignored layers, platform guards, and weight loading |
+| Checkpoint metadata | Auto-detection, method mismatch errors, serialized flags, and per-transformer metadata |
+| Model integration | Correct quantization scope, unquantized components, and parallel weight layouts |
+| Numerical quality | Same-seed BF16 versus quantized output comparison at representative resolutions and step counts |
+| Performance | Peak device memory, generation latency, and any backend-specific throughput metric |
+
+The unit tests live under `tests/diffusion/quantization/`. The trajectory
+similarity tool in
+`vllm_omni/quantization/tools/compare_diffusion_trajectory_similarity.py`
+provides a repeatable quality and memory comparison for diffusion models.
+Method support and feature combinations remain model- and hardware-dependent;
+passing factory tests alone does not establish production support.
+
+## Design invariants
+
+- `quantization_config` is the single runtime source of truth after config
+  normalization.
+- A checkpoint method mismatch fails loudly instead of risking weight
+  corruption.
+- Components without an explicit supported quantization path remain in their
+  original precision.
+- Optional backend dependencies are imported lazily.
+- Method implementations own platform-specific capability checks and kernels.
+- User-facing support claims require model, hardware, numerical, and memory
+  validation in addition to configuration-level tests.
@@ -1,59 +0,0 @@
-# Distributed utils
-
-This directory (vllm_omni/distributed/ray_utils) contains utilities for distributed execution in vllm-omni, supporting both **Ray** and **Multiprocessing** backends.
-## 1. Installation
-```bash
-pip install "ray[default]"
-```
-## 2. Ray Utils
-
-The `ray_utils` module provides helper functions for managing Ray clusters and actors, which is essential for:
-*   **Multi-node deployment**: Running pipeline stages across different physical machines.
-*   **Resource management**: Efficient GPU/CPU allocation.
-
-### 2.1 Basic Usage
-
-To use the Ray backend, specify `worker_backend="ray"` when initializing the engine.
-
-**Command Line Example:**
-```bash
-vllm serve Qwen/Qwen2.5-Omni-7B \
-  --omni \
-  --port 8091 \
-  --worker-backend ray \
-  --ray-address auto
-```
-
-### 2.2 Cluster Setup
-
-**Step 1: Start Head Node**
-Run this on your primary machine:
-```bash
-ray start --head --port=6399
-```
-
-**Step 2: Connect Worker Nodes**
-Run this on each worker machine:
-```bash
-ray start --address=<HEAD_NODE_IP>:6399
-```
-
-> **Tip**: For a complete cluster setup script, refer to the vLLM example:
-> [run_cluster.sh](https://github.com/vllm-project/vllm/blob/main/examples/online_serving/run_cluster.sh)
-
-### 2.3 Distributed Connector Support
-
-When running on Ray, the system automatically adapts its communication strategy:
-
-*   **Cross-Node**: Recommended to use `MooncakeTransferEngineConnector` (RDMA, fastest) or `MooncakeStoreConnector` (TCP fallback).
-*   **Same-Node**: Can still use `SharedMemoryConnector` for efficiency, or Ray's native object store (plasma).
-*   **SHM payload routing**: `SharedMemoryConnector` always stores payloads in `/dev/shm`.
-
-### 2.4 Internal Helpers
-
-*   **`initialize_ray_cluster`**: Connects to an existing Ray cluster or starts a local one.
-
-## 3. Troubleshooting
-
-*   **Connection Issues**: Ensure the Ray head node is accessible and ports (default 6399 in this example) are open.
-*   **Version Mismatch**: Ensure all nodes run the same version of Ray and Python.
@@ -1,6 +1,9 @@
 # Design Documents

-This section contains design documents and architecture specifications for vLLM-Omni.
+This section contains design documents and architecture specifications for
+vLLM-Omni. The sidebar groups documents by the system concern they describe;
+retired module pages are preserved in the legacy archive instead of remaining
+in the active navigation.

 ## Architecture Documents

@@ -8,19 +11,80 @@ This section contains design documents and architecture specifications for vLLM-

 ## Feature Design Documents

- [Disaggregated Inference](feature/disaggregated_inference.md)
- [Ray-based Execution](feature/ray_based_execution.md)
- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
- [Adding Step Execution Support for Diffusion Pipelines](feature/diffusion_step_execution.md)
- [Request-Level Batching for Diffusion](feature/diffusion_request_level_batching.md)
- [Continuous Batching for Step-Wise Diffusion](feature/diffusion_continuous_batching.md)
+### Runtime and stage execution

-## Infrastructure Design Documents
+- [Disaggregated Inference](feature/disaggregated_inference.md)
+- [Async Chunk](feature/async_chunk.md)
+- [Async Diffusion Output](feature/async_diffusion_output.md)
+- [Async Omni Output Materialization](feature/omni_async_output_materialization.md)
+- [Automatic Prefix Caching in Omni Models](feature/prefix_caching.md)
+
+### Communication
+
+#### OmniConnector implementations
+
+- [Mooncake Store Connector](feature/omni_connectors/mooncake_store_connector.md)
+- [Mooncake Transfer Engine Connector](feature/omni_connectors/mooncake_transfer_engine_connector.md)
+- [Mori Transfer Engine Connector](feature/omni_connectors/mori_transfer_engine_connector.md)
+- [Shared Memory Connector](feature/omni_connectors/shared_memory_connector.md)
+- [Yuanrong Store Connector](feature/omni_connectors/yuanrong_connector.md)
+- [Yuanrong Transfer Engine Connector](feature/omni_connectors/yuanrong_transfer_engine_connector.md)
+
+### Diffusion acceleration
+
+#### Parallelism
+
+- [CFG-Parallel](feature/cfg_parallel.md)
+- [Expert Parallel](feature/expert_parallel.md)
+- [Hybrid Sharded Data Parallel (HSDP)](feature/hsdp.md)
+- [Pipeline Parallel](feature/pipeline_parallel.md)
+- [Sequence Parallel](feature/sequence_parallel.md)
+- [Tensor Parallel](feature/tensor_parallel.md)
+- [VAE Patch Parallelism](feature/vae_parallel.md)
+
+#### Attention Backends
+
+- [Skip-Softmax](feature/skip_softmax.md)
+
+#### Quantization
+
+- [Quantization](feature/quantization.md)
+
+- [Cache-DiT](feature/cache_dit.md)
+- [TeaCache](feature/teacache.md)
+- [Diffusion Continuous Batching](feature/diffusion_continuous_batching.md)
+- [Distributed Layerwise Offload](feature/distributed_layerwise_offload.md)
+
+## Infrastructure and Performance

 - [Prometheus Metrics](metrics.md)
+- [Speech Generation Performance Optimizations](qwen3_omni_tts_performance_optimization.md)

 ## Module Design Documents

- [AR Module](module/ar_module.md)
- [DIT Module](module/dit_module.md)
- [Entrypoint Module](module/entrypoint_module.md)
+- [Entrypoints and Serving Boundaries](module/entrypoints.md)
+- [vLLM-Omni Configuration](module/vllm_omni_config.md)
+- [Input, Output, and Modality Contracts](module/input_output_modality_contracts.md)
+- [Error Classification, Propagation, and Rendering](module/error_contracts.md)
+- [Engine Orchestration](module/engine_orchestration.md)
+- [Stage Runtime and Replica Lifecycle](module/stage_runtime.md)
+- [OmniConnector](module/omni_connector.md)
+- [Model Integration](module/model_integration.md)
+- [Autoregressive Runtime](module/ar_runtime.md)
+- Diffusion
+  - [Overview](module/diffusion/index.md)
+  - [Runtime](module/diffusion/diffusion_runtime.md)
+  - [Model Integration](module/diffusion/diffusion_model_integration.md)
+  - [Continuous Batching](module/diffusion/continuous_batching.md)
+  - [Parallelism](module/diffusion/parallelism.md)
+  - [Offloader](module/diffusion/offloader.md)
+- [Execution Platforms](module/execution_platforms.md)
+- [Cache Management](module/cache_management.md)
+- [Quantization](module/quantization.md)
+- [Observability](module/observability.md)
+- [Profiling](module/profiling.md)
+- [Benchmarking](module/benchmarking.md)
+
+The pre-#5137 pages are preserved in the
+[legacy module archive](module/archive/README.md) for historical reference and
+are not active design contracts.
@@ -0,0 +1,52 @@
+---
+title: Autoregressive Runtime
+kind: module
+status: draft
+owners:
+  - "@tzhouam"
+  - "@yinpeiqi"
+  - "@fake0fan"
+  - "@Sy0307"
+primary_code_paths:
+  - vllm_omni/core/**
+  - vllm_omni/worker/**
+related_code_paths:
+  - vllm_omni/model_executor/**
+depends_on:
+  - engine_orchestration.md
+  - model_integration.md
+  - input_output_modality_contracts.md
+validation_paths:
+  - tests/core/**
+  - tests/worker/**
+upstream_refs:
+  - vllm.v1.core
+  - vllm.v1.worker
+last_reviewed: 2026-07-16
+---
+
+# Autoregressive runtime
+
+The AR runtime extends vLLM scheduling and worker execution for omni-stage
+inputs and outputs while preserving vLLM scheduling and cache semantics.
+
+## Candidate invariants
+
+### AR-INV-001: vLLM owns base scheduling semantics
+
+**Rule:** Omni schedulers MUST preserve upstream request-state and cache
+transitions unless an Omni-specific difference is documented and tested.
+
+### AR-INV-002: Omni data crosses explicit adapters
+
+**Rule:** Modality-specific stage data MUST be converted at an input or output
+adapter, not injected through unrelated scheduler state.
+
+### AR-INV-003: Workers execute assigned work
+
+**Rule:** Workers and model runners MUST NOT implement cross-stage routing.
+
+## Safe-change guide
+
+Test request lifecycle, abort, cache state, and every affected worker execution
+mode against the supported upstream vLLM contract.
@@ -0,0 +1,18 @@
+# Archived module design pages
+
+This directory preserves module-design pages that predate the structure
+proposed in [issue #5137](https://github.com/vllm-project/vllm-omni/issues/5137).
+They are historical snapshots, not active architectural contracts, and are
+intentionally excluded from the documentation navigation.
+
+| Archived page | Active destination |
+| --- | --- |
+| [AR Module](ar_module.md) | [Autoregressive Runtime](../ar_runtime.md) |
+| [AsyncOmni Architecture](async_omni_architecture.md) | [Engine Orchestration](../engine_orchestration.md) and [Stage Runtime](../stage_runtime.md) |
+| [DiT Module](dit_module.md) | [Diffusion module overview](../diffusion/index.md) |
+| [Entrypoint Module](entrypoint_module.md) | [Entrypoints and Serving Boundaries](../entrypoints.md) |
+
+Do not update these pages with current design guidance. Move still-useful
+material into the active destination and review it against current code and
+tests instead. No compatibility redirect pages are provided at the former
+paths.
@@ -12,7 +12,7 @@ The AR module of vLLM-Omni extends vLLM's core components to support:
 - **Hidden state exposure**: Exposing per-request hidden representations for downstream stages
 - **Basic generator support**: Support some basic heterogeneous architecture such as Convolution, LSTM, etc.

-As shown in the [end2end example](../../user_guide/examples/offline_inference/qwen3_omni.md), AR module can be widely applied across multiple stages, generating text tokens in thinker(AR), audio latent tokens in talker(AR) and audio wave in code2wav(Convolution).
+As shown in the [end2end example](../../../user_guide/examples/offline_inference/qwen3_omni.md), AR module can be widely applied across multiple stages, generating text tokens in thinker(AR), audio latent tokens in talker(AR) and audio wave in code2wav(Convolution).

 ## 2. Relationship with vLLM

@@ -252,8 +252,7 @@ while True:
  batch of compatible independent requests for one full pipeline forward when
  the pipeline supports request-level batching; the step-wise path can also
  admit or remove compatible requests between denoise steps. See
-  [Request-Level Batching](../feature/diffusion_request_level_batching.md) and
-  [Continuous Batching for Step-Wise Diffusion](../feature/diffusion_continuous_batching.md).
+  [Diffusion Continuous Batching](../../feature/diffusion_continuous_batching.md).

 ---

@@ -540,9 +539,9 @@ def get_attn_backend_for_role(role, head_size, attention_config=None, role_categ

 4. **Platform default** — `current_omni_platform.get_diffusion_attn_backend_cls(...)` picks the best-available kernel for the hardware.

-A backend that needs configuration exposes it as a typed field on `AttentionSpec`; `AttentionSpec.backend_kwargs()` serializes it into the `backend_kwargs` the impl consumes. Today the only such field is `skip_softmax` (TRTLLM_ATTN), so e.g. `target_sparsity` can be set without changing model code.
+A backend that needs configuration exposes it as a typed field on `AttentionSpec`; `AttentionSpec.backend_kwargs()` serializes it into the `backend_kwargs` the impl consumes. Today those fields are `skip_softmax` (TRTLLM_ATTN) and `block_sparse` (any block-sparse backend, currently RAINFUSION_ATTN), so e.g. `target_sparsity` or `sparsity` can be set without changing model code.

-For the user-facing CLI surface, see [Diffusion Attention Backends](../../user_guide/diffusion/attention_backends.md). For the role declaration contract on the model side, see [Adding a Diffusion Model](../../contributing/model/adding_diffusion_model.md).
+For the user-facing CLI surface, see [Diffusion Attention Backends](../../../user_guide/diffusion/attention_backends.md). For the role declaration contract on the model side, see [Adding a Diffusion Model](../../../contributing/model/adding_diffusion_model.md).

 **Backend Availability**:

@@ -0,0 +1,47 @@
+---
+title: Benchmarking
+kind: module
+status: draft
+owners:
+  - "@alex-jw-brooks"
+primary_code_paths:
+  - vllm_omni/benchmarks/**
+  - vllm_omni/entrypoints/cli/benchmark/**
+related_code_paths:
+  - benchmarks/**
+depends_on:
+  - entrypoints.md
+  - profiling.md
+validation_paths:
+  - tests/benchmarks/**
+  - tests/dfx/perf/**
+upstream_refs:
+  - vllm.benchmarks
+last_reviewed: 2026-07-16
+---
+
+# Benchmarking
+
+Benchmarking defines reproducible workloads, metrics, warm-up, measurement,
+and result metadata for comparing vLLM-Omni performance.
+
+## Candidate invariants
+
+### BENCH-INV-001: Results are reproducible
+
+**Rule:** Results MUST identify the model, revision, commit, hardware,
+configuration, workload, concurrency, warm-up, and measurement window.
+
+### BENCH-INV-002: Correctness precedes performance
+
+**Rule:** A benchmark MUST validate successful and semantically acceptable
+outputs before using timings as performance evidence.
+
+### BENCH-INV-003: Metric definitions are explicit
+
+**Rule:** Metrics MUST state their unit, population, and aggregation.
+
+## Safe-change guide
+
+Test workload parsing and metric calculations with deterministic fixtures. Do
+not compare materially different workloads or runtime settings.
@@ -0,0 +1,47 @@
+---
+title: Cache Management
+kind: module
+status: draft
+owners:
+  - "@Isotr0py"
+  - "@princepride"
+primary_code_paths:
+  - vllm_omni/diffusion/cache/**
+related_code_paths:
+  - vllm_omni/experimental/ar_diffusion/kv_cache/**
+  - vllm_omni/core/**
+depends_on:
+  - ar_runtime.md
+  - diffusion/diffusion_runtime.md
+validation_paths:
+  - tests/diffusion/cache/**
+upstream_refs:
+  - vllm.v1.core.kv_cache_manager
+last_reviewed: 2026-07-16
+---
+
+# Cache management
+
+Cache management defines reusable state, cache identity, validity, lifecycle,
+and eviction across AR and diffusion execution.
+
+## Candidate invariants
+
+### CACHE-INV-001: Cache identity is complete
+
+**Rule:** A cache key MUST include every input and execution property that can
+change the reused value.
+
+### CACHE-INV-002: Reuse preserves correctness
+
+**Rule:** A cache optimization MUST provide a disabled path and MUST NOT reuse
+state after its validity conditions change.
+
+### CACHE-INV-003: Ownership has an end
+
+**Rule:** Request-scoped and model-scoped cache entries MUST have explicit
+reset, eviction, or teardown behavior.
+
+## Safe-change guide
+
+Test hit, miss, invalidation, disabled, concurrent-request, and teardown paths.
@@ -0,0 +1,51 @@
+---
+title: Diffusion Continuous Batching
+kind: module
+status: draft
+owners:
+  - "@Isotr0py"
+  - "@princepride"
+primary_code_paths:
+  - vllm_omni/diffusion/sched/**
+related_code_paths:
+  - vllm_omni/diffusion/executor/**
+depends_on:
+  - diffusion_runtime.md
+validation_paths:
+  - tests/diffusion/batching/**
+upstream_refs: []
+last_reviewed: 2026-07-16
+---
+
+# Diffusion continuous batching
+
+Continuous batching defines when diffusion requests are compatible, how they
+share an execution step, and how each request advances independently.
+
+## Candidate invariants
+
+### BATCH-INV-001: Compatibility is explicit
+
+**Rule:** Requests MAY share a batch only when all properties required by the
+selected pipeline, scheduler step, shape, precision, and execution features are
+compatible.
+
+### BATCH-INV-002: Per-request state remains isolated
+
+**Rule:** Batch assembly MUST preserve each request's random generator, progress,
+conditioning, outputs, cancellation, and errors.
+
+### BATCH-INV-003: Admission is bounded
+
+**Rule:** Admission MUST respect configured capacity and MUST NOT rely on an
+unbounded waiting or active-request collection.
+
+### BATCH-INV-004: Batch order is not completion order
+
+**Rule:** Output association MUST use stable request identity rather than batch
+position across execution steps.
+
+## Safe-change guide
+
+Test heterogeneous requests, partial completion, cancellation, deterministic
+seeds, capacity limits, and batch sizes of one and many.
@@ -0,0 +1,58 @@
+---
+title: Diffusion Model Integration
+kind: module
+status: draft
+owners:
+  - "@Isotr0py"
+  - "@wtomin"
+  - "@ZJY0516"
+primary_code_paths:
+  - vllm_omni/diffusion/models/**
+  - vllm_omni/diffusion/model_loader/**
+related_code_paths:
+  - vllm_omni/diffusion/layers/**
+  - vllm_omni/diffusion/lora/**
+  - vllm_omni/diffusion/utils/**
+depends_on:
+  - diffusion_runtime.md
+  - ../input_output_modality_contracts.md
+validation_paths:
+  - tests/diffusion/models/**
+  - tests/diffusion/model_loader/**
+  - tests/diffusion/layers/**
+  - tests/diffusion/lora/**
+upstream_refs:
+  - diffusers.DiffusionPipeline
+last_reviewed: 2026-07-16
+---
+
+# Diffusion model integration
+
+Diffusion model integration owns pipeline contracts, registration, checkpoint
+loading, adapters, shared layers, and model-specific processing.
+
+## Candidate invariants
+
+### DIFF-MODEL-INV-001: Pipelines implement one runtime contract
+
+**Rule:** A pipeline MUST declare its supported modalities, configuration,
+inputs, outputs, loading path, and runtime capabilities.
+
+### DIFF-MODEL-INV-002: Registration is the selection boundary
+
+**Rule:** Runtime code MUST select model implementations through the registry or
+loader contract rather than scattered model-name conditionals.
+
+### DIFF-MODEL-INV-003: Model code does not schedule requests
+
+**Rule:** Pipeline code MUST NOT own admission, batching, cancellation, or
+cross-stage routing.
+
+### DIFF-MODEL-INV-004: Shared behavior stays shared
+
+**Rule:** Model directories SHOULD contain only genuine model differences.
+
+## Safe-change guide
+
+Test registry selection, checkpoint loading, minimal inference, input and output
+contracts, and every declared optional capability.
@@ -0,0 +1,58 @@
+---
+title: Diffusion Runtime
+kind: module
+status: draft
+owners:
+  - "@Isotr0py"
+  - "@princepride"
+  - "@SamitHuang"
+primary_code_paths:
+  - vllm_omni/diffusion/sched/**
+  - vllm_omni/diffusion/executor/**
+  - vllm_omni/diffusion/worker/**
+related_code_paths:
+  - vllm_omni/diffusion/hooks/**
+  - vllm_omni/diffusion/postprocess/**
+depends_on:
+  - ../engine_orchestration.md
+  - ../input_output_modality_contracts.md
+  - diffusion_model_integration.md
+validation_paths:
+  - tests/diffusion/diffusion_backend/**
+  - tests/diffusion/hooks/**
+upstream_refs:
+  - diffusers.DiffusionPipeline
+last_reviewed: 2026-07-16
+---
+
+# Diffusion runtime
+
+The diffusion runtime owns request admission, scheduling, execution, progress,
+output, cancellation, and cleanup inside a diffusion stage.
+
+## Candidate invariants
+
+### DIFF-RUNTIME-INV-001: One lifecycle owner
+
+**Rule:** Every admitted request MUST have exactly one scheduler-owned lifecycle
+until completion, cancellation, or failure.
+
+### DIFF-RUNTIME-INV-002: Execution follows scheduler output
+
+**Rule:** Executors and workers MUST execute scheduler decisions without
+admitting, reordering, or forwarding requests independently.
+
+### DIFF-RUNTIME-INV-003: Terminal cleanup is complete
+
+**Rule:** Every terminal path MUST release request state, temporary tensors,
+hooks, and runtime-owned resources.
+
+### DIFF-RUNTIME-INV-004: Optional features use runtime hooks
+
+**Rule:** Cache, profiling, offload, and parallel features SHOULD integrate at
+defined hooks instead of creating another request lifecycle.
+
+## Safe-change guide
+
+Test success, cancellation, failure, shutdown, multiple requests, and resource
+cleanup.
@@ -0,0 +1,42 @@
+---
+title: Diffusion Modules
+kind: module-index
+status: draft
+owners:
+  - "@Isotr0py"
+  - "@princepride"
+  - "@SamitHuang"
+  - "@wtomin"
+  - "@ZJY0516"
+  - "@RuixiangMa"
+  - "@david6666666"
+  - "@xuechendi"
+primary_code_paths:
+  - vllm_omni/diffusion/**
+related_code_paths:
+  - vllm_omni/platforms/**
+depends_on:
+  - ../engine_orchestration.md
+  - ../input_output_modality_contracts.md
+validation_paths:
+  - tests/diffusion/**
+upstream_refs:
+  - diffusers
+last_reviewed: 2026-07-16
+---
+
+# Diffusion modules
+
+Diffusion is a module family containing runtime, model integration, batching,
+distributed execution, and memory-management subsystems.
+
+## Module documents
+
+- [Diffusion runtime](diffusion_runtime.md)
+- [Diffusion model integration](diffusion_model_integration.md)
+- [Continuous batching](continuous_batching.md)
+- [Parallelism](parallelism.md)
+- [Offloader](offloader.md)
+
+Cache, quantization, profiling, and benchmarking follow their top-level module
+documents.
@@ -0,0 +1,53 @@
+---
+title: Diffusion Offloader
+kind: module
+status: draft
+owners:
+  - "@yuanheng-zhao"
+  - "@Isotr0py"
+primary_code_paths:
+  - vllm_omni/diffusion/offloader/**
+related_code_paths:
+  - vllm_omni/diffusion/models/**
+  - vllm_omni/diffusion/worker/**
+depends_on:
+  - diffusion_runtime.md
+  - ../execution_platforms.md
+validation_paths:
+  - tests/diffusion/offloader/**
+upstream_refs:
+  - torch.nn.Module.to
+last_reviewed: 2026-07-16
+---
+
+# Diffusion offloader
+
+The offloader owns component residency, transfer scheduling, memory accounting,
+prefetch, and teardown for diffusion execution.
+
+## Candidate invariants
+
+### OFFLOAD-INV-001: Residency is explicit
+
+**Rule:** Each managed component MUST have one known residency state and one
+owner responsible for transitions.
+
+### OFFLOAD-INV-002: Use follows readiness
+
+**Rule:** Runtime execution MUST NOT consume a component until its transfer to
+the required device is complete.
+
+### OFFLOAD-INV-003: Transfers preserve model state
+
+**Rule:** Offloading MUST preserve parameter and buffer identity, dtype, device,
+and correctness required by the selected execution mode.
+
+### OFFLOAD-INV-004: Memory is bounded and released
+
+**Rule:** Retained host and device copies MUST respect configured limits and
+provide deterministic teardown.
+
+## Safe-change guide
+
+Test enabled and disabled paths, repeated execution, asynchronous transfer,
+memory limits, failure, teardown, and numerical equivalence.
@@ -0,0 +1,56 @@
+---
+title: Diffusion Parallelism
+kind: module
+status: draft
+owners:
+  - "@princepride"
+  - "@yuanheng-zhao"
+  - "@xuechendi"
+primary_code_paths:
+  - vllm_omni/diffusion/distributed/**
+  - vllm_omni/diffusion/attention/parallel/**
+related_code_paths:
+  - vllm_omni/distributed/**
+  - vllm_omni/config/composable_parallel/**
+depends_on:
+  - diffusion_runtime.md
+  - ../vllm_omni_config.md
+validation_paths:
+  - tests/diffusion/distributed/**
+  - tests/diffusion/attention/**
+upstream_refs:
+  - torch.distributed
+last_reviewed: 2026-07-16
+---
+
+# Diffusion parallelism
+
+Parallelism owns diffusion rank topology, process groups, tensor and sequence
+sharding, collectives, and distributed execution.
+
+## Candidate invariants
+
+### PARALLEL-INV-001: Topology has one source of truth
+
+**Rule:** Rank coordinates, group membership, and parallel dimensions MUST be
+derived from validated configuration.
+
+### PARALLEL-INV-002: Sharding contracts are explicit
+
+**Rule:** Every distributed boundary MUST define tensor shape, sharded
+dimension, placement, dtype, producer, and consumer.
+
+### PARALLEL-INV-003: Collectives are symmetric
+
+**Rule:** Process-group members MUST invoke compatible collectives in the same
+logical order.
+
+### PARALLEL-INV-004: Single-rank behavior remains valid
+
+**Rule:** Distributed implementations SHOULD preserve a supported single-rank
+path without distributed initialization.
+
+## Safe-change guide
+
+Test topology validation, single-rank execution, affected parallel modes,
+combined modes, invalid shapes, and orderly teardown.
@@ -0,0 +1,122 @@
+---
+title: Engine Orchestration
+kind: module
+status: draft
+architecture_state: current-plus-in-flight
+owners:
+  - "@tzhouam"
+  - "@fake0fan"
+document_stewards:
+  - "@hsliuustc0106"
+  - "@Gaohan123"
+  - "@david6666666"
+required_reviewers:
+  - "@yinpeiqi"
+  - "@Sy0307"
+  - "@yenuo26"
+  - "@NickCao"
+primary_code_paths:
+  - vllm_omni/engine/async_omni_engine.py
+  - vllm_omni/engine/async_engine_utils.py
+  - vllm_omni/engine/orchestrator.py
+  - vllm_omni/engine/cfg_companion_tracker.py
+  - vllm_omni/engine/rpc_result_router.py
+related_code_paths:
+  - vllm_omni/engine/messages.py
+  - vllm_omni/engine/orchestrator_monitor.py
+  - vllm_omni/engine/stage_client.py
+  - vllm_omni/engine/stage_pool.py
+  - vllm_omni/outputs/output_processor.py
+  - vllm_omni/distributed/omni_coordinator/**
+depends_on:
+  - input_output_modality_contracts.md
+  - error_contracts.md
+  - stage_runtime.md
+  - omni_connector.md
+validation_paths:
+  - tests/engine/test_async_omni_engine_input.py
+  - tests/engine/test_async_omni_engine_outputs.py
+  - tests/engine/test_async_omni_engine_abort.py
+  - tests/engine/test_async_omni_engine_stage_init.py
+  - tests/engine/test_orchestrator.py
+  - tests/engine/test_orchestrator_error_handling.py
+  - tests/engine/test_orchestrator_stage_input_bridge.py
+  - tests/engine/test_cfg_companion_lifecycle.py
+  - tests/engine/test_rpc_result_router.py
+  - tests/e2e/features/fullduplex/engine/**
+upstream_refs:
+  - vllm.v1.engine.EngineCoreRequest
+  - vllm.v1.engine.EngineCoreOutputs
+  - vllm.v1.engine.exceptions.EngineDeadError
+  - vllm.outputs.RequestOutput
+invariant_namespace: ORCH-INV
+last_reviewed: 2026-08-07
+last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
+---
+
+# Engine orchestration
+
+Engine orchestration coordinates configured AR and diffusion stages while
+keeping public clients independent from stage processes and transports.
+
+## Contract status
+
+This document is a draft description of current behavior. It also identifies
+the boundary affected by the in-flight stage client/process refactor in
+[#5441](https://github.com/vllm-project/vllm-omni/pull/5441). Names and
+responsibilities proposed only by that PR are not current contracts.
+
+## Ownership boundary
+
+This document owns `AsyncOmniEngine`, `Orchestrator`, request-state creation,
+cross-stage routing, output ordering, companion tracking, control/RPC
+correlation, cancellation propagation, and terminal-state convergence.
+
+It does not own stage placement, replica selection, stage-process startup,
+payload schemas, public protocol rendering, connector transport, or semantic
+error classification. Those responsibilities belong to the stage runtime,
+I/O, entrypoint, connector, and error contracts.
+
+## Candidate invariants
+
+These identifiers are proposals while the document is `draft`.
+
+### ORCH-INV-001: The orchestrator owns cross-stage routing
+
+**Rule:** Entrypoints and stage clients MUST NOT independently forward a
+request to a downstream logical stage.
+
+### ORCH-INV-002: Stage clients do not own routing policy
+
+**Rule:** Stage clients MUST implement communication and lifecycle operations
+without selecting the next logical stage.
+
+### ORCH-INV-100: Terminal state is monotonic
+
+**Rule:** Once a request reaches a terminal state, orchestration MUST NOT
+forward new work for that request.
+
+## Invariant namespace
+
+`ORCH-INV` reserves `001-099` for dependency direction, `100-199` for
+request state and ordering, `200-299` for failure/cancellation/cleanup, and
+`300-399` for extension points, shutdown ordering, and upstream alignment.
+Numbers become append-only after normative promotion.
+
+## Safe-change guide
+
+Test routing, output ordering, queue correlation, cancellation, failure
+propagation, shutdown ordering, and representative multi-stage execution.
+Changes that cross into stage runtime or public error behavior require review
+from that contract's owners.
+
+## Promotion gate
+
+- Reconcile current names and boundaries after #5441 merges, closes, or is
+  replaced.
+- Demonstrate one terminal outcome per request across success, abort,
+  request-scoped failure, and fatal engine failure.
+- Verify queue correlation and orchestrator-before-runtime shutdown ordering
+  in the cited validation paths.
+- Obtain approval from a technical owner and an independent validation
+  reviewer.
@@ -0,0 +1,134 @@
+---
+title: Entrypoints and Serving Boundaries
+kind: module
+status: draft
+architecture_state: current-plus-in-flight
+owners:
+  - "@alex-jw-brooks"
+  - "@linyueqian"
+document_stewards:
+  - "@hsliuustc0106"
+  - "@Gaohan123"
+  - "@david6666666"
+required_reviewers:
+  - "@tzhouam"
+  - "@herotai214"
+  - "@yenuo26"
+  - "@NickCao"
+primary_code_paths:
+  - vllm_omni/entrypoints/omni.py
+  - vllm_omni/entrypoints/async_omni.py
+  - vllm_omni/entrypoints/omni_base.py
+  - vllm_omni/entrypoints/cli/**
+  - vllm_omni/entrypoints/openai/**
+  - vllm_omni/entrypoints/openpi/**
+  - vllm_omni/entrypoints/client_request_state.py
+  - vllm_omni/entrypoints/stage_utils.py
+  - vllm_omni/entrypoints/utils.py
+primary_path_exceptions:
+  - path: vllm_omni/entrypoints/openai/errors.py
+    owner: error_contracts.md
+related_code_paths:
+  - vllm_omni/errors.py
+  - vllm_omni/inputs/**
+  - vllm_omni/outputs/**
+  - vllm_omni/engine/async_omni_engine.py
+  - vllm_omni/config/**
+  - vllm_omni/deploy/**
+depends_on:
+  - input_output_modality_contracts.md
+  - error_contracts.md
+  - engine_orchestration.md
+validation_paths:
+  - tests/entrypoints/test_omni_entrypoints.py
+  - tests/entrypoints/test_async_omni.py
+  - tests/entrypoints/test_async_omni_duplex.py
+  - tests/entrypoints/test_serve.py
+  - tests/entrypoints/test_stream_finish_reason.py
+  - tests/entrypoints/openai/**
+  - tests/entrypoints/openai_api/**
+  - tests/e2e/online_serving/**
+upstream_refs:
+  - vllm.engine.protocol.EngineClient
+  - vllm.entrypoints.openai/**
+  - vllm.entrypoints.serve/**
+  - vllm.renderers/**
+  - vllm.v1.engine.exceptions/**
+invariant_namespace: ENTRY-INV
+last_reviewed: 2026-08-07
+last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
+---
+
+# Entrypoints and serving boundaries
+
+Entrypoints translate offline, CLI, and serving requests into stable engine
+operations and translate engine outputs into public responses.
+
+## Contract status
+
+This document describes current entrypoint responsibilities plus the boundary
+under review in the open roadmap
+[#5227](https://github.com/vllm-project/vllm-omni/issues/5227) and helper-move
+PR [#5453](https://github.com/vllm-project/vllm-omni/pull/5453). In-flight
+helper locations are not treated as current paths.
+
+## Ownership boundary
+
+This document owns offline API semantics, CLI and serve composition,
+supported OpenAI-compatible routes, request validation and normalization,
+response conversion, streaming/session behavior, and engine handoff.
+
+It does not own configuration precedence, cross-stage routing, stage
+placement, payload implementation, or semantic error classification.
+`entrypoints/openai/errors.py` is an explicit primary-path exception owned by
+`error_contracts.md`.
+
+## Candidate invariants
+
+These identifiers are proposals while the document is `draft`.
+
+### ENTRY-INV-001: Entrypoints adapt but do not orchestrate
+
+**Rule:** Entrypoints MUST NOT implement cross-stage routing or stage lifecycle
+policy.
+
+### ENTRY-INV-100: Public requests are normalized once
+
+**Rule:** Public protocol values MUST be validated and converted to internal
+request contracts before engine submission.
+
+### ENTRY-INV-101: Streaming preserves request identity
+
+**Rule:** Every streamed response MUST remain associated with the request and
+output modality that produced it.
+
+The reviewer-proposed rule that model-specific behavior should stay behind a
+common adapter or processor abstraction remains an unnumbered candidate until
+the entrypoint refactor makes that boundary enforceable.
+
+## Invariant namespace
+
+`ENTRY-INV` reserves `001-099` for boundary and dependency direction,
+`100-199` for normalization, sessions, and streaming, `200-299` for
+disconnect, cancellation, rendering, and cleanup, and `300-399` for upstream
+route and renderer compatibility. Numbers become append-only after normative
+promotion.
+
+## Safe-change guide
+
+Test request validation, protocol conversion, model-adapter routing,
+streaming, session identity, disconnect/cancellation, and error mapping for
+each affected offline or serving entrypoint.
+
+## Promotion gate
+
+- Reconcile the page after #5227's P0 ownership work and #5453's final
+  disposition.
+- Publish a supported route/transport matrix rather than claiming blanket
+  OpenAI parity.
+- Verify one normalization and handoff path per route family and one terminal
+  outcome per streaming transport.
+- Promote the `api_server.py` composition-root rule only after helper moves and
+  import boundaries enforce it.
+- Obtain approval from a technical owner and an independent validation
+  reviewer.
@@ -0,0 +1,109 @@
+---
+title: Error Classification, Propagation, and Rendering
+kind: module
+status: draft
+architecture_state: current-plus-rfc
+owners:
+  - "@alex-jw-brooks"
+  - "@NickCao"
+document_stewards:
+  - "@hsliuustc0106"
+  - "@Gaohan123"
+  - "@david6666666"
+required_reviewers:
+  - "@yinpeiqi"
+  - "@tzhouam"
+  - "@yenuo26"
+primary_code_paths:
+  - vllm_omni/errors.py
+  - vllm_omni/entrypoints/openai/errors.py
+related_code_paths:
+  - vllm_omni/engine/messages.py
+  - vllm_omni/engine/orchestrator.py
+  - vllm_omni/entrypoints/omni_base.py
+  - vllm_omni/entrypoints/async_omni.py
+  - vllm_omni/entrypoints/openai/api_server.py
+  - vllm_omni/entrypoints/openai/serving_chat.py
+  - vllm_omni/entrypoints/openai/serving_speech.py
+  - vllm_omni/entrypoints/openai/serving_audio_generate.py
+  - vllm_omni/entrypoints/openai/protocol/images.py
+  - vllm_omni/entrypoints/openai/protocol/videos.py
+depends_on: []
+validation_paths:
+  - tests/engine/test_orchestrator_error_handling.py
+  - tests/engine/test_async_omni_engine_outputs.py
+  - tests/entrypoints/test_omni_entrypoints.py
+  - tests/entrypoints/test_stream_finish_reason.py
+  - tests/entrypoints/openai_api/test_image_server.py
+  - tests/entrypoints/openai_api/test_video_server.py
+  - tests/entrypoints/openai_api/test_serving_audio_generate.py
+  - tests/entrypoints/openai_api/test_serving_speech.py
+  - tests/metrics/test_prometheus.py
+upstream_refs:
+  - vllm.v1.engine.exceptions.EngineDeadError
+  - vllm.v1.engine.exceptions.EngineGenerateError
+  - vllm.entrypoints.serve.utils.error_response
+  - vLLM RFC #48227
+  - vLLM PR #49665
+invariant_namespace: ERR-INV
+last_reviewed: 2026-08-07
+last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
+---
+
+# Error classification, propagation, and rendering
+
+This contract separates the meaning of a failure from the transport used to
+carry it and the public protocol used to render it.
+
+## Contract status
+
+Current behavior is distributed across engine, message, and entrypoint code.
+The semantic hierarchy proposed by
+[#5570](https://github.com/vllm-project/vllm-omni/issues/5570) remains an open
+RFC. Proposed types and wire records from that RFC are not current contracts.
+
+## Ownership boundary
+
+This document owns semantic classification, fatal versus request-scoped
+failure, cross-process error context, compatibility policy, sanitization, and
+offline, HTTP, SSE, WebSocket, and asynchronous-job rendering rules.
+
+`engine/messages.py` remains primarily owned by the I/O contract; this page
+owns the semantics of its error fields. Route and serving files remain
+primarily owned by `entrypoints.md`; this page owns their rendering
+constraints.
+
+## Candidate invariants
+
+The following topics require owner review before receiving stable IDs:
+
+- A request-scoped failure terminates only the affected request unless the
+  underlying engine is no longer usable.
+- Every public rendering path preserves a stable classification while
+  removing sensitive internal context.
+- Cancellation remains distinguishable from internal failure across process
+  and protocol boundaries.
+
+## Invariant namespace
+
+`ERR-INV` reserves `001-099` for classification ownership, `100-199` for wire
+context and compatibility, `200-299` for fatality, termination, sanitization,
+and rendering, and `300-399` for upstream alignment and transport extension
+points. Numbers become append-only after normative promotion.
+
+## Safe-change guide
+
+Exercise exception classification, stage wire round trips, fatal and
+request-scoped propagation, cancellation, sanitization, offline exceptions,
+HTTP status/body consistency, streaming terminal events, asynchronous-job
+persistence, and metrics classification.
+
+## Promotion gate
+
+- Resolve #5570's hierarchy, wire record, sanitization, and mixed-version
+  questions.
+- Add table-driven exception/status/type tests and stage wire round trips.
+- Verify all public transports terminate once and do not leak sensitive
+  payload data.
+- Obtain approval from both technical owners and an independent validation
+  reviewer.
@@ -0,0 +1,48 @@
+---
+title: Execution Platforms
+kind: module
+status: draft
+owners:
+  - "@gcanlin"
+  - "@tjtanaa"
+  - "@xuechendi"
+primary_code_paths:
+  - vllm_omni/platforms/**
+related_code_paths:
+  - vllm_omni/attention/**
+depends_on:
+  - ar_runtime.md
+  - diffusion/index.md
+validation_paths:
+  - tests/platforms/**
+upstream_refs:
+  - vllm.platforms
+last_reviewed: 2026-07-16
+---
+
+# Execution platforms
+
+Execution platforms isolate hardware-specific capability detection, worker
+selection, patches, kernels, and configuration adjustments.
+
+## Candidate invariants
+
+### PLATFORM-INV-001: Capabilities are explicit
+
+**Rule:** Hardware-dependent behavior MUST be guarded by platform selection or
+capability detection and MUST fail clearly when unsupported.
+
+### PLATFORM-INV-002: Portable code remains portable
+
+**Rule:** Platform-neutral modules MUST NOT import a vendor implementation
+directly when a common selection interface exists.
+
+### PLATFORM-INV-003: Overrides are minimal
+
+**Rule:** A platform implementation SHOULD override only behavior that differs
+from the common or upstream implementation.
+
+## Safe-change guide
+
+Validate common import paths without the target accelerator and run focused
+tests in a fresh `uv` environment on every affected device platform.
@@ -0,0 +1,133 @@
+---
+title: Input, Output, and Modality Contracts
+kind: module
+status: draft
+architecture_state: current
+owners:
+  - "@Sy0307"
+  - "@amy-why-3459"
+document_stewards:
+  - "@hsliuustc0106"
+  - "@Gaohan123"
+  - "@david6666666"
+required_reviewers:
+  - "@tzhouam"
+  - "@linyueqian"
+  - "@david6666666"
+  - "@yenuo26"
+  - "@NickCao"
+primary_code_paths:
+  - vllm_omni/inputs/**
+  - vllm_omni/outputs/**
+  - vllm_omni/request.py
+  - vllm_omni/data_entry_keys.py
+  - vllm_omni/engine/messages.py
+  - vllm_omni/engine/serialization.py
+  - vllm_omni/engine/mm_outputs.py
+  - vllm_omni/engine/output_modality.py
+  - vllm_omni/engine/output_processor.py
+related_code_paths:
+  - vllm_omni/model_executor/stage_input_processors/**
+  - vllm_omni/core/sched/**
+  - vllm_omni/distributed/omni_connectors/**
+  - vllm_omni/diffusion/request.py
+  - vllm_omni/diffusion/worker/request_batch.py
+  - vllm_omni/errors.py
+depends_on:
+  - error_contracts.md
+validation_paths:
+  - tests/inputs/**
+  - tests/engine/test_data_entry_keys.py
+  - tests/engine/test_omni_request.py
+  - tests/engine/test_omni_request_output.py
+  - tests/engine/test_multimodal_accumulation.py
+  - tests/engine/test_output_modality.py
+  - tests/engine/test_output_processor.py
+  - tests/engine/test_wire_multimodal_output.py
+  - tests/utils/test_mm_outputs.py
+  - tests/utils/test_mm_outputs_partition.py
+  - tests/diffusion/test_diffusion_output_metadata.py
+upstream_refs:
+  - vllm.inputs/**
+  - vllm.outputs/**
+  - vllm.v1.request.Request
+  - vllm.v1.engine.EngineCoreRequest
+  - vllm.v1.engine.EngineCoreOutput
+  - vllm.v1.engine.output_processor.OutputProcessor
+invariant_namespace: IO-INV
+last_reviewed: 2026-08-07
+last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
+---
+
+# Input, output, and modality contracts
+
+These contracts define the data that may cross entrypoint, orchestration,
+stage, connector, and model boundaries.
+
+## Contract status
+
+This document is a draft description of the current request, message,
+serialization, modality, accumulation, and output types. It does not assign
+semantic error ownership to the wire schema.
+
+## Ownership boundary
+
+This document owns request identity, prompt/input types, modality keys and
+metadata, queue and wire message schemas, serialization, output types,
+accumulation, completion semantics, and compatibility shims.
+
+It does not own route-specific validation or rendering, model-specific
+interpretation, connector transport mechanics, scheduling policy, or semantic
+error classification. The `ErrorMessage` schema belongs here; the meaning and
+public rendering of its error fields belong to `error_contracts.md`.
+
+## Candidate invariants
+
+These identifiers are proposals while the document is `draft`.
+
+### IO-INV-001: Boundary data has an explicit modality
+
+**Rule:** Data crossing a module or stage boundary MUST identify its modality
+and use the corresponding validated contract.
+
+### IO-INV-100: Request identity is stable
+
+**Rule:** Request identity MUST be preserved across conversions, stages,
+streaming updates, cancellation, and errors.
+
+### IO-INV-101: Completion is explicit
+
+**Rule:** Producers MUST distinguish partial updates from the terminal update
+for every output modality.
+
+### IO-INV-300: Internal objects do not leak into public protocols
+
+**Rule:** Entrypoints MUST explicitly translate internal output objects into
+public response types.
+
+## Invariant namespace
+
+`IO-INV` reserves `001-099` for schema ownership and dependency direction,
+`100-199` for identity, modality, accumulation, ordering, and completion,
+`200-299` for serialization failure and payload cleanup, and `300-399` for
+upstream extension, deprecation, and compatibility. Numbers become
+append-only after normative promotion.
+
+## Safe-change guide
+
+Test construction, validation, serialization, round trips, streaming,
+accumulation, completion, optional fields, unknown fields, and compatibility
+at every affected producer-consumer boundary.
+
+## Promotion gate
+
+- Add explicit CODEOWNERS for the primary I/O paths or confirm that fallback
+  ownership is intentional.
+- Document the compatibility window for deprecated `vllm_omni.engine`
+  I/O-related imports.
+- Verify wire round trips and rejection of unknown or incompatible schema
+  fields.
+- Define payload-versus-metadata rules and completion semantics for streaming
+  and batched outputs.
+- Obtain approval from a technical owner and independent AR, diffusion,
+  connector, entrypoint, and validation reviewers as applicable.
@@ -0,0 +1,51 @@
+---
+title: Model Integration
+kind: module
+status: draft
+owners:
+  - "@tzhouam"
+  - "@gcanlin"
+primary_code_paths:
+  - vllm_omni/model_executor/**
+  - vllm_omni/model_extras/**
+  - vllm_omni/plugins/**
+related_code_paths:
+  - vllm_omni/transformers_utils/**
+  - vllm_omni/tokenizers/**
+depends_on:
+  - input_output_modality_contracts.md
+  - ar_runtime.md
+validation_paths:
+  - tests/model_executor/**
+  - tests/model_extras/**
+  - tests/plugin/**
+upstream_refs:
+  - vllm.model_executor
+last_reviewed: 2026-07-16
+---
+
+# Model integration
+
+Model integration adapts model-specific configuration, preprocessing, loading,
+and execution behavior to stable runtime contracts.
+
+## Candidate invariants
+
+### MODEL-INV-001: Registration is explicit
+
+**Rule:** A model integration MUST declare how its model class, loader, input
+processor, and stage configuration are selected.
+
+### MODEL-INV-002: Model code does not route stages
+
+**Rule:** Model-specific code MUST NOT select or invoke a downstream omni stage.
+
+### MODEL-INV-003: Upstream behavior is reused deliberately
+
+**Rule:** An upstream vLLM implementation SHOULD be reused when its contract is
+sufficient; an override MUST document the behavioral difference.
+
+## Safe-change guide
+
+Test registration, checkpoint loading, input conversion, and representative
+model execution. Test shared utilities against more than one integration.
@@ -0,0 +1,46 @@
+---
+title: Observability
+kind: module
+status: draft
+owners:
+  - "@lishunyang12"
+primary_code_paths:
+  - vllm_omni/metrics/**
+related_code_paths:
+  - vllm_omni/engine/**
+  - vllm_omni/entrypoints/**
+depends_on:
+  - engine_orchestration.md
+validation_paths:
+  - tests/metrics/**
+upstream_refs:
+  - vllm.v1.metrics
+last_reviewed: 2026-07-16
+---
+
+# Observability
+
+Observability defines stable metrics, logs, and correlation fields for tracing a
+request across entrypoints, orchestration, stages, and workers.
+
+## Candidate invariants
+
+### OBS-INV-001: Request correlation survives stage boundaries
+
+**Rule:** Logs and metrics for the same request SHOULD carry a stable request
+identifier and stage identity.
+
+### OBS-INV-002: Metric meaning is stable
+
+**Rule:** A metric's unit, labels, aggregation, and lifecycle MUST be documented;
+changing any of them is a contract change.
+
+### OBS-INV-003: Labels are bounded
+
+**Rule:** Metrics MUST NOT use request IDs, prompts, outputs, or other unbounded
+values as labels.
+
+## Safe-change guide
+
+Test registration, label sets, units, lifecycle resets, and representative
+multi-stage emission.
@@ -0,0 +1,50 @@
+---
+title: OmniConnector
+kind: module
+status: draft
+owners:
+  - "@princepride"
+  - "@yuanheng-zhao"
+  - "@xuechendi"
+  - "@natureofnature"
+  - "@fake0fan"
+primary_code_paths:
+  - vllm_omni/distributed/omni_connectors/**
+related_code_paths:
+  - vllm_omni/platforms/*/omni_connectors/**
+depends_on:
+  - input_output_modality_contracts.md
+  - vllm_omni_config.md
+validation_paths:
+  - tests/distributed/omni_connectors/**
+upstream_refs:
+  - vllm.distributed
+last_reviewed: 2026-07-16
+---
+
+# OmniConnector
+
+OmniConnector defines model-agnostic transport and synchronization contracts
+for data exchanged across stages, processes, devices, and nodes.
+
+## Candidate invariants
+
+### CONNECTOR-INV-001: Connectors transport but do not route
+
+**Rule:** A connector MUST NOT select the next stage or implement model-specific
+execution policy.
+
+### CONNECTOR-INV-002: Producer and consumer contracts agree
+
+**Rule:** Both ends MUST agree on data identity, shape, dtype, placement,
+ownership, and completion semantics.
+
+### CONNECTOR-INV-003: Resources have deterministic cleanup
+
+**Rule:** Connections, buffers, handles, and background work MUST be released on
+normal completion, cancellation, failure, and shutdown.
+
+## Safe-change guide
+
+Test setup, transfer, synchronization, timeout, cancellation, failure, and
+cleanup across every affected backend.
@@ -0,0 +1,46 @@
+---
+title: Profiling
+kind: module
+status: draft
+owners:
+  - "@gcanlin"
+primary_code_paths:
+  - vllm_omni/profiler/**
+  - vllm_omni/diffusion/profiler/**
+related_code_paths:
+  - vllm_omni/entrypoints/**
+depends_on:
+  - observability.md
+validation_paths:
+  - tests/profile/**
+upstream_refs:
+  - vllm.profiler
+last_reviewed: 2026-07-16
+---
+
+# Profiling
+
+Profiling owns opt-in instrumentation and trace collection used to locate
+runtime bottlenecks without changing model results or execution policy.
+
+## Candidate invariants
+
+### PROFILE-INV-001: Profiling is opt-in
+
+**Rule:** Expensive trace collection MUST be disabled by default and bounded by
+an explicit start and stop lifecycle.
+
+### PROFILE-INV-002: Instrumentation preserves semantics
+
+**Rule:** Profiling MUST NOT change scheduling decisions, generated outputs, or
+resource ownership beyond documented synchronization overhead.
+
+### PROFILE-INV-003: Trace context is sufficient
+
+**Rule:** Trace events SHOULD identify stage, rank, worker, operation, and
+request where available without embedding user payloads.
+
+## Safe-change guide
+
+Test disabled overhead, repeated start and stop, cleanup, multi-rank trace
+naming, and each supported profiler backend.
@@ -0,0 +1,50 @@
+---
+title: Quantization
+kind: module
+status: draft
+owners:
+  - "@david6666666"
+  - "@Isotr0py"
+  - "@lishunyang12"
+primary_code_paths:
+  - vllm_omni/quantization/**
+  - vllm_omni/diffusion/quantization/**
+related_code_paths:
+  - vllm_omni/platforms/*/quant/**
+depends_on:
+  - model_integration.md
+  - diffusion/diffusion_model_integration.md
+validation_paths:
+  - tests/quantization/**
+  - tests/diffusion/quantization/**
+upstream_refs:
+  - vllm.model_executor.layers.quantization
+last_reviewed: 2026-07-16
+---
+
+# Quantization
+
+Quantization owns method registration, checkpoint interpretation, layer
+replacement, and runtime constraints for reduced-precision execution.
+
+## Candidate invariants
+
+### QUANT-INV-001: Checkpoint and runtime methods agree
+
+**Rule:** The selected method MUST validate checkpoint metadata, supported
+layers, device capabilities, and required dependencies before use.
+
+### QUANT-INV-002: Unsupported combinations fail clearly
+
+**Rule:** Quantization MUST NOT silently fall back to a different precision or
+method when a requested combination is unsupported.
+
+### QUANT-INV-003: Numerical validation is required
+
+**Rule:** A new method or layer mapping MUST include correctness comparison
+against a defined reference result.
+
+## Safe-change guide
+
+Test loading, representative layers, metadata, platform guards, and numerical
+tolerances separately from performance.
@@ -0,0 +1,126 @@
+---
+title: Stage Runtime and Replica Lifecycle
+kind: module
+status: draft
+architecture_state: current-plus-in-flight
+owners:
+  - "@tzhouam"
+  - "@fake0fan"
+document_stewards:
+  - "@hsliuustc0106"
+  - "@Gaohan123"
+  - "@david6666666"
+required_reviewers:
+  - "@Sy0307"
+  - "@princepride"
+  - "@chickeyton"
+  - "@yenuo26"
+  - "@NickCao"
+primary_code_paths:
+  - vllm_omni/engine/stage_runtime.py
+  - vllm_omni/engine/stage_pool.py
+  - vllm_omni/engine/stage_client.py
+  - vllm_omni/engine/membership_controller.py
+  - vllm_omni/engine/stage_engine_startup.py
+  - vllm_omni/engine/stage_engine_core_client.py
+  - vllm_omni/engine/stage_engine_core_proc.py
+  - vllm_omni/engine/stage_engine_core_proc_manager.py
+  - vllm_omni/engine/stage_init_utils.py
+related_code_paths:
+  - vllm_omni/distributed/omni_coordinator/**
+  - vllm_omni/distributed/ray_utils/**
+  - vllm_omni/diffusion/inline_stage_diffusion_client.py
+  - vllm_omni/diffusion/stage_diffusion_client.py
+  - vllm_omni/diffusion/stage_diffusion_proc.py
+  - vllm_omni/config/**
+depends_on:
+  - input_output_modality_contracts.md
+  - error_contracts.md
+  - omni_connector.md
+validation_paths:
+  - tests/engine/test_async_omni_engine_stage_init.py
+  - tests/engine/test_membership_controller.py
+  - tests/engine/test_single_stage_mode.py
+  - tests/engine/test_stage_engine_core_client.py
+  - tests/engine/test_stage_engine_core_proc.py
+  - tests/diffusion/test_inline_stage_diffusion_client.py
+  - tests/diffusion/test_stage_diffusion_proc.py
+  - tests/distributed/omni_coordinator/**
+upstream_refs:
+  - vllm.v1.engine.core_client.AsyncMPClient
+  - vllm.v1.engine.core_client.DPLBAsyncMPClient
+  - vllm.v1.engine.core.EngineCoreProc
+  - vllm.v1.engine.utils.CoreEngineProcManager
+  - vllm.v1.engine.coordinator.DPCoordinator
+invariant_namespace: STAGE-INV
+last_reviewed: 2026-08-07
+last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
+---
+
+# Stage runtime and replica lifecycle
+
+The stage runtime turns a logical stage definition into ready local or
+distributed replicas and provides the lifecycle boundary used by the
+orchestrator.
+
+## Contract status
+
+This document describes the current `StageRuntime`, `StagePool`, stage-client,
+and stage-process layout. The unified LLM/diffusion direction in
+[#5441](https://github.com/vllm-project/vllm-omni/pull/5441) remains in flight
+and is not presented as current behavior.
+
+## Ownership boundary
+
+This document owns local and distributed stage placement, startup and
+readiness, one pool per logical stage, replica identity and membership,
+selection and affinity, stage-client/process lifecycle, liveness, draining,
+and shutdown.
+
+It does not own cross-stage request policy, model scheduler policy, connector
+transfer semantics, configuration precedence, or public error rendering.
+
+## Candidate invariants
+
+These identifiers are proposals while the document is `draft`.
+
+### STAGE-INV-001: The runtime owns replica lifecycle
+
+**Rule:** Orchestration MUST acquire stage capacity through the stage runtime
+and MUST NOT construct or retire backend replicas directly.
+
+### STAGE-INV-100: Affinity uses stable replica identity
+
+**Rule:** Follow-up operations for a request with replica affinity MUST resolve
+the same logical replica until the request terminates or the replica is
+declared lost.
+
+### STAGE-INV-200: Shutdown is idempotent
+
+**Rule:** Repeated shutdown or cleanup signals MUST NOT leak stage clients,
+process managers, or membership registrations.
+
+## Invariant namespace
+
+`STAGE-INV` reserves `001-099` for placement and ownership, `100-199` for
+replica identity/readiness/affinity, `200-299` for liveness loss, draining and
+cleanup, and `300-399` for local/distributed and upstream compatibility.
+Numbers become append-only after normative promotion.
+
+## Safe-change guide
+
+Exercise local and distributed startup, readiness, replica selection,
+affinity, membership changes, abort routing, process failure, draining, and
+repeated shutdown. Cross-stage policy changes belong in
+`engine_orchestration.md`.
+
+## Promotion gate
+
+- Reconcile terminology and ownership after #5441 reaches a final state.
+- Add or identify a dedicated StageRuntime/StagePool lifecycle suite; current
+  evidence is distributed across the cited tests.
+- Demonstrate affinity preservation for update, interaction, and abort paths.
+- Demonstrate membership loss and repeated shutdown without leaked clients or
+  processes.
+- Obtain approval from a technical owner and an independent validation
+  reviewer.
@@ -0,0 +1,56 @@
+---
+title: vLLM-Omni Configuration
+kind: module
+status: draft
+architecture_state: deferred-pending-refactor
+ownership_status: provisional
+owners:
+  - "@lishunyang12"
+  - "@alex-jw-brooks"
+primary_code_paths:
+  - vllm_omni/config/**
+  - vllm_omni/deploy/**
+  - vllm_omni/model_executor/stage_configs/**
+related_code_paths:
+  - vllm_omni/platforms/*/stage_configs/**
+depends_on: []
+validation_paths:
+  - tests/config/**
+upstream_refs:
+  - vllm.config
+last_reviewed: 2026-08-07
+last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
+---
+
+# vLLM-Omni configuration
+
+This page intentionally records discovery paths only. Substantive
+configuration ownership and contract work is deferred until the active
+configuration refactoring has settled.
+
+## Deferred contract scope
+
+The current code and tests remain authoritative for parsing, defaults,
+overrides, deployment topology, stage construction, and runtime projection.
+This draft does not define a stable precedence model, environment-variable
+capture rule, canonical configuration object, or invariant namespace.
+
+The PR review proposal to capture environment-derived values in configuration
+objects at initialization is an open design question, not a current
+invariant.
+
+## Safe-change guide while deferred
+
+Changes should trace every affected producer to its runtime consumer and test
+the structured, legacy, CLI, and deployment paths that are actually supported.
+Do not infer a stable contract from this placeholder page.
+
+## Promotion gate
+
+- Finish or stabilize the configuration refactor and identify the canonical
+  runtime configuration representation.
+- Confirm technical owners, primary path exceptions, and validation owners.
+- Document parsing and override precedence, including when environment values
+  are captured.
+- Allocate an invariant namespace only after those behaviors are enforced and
+  owner-approved.
@@ -87,9 +87,8 @@ if __name__ == "__main__":

 !!! info

-    For diffusion request-level batching controls such as `max_num_seqs` and
-    `request_batch_max_wait_ms`, see
-    [Request-Level Batching](../user_guide/diffusion/request_batching.md).
+    For diffusion request batching, step execution, and streaming controls, see
+    [Diffusion Execution Modes](../user_guide/diffusion/execution_modes.md).

 For more usages, please refer to [offline inference](../user_guide/examples/offline_inference/qwen2_5_omni.md)

@@ -44,7 +44,7 @@ th {
 | `LTX2Pipeline` | LTX-2 / LTX-2.3 one-stage T2V and I2V | `Lightricks/LTX-2`, `diffusers/LTX-2.3-Diffusers` | ✅︎ | ✅︎ | | |
 | `LTX2DistilledPipeline` | LTX-2 distilled two-stage T2V and I2V | `rootonchair/LTX-2-19b-distilled` | ✅︎ | ✅︎ | | |
 | `LingBotVideoPipeline` | LingBot-Video dense and MoE T2V | `robbyant/lingbot-video-dense-1.3b`, `robbyant/lingbot-video-moe-30b-a3b` | ✅︎ | | | |
-| `MiniMaxH3Pipeline` | MiniMax H3 T2VA, FL2VA, and Ref2VA | `MiniMaxAI/MiniMax-H3` | ✅︎ | | | |
+| `MiniMaxH3Pipeline` | MiniMax H3 T2VA, first/last-keyframe FL2VA, and mixed-reference Ref2VA | `MiniMaxAI/MiniMax-H3` | ✅︎ | | | |
 | `DreamZeroPipeline` | DreamZero-DROID | `GEAR-Dreams/DreamZero-DROID` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `HeliosPipeline`, `HeliosPyramidPipeline` | Helios | `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled` | ✅︎ | ✅︎ | ✅︎ | |
 | `MagiHumanPipeline` | MagiHuman | `SII-GAIR/daVinci-MagiHuman-Base-1080p` | ✅︎ | ✅︎ | | |
@@ -14,7 +14,7 @@ The full set of backends and their platform defaults is in the **Backend Options

 | Value | Notes |
 |---|---|
-| `TRTLLM_ATTN` | FlashInfer's trtllm-gen FMHA (TensorRT-LLM's generated kernels, vendored by FlashInfer). BF16, GQA native, `head_dim=128`. Datacenter Blackwell only (sm_100 / sm_103). Supports **Skip-Softmax** sparse attention — see below. Requires `flashinfer`. |
+| `TRTLLM_ATTN` | FlashInfer's trtllm-gen FMHA (TensorRT-LLM's generated kernels, vendored by FlashInfer). Dense BF16, GQA native, `head_dim=128`. Datacenter Blackwell only (sm_100 / sm_103). Packed paths can provide `cu_seqlens` directly. Supports optional **Skip-Softmax** sparse attention — see below. Requires `flashinfer`. |
 | `FLASH_ATTN` | Wraps FlashAttention 4 on Blackwell when `flash-attn-4` is installed, then falls back to FlashAttention 3/2. Default on Hopper / Ada / Ampere when a compatible FlashAttention package is installed. |
 | `CUDNN_ATTN` | Pins `sdpa_kernel([CUDNN_ATTENTION])`. Default on Blackwell (sm_10x / sm_12x) with cuDNN ≥ 9.5. Wins on mask-heavy DiTs (HunyuanVideo-1.5: 2× e2e vs SDPA). |
 | `FLASHINFER_ATTN` | Calls FlashInfer's dense `single_prefill_with_kv_cache` directly with `custom_mask` for non-causal masked attention. Used as Blackwell fallback when cuDNN is unavailable. Requires `flashinfer`. |
@@ -23,6 +23,7 @@ The full set of backends and their platform defaults is in the **Backend Options
 | `SAGE_ATTN_3` | Requires `sageattn3` from `SageAttention/sageattention3_blackwell`. CUDA only, intended for Blackwell GPUs, with GQA/MQA requests falling back to PyTorch SDPA. |
 | `FLASH_ATTN_HUB` | FlashAttention 2 from HuggingFace `kernels` library. Useful for train/rollout alignment. |
 | `FLASH_ATTN_3_HUB` | FlashAttention 3 from HuggingFace `kernels` library. CUDA Hopper (sm_90+) only; falls back to `FLASH_ATTN_HUB` on older GPUs. |
+| `RAINFUSION_ATTN` | MindIE-SD **RainFusion** block-sparse video attention — see [below](#rainfusion_attn-backend-and-block-sparse-video-attention). Ascend NPU only; requires `mindiesd`. Delegates to `FLASH_ATTN` for anything that is not a packed video sequence. |


 ## Configuration
@@ -69,8 +70,9 @@ vllm-omni serve <model> \
    --diffusion-attention-config '{"default":{"backend":"FLASH_ATTN"},"per_role":{"cross":{"backend":"TORCH_SDPA"}}}'
 ```

-A backend that needs configuration exposes it as a typed field on the spec. Today the only
-one is `TRTLLM_ATTN`'s Skip-Softmax (see [below](#trtllm_attn-backend-and-skip-softmax)):
+A backend that needs configuration exposes it as a typed field on the spec:
+`TRTLLM_ATTN`'s Skip-Softmax (see [below](#trtllm_attn-backend-and-skip-softmax)) and
+`RAINFUSION_ATTN`'s RainFusion block (see [below](#rainfusion_attn-backend-and-block-sparse-video-attention)):

 ```bash
 --diffusion-attention-config.default.backend TRTLLM_ATTN \
@@ -103,7 +105,7 @@ A plain dict is also accepted and normalized to `AttentionConfig`.

 Auto-route preference, in order:

-1. `TRTLLM_ATTN` — on **datacenter** Blackwell (sm_100 / sm_103) when `flashinfer` is installed, the model's `head_dim` is 128, **and the model is mask-free** (see below)
+1. `TRTLLM_ATTN` — on **datacenter** Blackwell (sm_100 / sm_103) when `flashinfer` is installed, the model's `head_dim` is 128, **and the model declares a compatible packed/mask-free path**
 2. `CUDNN_ATTN` — when cuDNN ≥ 9.5 is available (ships in PyTorch 2.5+ wheels)
 3. `FLASHINFER_ATTN` — when `flashinfer` is installed but cuDNN < 9.5
 4. `FLASH_ATTN` — when `flash-attn` is installed with the Blackwell CUTE kernel
@@ -111,7 +113,7 @@ Auto-route preference, in order:

 `TRTLLM_ATTN` is skipped on workstation Blackwell (sm_120 / sm_121) and for any `head_dim != 128`, so those GPUs keep the `CUDNN_ATTN` route described below.

-`TRTLLM_ATTN` outranks `CUDNN_ATTN` on datacenter Blackwell because it is the only backend that can enable Skip-Softmax, not because its dense kernel is faster — dense, the two are comparable. It cannot honor an attention mask, so the auto-default only applies to pipelines verified mask-free (the Wan family); mask-using pipelines keep `CUDNN_ATTN`. `TRTLLM_ATTN` stays available everywhere via explicit `--diffusion-attention-backend TRTLLM_ATTN` (which raises clearly if a mask is received).
+`TRTLLM_ATTN` outranks `CUDNN_ATTN` on datacenter Blackwell for compatible packed/mask-free pipelines. Workstation Blackwell (sm_120 / sm_121) and pipelines that require attention masks retain their normal fallback.

 The startup log line `Defaulting to diffusion attention backend CUDNN_ATTN (Blackwell sm_120, cuDNN 91002)` confirms the route.

@@ -204,11 +206,80 @@ AttentionConfig(
 )
 ```

-
 **Requirements.** Needs FlashInfer ≥ 0.6.16rc1. Kernel
 availability is arch-dependent: `fp8_e4m3` QK has kernels on both **SM100** (B200) and **SM103**
 (B300); `int8` QK kernels are compiled for **SM100 only**.

+## RAINFUSION_ATTN Backend and Block-Sparse Video Attention
+
+`RAINFUSION_ATTN` runs MindIE-SD's RainFusion (`rf_v2`) kernel on Ascend NPU. It pools each
+128-token block of keys, ranks them per query block, and computes attention against only the
+top-scoring ones. Video tokens are rearranged into `(t, h, w)` order first, so the kept blocks are
+spatiotemporal neighbours rather than arbitrary rows of the packed sequence.
+
+Only the video segment is sparsified. The prefix rows (text, visual conditions, audio) and the
+first-frame blocks are always kept dense, which is why the realized sparsity is lower than the
+nominal `sparsity` you configure. Anything the kernel cannot handle — a warmup denoise step, an
+exempt layer, a sequence with no published video geometry, a video segment under 32 blocks —
+delegates to `FLASH_ATTN`, so a model can select this backend unconditionally.
+
+Enable it through the typed `block_sparse` block on the attention spec, shared by every
+block-sparse backend:
+
+| Key | Valid values | Meaning |
+|---|---|---|
+| `sparsity` | finite, `[0, 1]` | Nominal fraction of key blocks dropped per query block. `0` disables sparsity. Defaults to `0.8`. |
+| `start_step` | `≥ 0` | Keep the first N denoise steps dense. Layout is decided early, so these steps dominate structural fidelity. |
+| `skip_layers` | index selector, e.g. `"0-3,38"` | DiT blocks that always stay dense. |
+
+```bash
+vllm-omni serve MiniMaxAI/MiniMax-H3 \
+  --diffusion-attention-config '{"default": {"backend": "RAINFUSION_ATTN",
+      "block_sparse": {"sparsity": 0.8, "start_step": 0}}}'
+```
+
+Programmatically the same block is a typed `BlockSparseSpec` (values validated at construction):
+
+```python
+from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, BlockSparseSpec
+
+AttentionConfig(
+    default=AttentionSpec(
+        backend="RAINFUSION_ATTN",
+        block_sparse=BlockSparseSpec(sparsity=0.8, start_step=0, skip_layers="0-1"),
+    ),
+)
+```
+
+**Tune in the order `start_step` → `sparsity` → `skip_layers`.** Raise `start_step` first: it is the
+cheapest way to recover structure, because the early high-noise steps decide global layout while the
+later steps only refine texture. Then walk `sparsity` up to your quality bar. Reach for
+`skip_layers` last, once an A/B against dense at the same seed points at specific blocks.
+
+Requires Ascend NPU with `mindiesd`; selecting it on any other platform raises. It is also
+incompatible with ring sequence parallelism, since `rf_v2` needs the whole key sequence to rank
+blocks — use Ulysses SP (`ring_degree=1`).
+
+### Which geometries run sparse
+
+Sparsity is only applied when the video segment is a **multiple of 128 rows**, where the row count
+is `latent_t × (height / 32) × (width / 32)`. Otherwise `rf_v2`'s block mask and the kernel's own
+tiling disagree on the block count, and the block straddling the seam mixes video and prefix rows
+that selection may then drop. Any resolution still runs — an unaligned geometry falls back to dense
+attention and logs `RAINFUSION_ATTN staying dense` with the row count it computed — but it gets no
+speedup, so pick an aligned geometry when you want one.
+
+Alignment is necessary, not sufficient, for good quality. `rf_v2` groups video positions into 8x8
+spatial tiles (two tiles fill one 128-row block), which is what makes a selected block a compact
+patch of the frame. When the latent `h` or `w` is not a multiple of 8, the leftover rows or columns
+are peeled off and appended as a flat run instead of being tiled, so they get pooled with spatially
+distant positions and selection can no longer rank them meaningfully. This is silent, and invisible
+to a `sparsity=0` check, because a fully populated mask does not care how blocks are grouped.
+
+"Multiple of 8" on the latent grid means **width and height that are multiples of 256**. Aligned
+resolutions off that grid still run sparse; they just lose more fidelity at the same `sparsity`,
+which you can buy back with `start_step`.
+
 ## End-to-End Benchmark (BF16, sm_120 RTX Pro 6000 Blackwell)

 Same prompt and seed across runs. `Total generation time` from `text_to_video.py` / `text_to_image.py`.
@@ -184,10 +184,15 @@ class Flux2Transformer2DModel(nn.Module):
 ### How It Works

 Distributed layerwise offloading extends single-GPU layerwise offloading to
-multi-device deployments.  Each DP rank stores only **1/dp_size** of the model
-weights in host memory; full layer weights are reconstructed at runtime via
-**AllGather** on a dedicated communication stream, overlapped with computation
-via a fixed double-buffer scheme.
+multi-device deployments. In the default mode, each rank stores only
+**1/dp_size** of the model weights in host memory; full layer weights are
+reconstructed at runtime via **AllGather** on a dedicated communication stream,
+overlapped with computation via a fixed double-buffer scheme. The
+`--dlo-no-use-allgather` mode instead streams standard-loader rank-local
+weights independently and does not shard weights across ranks.
+
+For the implementation design and compatibility matrix, see
+[Distributed Layerwise Offload](../../design/feature/distributed_layerwise_offload.md).

 **Key features:**
 - **Weight sharding + AllGather**: each rank stores 1/dp_size of weights,
@@ -219,7 +224,7 @@ vllm serve /path/to/model --omni \
  --enable-distributed-layerwise-offload \
  --data-parallel-size 4

-# Without AllGather (each rank loads full weights, no sharding)
+# Without DLO AllGather (each rank streams standard-loader rank-local weights)
 vllm serve /path/to/model --omni \
  --enable-distributed-layerwise-offload \
  --data-parallel-size 4 \
@@ -247,9 +252,9 @@ m = Omni(
 | Flag | Description | Default |
 |------|-------------|---------|
 | `--enable-distributed-layerwise-offload` | Enable DLO | `false` |
-| `--data-parallel-size N` | Number of DP ranks for weight sharding | `1` |
+| `--data-parallel-size N` | Number of DP ranks; DLO weight-sharding group when AllGather is enabled | `1` |
 | `--dlo-use-allgather` | Shard + AllGather (saves CPU, requires concurrent requests) | `true` |
-| `--dlo-no-use-allgather` | Full weights per rank (no sharding, no AllGather) | `false` |
+| `--dlo-no-use-allgather` | Use standard-loader rank-local weights; no DLO sharding or AllGather | `false` |

 ### How model weights are loaded (mmap path)

@@ -303,9 +308,15 @@ request while AllGather synchronizes only weight shards (request-independent).

 ### Limitations
 - Online quantization (FP8) is incompatible with mmap loading — falls back to
-  regular `load_weights()` automatically
- Tensor Parallel is not supported (DLO uses DP-based sharding)
- HSDP + AllGather is rejected (would double-shard weights)
+  regular loading only when the model is outside the DLO+AllGather mmap path;
+  online quantization with DLO+AllGather is rejected. Use
+  `--dlo-no-use-allgather` or disable online quantization.
+- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path because TP-aware
+  weight-loader callbacks are bypassed. The no-AllGather path retains the
+  standard TP loader and is experimental.
+- HSDP + AllGather is rejected because it would double-shard parameters. HSDP
+  with no-AllGather is accepted at configuration level, but has limited
+  end-to-end validation.
 - `num_inference_steps=None` is not allowed in DP multi-concurrency mode

 **Module Discovery**
@@ -338,7 +349,7 @@ OffloadBackend (base class)
 ├── LayerWiseOffloadBackend → uses LayerwiseOffloadHook
 │                          (single-GPU, full weights on host)
 └── DistributedLayerwiseOffloadBackend → uses DistributedLayerwiseOffloadHook
-                                         (multi-GPU, 1/dp_size sharded weights + AllGather)
+                                         (multi-GPU, sharded AllGather or rank-local mode)
 ```

 Factory function `get_offload_backend()` selects the appropriate backend based on
@@ -369,4 +380,10 @@ reasoner/generator components inside the model forward pass.
 **Notes:**
 - Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
 - Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
- Distributed Layerwise Offloading works with any model that supports Layerwise Offloading — no additional model changes required.  See [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md) for usage examples.
+- Distributed Layerwise Offloading reuses the layerwise block topology, but the
+  model and parallelism combination still needs validation. AllGather mode
+  additionally requires a compatible checkpoint/mmap layout; no-AllGather mode
+  retains the standard loader's rank-local tensors. See the
+  [DLO design](../../design/feature/distributed_layerwise_offload.md) and
+  [Cosmos3 DistOffload recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/cosmos3/Cosmos3-DistOffload.md)
+  for usage examples.
@@ -0,0 +1,252 @@
+# Diffusion Execution Modes
+
+vLLM-Omni supports complete-request execution and step-wise diffusion
+execution. Both modes use the same asynchronous engine output stream; the
+configuration controls how work is scheduled and whether intermediate outputs
+are exposed.
+
+## Choose a Mode
+
+| Goal | CLI configuration |
+|---|---|
+| Serial request execution | `--max-num-seqs 1` |
+| Fused request-level batching | `--max-num-seqs N` |
+| Single-request step execution | `--step-execution --max-num-seqs 1` |
+| Step-wise continuous batching | `--step-execution --max-num-seqs N` |
+| Chunked diffusion output | `--diffusion-streaming-output` |
+
+`N` must be greater than `1` to allow batching. The selected pipeline must
+support request-level batching or batched-step execution as appropriate;
+single-request step support alone is not sufficient for step-wise continuous
+batching.
+
+## Request Execution
+
+Request execution is the default when `--step-execution` is omitted. The
+pipeline performs a complete `forward()` for each scheduler wave.
+
+### Serial Requests
+
+Use the conservative serial path with any request-mode pipeline:
+
+```bash
+vllm serve MODEL --omni \
+  --port 8091 \
+  --max-num-seqs 1
+```
+
+### Request-Level Batching
+
+Set `max_num_seqs` above one to combine compatible independent requests into a
+single pipeline forward:
+
+```bash
+vllm serve Qwen/Qwen-Image --omni \
+  --port 8091 \
+  --max-num-seqs 4
+```
+
+For bursty traffic, add a small admission window:
+
+```bash
+vllm serve Qwen/Qwen-Image --omni \
+  --port 8091 \
+  --max-num-seqs 4 \
+  --request-batch-max-wait-ms 20
+```
+
+`--request-batch-max-wait-ms 0` is the default. A nonzero value can improve
+batch formation but adds up to that much latency before a new scheduler wave.
+It has no effect in step mode.
+
+Each prompt remains a separate logical request. Do not submit a top-level list
+as one packed prompt. Concurrent serving requests are batched internally when
+their shapes, guidance settings, output counts, and LoRA settings are
+compatible.
+
+Only pipelines that declare request-batch support accept
+`max_num_seqs>1` in request mode. Unsupported pipelines fail during engine
+initialization; use `max_num_seqs=1` instead.
+
+## Step Execution
+
+Step execution lets the scheduler advance and abort requests between denoise
+steps:
+
+```bash
+vllm serve Qwen/Qwen-Image --omni \
+  --port 8091 \
+  --step-execution \
+  --max-num-seqs 1
+```
+
+Set a larger capacity to allow compatible requests to share denoise waves:
+
+```bash
+vllm serve Qwen/Qwen-Image --omni \
+  --port 8091 \
+  --step-execution \
+  --max-num-seqs 8
+```
+
+Step continuous batching is experimental. Start with `max_num_seqs=1` when
+validating a model or debugging correctness, then increase it for
+multi-request throughput.
+
+Step execution is capability-based, not a generic switch for every diffusion
+model. Qwen-Image supports step-wise continuous batching. HunyuanImage3 also
+supports it, but only when its resolved self-attention backend is `TORCH_SDPA`;
+set `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA` or configure
+`diffusion_attention_config.default.backend=TORCH_SDPA` before using
+`--max-num-seqs >1`. See the
+[HunyuanImage-3.0 recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/HunyuanImage-3.0-Instruct.md)
+for its validated configuration. Helios supports single-request step execution only: use
+`--step-execution --max-num-seqs 1` for Helios. Consult the selected pipeline's
+documentation and source for the latest support status.
+
+## Streaming Output
+
+Use `--diffusion-streaming-output` for a pipeline that can produce intermediate
+diffusion outputs:
+
+```bash
+vllm serve BestWishYsh/Helios-Distilled --omni \
+  --port 8000 \
+  --diffusion-streaming-output
+```
+
+Streaming output requires step execution. If
+`--diffusion-streaming-output` is set without `--step-execution`, the engine
+enables step execution automatically. Model initialization fails if the
+pipeline does not implement step execution.
+
+Chunk-capable pipelines emit intermediate and final outputs through the same
+request stream. Final-only step pipelines emit only the final result even when
+streaming output is enabled.
+
+The normal non-streaming serving path uses the same internal stream but drains
+it and returns only the final output. Users do not need to enable
+`--diffusion-streaming-output` to benefit from the unified engine lifecycle.
+
+## Send Requests
+
+Execution-mode flags configure the server. They do not change the
+OpenAI-compatible client request format. For example, after starting any of
+the non-streaming configurations above:
+
+```bash
+curl -X POST http://localhost:8091/v1/images/generations \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "a cat sitting on a windowsill",
+    "size": "1024x1024",
+    "num_inference_steps": 50,
+    "seed": 42
+  }' | jq -r '.data[0].b64_json' | base64 -d > cat.png
+```
+
+To create a request-level batch, send multiple independent requests
+concurrently. Do not put several prompts into one `prompt` field:
+
+```bash
+for prompt in "a red fox" "a blue bird" "a green frog"; do
+  curl -s -X POST http://localhost:8091/v1/images/generations \
+    -H "Content-Type: application/json" \
+    -d "{\"prompt\":\"${prompt}\",\"size\":\"1024x1024\",\"seed\":42}" \
+    > /dev/null &
+done
+wait
+```
+
+The scheduler may batch these requests when their sampling parameters are
+compatible. See the
+[Image Generation API](../../serving/image_generation_api.md) for response
+formats and additional client examples. Streaming models can use
+model-specific streaming endpoints documented by their serving guide.
+
+## Python API
+
+The Python arguments mirror the CLI flags:
+
+```python
+from vllm_omni import Omni
+
+omni = Omni(
+    model="Qwen/Qwen-Image",
+    max_num_seqs=4,
+    request_batch_max_wait_ms=20.0,
+)
+
+outputs = omni.generate(
+    [
+        "a cup of coffee on a table",
+        "a toy dinosaur on a sandy beach",
+        "a fox waking up in bed and yawning",
+    ]
+)
+```
+
+Each list item passed to `Omni.generate()` becomes an independent logical
+request that the scheduler may batch with compatible items.
+
+For step execution:
+
+```python
+from vllm_omni import Omni
+from vllm_omni.inputs.data import OmniDiffusionSamplingParams
+
+omni = Omni(
+    model="Qwen/Qwen-Image",
+    step_execution=True,
+    max_num_seqs=1,
+)
+
+outputs = omni.generate(
+    "A cat sitting on a windowsill",
+    OmniDiffusionSamplingParams(num_inference_steps=50),
+)
+```
+
+## Stage Configuration
+
+Use the equivalent engine arguments in a deployment YAML:
+
+```yaml
+stage_args:
+  - stage_id: 0
+    stage_type: diffusion
+    engine_args:
+      step_execution: false
+      max_num_seqs: 4
+      request_batch_max_wait_ms: 20
+```
+
+For step execution, set `step_execution: true` and remove
+`request_batch_max_wait_ms`.
+
+## CLI Reference
+
+| Flag | Default | Effect |
+|---|---:|---|
+| `--step-execution` | disabled | Select step-wise scheduling |
+| `--max-num-seqs` | `1` for diffusion stages | Set request- or step-scheduler capacity |
+| `--request-batch-max-wait-ms` | `0` | Wait for burst coalescing in request mode |
+| `--diffusion-streaming-output` | disabled | Expose supported intermediate diffusion outputs and require step execution |
+
+## Limitations and Troubleshooting
+
+- Requests batch only when their compatibility-sensitive parameters match.
+- Different LoRA adapters or scales run in separate batches.
+- FIFO scheduling can cause an incompatible request to block later compatible
+  requests.
+- All diffusion cache backends are unsupported in step mode. KV transfer is
+  supported for newly admitted step requests; some other request-mode extras
+  remain unsupported.
+- If request-mode startup reports that the pipeline does not support batching,
+  use `--max-num-seqs 1`.
+- If step-mode startup mentions `prepare_encode()`, `denoise_step()`,
+  `step_scheduler()`, or `post_decode()`, the pipeline does not implement the
+  required step contract.
+
+For implementation details and model-author guidance, see
+[Diffusion Continuous Batching](../../design/feature/diffusion_continuous_batching.md).
@@ -1,119 +0,0 @@
-# Request-Level Batching
-
-Request-level batching lets diffusion serving combine multiple compatible
-logical requests into one pipeline `forward()` call. Each prompt remains a
-separate request with its own `request_id`, sampling parameters, seed, output,
-error, and abort state. The scheduler decides which requests can run together.
-
-!!! warning "Prompt List Semantics"
-
-    Diffusion request-level batching does not support a top-level packed
-    list-prompt request. Submit multiple prompts as independent requests and let
-    the scheduler batch compatible in-flight requests. Multimodal payloads stay
-    inside a single prompt dict, for example
-    `{"prompt": "...", "multi_modal_data": {"image": image}}`.
-
-## Enablement
-
-Request-level execution is the default diffusion request mode when
-`step_execution=False`. `max_num_seqs` controls the maximum number of compatible
-requests in one scheduler wave:
-
- `max_num_seqs=1` is the serial request path.
- `max_num_seqs>1` allows fused request-level batching for pipelines that
-  declare request-batch support.
-
-Increase `max_num_seqs` above `1` to allow the request scheduler to keep more
-than one compatible request active:
-
-```bash
-vllm serve Qwen/Qwen-Image --omni \
-  --port 8091 \
-  --max-num-seqs 4
-```
-
-For bursty online traffic, you can also set a small admission wait window. This
-lets the engine wait briefly before the first `schedule()` of a new wave so
-nearby compatible requests can arrive and share the same fused forward pass:
-
-```bash
-vllm serve Qwen/Qwen-Image --omni \
-  --port 8091 \
-  --max-num-seqs 4 \
-  --request-batch-max-wait-ms 20
-```
-
-`--request-batch-max-wait-ms 0` is the default and disables admission waiting,
-so there is no added wait latency.
-
-For deploy YAMLs, configure the diffusion stage engine args:
-
-```yaml
-stage_args:
-  - stage_id: 0
-    stage_type: diffusion
-    engine_args:
-      max_num_seqs: 4
-      request_batch_max_wait_ms: 20
-```
-
-## Compatibility
-
-Only pipelines that declare request-batch support can use the fused
-multi-request `pipeline.forward(batch)` path. Pipelines that do not support this
-contract can still use request-level execution with `max_num_seqs=1`, which is
-the serial request path. Setting `max_num_seqs>1` for a pipeline without
-request-batch support fails early during engine initialization.
-
-The scheduler batches only compatible requests. Compatibility is based on
-shape-sensitive and guidance-sensitive sampling fields, including resolution,
-frame count, CFG settings, output count, and LoRA identity. Requests with
-different LoRA adapters or scales are kept in separate batches so the worker
-activates one adapter per batch.
-
-Request-level batching applies when `step_execution=False`. For the separate
-step-wise runtime, see [Step Execution](step_execution.md).
-
-## Tuning
-
- `max_num_seqs` caps the number of active compatible requests in one scheduler
-  wave. `1` means serial request execution; values above `1` enable fused
-  request-level batching when the pipeline supports it.
- `request_batch_max_wait_ms` is an upper bound on extra admission wait before a
-  new wave starts. Keep it small for latency-sensitive serving; values such as
-  `10` to `50` ms are a practical starting range for bursty HTTP ingress.
- `0` disables admission waiting and preserves the lowest first-request latency.
- FIFO ordering is conservative: an incompatible request at the front of the
-  waiting queue can block later compatible requests from joining the current
-  batch.
-
-## Python API
-
-When constructing `Omni`, pass the same engine arguments:
-
-```python
-from vllm_omni.entrypoints.omni import Omni
-
-omni = Omni(
-    model="Qwen/Qwen-Image",
-    max_num_seqs=4,
-    request_batch_max_wait_ms=20.0,
-)
-
-outputs = omni.generate(
-    [
-        "a cup of coffee on a table",
-        "a toy dinosaur on a sandy beach",
-        "a fox waking up in bed and yawning",
-    ]
-)
-```
-
-`Omni.generate([...])` submits each list item as its own logical diffusion
-request. The runtime may batch those requests internally when their sampling
-parameters are compatible.
-
-## For Contributors
-
-For implementation details and model-author guidance, see
-[Request-Level Batching for Diffusion](../../design/feature/diffusion_request_level_batching.md).
@@ -1,100 +0,0 @@
-# Step Execution
-
-Step execution is an opt-in diffusion execution mode enabled with
-`step_execution=True` when constructing `Omni`.
-
-It is not a generic diffusion toggle for every pipeline. Only pipelines that
-implement the stepwise contract support it today.
-
-## Quick Start
-
-### Python API
-
-```python
-from vllm_omni import Omni
-from vllm_omni.inputs.data import OmniDiffusionSamplingParams
-
-omni = Omni(
-    model="Qwen/Qwen-Image",
-    step_execution=True,
-)
-
-outputs = omni.generate(
-    "A cat sitting on a windowsill",
-    OmniDiffusionSamplingParams(
-        num_inference_steps=50,
-    ),
-)
-```
-
-### Serving
-
-```bash
-vllm serve Qwen/Qwen-Image --omni \
-  --port 8091 \
-  --step-execution \
-  --max-num-seqs 8
-```
-
-For serving, `--step-execution` enables the step-wise runtime. Continuous
-batching only becomes relevant when `--max-num-seqs > 1`.
-
-`max_num_seqs` controls the step scheduler capacity:
-
- `max_num_seqs=1` runs one request through the step-wise state machine.
- `max_num_seqs>1` allows compatible active requests to share step waves.
-
-## Supported Pipelines
-
-| Pipeline | Example models | Step execution |
-|----------|----------------|----------------|
-| `QwenImagePipeline` | `Qwen/Qwen-Image`, `Qwen/Qwen-Image-2512` | Yes |
-| All other diffusion pipelines | `QwenImageEditPipeline`, `QwenImageEditPlusPipeline`, `QwenImageLayeredPipeline`, GLM-Image, Wan, Flux, etc. | No |
-
-!!! warning "Experimental continuous batching"
-    When `--step-execution` is enabled and `max_num_seqs > 1` is configured,
-    the step-wise path can batch
-    compatible requests together. This is experimental. Requests with
-    incompatible sampling parameters are intentionally kept in separate batches,
-    and `max_num_seqs=1` remains the conservative single-request step path.
-
-## Current Limitations
-
- Continuous batching under `step_execution` is experimental and only batches
-  compatible requests.
- `cache_backend` is not supported together with step execution.
- Unsupported pipelines fail early during model loading.
- Request-mode extras such as KV transfer are not wired into step mode yet.
- LoRA is supported in step mode, but each batch must use a single adapter:
-  requests with different `lora_request` or `lora_scale` are scheduled into
-  separate batches.
-
-## When To Use It
-
-Use step execution only when you specifically need the pipeline to run through
-its stepwise request state machine. For normal diffusion inference, leave it
-disabled unless your workflow depends on this mode.
-
-For Qwen-Image online serving, the usual progression is:
-
- start with `--step-execution --max-num-seqs 1` if you only need the step-wise path
- increase `--max-num-seqs` after that if you want the experimental compatible-request batching behavior
-
-If you are looking for general diffusion speedups, see
-[Diffusion Features Overview](../diffusion_features.md).
-
-## Troubleshooting
-
-If model loading fails with a message mentioning `prepare_encode()`,
-`denoise_step()`, `step_scheduler()`, and `post_decode()`, the selected
-pipeline does not support step execution.
-
-## For Model Authors
-
-If you want to add step execution support to a new diffusion pipeline, see the
-implementation guide:
-[Diffusion Step Execution Design](../../design/feature/diffusion_step_execution.md).
-
-If you also want that pipeline to participate in the experimental batched
-step-wise path, see:
-[Continuous Batching for Step-Wise Diffusion](../../design/feature/diffusion_continuous_batching.md).
@@ -59,7 +59,7 @@ Memory optimization methods help reduce GPU memory usage, enabling inference on

 | Method | Description | Best For |
 |--------|-------------|----------|
-| **[CPU Offload](diffusion/cpu_offload_diffusion.md)** | Offloads model components to CPU memory | Limited VRAM, large models on consumer GPUs |
+| **[CPU Offload](diffusion/cpu_offload.md)** | Offloads model components to CPU memory | Limited VRAM, large models on consumer GPUs |
 | **[Quantization](quantization/overview.md)** | Reduces transformer stages from BF16 to FP8/INT8/etc. | Limited VRAM, minimal accuracy loss    |
 | **[VAE Parallelism](diffusion/parallelism/vae_parallelism.md)** | Distributes VAE decode work across GPUs | High-resolution generation with reduced VAE memory peak |

@@ -80,12 +80,11 @@ denoise steps.

 | Method | Description | Best For |
 |--------|-------------|----------|
-| **[Request-Level Batching](diffusion/request_batching.md)** | Scheduler batches compatible independent diffusion requests into one pipeline forward pass | Bursty online serving and multi-request throughput |
-| **[Step Execution](diffusion/step_execution.md)** | Per-step denoise execution with mid-request abort support | Request cancellation between denoise steps, fine-grained execution control |
+| **[Diffusion Execution Modes](diffusion/execution_modes.md)** | Configures serial requests, request batching, step execution, continuous batching, and streaming output | Matching latency, throughput, cancellation, and output-delivery requirements |

-**Note:** Request-level batching is available for pipelines that declare the
-request-batch forward contract. Step execution is currently supported by
-QwenImagePipeline only. See [Supported Models](#supported-models) for details.
+**Note:** Request-level batching and step execution are capability-based.
+Consult the execution guide and the selected pipeline's documentation for
+current support.

 ### Quantization Methods

@@ -105,8 +104,8 @@ The following tables show which models support each feature:

 > Notes:

-> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload_diffusion.md).
-> 2. The **💾Quantization** column is collapsed for readability. See [Quantization Overview](quantization/overview.md) for per-method and per-model support details.
+> 1. CPU Offload has two methods: Module-wise (default for models with DiT + text encoder) and Layerwise. The tables below show **Layerwise support** only. Split models like Cosmos3 (no separate text encoder) swap their reasoner/generator components for module-wise offload; see the [CPU Offload Guide](diffusion/cpu_offload.md).
+> 2. The **💾Quantization** column is collapsed for readability. See [Quantization](quantization/overview.md) for per-method and per-model support details.

 ### ImageGen

@@ -160,7 +159,7 @@ The following tables show which models support each feature:
 | **HunyuanVideo-1.5 T2V I2V** |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ❌         |   ✅    |             ✅             |  ✅ (encode/decode)   |       ✅        |        ❌         |
 | **DreamID-Omni**             |     ❌     |     ❌      |           ❌           |       ✅        |         ❌         |         ❌         |   ✅    |             ✅             |          ❌           |       ❌        |        ❌         |
 | **Cosmos3**                  |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ❌         |   ✅    |             ✅             |  ✅ (encode/decode)   |       ✅        |        ❌         |
-
+| **MiniMax-H3**               |     ❌     |     ✅      |           ✅           |       ❌        |       ✅ (DiT/TE)  |         ❌         |   ✅    |             ✅             |       ✅ (tile)       |      ✅ (DiT)      |        ❌         |

 **Frame Interpolation Support**

@@ -206,7 +205,7 @@ The following tables show which models support each feature:
    3. CPU Offloading (Layerwise) and CPU Offloading (Module-wise) are not compatible.
    4. CPU Offloading (Layerwise) supports single-card for now.
    5. Using FP8-Quant as an example of qunatization methods.
-    6. Step Execution is not compatible with cache backends (TeaCache, Cache-DiT). LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).
+    6. Step Execution is not compatible with any diffusion cache backend. LoRA is supported, but each scheduled batch must use a single adapter (requests with different `lora_request` or `lora_scale` are kept in separate batches).


 ## Multi-Thread Weight Loading
@@ -275,9 +274,9 @@ Measured on NVIDIA H800:

 **Memory Optimization:**

- **[CPU Offload Guide](diffusion/cpu_offload_diffusion.md)** - Offload model components to CPU, reduce GPU memory usage
+- **[CPU Offload Guide](diffusion/cpu_offload.md)** - Offload model components to CPU, reduce GPU memory usage
 - **[VAE Parallelism Guide](diffusion/parallelism/vae_parallelism.md)** - Distribute VAE decode work across GPUs for high-resolution images and videos
- **[Quantization Overview](quantization/overview.md)** - Overview of quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models
+- **[Quantization](quantization/overview.md)** - Quantization methods for diffusion, multi-stage omni/TTS, and multi-stage diffusion models

 **Extensions:**

@@ -286,7 +285,7 @@ Measured on NVIDIA H800:

 **Execution Modes:**

- **[Step Execution Guide](diffusion/step_execution.md)** - Per-step denoise execution with mid-request abort support
+- **[Diffusion Execution Modes](diffusion/execution_modes.md)** - Configure request batching, step execution, continuous batching, and streaming output

 **Startup Optimization:**

@@ -184,7 +184,7 @@ if __name__ == "__main__":
 !!! info

    For diffusion request-level batching controls such as `max_num_seqs`, see
-    [Request-Level Batching](../../diffusion/request_batching.md).
+    [Diffusion Execution Modes](../../diffusion/execution_modes.md).

 ### Negative Prompts

@@ -5,6 +5,9 @@ path. The same configuration entrypoint is used across diffusion-only models,
 multi-stage omni/TTS models, and multi-stage diffusion models, but each model
 type has a different quantization scope.

+For the internal architecture and backend extension points, see the
+[quantization design document](../../design/feature/quantization.md).
+
 ## Quantization Modes

 | Mode | Guide | Description | Methods |
@@ -0,0 +1,77 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Offline PersonaPlex inference through the vllm-omni fullduplex backend.
+
+This is the runnable end-to-end proof for the integration: it drives a user WAV
+through :class:`PersonaPlexEngine` + :class:`PersonaPlexSession` (our wrappers
+around the Moshi/Mimi lockstep loop) and writes the agent's audio + inner-monologue
+text. It mirrors ``python -m moshi.offline`` but goes through our ``FrameStepper``
+seam, so a green run validates that the session/adapter plumbing is faithful.
+
+Prereqs: HF access to ``nvidia/personaplex-7b-v1`` (``export HF_TOKEN=...``). The
+stack is moshi-free (native temporal + depformer + streaming Mimi via
+``transformers``); decoding is greedy.
+
+Example:
+    python examples/offline_inference/personaplex/personaplex_offline.py \\
+        --input-wav input_assistant.wav --output-wav out.wav \\
+        --voice-prompt NATF2.pt --persona "You are a wise and friendly teacher."
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+
+import numpy as np
+
+from vllm_omni.experimental.fullduplex.personaplex import (
+    PersonaPlexConfig,
+    PersonaPlexEngine,
+    PersonaPlexSession,
+)
+from vllm_omni.experimental.fullduplex.personaplex.config import DEFAULT_PERSONA
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--input-wav", required=True, help="User-side input WAV")
+    parser.add_argument("--output-wav", required=True, help="Agent audio output WAV")
+    parser.add_argument("--output-text", default=None, help="Optional JSON for agent text")
+    parser.add_argument("--voice-prompt", default="NATF2.pt", help="Voice clone (bundled .pt or .wav)")
+    parser.add_argument("--persona", default=DEFAULT_PERSONA, help="System role text")
+    parser.add_argument("--hf-repo", default="nvidia/personaplex-7b-v1")
+    parser.add_argument("--device", default="cuda")
+    parser.add_argument("--cpu-offload", action="store_true")
+    args = parser.parse_args()
+
+    import sphn  # local import: only the runnable driver needs the audio I/O dep
+
+    config = PersonaPlexConfig(
+        hf_repo=args.hf_repo,
+        voice_prompt=args.voice_prompt,
+        persona=args.persona,
+        device=args.device,
+        cpu_offload=args.cpu_offload,
+    )
+
+    engine = PersonaPlexEngine(config).load()
+    session = PersonaPlexSession(engine, config)
+    session.open()
+
+    # Load exactly like the reference (moshi.lm.load_audio): native read then
+    # explicit resample, so greedy runs stay bit-faithful to `python -m moshi.offline`.
+    user_pcm, src_sr = sphn.read(args.input_wav)
+    user_pcm = sphn.resample(user_pcm, src_sample_rate=src_sr, dst_sample_rate=config.sample_rate)
+    audio, text = session.run(np.asarray(user_pcm[0], dtype=np.float32))
+
+    sphn.write_wav(args.output_wav, audio, config.sample_rate)
+    print(f"wrote {audio.shape[0] / config.sample_rate:.2f}s of agent audio to {args.output_wav}")
+    print(f"agent text: {text!r}")
+    if args.output_text:
+        with open(args.output_text, "w") as f:
+            json.dump({"text": text}, f, ensure_ascii=False)
+
+
+if __name__ == "__main__":
+    main()
@@ -260,7 +260,7 @@ if __name__ == "__main__":
    For diffusion pipelines, the input list is sliced into single-item requests
    before feeding into the diffusion pipeline. For request-level batching
    controls such as `max_num_seqs`, see
-    [Request-Level Batching](../../../docs/user_guide/diffusion/request_batching.md).
+    [Diffusion Execution Modes](../../../docs/user_guide/diffusion/execution_modes.md).

 ### Negative Prompts

@@ -0,0 +1,84 @@
+# PersonaPlex full-duplex online serving
+
+Serve [`nvidia/personaplex-7b-v1`](https://huggingface.co/nvidia/personaplex-7b-v1)
+(a Moshi-based full-duplex speech-to-speech model) with the native vLLM-Omni engine.
+
+The server hosts the **official PersonaPlex web client** at `/` (auto-downloaded from
+the model repo) and implements its WebSocket protocol at `/api/chat`, so you talk to the
+model live in the browser — mic in, agent speech + inner-monologue text out.
+
+> Experimental. Requires a GPU and Hugging Face access to the gated repo
+> (`HF_TOKEN` with access to `nvidia/personaplex-7b-v1`).
+
+## Start the server
+
+```bash
+HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 bash run_server.sh
+# or directly:
+HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \
+    vllm_omni.experimental.fullduplex.personaplex.serving.server --port 8124
+```
+
+## Talk to it
+
+- **Browser (recommended):** open `http://localhost:8124/` and allow the microphone.
+  Use headphones so the agent does not hear itself.
+- **Headless:** stream a 24 kHz mono WAV and save the reply:
+
+  ```bash
+  python duplex_client.py --url ws://localhost:8124/api/chat --input user.wav --out reply.wav
+  ```
+
+## Unified Realtime API
+
+The default `vllm_omni/deploy/personaplex.yaml` also enables main's engine-owned
+full-duplex control plane:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
+  /path/to/personaplex-7b-v1 \
+  --omni \
+  --stage-configs-path vllm_omni/deploy/personaplex.yaml
+```
+
+Validate the actual `/v1/realtime?duplex=1` scheduler path with paced 24 kHz
+PCM, two concurrent sessions, overflow admission, per-session slot recycling,
+and non-silent output:
+
+```bash
+python tests/e2e/online_serving/personaplex_realtime_duplex.py \
+  --model /path/to/personaplex-7b-v1 \
+  --input-wav /path/to/speech.wav \
+  --output-dir /tmp/personaplex-realtime-duplex
+```
+
+This is distinct from the standalone `/api/chat` compatibility server above.
+The unified endpoint advertises `supports_barge_in=false`: overlapping speech
+is native model behavior, but destructive output interruption and model-state
+rewind have not been validated for PersonaPlex.
+
+## Protocol (`/api/chat`)
+
+Binary WebSocket messages, first byte is a tag (identical to `moshi.server`):
+
+| Direction | Tag | Payload |
+| --- | --- | --- |
+| server → client | `\x00` | ready handshake (after the system-prompt prefill) |
+| client → server | `\x01` | Opus-encoded mic audio (24 kHz) |
+| server → client | `\x01` | Opus-encoded agent audio |
+| server → client | `\x02` | UTF-8 inner-monologue text |
+
+Connect with query params `text_prompt` (persona) and `voice_prompt` (voice file).
+A raw-PCM endpoint (`/v1/audio/duplex`, JSON `open` + float32 frames) is also available
+for clients that do not want an Opus dependency.
+
+## Notes
+
+- **Run the client near the server.** Real-time 80 ms audio is sensitive to network
+  latency/jitter; over a high-latency remote link playback can stutter regardless of
+  engine speed. On localhost it is smooth.
+- Greedy decoding by default. One conversation per server by default (a second
+  concurrent connection is rejected with WS close 1013 until the first ends);
+  pass `--batch-size N` to host N concurrent conversations on one engine
+  (elastic batching; connections beyond N are rejected until a slot frees).
+- Full runbook: [`recipes/NVIDIA/PersonaPlex.md`](../../../recipes/NVIDIA/PersonaPlex.md).
@@ -0,0 +1,114 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Headless client for the PersonaPlex duplex server (official Moshi protocol).
+
+Streams a 24 kHz mono WAV to ``/api/chat`` exactly as the browser client does
+(Opus over WebSocket, tags ``\\x00`` ready / ``\\x01`` audio / ``\\x02`` text),
+collects the agent's reply audio + inner-monologue text, and writes a WAV.
+
+The browser UI at ``http://<host>:<port>/`` is the primary demo; this script is
+for headless testing / scripting.
+
+    python duplex_client.py --url ws://localhost:8124/api/chat \
+        --input user.wav --out reply.wav
+
+Deps: ``websockets``, ``sphn``, ``soundfile``, ``numpy``.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+
+import numpy as np
+import soundfile as sf
+import sphn
+
+SR = 24000  # PersonaPlex / Mimi codec rate
+
+
+async def converse(url: str, pcm: np.ndarray) -> tuple[np.ndarray, str]:
+    import websockets
+
+    reader = sphn.OpusStreamReader(SR)  # decode agent audio
+    writer = sphn.OpusStreamWriter(SR)  # encode our mic audio
+    text: list[str] = []
+    ready = asyncio.Event()
+    last = {"t": 0.0}
+
+    async with websockets.connect(url, max_size=None) as ws:
+
+        async def receive() -> None:
+            loop = asyncio.get_event_loop()
+            while True:
+                try:
+                    msg = await ws.recv()
+                except Exception:
+                    return
+                if not isinstance(msg, (bytes, bytearray)) or not msg:
+                    continue
+                tag, payload = msg[0], msg[1:]
+                if tag == 0:
+                    ready.set()
+                elif tag == 1:
+                    reader.append_bytes(bytes(payload))
+                    last["t"] = loop.time()
+                elif tag == 2:
+                    text.append(payload.decode("utf8", errors="replace"))
+
+        recv_task = asyncio.create_task(receive())
+        await asyncio.wait_for(ready.wait(), timeout=120)  # waits for system-prompt prefill
+
+        frame = 1920  # 80 ms
+        for i in range(0, len(pcm), frame):
+            chunk = pcm[i : i + frame]
+            if len(chunk) < frame:
+                chunk = np.concatenate([chunk, np.zeros(frame - len(chunk), dtype=np.float32)])
+            writer.append_pcm(chunk)
+            data = writer.read_bytes()
+            if data:
+                await ws.send(b"\x01" + data)
+            await asyncio.sleep(frame / SR)  # pace at real time
+
+        loop = asyncio.get_event_loop()
+        last["t"] = loop.time()
+        while loop.time() - last["t"] < 2.5:  # drain until the agent stops talking
+            await asyncio.sleep(0.1)
+        recv_task.cancel()
+
+    chunks: list[np.ndarray] = []
+    while True:
+        out = reader.read_pcm()
+        if out is None or out.shape[-1] == 0:
+            break
+        chunks.append(np.asarray(out, dtype=np.float32))
+    audio = np.concatenate(chunks) if chunks else np.zeros(0, dtype=np.float32)
+    return audio, "".join(text)
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", default="ws://localhost:8124/api/chat")
+    ap.add_argument("--input", required=True, help="24 kHz mono WAV of the user turn")
+    ap.add_argument("--out", default="personaplex_reply.wav")
+    ap.add_argument("--tail", type=float, default=10.0, help="trailing silence (s) for the agent reply")
+    ap.add_argument("--persona", default="")
+    ap.add_argument("--voice", default="NATF2.pt")
+    args = ap.parse_args()
+
+    wav, sr = sf.read(args.input, dtype="float32")
+    if wav.ndim > 1:
+        wav = wav.mean(axis=1)
+    if sr != SR:
+        raise SystemExit(f"input must be {SR} Hz mono; got {sr} Hz. Resample it first.")
+    wav = np.concatenate([wav, np.zeros(int(args.tail * SR), dtype=np.float32)])
+
+    url = f"{args.url}?text_prompt={args.persona}&voice_prompt={args.voice}"
+    audio, text = asyncio.run(converse(url, wav))
+    sf.write(args.out, audio if audio.size else np.zeros(SR, dtype=np.float32), SR)
+    print(f"agent reply: {audio.size / SR:.2f}s -> {args.out}")
+    print(f"inner-monologue: {text.strip()!r}")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,15 @@
+#!/bin/bash
+# PersonaPlex full-duplex online serving.
+#
+# Serves the OFFICIAL PersonaPlex web client at / (auto-downloaded from
+# nvidia/personaplex-7b-v1) and the Moshi WebSocket protocol at /api/chat,
+# backed by the native vLLM-Omni engine. Requires HF access to the gated repo.
+
+HOST="${HOST:-0.0.0.0}"
+PORT="${PORT:-8124}"
+
+echo "Starting PersonaPlex duplex server on ${HOST}:${PORT}..."
+echo "Open http://localhost:${PORT}/ for the official web client (run it near the server)."
+
+python -m vllm_omni.experimental.fullduplex.personaplex.serving.server \
+    --host "$HOST" --port "$PORT"
@@ -71,6 +71,14 @@ plugins:
  - search
  - autorefs
  - awesome-nav
+  - redirects:
+      redirect_maps:
+        user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
+        user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
+        user_guide/diffusion/cpu_offload_diffusion.md: user_guide/diffusion/cpu_offload.md
+        user_guide/quantization/index.md: user_guide/quantization/overview.md
+        design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
+        design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
  - glightbox
  - git-revision-date-localized:
      # exclude files
@@ -70,8 +70,7 @@ dev = [
    "FlagEmbedding==1.4.0", # for hunyuan_image3 tests
    "kernels==0.14.1", # for accuracy tests comparison with diffusers output (0.15.0 incompatible with diffusers 0.38.0)
    "s3tokenizer==0.3.0", # for tts test
-    "step-audio2==1.0.0", # Step-Audio2 Token2Wav (flashcosyvoice / hyperpyyaml)
-    "stepaudio2-minicpmo", # for MiniCPM-o 4.5 talker vocoder (stepaudio2-minicpmo Token2wav)
+    "step-audio2==1.0.0", # Step-Audio2 / MiniCPM-o 4.5 Token2Wav (flashcosyvoice / hyperpyyaml)
 ]

 demo = [
@@ -109,6 +108,7 @@ docs = [
    "mkdocstrings-python",
    "mkdocs-gen-files",
    "mkdocs-awesome-nav",
+    "mkdocs-redirects",
    "mkdocs-glightbox",
    "mkdocs-git-revision-date-localized-plugin",
    "mkdocs-minify-plugin",
@@ -250,13 +250,16 @@ markers = [
    "npu: Tests that run on NPU/Ascend (auto-added)",
    "musa: Tests that run on MUSA/Moore Threads (auto-added)",
    # specified computation resources marks (auto-added)
-    "H100: Tests that require H100 GPU",
-    "L4: Tests that require L4 GPU",
-    "MI325: Tests that require MI325 GPU (AMD/ROCm)",
-    "B60: Tests that require Intel Arc Pro B60 XPU",
-    "S5000: Tests that require S5000 GPU (Moore Threads/MUSA)",
-    "A2: Tests that require A2 NPU",
-    "A3: Tests that require A3 NPU",
+    # Tag with [hardware-resource] for tests.helpers.mark.get_hardware_mark_list().
+    "H100: [hardware-resource] Tests that require H100 GPU",
+    "H200: [hardware-resource] Tests that require H200 GPU",
+    "L4: [hardware-resource] Tests that require L4 GPU",
+    "B200: [hardware-resource] Tests that require B200 GPU",
+    "MI325: [hardware-resource] Tests that require MI325 GPU (AMD/ROCm)",
+    "B60: [hardware-resource] Tests that require Intel Arc Pro B60 XPU",
+    "S5000: [hardware-resource] Tests that require S5000 GPU (Moore Threads/MUSA)",
+    "A2: [hardware-resource] Tests that require A2 NPU",
+    "A3: [hardware-resource] Tests that require A3 NPU",
    "distributed_cuda: Tests that require multi cards on CUDA platform",
    "distributed_rocm: Tests that require multi cards on ROCm platform",
    "distributed_xpu: Tests that require multi cards on XPU platform",
@@ -332,6 +332,6 @@ bundled offline CLI do not currently expose `sigmas`.
 - <https://huggingface.co/Lightricks/LTX-2.3>
 - <https://huggingface.co/diffusers/LTX-2.3-Diffusers>
 - [Online video generation](../../docs/user_guide/examples/online_serving/text_to_video.md)
- [Request batching](../../docs/user_guide/diffusion/request_batching.md)
+- [Diffusion execution modes](../../docs/user_guide/diffusion/execution_modes.md)
 - [T2V offline example](../../examples/offline_inference/text_to_video/text_to_video.md)
 - [I2V offline example](../../examples/offline_inference/image_to_video/README.md)
@@ -0,0 +1,61 @@
+# MiniMax-H3 on RTX 5090
+
+This recipe uses BF16 weights, tiled VAE decode, tensor parallelism where a
+second GPU is available, and distributed layerwise offload (DLO). It is a
+memory-first serving configuration; lower resident counts reduce HBM use and
+increase CPU-to-GPU transfer time.
+
+## Capacity requirements
+
+| Resource | One RTX 5090 | Two RTX 5090s |
+| --- | ---: | ---: |
+| GPU HBM | 32 GiB | 32 GiB per GPU |
+| Checkpoint storage | 135 GiB per partition | 135 GiB per partition |
+| Available system RAM | 200 GiB minimum | 200 GiB minimum |
+| Recommended system RAM | 384 GiB | 384 GiB |
+
+`FL2VA` and `Ref2VA` are separate 135 GiB checkpoint partitions. Start one
+server at a time. DLO keeps rank-local weights in pinned host memory; increasing
+`--dlo-resident-layers` improves latency but does **not** reduce host RAM in the
+current implementation because resident layers retain pinned CPU master copies.
+
+> **Modular H3:** after #5720 lands, preserve this recipe's one-partition
+> behavior with `--task-type fl2va` or `--task-type ref2va`.
+
+## One RTX 5090: 1344x768, 5 seconds
+
+Use 12 resident DiT layers. A 50-step B300 allocation test with this exact
+single-rank topology peaked at 26.50 GiB; re-measure peak HBM on the target
+card before increasing the resident count.
+
+```bash
+CUDA_VISIBLE_DEVICES=0 vllm serve /path/to/MiniMax-H3/FL2VA \
+  --omni --trust-remote-code --host 0.0.0.0 --port 8000 \
+  --num-gpus 1 --tensor-parallel-size 1 --text-encoder-tp-size 1 \
+  --usp 1 --ring 1 --vae-patch-parallel-size 1 \
+  --vae-parallel-mode tile --vae-use-tiling \
+  --enable-distributed-layerwise-offload --dlo-no-use-allgather \
+  --dlo-resident-layers 12 --enforce-eager \
+  --diffusion-attention-backend CUDNN_ATTN
+```
+
+## Two RTX 5090s: 1344x768, 5 seconds
+
+Use TP2 and 20 resident DiT layers. The two-rank B300 capacity run peaked at
+27,726 MiB per rank for this shape and 50 steps. This is a memory/correctness
+proxy, not a consumer-GPU latency claim.
+
+```bash
+CUDA_VISIBLE_DEVICES=0,1 vllm serve /path/to/MiniMax-H3/FL2VA \
+  --omni --trust-remote-code --host 0.0.0.0 --port 8000 \
+  --num-gpus 2 --tensor-parallel-size 2 --text-encoder-tp-size 2 \
+  --usp 1 --ring 1 --vae-patch-parallel-size 2 \
+  --vae-parallel-mode tile --vae-use-tiling \
+  --enable-distributed-layerwise-offload --dlo-no-use-allgather \
+  --dlo-resident-layers 20 --enforce-eager \
+  --diffusion-attention-backend CUDNN_ATTN
+```
+
+For Ref2VA, stop the FL2VA server and restart the same command with
+`/path/to/MiniMax-H3/Ref2VA`. Ref2VA reference video count and prompt length
+can increase activation memory; begin with one request at a time.
@@ -0,0 +1,146 @@
+# MiniMax H3 on Moore Threads MUSA
+
+> Joint video and audio generation on Moore Threads GPUs
+
+## Summary
+
+- Vendor: MiniMaxAI
+- Model: [`MiniMaxAI/MiniMax-H3`](https://huggingface.co/MiniMaxAI/MiniMax-H3)
+- Tasks: T2VA, FL2VA, and Ref2VA
+- Mode: OpenAI-compatible `/v1/videos` HTTP serving
+- Hardware: 4x Moore Threads MTT S5000 for the validated Ref2VA profile
+- Maintainer: Community
+
+This recipe adapts [MiniMax-H3.md](MiniMax-H3.md) for MUSA environments.
+The modular H3 pipeline can load both task-specific DiTs, but this MUSA recipe
+uses a **single task partition**: `FL2VA` for `t2va`/`fl2va`, or `Ref2VA` for
+`ref2va`. This avoids loading both DiTs.
+
+## Prerequisites
+
+### Checkpoint
+
+Download the partition required by the selected startup task. The commands below
+run a single-task server; do not use the modular default (combined) mode on MUSA
+until its two-DiT memory and performance profile is validated.
+
+For T2VA and FL2VA:
+
+```bash
+python -m pip install modelscope
+export MODEL_ROOT=/path/to/MiniMax-H3
+modelscope download MiniMax/MiniMax-H3 \
+  --local_dir "${MODEL_ROOT}" \
+  --max-workers 16 \
+  --include 'FL2VA/**'
+```
+
+For Ref2VA, download `Ref2VA/**` instead.
+
+### Environment
+
+Install a compatible PyTorch/torch-musa, torchada, MATE, and vLLM-MUSA
+stack before installing vLLM-Omni. Importing vLLM-Omni should report that the
+`musa` vLLM platform plugin is active.
+
+Install vLLM-Omni from a checkout containing MiniMax H3 support:
+
+```bash
+python -m pip install -e .
+```
+
+MiniMax H3 reference inputs and MP4 output require `soundfile`, `ffmpeg`, and
+`ffprobe`. The Python dependency is installed by vLLM-Omni; install the two
+executables with the operating system package manager and verify them:
+
+```bash
+python -c 'import soundfile; print(soundfile.__version__)'
+ffmpeg -version
+ffprobe -version
+```
+
+When torchaudio cannot load TorchCodec, vLLM-Omni automatically falls back to
+soundfile. Formats that libsndfile cannot read are demuxed through ffmpeg.
+
+## Start a server
+
+Use the partition path and set `--task-type` explicitly: `fl2va` for T2VA/FL2VA
+or `ref2va` for Ref2VA.
+
+The validated Ref2VA configuration uses tensor parallelism across four MTT S5000 GPUs and offloads inactive model components to CPU:
+
+```bash
+export MODEL="${MODEL_ROOT}/Ref2VA"
+export PORT=8091
+
+MUSA_VISIBLE_DEVICES=0,1,2,3 \
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
+VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
+vllm serve "${MODEL}" \
+  --omni \
+  --host 0.0.0.0 \
+  --port "${PORT}" \
+  --trust-remote-code \
+  --num-gpus 4 \
+  --task-type ref2va \
+  --tensor-parallel-size 4 \
+  --vae-patch-parallel-size 4 \
+  --enable-cpu-offload \
+  --vae-use-tiling \
+  --diffusion-attention-backend FLASH_ATTN
+```
+
+On MUSA, `FLASH_ATTN` selects the MATE/FlashAttention-3 implementation. Do
+not install the CUDA-only `fa4` optional dependency.
+
+For request examples and parameter definitions, see
+[MiniMax-H3.md](MiniMax-H3.md#http-api-examples).
+
+## MUSA validation
+
+The following weight-independent gates have been validated on one MTT S5000:
+
+- packed variable-length attention selects MATE FlashAttention-3 and returns
+  finite BF16 output;
+- MiniMax H3 RoPE casts CPU FP64 position metadata to FP32 before device
+  arithmetic and returns finite output;
+- the video-VAE seeded RNG context is deterministic on MUSA and restores both
+  CPU and MUSA RNG states;
+- the soundfile fallback preserves stereo channel layout, FP32 samples, and
+  the native sample rate when torchaudio/TorchCodec is unavailable.
+
+Ref2VA image+audio generation was also validated end to end on one MTT S5000
+with CPU offload and VAE tiling. The smoke used a 448x256 reference image, a
+four-second 32 kHz stereo audio reference, two denoising steps, and seed 42.
+The synchronous HTTP request completed in 50.6 seconds and returned an MP4
+with the following decoded properties:
+
+- 107 video frames at 448x256 and 24 FPS;
+- non-static video (`pixel_std=13.07`, `temporal_delta=0.49`);
+- finite stereo audio at 32 kHz (`audio_rms=0.0458`).
+
+Official Ref2VA video-reference serving was validated with the expanded
+MiniMax prompt at 1344x768, 5 seconds, seed 0, and 50 steps. A true TP4 run on
+four MTT S5000 GPUs returned HTTP 200 in 638.6 seconds and produced 124 H.264
+frames at 24 FPS with AAC stereo audio at 32 kHz.
+
+Keep `--vae-use-tiling` enabled for this serving profile.
+
+## Known limitations
+
+- Keep Ring Attention at degree 1. The current Ring path does not preserve
+  MiniMax H3 packed padding boundaries.
+- H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1.
+- This MUSA recipe uses an explicitly selected single task partition; combined
+  serving loads both DiTs.
+- H3 currently executes one generation request per diffusion batch.
+- FP8 quantization has not been enabled for MiniMax H3.
+- MP3, M4A, MP4, and reference-video audio fallback requires `ffmpeg` on
+  `PATH`; WAV inputs can be read directly through soundfile.
+
+## Additional resources
+
+- [MiniMax-H3.md](MiniMax-H3.md) — full model and API guide
+- [MiniMax-H3-NPU.md](MiniMax-H3-NPU.md) — Ascend NPU deployment guide
+- [Supported models](../../docs/models/supported_models.md)
+- [Video API](../../docs/serving/videos_api.md)
@@ -1,7 +1,7 @@
 # MiniMax H3 on Ascend NPU

-> Joint video and audio generation with text, first-frame, image/audio, or
-> multi-video conditions — Ascend NPU deployment guide
+> Joint video and audio generation with text, first/last keyframes, and mixed
+> image/video/audio references — Ascend NPU deployment guide

 ## Summary

@@ -26,12 +26,12 @@ environments. Differences from the GPU path:

 ### Checkpoint

-Same as the GPU recipe — Hugging Face access approval is required:
+Same as the GPU recipe — Hugging Face access approval is required. Authenticate
+once; `vllm serve` downloads the required nested components automatically:

 ```bash
 hf auth login
-export MODEL_ROOT=/path/to/MiniMax-H3
-hf download MiniMaxAI/MiniMax-H3 --local-dir "${MODEL_ROOT}"
+export MODEL=MiniMaxAI/MiniMax-H3
 ```

 ### Environment
@@ -50,7 +50,8 @@ uv pip install -e .
 ```

 Install the **mindie-sd** operator library to enable Ascend-optimized fused
-operators (`adalayernorm`, etc.):
+operators (`adalayernorm`, etc.) and the RainFusion `rf_v2` block-sparse
+attention kernel:

 ```bash
 git clone https://gitcode.com/Ascend/MindIE-SD.git && cd MindIE-SD
@@ -72,24 +73,21 @@ pip install mindiesd-*.whl

 ## Start a server

-One server loads one checkpoint partition. Set `MODEL` to `FL2VA` for T2VA
-and FL2VA requests, or to `Ref2VA` for Ref2VA requests.
+Pass the repository ID directly. The pipeline loads the two nested DiTs while
+sharing the tokenizer, processor, text encoder, and VAEs from `FL2VA`.

-### Multi-NPU: 768P validated configuration
+### Multi-NPU: 768P combined-service configuration

-Validated on eight NPUs of an Atlas 800I A3 server with Ulysses sequence
-parallelism degree 8, text-encoder tensor parallelism degree 8, native tiled
-VAE patch parallelism degree 8, and layerwise offload:
+Use Ulysses sequence parallelism degree 8, text-encoder tensor parallelism
+degree 8, native tiled VAE patch parallelism degree 8, and layerwise offload:

 ```bash
 export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
 export PORT=9098
-export MODEL="${MODEL_ROOT}/FL2VA"
+export MODEL=MiniMaxAI/MiniMax-H3
 export VLLM_WORKER_MULTIPROC_METHOD=spawn
 export VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800
 export PYTHONDONTWRITEBYTECODE=1
-export HF_HUB_OFFLINE=1
-export TRANSFORMERS_OFFLINE=1

 vllm serve "${MODEL}" \
  --omni \
@@ -111,12 +109,31 @@ Do not add `--enforce-eager`. The first request includes regional
 compilation; warm the server once before measuring steady-state latency.
 H3 is CFG-distilled, so `--cfg-parallel-size` must remain 1.

-To serve Ref2VA, stop the FL2VA server and restart with:
+The same endpoint accepts `task=t2va`, `task=fl2va`, and `task=ref2va`; no
+partition restart is required. Layerwise offload applies to both DiTs.
+
+### Optional optimizations
+
+Two independent optimizations may be enabled on top of the configuration
+above. Both are validated for T2VA only.
+
+**RainFusion block-sparse attention** — switch the attention backend, keeping
+every other flag unchanged:

 ```bash
-export MODEL="${MODEL_ROOT}/Ref2VA"
+  --diffusion-attention-backend RAINFUSION_ATTN
 ```

+**INT8 online quantization** — add one flag to the server command above:
+
+```bash
+  --quantization int8
+```
+
+Keep `--ring 1` when using RainFusion: the `rf_v2` kernel ranks key blocks
+over the whole sequence, so ring parallelism would split away the keys it
+needs. Scale with `--usp` instead.
+
 ## HTTP API examples

 The request format is identical to the GPU recipe; see
@@ -145,18 +162,26 @@ throughput guarantee.

 ## Known limitations

- Each server process loads only one checkpoint partition.
+- Combined serving requires sibling `FL2VA` and `Ref2VA` directories, loads
+  both task-specific DiTs, and loads one copy of every shared component.
 - H3 currently executes one generation request per diffusion batch.
 - The first regional-compile request is a warmup and should not be included
  in steady-state performance measurements.
- Image+audio Ref2VA accepts exactly one image and one audio reference.
- Video Ref2VA accepts one or more video files, but not an additional
-  standalone audio reference.
+- The official H3 input matrix and media limits are documented in the [GPU
+  recipe](MiniMax-H3.md#official-input-matrix-and-limits); this NPU path uses
+  the same HTTP request contract.
 - VAE patch parallelism requires size 1 or the full DiT group size and
  supports the H3 native `tile` mode only.
+- RainFusion block-sparse attention and INT8 quantization are validated for
+  T2VA only; use the BF16 dense configuration for FL2VA and Ref2VA.
+- Online quantization cannot be combined with distributed layerwise offload
+  while AllGather is enabled; pass `--dlo-no-use-allgather` in that case.

 ## Additional resources

 - [MiniMax-H3.md](MiniMax-H3.md) — full GPU guide
+- [Attention backends § RAINFUSION_ATTN](../../docs/user_guide/diffusion/attention_backends.md#rainfusion_attn-backend-and-block-sparse-video-attention)
+  — RainFusion knobs and tuning
+- [Int8 quantization](../../docs/user_guide/quantization/int8.md)
 - [Supported models](../../docs/models/supported_models.md)
 - [Video API](../../docs/serving/videos_api.md)
@@ -1,7 +1,7 @@
 # MiniMax H3

-> Joint video and audio generation with text, first-frame, image/audio, or
-> multi-video conditions
+> Joint video and audio generation with text, first/last keyframes, and
+> mixed image/video/audio references

 ## Summary

@@ -12,33 +12,49 @@
 - Maintainer: Community

 MiniMax H3 is a CFG-distilled joint video/audio diffusion transformer. Its
-checkpoint has two independently served partitions:
+checkpoint has two task-specific DiT partitions:

 - `FL2VA`: text-to-video+audio (`t2va`) and first-frame-to-video+audio
  (`fl2va`)
- `Ref2VA`: image+audio or one-or-more-video reference-to-video+audio
-  (`ref2va`)
+- `Ref2VA`: up to 9 images, 3 videos, and 3 audio references in supported
+  image/video/audio combinations (`ref2va`; audio-only is rejected)
+
+One vLLM-Omni diffusion stage can load both DiTs while instantiating the
+tokenizer, processor, Qwen3-VL text encoder, video VAE, and audio VAE only
+once. Requests select the DiT with `extra_params.task`.

 The generated MP4 contains H.264 video and synchronized stereo audio.

 ## Prerequisites

-The checkpoint requires Hugging Face access approval. Download it and point
-`MODEL_ROOT` at the directory containing the `FL2VA` and `Ref2VA`
-subdirectories:
+The checkpoint requires Hugging Face access approval. Authenticate once;
+`vllm serve` downloads the required components automatically:

 ```bash
 hf auth login
-export MODEL_ROOT=/path/to/MiniMax-H3
-hf download MiniMaxAI/MiniMax-H3 --local-dir "${MODEL_ROOT}"
+export MODEL=MiniMaxAI/MiniMax-H3
 ```

-Install vLLM-Omni from the checkout containing MiniMax H3 support. On
-Blackwell, install the optional FlashAttention-4 dependency:
+The vLLM-Omni pipeline downloads `FL2VA/**`, `Ref2VA/model_index.json`, and
+`Ref2VA/transformer/**`. It does not download or load the diffusers-format
+`transformer`, `transformer_ref`, or `vae` weights at the repository root, nor
+duplicate Ref2VA copies of shared components.
+
+Install vLLM-Omni from the checkout containing MiniMax H3 support. The
+two-GPU RTX 5090/4090 profiles use cuDNN attention and do not need
+FlashAttention-4. Install the optional dependency only for the four-GPU
+B300/GB200 `FLASH_ATTN` profile:

 ```bash
 uv venv
 source .venv/bin/activate
+uv pip install -e .
+```
+
+To keep FA4 available as an explicit option on Blackwell, install the
+FlashAttention-4 extra:
+
+```bash
 uv pip install -e '.[fa4]'
 ```

@@ -47,8 +63,29 @@ reference-video preparation and MP4 output.

 ## Start a server

-One server loads one checkpoint partition. Set `MODEL` to `FL2VA` for T2VA
-and FL2VA requests, or to `Ref2VA` for either Ref2VA request.
+Pass the repository ID directly. The pipeline uses `FL2VA` for model discovery
+and shared components, and loads the second DiT from `Ref2VA/transformer`.
+
+### Memory and storage requirements
+
+Treat GPU HBM, host RAM, and checkpoint storage as separate requirements. Each
+H3 checkpoint partition (`FL2VA` or `Ref2VA`) contains about **134 GiB** of
+BF16 safetensors (about **135 GiB** on disk). Keeping both partitions
+locally therefore needs roughly **270 GiB** of model storage. A combined
+service downloads both; `--task-type fl2va` or `--task-type ref2va` downloads
+only the selected partition.
+
+CPU offload and distributed layerwise offload reduce GPU residency; they do
+not make the model weights disappear. With `--dlo-no-use-allgather`, each
+worker retains its standard-loader rank-local weights in host memory, including
+pinned CPU buffers used for H2D streaming. Use at least **200 GiB available
+system RAM** before starting the two-GPU recipe; a **384 GiB host is
+recommended** to leave room for the OS, CUDA/PyTorch allocations, request
+inputs, and filesystem cache. Do not run the FL2VA and Ref2VA servers at the
+same time on a host sized for this minimum.
+
+The consumer-GPU profiles below are HBM budgets only. They still require the
+host-RAM budget above.

 ### Single GPU: accuracy and memory first

@@ -57,7 +94,7 @@ This matches the accuracy-qualified reference path and prevents the Qwen3-VL
 encoder and DiT from being resident on the GPU at the same time.

 ```bash
-export MODEL="${MODEL_ROOT}/FL2VA"
+export MODEL=MiniMaxAI/MiniMax-H3
 export PORT=8091

 CUDA_VISIBLE_DEVICES=0 \
@@ -74,25 +111,120 @@ vllm serve "${MODEL}" \
 ```

 Use a GPU with enough memory for the active H3 component and enough system RAM
-for the offloaded components. CPU offload reduces GPU memory pressure but adds
-PCIe/NVLink transfer latency.
+for both offloaded DiTs plus the shared components. Model-level offload keeps
+the two DiTs mutually exclusive on GPU, but adds PCIe/NVLink transfer latency.

-### Four GPUs: measured best-practice throughput
+### Two 24/32 GB GPUs: TP2 distributed layerwise offload

-The best validated four-GPU configuration on four NVIDIA B300 GPUs is:
+For two PCIe consumer GPUs, combine TP2 with distributed layerwise offload
+(DLO). The standard loader first creates the rank-local TP shard. DLO keeps
+that shard in pinned host memory and streams the 30 tail DiT blocks through a
+shared two-buffer window without a DP AllGather. The first 20 DiT blocks are
+copied to the GPUs once per denoise stage, reused by every sampling step, and
+released before VAE decode so the decoder can reuse their HBM.
+
+```bash
+export MODEL=MiniMaxAI/MiniMax-H3
+export PORT=8091
+
+CUDA_VISIBLE_DEVICES=0,1 \
+VLLM_WORKER_MULTIPROC_METHOD=spawn \
+VLLM_OMNI_VIDEO_SYNC_TIMEOUT=14400 \
+vllm serve "${MODEL}" \
+  --omni \
+  --host 0.0.0.0 \
+  --port "${PORT}" \
+  --trust-remote-code \
+  --task-type fl2va \
+  --num-gpus 2 \
+  --tensor-parallel-size 2 \
+  --usp 1 \
+  --ring 1 \
+  --text-encoder-tp-size 2 \
+  --vae-patch-parallel-size 2 \
+  --vae-parallel-mode tile \
+  --vae-use-tiling \
+  --enable-distributed-layerwise-offload \
+  --dlo-no-use-allgather \
+  --dlo-resident-layers 20 \
+  --enforce-eager \
+  --diffusion-attention-backend CUDNN_ATTN
+```
+
+Use the profile that matches the per-GPU memory capacity:
+
+| Profile | GPUs | Starting shape | Resident DiT blocks | Attention | Execution | Status |
+|---|---:|---:|---:|---|---|---|
+| `rtx5090` | 2 x 32 GB | 1344x768 | 20 | cuDNN attention | eager | Target-hardware validated |
+| `rtx4090` | 2 x 24 GB | 1024x576 | 12 | cuDNN attention | eager | Capacity-proxy starting point |
+
+This topology uses all available parallel capacity: TP2 shards both the DiT
+and text encoder, `--dlo-no-use-allgather` streams each rank's local TP shard
+without reconstructing full blocks, and VAE patch parallelism splits tiled
+decode across both GPUs. cuDNN attention is selected explicitly for the RTX
+consumer path; the server stays eager to avoid an unqualified compile path.
+
+The resident count changes placement and transfer frequency only; it does not
+quantize or change the BF16/FP32 denoise math. Re-measure peak memory before
+increasing it on a different request shape.
+
+### RTX 5090 target-hardware validation
+
+At vLLM-Omni commit `ae6577ea`, one full 50-step T2VA request completed on
+2 x RTX 5090 without OOM:
+
+| Shape | Frames | Client E2E | Sampled peak/GPU | Output validation |
+|---:|---:|---:|---:|---|
+| 1344x768 | 124 at 24 FPS | 8 min 38 s | approximately 22.6 GiB | H.264 video + 32 kHz stereo AAC; full `ffmpeg` decode passed |
+
+This is a single end-to-end validation run, not a warmed multi-run latency
+benchmark. The sampled `nvidia-smi` peak is also not a CUDA allocator
+high-water mark. The environment used vLLM 0.26.0, vLLM-Omni
+`0.26.1.dev14+gae6577ea`, and PyTorch 2.11.0+cu130. The
+[run record](https://github.com/lishunyang12/vllm-omni-rankings/blob/dcd06d7e83cb069842535918c0169ee9f3f29ba0/scripts/%E5%BE%AE%E4%BF%A1%E5%9B%BE%E7%89%87_20260805000034_86_237.png)
+captures the environment, output contract, elapsed time, and sampled peak.
+
+Before the target run, both profiles were exercised on two B300 ranks as an
+allocation and correctness proxy. At 1344x768, 124 frames, and 50 steps, the
+20-layer profile peaked at 27,726 MiB per rank. At 1024x576, the 12-layer
+profile peaked at 18,888 MiB per rank in a 5-step capacity run. The resident
+and fully streamed placements produced identical decoded video-frame and audio
+hashes for the same shape, step count, prompt, and seed. The B300 result does
+not establish RTX 4090 PCIe latency; treat the 4090 profile as a conservative
+starting point until it is measured on that GPU.
+
+To run T2VA, FL2VA, image+audio Ref2VA, and two-video Ref2VA in order, validate
+every MP4's H.264/AAC streams, and retain live server and GPU-memory logs:
+
+```bash
+RUN_ROOT=/path/to/run-root \
+MODEL_ROOT=/path/to/MiniMax-H3 \
+GPU_IDS=0,1 \
+PROFILE=rtx5090 \
+bash examples/offline_inference/minimax_h3/run_h3_2gpu_all_tasks.sh
+```
+
+The script selects 20 resident layers for `PROFILE=rtx5090` and 12 for
+`PROFILE=rtx4090`; `DLO_RESIDENT_LAYERS=N` overrides either default.
+
+### Four GPUs: throughput-oriented combined service
+
+For a combined service on four high-memory GPUs, use:

 - no CPU or layerwise offload;
 - Ulysses sequence parallelism degree 4;
 - native tiled VAE patch parallelism degree 4;
 - regional `torch.compile` for the repeated DiT blocks;
- FlashAttention, with Ring and TP left at 1.
+- dense BF16 `TRTLLM_ATTN`, with Ring and TP left at 1.
+
+Both DiTs remain resident in this no-offload configuration. If they do not fit,
+use model-level CPU offload.

 ```bash
-export MODEL="${MODEL_ROOT}/FL2VA"
+export MODEL=MiniMaxAI/MiniMax-H3
 export PORT=8091

 CUDA_VISIBLE_DEVICES=0,1,2,3 \
-FLASHINFER_DISABLE_VERSION_CHECK=1 \
 VLLM_WORKER_MULTIPROC_METHOD=spawn \
 VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
 vllm serve "${MODEL}" \
@@ -105,8 +237,7 @@ vllm serve "${MODEL}" \
  --ring 1 \
  --vae-patch-parallel-size 4 \
  --vae-parallel-mode tile \
-  --vae-use-tiling \
-  --diffusion-attention-backend FLASH_ATTN
+  --vae-use-tiling
 ```

 Do not add `--enforce-eager` to this performance configuration. The first
@@ -115,6 +246,67 @@ steady-state latency. H3 is CFG-distilled, so `--cfg-parallel-size` must remain
 1. The H3 VAE supports its native `tile` mode, not
 `spatial_shard_height` or `spatial_shard_width`.

+### Attention Backends
+
+On datacenter Blackwell GPUs, MiniMax H3 defaults to dense BF16
+`TRTLLM_ATTN`; no attention backend flag is required. To select it explicitly,
+use:
+
+```bash
+--diffusion-attention-backend TRTLLM_ATTN
+```
+
+Stable measurements with the four-GPU profile above put dense `TRTLLM_ATTN`
+and FA4 within 2% of each other. `TRTLLM_ATTN` remains the datacenter Blackwell
+default and enables the optional optimizations below. Confirm the server log
+contains `Defaulting to diffusion attention backend TRTLLM_ATTN` before
+recording measurements when using the default selection.
+
+FA4 remains available by explicitly selecting the `FLASH_ATTN` backend:
+
+```bash
+--diffusion-attention-backend FLASH_ATTN
+```
+
+On Blackwell, `FLASH_ATTN` selects FA4. Confirm the server log contains
+`Using CuTe FlashAttention-4 on Blackwell` before recording FA4 measurements.
+
+`TRTLLM_ATTN` additionally supports two **lossy** optimizations for the long main
+DiT attention sequence: SAGE attention quantization and Skip-Softmax Sparse
+Attention. SAGE quantizes Q/K to the configured dtype and V to FP8. This example uses
+`fp8_e4m3` for Q/K; B200 also supports `int8` Q/K. The TRTLLM SAGE path fixes V
+to FP8, so vLLM-Omni only exposes the Q/K dtype. The token refiner is a short
+attention path, so the `per_role` override leaves SAGE and Skip-Softmax disabled
+for it. The example enables the calibration-free Skip-Softmax path with
+`threshold=0.05`, after the normalized timestep reaches `0.97`:
+
+```bash
+--diffusion-attention-config '{
+  "default": {
+    "backend": "TRTLLM_ATTN",
+    "quant": {
+      "dtype_qk": "fp8_e4m3",
+      "q_block_size": 1,
+      "k_block_size": 16
+    },
+    "skip_softmax": {
+      "threshold": 0.05,
+      "disabled_until_timestep": 0.97
+    }
+  },
+  "per_role": {
+    "minimax_h3.token_refiner": {
+      "backend": "TRTLLM_ATTN"
+    }
+  }
+}'
+```
+
+For configuration details, see
+[TRTLLM_ATTN Backend and Skip-Softmax](https://github.com/vllm-project/vllm-omni/blob/main/docs/user_guide/diffusion/attention_backends.md#trtllm_attn-backend-and-skip-softmax)
+and
+[TRTLLM_ATTN SAGE Quantization](https://github.com/vllm-project/vllm-omni/blob/main/docs/user_guide/diffusion/attention_backends.md#trtllm_attn-sage-quantization).
+
 ### Text encoder tensor parallelism

 The Qwen3-VL text encoder (~51.5 GB in BF16 for the retained 50 layers) is by
@@ -125,11 +317,10 @@ vLLM-style tensor-parallel layers and runs with distributed collectives over
 its own encoder process group):

 ```bash
-export MODEL="${MODEL_ROOT}/FL2VA"
+export MODEL=MiniMaxAI/MiniMax-H3
 export PORT=8091

 CUDA_VISIBLE_DEVICES=0,1,2,3 \
-FLASHINFER_DISABLE_VERSION_CHECK=1 \
 VLLM_WORKER_MULTIPROC_METHOD=spawn \
 VLLM_OMNI_VIDEO_SYNC_TIMEOUT=1800 \
 vllm serve "${MODEL}" \
@@ -143,8 +334,7 @@ vllm serve "${MODEL}" \
  --text-encoder-tp-size 4 \
  --vae-patch-parallel-size 4 \
  --vae-parallel-mode tile \
-  --vae-use-tiling \
-  --diffusion-attention-backend FLASH_ATTN
+  --vae-use-tiling
 ```

 `N` must divide the Qwen3-VL head counts (64 attention heads / 8 KV heads), so
@@ -157,12 +347,9 @@ the reference path within bf16 rounding: every encoder rank all-reduces the
 row-parallel projections, so the full `[seq, 5120]` layer-50 hidden state is
 replicated on every rank.

-To serve Ref2VA, stop the FL2VA server and restart the same one- or four-GPU
-command with:
-
-```bash
-export MODEL="${MODEL_ROOT}/Ref2VA"
-```
+No restart is needed: `task=fl2va` routes to `FL2VA/transformer`, while
+`task=ref2va` routes to
+`Ref2VA/transformer`. T2VA uses the FL2VA DiT.

 ### Online FP8 quantization

@@ -218,13 +405,14 @@ export API_URL="http://127.0.0.1:${PORT}/v1/videos/sync"

 ### 1. T2VA: text to video and audio

-Run this request against the `FL2VA` partition:
+Run this request against the combined service:

 ```bash
 curl -sS -X POST "${API_URL}" \
  -F 'prompt=In a snowy blue-purple forest, Ori carefully walks past a sleeping giant; footsteps crunch in the snow while the creature breathes and softly snorts.' \
  -F 'width=1344' \
  -F 'height=768' \
+  -F 'aspect_ratio=16:9' \
  -F 'fps=24' \
  -F 'num_inference_steps=50' \
  -F 'flow_shift=12' \
@@ -235,7 +423,7 @@ curl -sS -X POST "${API_URL}" \

 ### 2. FL2VA: first frame to video and audio

-Run this request against the `FL2VA` partition. When width and height are
+Run this request against the combined service. When width and height are
 omitted, H3 preserves the first-frame aspect ratio and uses a 768-pixel short
 edge.

@@ -253,11 +441,32 @@ curl -sS -X POST "${API_URL}" \
  -o fl2va.mp4
 ```

-### 3. Ref2VA: image and audio reference
+Use the same `FL2VA` partition for the official tail-keyframe forms. A single
+image with `frame_indices=[-1]` conditions the last frame; two ordered images
+with `frame_indices=[0,-1]` condition the first and last frames:

-Run this request against the `Ref2VA` partition. `audio_reference` accepts an
-HTTP(S) URL or a `data:` URL. In one terminal, expose the local reference
-assets to the serving host:
+```bash
+export LAST_FRAME=/path/to/fl2va_last_frame.png
+export FIRST_FRAME=/path/to/fl2va_first_frame.png
+
+curl -sS -X POST "${API_URL}" \
+  -F 'prompt=The subject moves naturally from the first image to the last image.' \
+  -F 'num_inference_steps=50' \
+  -F 'flow_shift=12' \
+  -F 'seed=2102' \
+  -F 'extra_params={"task":"fl2va","duration":8.7,"frame_indices":[0,-1],"audio_flow_shift":3.0}' \
+  -F "input_references=@${FIRST_FRAME};type=image/png" \
+  -F "input_references=@${LAST_FRAME};type=image/png" \
+  -o fl2va_first_last.mp4
+```
+
+### 3. Ref2VA: image-only, image/audio, or mixed references
+
+Run these requests against the combined service or a Ref2VA-only service.
+Image-only Ref2VA omits `audio_reference`; adding one or more audio references
+is optional. The typed fields accept one object or an ordered JSON list.
+`audio_reference` accepts an HTTP(S) URL or a `data:` URL. In one terminal,
+expose the local reference assets to the serving host:

 ```bash
 python -m http.server 8092 \
@@ -265,7 +474,24 @@ python -m http.server 8092 \
  --directory /path/to/reference_assets
 ```

-Then submit the image and audio request from another terminal:
+Then submit an image-only request from another terminal:
+
+```bash
+export REF_IMAGE=/path/to/reference_assets/ref2va_image.png
+
+curl -sS -X POST "${API_URL}" \
+  -F 'prompt=A white cat sits on a beige couch and slowly looks toward the camera.' \
+  -F 'aspect_ratio=adaptive' \
+  -F 'short_edge=768' \
+  -F 'num_inference_steps=50' \
+  -F 'flow_shift=12' \
+  -F 'seed=3100' \
+  -F 'extra_params={"task":"ref2va","duration":8.0,"audio_flow_shift":3.0}' \
+  -F "input_reference=@${REF_IMAGE};type=image/png" \
+  -o ref2va_image_only.mp4
+```
+
+An image-plus-audio request is:

 ```bash
 export REF_IMAGE=/path/to/reference_assets/ref2va_image.png
@@ -288,9 +514,9 @@ curl -sS -X POST "${API_URL}" \
 The requested duration should cover the complete audio. If `duration` is
 shorter, the reference soundtrack is truncated to the generated clip.

-### 4. Ref2VA: two video references
+### 4. Ref2VA: video, separate audio, and mixed references

-Run this request against the `Ref2VA` partition. Repeat the
+Run this request against the combined service. Repeat the
 `input_references` multipart field once per source video. H3 consumes the
 videos in form order and preserves their original soundtracks during
 conditioning.
@@ -313,23 +539,61 @@ curl -sS -X POST "${API_URL}" \
  -o ref2va_video_video.mp4
 ```

-The server stores uploaded multi-video references only for the lifetime of the
-request and deletes the temporary files after generation. Video Ref2VA uses
-the source-video soundtracks and does not accept a separate
-`audio_reference`.
+The server stores uploaded references only for the lifetime of the request and
+deletes temporary files after generation. A video may use its embedded
+soundtrack, a separate `audio_reference`, or both. To send a mixed multipart
+request, repeat `input_references` for each image, video, or audio file; the
+server classifies them by MIME type and preserves the per-type order.
+
+Reference videos must be MP4/MOV with H.264/H.265 video, optional AAC/MP3
+audio, 2–15 seconds each, and at most 15 seconds combined. They may still be
+longer than the generated clip. Use `start_time_seconds` to select a
+synchronized segment; for multiple typed video references, pass one value per
+video in `extra_params.start_time_seconds`.
+
+Reference images accept JPG/JPEG, PNG, WEBP, HEIC, or HEIF up to 30 MiB. Standalone
+audio references accept WAV or MP3 up to 15 MiB, with 2–15 seconds per file and
+at most 15 seconds combined.
+
+## Official input matrix and limits
+
+| Task | Supported references | Limits |
+|------|----------------------|--------|
+| T2VA | text only | prompt must be non-empty |
+| FL2VA | first image, last image, or ordered first+last images | at most 2 images; `frame_indices` is `[0]`, `[-1]`, or `[0,-1]` |
+| Ref2VA | image-only, image+image, image+video, video+audio, and mixed image/video/audio | images ≤9, videos ≤3, audios ≤3, total references ≤12; audio requires a visual reference |
+
+The H3 output contract is 4–15 seconds at 24 FPS, stereo 32 kHz audio, and a
+32-pixel canvas multiple. T2VA requires one named output ratio from `21:9`,
+`16:9`, `4:3`, `1:1`, `3:4`, or `9:16`. FL2VA always follows the first input
+image's ratio and ignores a generic `aspect_ratio` override. Ref2VA defaults to
+`16:9`; `adaptive` and SGLang's `auto` spelling are accepted aliases for that
+default. `short_edge` controls the 768-pixel canvas and must be `768`.
+`num_outputs_per_prompt` accepts 1–10 and derives each output seed as
+`seed + output_index`. The asynchronous endpoint returns all
+outputs; the synchronous raw-MP4 endpoint returns the first output when more
+than one is requested.

 ## Key parameters

 | Parameter | Recommended value | Notes |
 |-----------|-------------------|-------|
-| `task` | `t2va`, `fl2va`, or `ref2va` | Passed in `extra_params`; must match the served partition |
+| `task` | `t2va`, `fl2va`, or `ref2va` | Passed in `extra_params`; selects the task-specific DiT |
 | `duration` | Workload-specific | Decimal seconds in `extra_params`; converted to H3-compatible frame count |
 | `fps` | `24` | H3 output FPS is fixed |
 | `num_inference_steps` | `50` | Matches the reference accuracy workloads |
 | `flow_shift` | `12` | Video sigma shift |
 | `audio_flow_shift` | `3` | Audio sigma shift, passed in `extra_params` |
 | `seed` | Task-specific | Use a fixed value for reproducibility |
-| `width`, `height` | Multiples of 32 | Aspect ratio must be between 1:4 and 4:1 |
+| `aspect_ratio` | Task-specific | T2VA requires a named ratio; FL2VA follows the input image; Ref2VA defaults to `16:9` |
+| `short_edge` | `768` | H3 shape policy requires exactly 768 when `width`/`height` are omitted |
+| `num_outputs_per_prompt` | `1` | 1–10; async API returns every output |
+| `start_time_seconds` | `0` | Reference-video segment start; use a list in `extra_params` for multiple videos |
+| `width`, `height` | Multiples of 32 | Output aspect ratio must be between 1:4 and 4:1 |
+
+## ComfyUI Frontend
+
+Users can also use a ComfyUI frontend to interact with a hosted MiniMax-H3 service. The ComfyUI frontend can run in a separate environment or machine. Refer to [vLLM-Omni ComfyUI Integration](../../docs/features/comfyui.md) for details.

 ## Validated four-GPU evidence

@@ -356,7 +620,8 @@ reduction.

 ## Known limitations

- Each server process loads only one checkpoint partition.
+- Combined serving requires sibling `FL2VA` and `Ref2VA` directories, loads
+  both task-specific DiTs, and loads shared components once from `FL2VA`.
 - H3 currently executes one generation request per diffusion batch.
 - The first regional-compile request is a warmup and should not be included in
  steady-state performance measurements.
@@ -0,0 +1,174 @@
+# PersonaPlex
+
+> Live full-duplex speech-to-speech (talk to the model in your browser, it listens
+> while it speaks) with persona and voice control, on the native vLLM-Omni engine.
+
+## Summary
+
+- Vendor: NVIDIA
+- Model: [`nvidia/personaplex-7b-v1`](https://huggingface.co/nvidia/personaplex-7b-v1)
+  (gated; a Moshi finetune)
+- Task: Full-duplex speech-to-speech. 24 kHz mic audio in, agent speech +
+  inner-monologue text out, in 80 ms lockstep. Persona (role text) and voice
+  (zero-shot clone) are set per session.
+- Mode: Live duplex WebSocket server with the official browser client, single or
+  multi-session (`--batch-size`), plus an offline WAV-in / WAV-out example
+- Maintainer: [`@linyueqian`](https://github.com/linyueqian)
+
+## When to use this recipe
+
+Use this recipe to run `nvidia/personaplex-7b-v1` as a live voice agent on one GPU.
+The integration is **moshi-free**: the temporal transformer, depformer, and
+streaming Mimi codec are all native vLLM-Omni modules (Mimi runs on
+`transformers.MimiModel`), so no vendored fork is installed. Decoding is greedy,
+matching the reference implementation frame for frame on the golden replays used
+as the acceptance gate.
+
+PersonaPlex is the first Moshi-class (pure-lockstep) model in the experimental
+full-duplex framework (`vllm_omni/experimental/fullduplex/`), and the first with
+elastic multi-session batching: one engine hosts `--batch-size` concurrent
+conversations, callers join or leave on any 80 ms tick without disturbing live
+slots (greedy isolation is bit-exact).
+
+## References
+
+- Serving package:
+  [`vllm_omni/experimental/fullduplex/personaplex/`](../../vllm_omni/experimental/fullduplex/personaplex/)
+- Native model modules (temporal / depformer / streaming Mimi):
+  [`vllm_omni/model_executor/models/personaplex/`](../../vllm_omni/model_executor/models/personaplex/)
+- Online example (server + headless client):
+  [`examples/online_serving/personaplex/`](../../examples/online_serving/personaplex/)
+- Offline example:
+  [`examples/offline_inference/personaplex/personaplex_offline.py`](../../examples/offline_inference/personaplex/personaplex_offline.py)
+- Staged offline pipeline (talker -> Mimi code2wav, async-chunk streaming):
+  [`vllm_omni/deploy/personaplex.yaml`](../../vllm_omni/deploy/personaplex.yaml)
+- Integration PR: [vllm-project/vllm-omni#4771](https://github.com/vllm-project/vllm-omni/pull/4771)
+- Framework context: [RFC #3745](https://github.com/vllm-project/vllm-omni/issues/3745)
+  (duplex adapter patterns), [#1335](https://github.com/vllm-project/vllm-omni/issues/1335)
+  (full-duplex target), [PR #3907](https://github.com/vllm-project/vllm-omni/pull/3907)
+  (fullduplex core contracts this adapter conforms to)
+- Upstream model card:
+  [`nvidia/personaplex-7b-v1`](https://huggingface.co/nvidia/personaplex-7b-v1)
+
+## Why it is different from MiniCPM-o / JoyVL duplex
+
+| | MiniCPM-o 4.5 (#3907) | JoyVL | **PersonaPlex (this)** |
+|---|---|---|---|
+| Cadence | 1 s chunk groups | ~1 fps frames | **80 ms lockstep (12.5 Hz)** |
+| Turn control | learned `⟨listen⟩`/`⟨speak⟩` | `</silence>`/`</response>` | **none, pure lockstep** |
+| Per step | variable-length token group | text decision | **1 user frame in -> 1 agent frame + 1 text token** |
+| Barge-in | at chunk boundary | n/a | **native (model always hears the user)** |
+| Session state | chunk-group KV | per-tick HTTP | **persistent ring KV + streaming Mimi state** |
+
+This is the lockstep ("parallel-frame joint") shape of the duplex adapter
+patterns: the adapter declares `DuplexCapability.continuous = True`, and
+`core.DuplexRuntime` runs ONE eternal response that consumes input frames as they
+arrive and drains on close. The flag is a small, default-off lifecycle mode, so
+turn-based adapters (JoyVL, MiniCPM-o) are unaffected.
+
+## Architecture (Moshi RQ-Transformer, all native)
+
+- **Mimi codec** 24 kHz, 12.5 Hz, 1920 samples/frame, 8 active codebooks
+  (card 2048). Streaming encode + decode via `transformers.MimiModel` with the
+  PersonaPlex reference checkpoint weights.
+- **Temporal transformer** (Helium backbone) 4096-d / 32 layers / 32 heads,
+  sliding window 3000 frames (a 4-minute rolling context) held in a ring KV cache
+  with per-slot recycle.
+- **Depformer** 1024-d / 6 layers / 16 heads, autoregressive over 16 codebooks
+  per frame (8 vocoded), reset each frame.
+- Per-frame token column `[B, 17, 1]`: row 0 = inner-monologue text,
+  rows 1-8 = agent audio, rows 9-16 = user audio.
+- Persona + voice are injected once at session open through the same lockstep
+  step: voice clone forces the agent stream from reference-audio Mimi codes;
+  persona forces the text stream from `<system> ... <system>` tokens.
+
+## Hardware Support
+
+Verified on GPU (Hopper-class). The serving path is plain PyTorch eager plus the
+native modules, so other CUDA GPUs with enough memory are expected to work.
+
+## GPU
+
+### 1x 141 GB Hopper-class GPU (H20-class, verified)
+
+#### Environment
+
+- OS: Linux
+- Python: 3.10+ (CI covers 3.11 / 3.12)
+- vLLM-Omni: PR #4771 branch or current `main` once merged
+- Extra dependency for the live server: `pip install sphn` (Opus framing;
+  `aiohttp` already ships with vLLM)
+- `export HF_TOKEN=...` with access to the gated
+  `nvidia/personaplex-7b-v1` repo (accept the license on the model page)
+
+First run auto-downloads from the model repo: `model.safetensors`,
+`tokenizer_spm_32k_3.model`, `voices.tgz`, and the Mimi reference checkpoint;
+`kyutai/mimi` is fetched via `transformers` for the codec module graph.
+
+#### Command
+
+Live duplex server (official browser client served at `/`):
+
+```bash
+HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 \
+python -m vllm_omni.experimental.fullduplex.personaplex.serving.server \
+    --port 8124 --voice NATF2.pt
+```
+
+Multi-session on the same GPU (elastic batching, callers join/leave any tick):
+
+```bash
+HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 \
+python -m vllm_omni.experimental.fullduplex.personaplex.serving.server \
+    --port 8124 --batch-size 4
+```
+
+Then open `http://localhost:8124/` and allow the microphone (use headphones so
+the agent does not hear itself). Headless alternative:
+
+```bash
+python examples/online_serving/personaplex/duplex_client.py \
+    --url ws://localhost:8124/api/chat --input user.wav --out reply.wav
+```
+
+Offline WAV-in / WAV-out (no server):
+
+```bash
+python examples/offline_inference/personaplex/personaplex_offline.py \
+    --input-wav question.wav --output-wav reply.wav --output-text reply.json \
+    --voice-prompt NATF2.pt --persona "You are a wise and friendly teacher."
+```
+
+#### Verification
+
+```bash
+# GPU-free contract tests (stubbed FrameStepper)
+pytest tests/e2e/features/fullduplex/test_personaplex_adapter.py \
+       tests/e2e/features/fullduplex/test_personaplex_session.py \
+       tests/e2e/features/fullduplex/test_personaplex_batched.py -q
+
+# GPU e2e: batched isolation + slot recycle (coherence check needs a real
+# question WAV via PPLEX_QUESTION_WAV)
+pytest tests/e2e/features/fullduplex/test_personaplex_elastic.py -q
+```
+
+A green offline run producing a coherent spoken reply (and inner-monologue text
+in `reply.json`) is the quickest end-to-end check.
+
+#### Notes
+
+- Realtime budget is 80 ms/frame. Verified eager per-tick latency at
+  `--batch-size 4` is ~70-74 ms on this hardware class, i.e. all four
+  conversations stay realtime; single-session has comfortable headroom.
+- Decoding is greedy only (temperature/top-k knobs are intentionally not
+  exposed): greedy is what the parity gates pin against the reference
+  implementation, and sampling amplifies sub-bit numeric drift into divergence.
+- With `--batch-size N`, connections beyond N are rejected until a slot frees;
+  a recycled slot is bit-exact with a fresh engine for the same inputs.
+- Voice prompts: bundled `voices.tgz` names (`NATF*`/`NATM*` natural,
+  `VARF*`/`VARM*` varied) or a path to your own `.pt`/`.wav`.
+- Run the browser client near the server. The 80 ms cadence is sensitive to
+  network jitter; over a high-latency link playback can stutter regardless of
+  engine speed. On localhost it is smooth.
+- Sliding window is 3000 frames (~4 min). The ring KV recycles beyond that;
+  very long sessions keep running with a rolling context.
@@ -46,6 +46,7 @@ recipes/
 | [`inclusionAI/Ming-omni-tts.md`](./inclusionAI/Ming-omni-tts.md) | Offline + online dense Ming TTS/audio generation | 1x H100 80GB / 1x AMD MI300X (ROCm 7.2) |
 | [`IndexTeam/IndexTTS-2.md`](./IndexTeam/IndexTTS-2.md) | Online serving for voice-cloned TTS with optional emotion control | 1x L4 24GB or larger CUDA GPU |
 | [`MiniMaxAI/MiniMax-H3.md`](./MiniMaxAI/MiniMax-H3.md) | T2VA, FL2VA, image+audio Ref2VA, and multi-video Ref2VA serving | 1x GPU with CPU offload / 4x B300 |
+| [`MiniMaxAI/MiniMax-H3-MUSA.md`](./MiniMaxAI/MiniMax-H3-MUSA.md) | MiniMax H3 T2VA, FL2VA, and Ref2VA on MUSA | 4x MTT S5000 (TP4 Ref2VA profile) |
 | [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB |
 | [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU |
 | [`MammothModa2/MammothModa2.md`](./MammothModa2/MammothModa2.md) | Preview and Dev text-to-image (AR → DiT); Dev text/image understanding | Preview: 1x L40S 48GB / 1x ≥40GB GPU; Dev: 1x NVIDIA GPU with sufficient cache headroom |
@@ -164,10 +164,11 @@ curl -s http://localhost:8091/v1/chat/completions \
 PR [#2495](https://github.com/vllm-project/vllm-omni/pull/2495) adds
 performance CI configs for the same DiT-only settings. The CI step is
 currently opt-in (gated by `RUN_HUNYUAN_IMAGE3_PERF=1`) with `soft_fail`
-enabled, intended for initial data collection. Performance assertions are
-skipped (`skip-performance-assertion: true`); the baseline values in the
-JSON configs are reference-only and will be promoted to regression gates
-once enough nightly data has been collected.
+enabled, intended for initial data collection. Perf runners no longer
+compare metrics against JSON baselines at assert time; the `baseline`
+blocks in the configs are reference-only and are copied into result JSON
+for reporting. Regression gates can be reintroduced once enough nightly
+data has been collected.

 The user-facing equivalent is to launch one of the CLI commands above and
 generate 1024x1024 images with 50 denoising steps.
@@ -56,10 +56,14 @@ vllm serve /path/to/Cosmos3-Super \
  --data-parallel-size 4
 ```

-### Without AllGather (full weights per rank, no sharding)
+### Without DLO AllGather (rank-local weights, no DLO sharding)

-Use `--dlo-no-use-allgather` when you want each rank to load full weights independently —
-no AllGather synchronization, no concurrent request requirement, but N× host memory.
+Use `--dlo-no-use-allgather` when you want each rank to stream the standard
+loader's rank-local weights independently — no DLO AllGather synchronization,
+no concurrent request requirement, but no DLO weight sharding and higher host
+memory use. In pure DP, this is equivalent to independent full-weight
+layerwise-offload replicas; with TP or HSDP, each rank retains the standard
+loader's local tensor layout.

 ```bash
 vllm serve /path/to/Cosmos3-Nano \
@@ -154,9 +158,9 @@ wait
 | Flag | Description | Default |
 |------|-------------|---------|
 | `--enable-distributed-layerwise-offload` | Enable DLO with H2D + AllGather overlap | `false` |
-| `--data-parallel-size N` | Number of DP ranks (weight sharding + concurrent requests) | `1` |
+| `--data-parallel-size N` | Number of DP ranks; DLO weight sharding + concurrent requests when AllGather is enabled | `1` |
 | `--dlo-use-allgather` | Use shard + AllGather for weight reconstruction (recommended) | `true` |
-| `--dlo-no-use-allgather` | Each rank loads full weights independently (no sharding, no AllGather) | `false` |
+| `--dlo-no-use-allgather` | Stream standard-loader rank-local weights independently (no DLO sharding, no DLO AllGather) | `false` |
 | `--usp N` | Use SP instead of DP for weight sharding (long sequences) | `1` |

 ## Mode comparison
@@ -166,7 +170,7 @@ wait
 | DLO + AllGather (DP4) | `--enable-distributed-layerwise-offload --data-parallel-size 4` | 1/4 model | 2 blocks | **3.3× HSDP** | Max throughput, short sequences |
 | DLO + AllGather (DP2) | `--enable-distributed-layerwise-offload --data-parallel-size 2` | 1/2 model | 2 blocks | 2× HSDP | Balanced throughput + memory |
 | DLO + AllGather (SP4) | `--enable-distributed-layerwise-offload --usp 4` | 1/4 model | 2 blocks | 1× (single req) | Long sequences (720p+) |
-| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Full model | 2 blocks | 0.4× HSDP | No AllGather overhead, high CPU |
+| DLO no-AllGather | `--enable-distributed-layerwise-offload --dlo-no-use-allgather` | Standard-loader rank-local weights | 2 blocks | 0.4× HSDP | No DLO AllGather overhead, high CPU |
 | HSDP | `--use-hsdp --hsdp-shard-size 4` | 0 | 1/4 model | 1× (baseline) | Weights fit in HBM |

 ## Memory expectations
@@ -185,7 +189,11 @@ wait

 - `--dlo-use-allgather` (default) requires all concurrent requests to have the same
  `num_inference_steps` (AllGather is a collective that needs all ranks synchronized)
- Online quantization (FP8) is incompatible with mmap loading — falls back to regular
-  `load_weights()` automatically
- Tensor Parallel is not supported (DLO uses DP-based sharding, a different dimension)
- HSDP + AllGather is rejected (would double-shard weights)
+- Online quantization (FP8) is incompatible with the DLO+AllGather mmap path;
+  use `--dlo-no-use-allgather` or disable online quantization
+- Tensor Parallel > 1 is rejected in the DLO+AllGather mmap path. TP with
+  `--dlo-no-use-allgather` retains the standard TP loader and should be treated
+  as an experimental compatibility path
+- HSDP + AllGather is rejected (would double-shard weights). HSDP with
+  `--dlo-no-use-allgather` is accepted at configuration level, but is not yet a
+  broadly validated production combination
@@ -423,6 +423,156 @@ ffprobe -v error -show_entries stream=codec_type,nb_frames,width,height cosmos3_
  once in `vllm_omni/model_extras/cosmos3.py` and forwarded through `--extra-body`;
  unknown keys for the model are dropped.

+## ROCm
+
+### 1x MI350X (Online serving)
+
+#### Environment
+
+- OS: Ubuntu 22.04+
+- Python: 3.12+
+- Driver / runtime: ROCm, HIP 7.2, **gfx942 or gfx950** (validated on gfx950 / MI350X,
+  which exposes 252 GiB of HBM to PyTorch)
+- PyTorch: 2.11.0 (ROCm build)
+- vLLM version: match the repository requirements from your current checkout
+- vLLM-Omni version or commit: use the commit you are deploying from
+
+No ROCm-specific patch or flag is needed: Cosmos3 runs on the stock code path.
+
+#### Command
+
+The NVIDIA command works unchanged. Guardrails need the gated
+`nvidia/Cosmos-1.0-Guardrail` repo, so the quick path is to disable them (you are
+then responsible for license compliance):
+
+```bash
+vllm serve nvidia/Cosmos3-Nano \
+  --omni \
+  --no-guardrails \
+  --host 0.0.0.0 --port 8000 \
+  --init-timeout 1800
+```
+
+For 720p video, add `--vae-use-tiling`: it cuts peak VRAM by ~68% for ~13%
+latency, which is the difference between needing a 120 GiB card and a 40 GiB one.
+`--quantization fp8` and `--enable-layerwise-offload` are also supported; see the
+Notes for the measured cost of each. For extra GPUs use `--ulysses-degree N` or
+`--tensor-parallel-size N` (not validated on ROCm yet).
+
+#### Verification
+
+Use the same `curl` requests as the CUDA section above. The ROCm images do not
+ship `ffprobe`, so the CUDA section's `ffprobe` line does not work here. `imageio`
+and `pyav` are already installed and report the same thing:
+
+```bash
+python -c "from PIL import Image; im=Image.open('cosmos3_t2i.png'); print(im.size, im.mode)"
+python -c "import imageio.v3 as iio; print(iio.improps('cosmos3_t2v.mp4', plugin='pyav').shape)"
+```
+
+Expect `(1024, 1024) RGB` and `(189, 720, 1280, 3)` for the requests above.
+Re-running the same request with the same seed reproduces a byte-identical mp4,
+so an md5 comparison across two runs also works as a smoke check.
+
+#### Notes
+
+- **Warm up once per output shape before timing anything.** The first generation
+  of a shape pays a one-time cost that is cached on disk and reused by later
+  processes in the same container: a ~250 s aiter attention-kernel JIT build plus
+  a ~390 s build of the VAE decode path *for that shape*, so warming 1024² images
+  does nothing for 189-frame video. At 720p / 189 frames / 35 steps this is
+  **540 s cold vs 161 s warm, with byte-identical output**.
+- **Measured on 1x MI350X (bf16, guardrails off, warm):** T2I 1024² @ 50 steps
+  **~2.7 s**; T2V 1280×720 / 189 frames @ 35 steps **~161 s**, of which ~92% is
+  the DiT denoise loop and ~4% VAE decode, so optimization effort belongs in the
+  denoise loop. The optional flags act on different terms of the memory bill and
+  are therefore complementary rather than redundant:
+
+  | Flag | Latency | Peak reserved | Peak allocated | Acts on |
+  |---|---|---|---|---|
+  | *(none)* | 161 s | 120 GiB | 95 GiB | — |
+  | `--vae-use-tiling` | 183 s (+13.5%) | **38 GiB** (−68%) | **36 GiB** (−62%) | decode activations |
+  | `--enable-layerwise-offload` | 162 s (+0.8%) | 84 GiB (−30%) | 69 GiB (−27%) | weights |
+  | `--quantization fp8` | 150 s (−6.6%) | 107 GiB (−11%) | 82 GiB (−14%) | weights (online, no calibration) |
+
+  Reserved is the caching-allocator high water mark and runs ~2–25 GiB above
+  allocated; allocated is the figure to compare against another platform.
+- **`--vae-use-tiling` is by far the biggest lever for video.** Of the 95 GiB
+  allocated by default only ~35 GiB is weights and fixed overhead; the remaining
+  **60 GiB is activations from decoding all 189 frames in one piece**, which is
+  why fp8 and offload cannot touch it and why tiling, which leaves under 5 GiB of
+  them, wins so much. `--vae-use-slicing` is a no-op for a single video request:
+  peak memory is bit-identical to the default.
+- **`--enable-layerwise-offload` is close to free on this part:** per-block DiT
+  compute is long enough to hide the weight transfer, so on a card this size it
+  buys headroom for concurrency or higher resolution rather than the
+  speed-for-memory trade it is on small cards.
+- **Attention backend:** the numbers above are aiter FlashAttention, not SDPA.
+  ROCm picks aiter when it is installed and the encoded device capability
+  satisfies `90 < major * 10 + minor < 100` (gfx942 and gfx950); otherwise it
+  falls back to `TORCH_SDPA` and says so only at debug level, which is easy to
+  miss. If you are reporting numbers, assert the choice instead of trusting the
+  default: `RocmOmniPlatform.get_diffusion_attn_backend_cls(None, 128)` returns
+  the resolved path.
+- **Determinism is per-configuration:** same seed and same flags reproduce a
+  byte-identical mp4 across processes, but different configurations do not match
+  each other — including offload, which nominally only relocates weights, so do
+  not use it as a bit-exact regression baseline. Quality was not evaluated for
+  any configuration.
+
+### 1x MI350X (Offline generation)
+
+#### Environment
+
+Same as the online serving section above.
+
+#### Command
+
+The standard task examples run unchanged:
+
+```bash
+# Text-to-image
+python examples/offline_inference/text_to_image/text_to_image.py \
+  --model nvidia/Cosmos3-Nano \
+  --prompt "A photorealistic red sports car at golden hour, cinematic lighting." \
+  --negative-prompt "blurry, distorted, low quality" \
+  --height 1024 --width 1024 --num-inference-steps 50 --guidance-scale 7.0 \
+  --extra-body '{"flow_shift": 3.0, "guardrails": false}' \
+  --output cosmos3_t2i.png
+
+# Text-to-video
+python examples/offline_inference/text_to_video/text_to_video.py \
+  --model nvidia/Cosmos3-Nano \
+  --prompt "A robot arm is cleaning a plate in the kitchen." \
+  --negative-prompt "blurry, distorted, low quality" \
+  --height 720 --width 1280 --num-frames 189 --fps 24 \
+  --num-inference-steps 35 --guidance-scale 6.0 --flow-shift 10.0 \
+  --extra-body '{"max_sequence_length": 4096, "guardrails": false,
+                 "use_resolution_template": false, "use_duration_template": false}' \
+  --output cosmos3_t2v.mp4
+```
+
+Both scripts print `Total generation time` and `Worker peak GPU memory
+(reserved)`, which is where the latency and reserved figures in the Notes above
+come from. Add `--vae-use-tiling`, `--quantization fp8` or
+`--enable-layerwise-offload` to reproduce the other rows. The allocated column
+needs `VLLM_LOGGING_LEVEL=DEBUG`, which enables a per-request line reporting
+reserved, allocated and pool overhead together.
+
+#### Verification
+
+Same as the online serving section above.
+
+#### Notes
+
+- Run each command twice and read the second run; the first pays the one-time
+  compilation described above. If you run in a container, keep the container
+  alive between runs (`docker run --rm` throws the JIT cache away and you pay
+  the ~250 s kernel build every time).
+- Latency and memory figures are identical to the online serving section — they
+  were taken from this path, because a single process has no client/server
+  timing ambiguity.
+
 ## NPU

 ### 1x Ascend 910B / 910C (Atlas A2 / A3) — Online serving
@@ -39,7 +39,7 @@ from vllm_omni.config.stage_config import (
    load_deploy_config,
    merge_pipeline_deploy,
 )
-from vllm_omni.engine.stage_init_utils import build_engine_args_dict
+from vllm_omni.engine.stage_init_utils import build_legacy_engine_args_dict

 pytestmark = [pytest.mark.core_model, pytest.mark.cpu]

@@ -81,6 +81,21 @@ def _from_pipeline_key(
    )


+def test_duplex_session_capacity_propagates_to_every_structured_stage() -> None:
+    omni_config = _from_pipeline_key("personaplex")
+
+    assert [stage.model_config.duplex_max_sessions for stage in omni_config.stage_configs] == [2, 2]
+
+
+def test_non_duplex_deploy_keeps_model_session_capacity_at_one(tmp_path: Path) -> None:
+    deploy_path = tmp_path / "personaplex-turn.yaml"
+    deploy_path.write_text("session_mode: turn\n", encoding="utf-8")
+
+    omni_config = _from_pipeline_key("personaplex", deploy_config_path=str(deploy_path))
+
+    assert [stage.model_config.duplex_max_sessions for stage in omni_config.stage_configs] == [1, 1]
+
+
@pytest.mark.parametrize("model_type", sorted(OMNI_PIPELINES))
 def test_vllm_omni_config_from_pipeline_config_matches_merge_pipeline_deploy(model_type: str):
    pipeline = _resolve_pipeline_or_skip(model_type)
@@ -105,13 +120,20 @@ def test_vllm_omni_config_from_pipeline_config_matches_merge_pipeline_deploy(mod
        assert omni_stage.runtime_config.num_replicas == legacy_stage.yaml_runtime.get("num_replicas", 1)

        engine_args = legacy_stage.yaml_engine_args
+        assert omni_stage.model_config.duplex_max_sessions == engine_args.get("duplex_max_sessions", 1)
        assert omni_stage.model_config.enforce_eager == engine_args.get("enforce_eager", False)
        assert omni_stage.load_config.load_format == engine_args.get("load_format", "auto")
        assert omni_stage.load_config.tokenizer_mode == engine_args.get("tokenizer_mode", "auto")
-        assert omni_stage.cache_config.gpu_memory_utilization == engine_args.get("gpu_memory_utilization", 0.90)
-        assert omni_stage.scheduler_config.max_num_seqs == engine_args.get("max_num_seqs", 128)
+        assert omni_stage.cache_config.gpu_memory_utilization == engine_args.get("gpu_memory_utilization")
+        assert omni_stage.cache_config.enable_prefix_caching == engine_args.get("enable_prefix_caching")
+        expected_disable_hybrid = engine_args.get("disable_hybrid_kv_cache_manager")
+        if omni_stage.stage_pipeline_config.execution_type == StageExecutionType.LLM_GENERATION:
+            expected_disable_hybrid = True if expected_disable_hybrid is None else expected_disable_hybrid
+        assert omni_stage.cache_config.disable_hybrid_kv_cache_manager == expected_disable_hybrid
+        assert omni_stage.scheduler_config.max_num_seqs == engine_args.get("max_num_seqs")
        assert omni_stage.scheduler_config.max_num_batched_tokens == engine_args.get("max_num_batched_tokens")
-        assert omni_stage.scheduler_config.async_scheduling == engine_args.get("async_scheduling", True)
+        assert omni_stage.scheduler_config.enable_chunked_prefill == engine_args.get("enable_chunked_prefill")
+        assert omni_stage.scheduler_config.async_scheduling == engine_args.get("async_scheduling")
        legacy_parallel_config = engine_args.get("parallel_config") or {}
        assert omni_stage.parallel_config.tensor_parallel_size == legacy_parallel_config.get(
            "tensor_parallel_size",
@@ -182,6 +204,64 @@ def test_from_pipeline_config_applies_cli_overrides_without_stage_config_runtime
    assert stage1.runtime_config.num_gpus == stage1.parallel_config.world_size


+@pytest.mark.parametrize(
+    "cli_overrides",
+    [
+        {"enable_lora": True},
+        {"stage_0_enable_lora": True},
+    ],
+    ids=["global", "stage-scoped"],
+)
+def test_from_pipeline_config_rejects_explicit_unowned_engine_cli_fields(cli_overrides):
+    with pytest.raises(ValueError, match=r"no structured config owner: enable_lora"):
+        _from_pipeline_key("qwen3_tts", cli_overrides=cli_overrides)
+
+
+@pytest.mark.parametrize(
+    ("engine_extras", "unowned_field"),
+    [
+        ({"enable_lora": True}, "enable_lora"),
+        ({"parallel_config": {"cfg_parallel_size": 2}}, "parallel_config.cfg_parallel_size"),
+    ],
+    ids=["top-level", "nested-parallel-config"],
+)
+def test_from_pipeline_config_rejects_unowned_deploy_engine_extras(engine_extras, unowned_field):
+    pipeline = _resolve_pipeline_or_skip("qwen3_tts")
+    deploy = DeployConfig(stages=[StageDeployConfig(stage_id=0, engine_extras=engine_extras)])
+
+    with pytest.raises(ValueError, match=rf"no structured config owner: {unowned_field}"):
+        VllmOmniConfig.from_pipeline_config(pipeline, user_deploy_config=deploy)
+
+
+@pytest.mark.parametrize(
+    ("cli_overrides", "stage_id"),
+    [
+        ({"kv_cache_dtype": "fp8"}, 0),
+        ({"stage_0_kv_cache_dtype": "fp8"}, 0),
+        ({"stage_1_kv_cache_dtype": "fp8"}, 1),
+    ],
+    ids=["global", "ar-stage", "generation-stage"],
+)
+def test_from_pipeline_config_rejects_diffusion_only_cli_fields_for_llm_stages(cli_overrides, stage_id):
+    with pytest.raises(ValueError, match=rf"Stage {stage_id} .*no structured config owner: kv_cache_dtype"):
+        _from_pipeline_key("qwen3_tts", cli_overrides=cli_overrides)
+
+
+def test_from_pipeline_config_accepts_diffusion_only_cli_fields_for_diffusion_stage():
+    stage = _from_pipeline_key(
+        "dreamzero",
+        deploy_config_path="dreamzero_tp1_cfg2",
+        cli_overrides={"kv_cache_dtype": "fp8"},
+    ).stage_by_id(0)
+
+    assert isinstance(stage, VllmOmniDiffusionStageConfig)
+    assert stage.diffusion_config.diffusion_kv_cache_dtype == "fp8"
+
+
+def test_stage_cli_field_selection_defers_ownership_validation_until_sources_are_merged():
+    assert omni_config_module._stage_cli_overrides(0, {"enable_lora": True}) == {"enable_lora": True}
+
+
 def test_runtime_num_gpus_is_derived_from_parallel_world_size():
    omni_config = _from_pipeline_key("hunyuan_image3_dit")
    stage = omni_config.stage_by_id(0)
@@ -364,8 +444,10 @@ def test_vllm_omni_stage_config_public_fields_use_typed_stage_realizations():
    assert {f.name for f in fields(VllmOmniGenerationStageConfig)} == public_fields


-def test_runtime_config_fields_match_rfc_runtime_scope():
+def test_runtime_config_fields_match_structured_runtime_scope():
    assert {f.name for f in fields(OmniStageRuntimeConfig)} == {
+        "distributed_executor_backend",
+        "worker_cls",
        "devices",
        "num_replicas",
        "env",
@@ -376,9 +458,23 @@ def test_runtime_config_fields_match_rfc_runtime_scope():
    }


-def test_sub_config_fields_match_rfc_scopes():
+def test_sub_config_fields_match_structured_scopes():
    assert {f.name for f in fields(OmniStageModelConfig)} == {
+        "model",
+        "model_arch",
+        "revision",
+        "tokenizer_revision",
+        "code_revision",
+        "seed",
+        "logits_processors",
+        "trust_remote_code",
+        "dtype",
+        "attention_backend",
+        "moe_backend",
+        "hf_overrides",
+        "limit_mm_per_prompt",
        "active_stream_window",
+        "duplex_max_sessions",
        "enable_sleep_mode",
        "default_sampling_params",
        "subtalker_sampling_params",
@@ -387,6 +483,7 @@ def test_sub_config_fields_match_rfc_scopes():
        "task_type",
        "codec_frame_rate_hz",
        "enforce_eager",
+        "max_cudagraph_capture_size",
        "enable_flashinfer_autotune",
        "compilation_config",
        "enable_multithread_weight_load",
@@ -399,12 +496,16 @@ def test_sub_config_fields_match_rfc_scopes():
        "tokenizer_subdir",
    }
    assert {f.name for f in fields(OmniStageLoadConfig)} == {
+        "tokenizer",
+        "download_dir",
+        "skip_tokenizer_init",
        "load_format",
        "tokenizer_mode",
        "config_format",
        "skip_mm_profiling",
    }
    assert {f.name for f in fields(OmniStageCacheConfig)} == {
+        "kv_cache_memory_bytes",
        "gpu_memory_utilization",
        "enable_prefix_caching",
        "disable_hybrid_kv_cache_manager",
@@ -418,6 +519,8 @@ def test_sub_config_fields_match_rfc_scopes():
        "async_scheduling",
    }
    assert {f.name for f in fields(OmniStageConnectorConfig)} == {
+        "async_chunk",
+        "omni_kv_config",
        "stage_connector",
        "output_connectors",
        "input_connectors",
@@ -810,7 +913,7 @@ def test_from_pipeline_config_matches_build_engine_args_dict_behavior_for_repres
    pipeline = _resolve_pipeline_or_skip("qwen3_tts")
    legacy_stage = merge_pipeline_deploy(pipeline, _load_default_deploy(pipeline))[0]
    omega_stage = legacy_stage.to_omegaconf()
-    legacy_engine_args = build_engine_args_dict(
+    legacy_engine_args = build_legacy_engine_args_dict(
        omega_stage,
        model="/tmp/qwen3-tts",
        stage_connector_spec={"name": "SharedMemoryConnector", "extra": {}},
@@ -905,6 +1008,8 @@ def test_from_pipeline_config_normalizes_diffusion_config_aliases_from_engine_ar


 def test_diffusion_config_field_classification_covers_current_fields():
+    from vllm_omni.diffusion.data import OmniDiffusionConfig
+
    classified_fields = (
        omni_config_module._DIFFUSION_SHARED_CONFIG_FIELDS
        | omni_config_module._DIFFUSION_RUNTIME_CONFIG_FIELDS
@@ -912,8 +1017,12 @@ def test_diffusion_config_field_classification_covers_current_fields():
    )

    assert classified_fields == {f.name for f in fields(omni_config_module._DiffusionConfigProjection)}
+    assert {f.name for f in fields(OmniDiffusionConfig)} <= (
+        classified_fields | omni_config_module._DIFFUSION_MOVED_SHARED_FIELDS
+    )
    assert {
        "enable_prompt_embed_cache",
+        "enable_session_state_manager",
        "prompt_embed_cache_size",
        "diffusion_kv_cache_dtype",
    } <= omni_config_module._DIFFUSION_ONLY_CONFIG_FIELDS
@@ -110,6 +110,9 @@ def test_resumable_generation_stop_marks_segment_boundary() -> None:
    session.status = RequestStatus.RUNNING
    session.resumable = True
    session.num_computed_tokens = len(session.prompt_token_ids)
+    # schedule() has counted this step's token as in flight; update_from_output()
+    # must settle it before handling the segment boundary.
+    session.num_in_flight_tokens = 1

    sched = MagicMock()
    sched.requests = {session.request_id: session}
@@ -118,12 +121,20 @@ def test_resumable_generation_stop_marks_segment_boundary() -> None:
        is_done_receiving_chunks=lambda _request_id: True,
        segment_finished_requests={session.request_id},
    )
-    sched._handle_stopped_request.return_value = False
+
+    def rearm_resumable_request(request: Request) -> bool:
+        request.status = RequestStatus.WAITING
+        return False
+
+    sched._handle_stopped_request.side_effect = rearm_resumable_request
    sched.running = [session]
    sched.waiting = MagicMock()
    sched.skipped_waiting = MagicMock()
    sched.structured_output_manager.should_advance.return_value = False
-    sched._pending_finish_reqs = []
+    # Async scheduling can observe the segment boundary in the next schedule()
+    # before this model output is applied and defer the same request for finish.
+    # update_from_output() must not re-arm the resumable request twice.
+    sched._pending_finish_reqs = [session]
    sched.recompute_kv_load_failures = False
    sched.connector = None
    sched.kv_cache_manager.take_events.return_value = None
@@ -155,6 +166,8 @@ def test_resumable_generation_stop_marks_segment_boundary() -> None:
    output = outputs[session.client_index].outputs[0]
    assert output.finish_reason is not None
    assert output.is_segment_finished is True
+    sched._handle_stopped_request.assert_called_once_with(session)
+    assert sched._pending_finish_reqs == []


 def test_async_chunk_resumable_stop_rearms_connector_polling() -> None:
@@ -1,3 +1,6 @@
+import math
+from typing import NamedTuple
+
 import pytest
 import torch

@@ -317,6 +320,103 @@ def test_deferred_multimodal_cache_write_commits_on_completion():
    assert torch.equal(rows[8:16], expected)


+def test_discard_deferred_mm_outputs_drops_pending_chunks():
+    """discard_deferred_mm_outputs is the abort path: a request that
+    leaves without a cache commit must not leak its staged chunks into
+    the cache if commit_deferred_mm_outputs is later called for it."""
+    cache = get_omni_pcache()
+    mm_key = "codes.audio"
+    block_table = torch.tensor(
+        [
+            [2, 3, 4, 5],
+            [6, 7, 8, 9],
+        ],
+        dtype=torch.long,
+    )
+    input_batch = MockInputBatch(
+        num_computed_tokens_cpu=torch.tensor([4, 0], dtype=torch.long),
+        block_table=block_table,
+    )
+
+    cache.update_omni_tensor_prefix_cache(
+        hidden_states=None,
+        multimodal_outputs={mm_key: torch.arange(8, dtype=torch.long).reshape(4, 2)},
+        num_tokens_unpadded=4,
+        slot_mapping=torch.arange(8, 12),
+        skip_mm_cache_keys={mm_key},
+    )
+
+    chunk = torch.arange(8, dtype=torch.long).reshape(4, 2)
+    cache.stage_deferred_mm_outputs(
+        query_start_loc=torch.tensor([0, 4], dtype=torch.long),
+        input_batch=input_batch,
+        multimodal_outputs={mm_key: chunk},
+        num_scheduled_tokens={"req1": 4, "req2": 0},
+        deferred_mm_cache_keys={mm_key},
+    )
+    assert "req1" in cache._deferred_mm_outputs
+
+    cache.discard_deferred_mm_outputs("req1")
+    assert "req1" not in cache._deferred_mm_outputs
+
+    # A later commit call for the now-discarded req_id must be a no-op,
+    # not a resurrection of the dropped chunk.
+    cache.commit_deferred_mm_outputs({"req1"}, input_batch)
+    rows = cache.mm_outputs_cache[mm_key].view(-1, 2)
+    assert torch.all(rows[8:12] == 0)
+
+
+def test_deferred_mm_outputs_request_id_reused_after_discard():
+    """A request ID reused after discard_deferred_mm_outputs must start
+    clean: the new occupant's committed data must not be mixed with the
+    previous (discarded) occupant's staged chunks."""
+    cache = get_omni_pcache()
+    mm_key = "codes.audio"
+    block_table = torch.tensor(
+        [
+            [2, 3, 4, 5],
+            [6, 7, 8, 9],
+        ],
+        dtype=torch.long,
+    )
+    input_batch = MockInputBatch(
+        num_computed_tokens_cpu=torch.tensor([4, 0], dtype=torch.long),
+        block_table=block_table,
+    )
+
+    cache.update_omni_tensor_prefix_cache(
+        hidden_states=None,
+        multimodal_outputs={mm_key: torch.arange(8, dtype=torch.long).reshape(4, 2)},
+        num_tokens_unpadded=4,
+        slot_mapping=torch.arange(8, 12),
+        skip_mm_cache_keys={mm_key},
+    )
+
+    stale_chunk = torch.full((4, 2), -1, dtype=torch.long)
+    cache.stage_deferred_mm_outputs(
+        query_start_loc=torch.tensor([0, 4], dtype=torch.long),
+        input_batch=input_batch,
+        multimodal_outputs={mm_key: stale_chunk},
+        num_scheduled_tokens={"req1": 4, "req2": 0},
+        deferred_mm_cache_keys={mm_key},
+    )
+    cache.discard_deferred_mm_outputs("req1")
+
+    # "req1" is reused by a brand-new request with different data.
+    fresh_chunk = torch.arange(8, dtype=torch.long).reshape(4, 2)
+    cache.stage_deferred_mm_outputs(
+        query_start_loc=torch.tensor([0, 4], dtype=torch.long),
+        input_batch=input_batch,
+        multimodal_outputs={mm_key: fresh_chunk},
+        num_scheduled_tokens={"req1": 4, "req2": 0},
+        deferred_mm_cache_keys={mm_key},
+    )
+    cache.commit_deferred_mm_outputs({"req1"}, input_batch)
+
+    rows = cache.mm_outputs_cache[mm_key].view(-1, 2)
+    assert torch.equal(rows[8:12], fresh_chunk)
+
+
 def test_deferred_multimodal_cache_can_be_merged_on_full_block_hit():
    cache = get_omni_pcache()
    mm_key = "codes.audio"
@@ -599,3 +699,219 @@ def test_get_merged_multimodal_outputs(feat_dims, num_tokens_padded, mocker):

            assert req2_merged_mm_outputs.shape == torch.Size([num_new_toks_req2, curr_feat_dim])
            assert torch.all(req2_merged_mm_outputs == req2_new_mm_outputs)
+
+
+@pytest.mark.parametrize(
+    "passthrough_shape",
+    [(5,), (5, 3)],
+    ids=["1d_token_metadata", "2d_feature_tensor"],
+)
+def test_get_merged_multimodal_outputs_slices_passthrough_tensor_per_request(passthrough_shape, mocker):
+    """A passthrough key that is itself a real tensor (not the string/list
+    passthrough already covered above) must be sliced per request through
+    to_payload_element's tensor branch, under the same mixed cache
+    hit/miss + uneven scheduled-length batch as test_get_merged_multimodal_outputs.
+
+    Covers both a 1D token-aligned metadata tensor (e.g. ref_code_len,
+    codec_streaming -- shape (total_tokens,)) and a 2D per-token feature
+    tensor (shape (total_tokens, feat_dim)), since W2's batching could
+    easily special-case one and silently mishandle the other.
+    """
+    feat_dims = {"foo": 100, "bar": 100}
+    cache = get_omni_pcache_with_mm_tensors(feat_dims, seq_len=DEFAULT_SEQ_LEN)
+
+    orig_num_tokens_unpadded = 8
+    slot_offset = 8
+    orig_slot_mapping = torch.arange(slot_offset, slot_offset + orig_num_tokens_unpadded)
+    feature_dims = {key: val.shape[-1] for key, val in cache.mm_outputs_cache.items()}
+    orig_mm_outputs = {
+        key: torch.rand((orig_num_tokens_unpadded, feature_dims[key]), dtype=DTYPE) for key in cache.mm_cache_keys
+    }
+
+    cache.update_omni_tensor_prefix_cache(
+        hidden_states=None,
+        multimodal_outputs=orig_mm_outputs,
+        num_tokens_unpadded=orig_num_tokens_unpadded,
+        slot_mapping=orig_slot_mapping,
+        num_tokens_padded=None,
+    )
+
+    num_new_toks_req1 = 3
+    num_new_toks_req2 = 2
+    cache.add_prefix_cached_new_req_id("req1")
+
+    num_scheduled_tokens = {
+        "req1": num_new_toks_req1,
+        "req2": num_new_toks_req2,
+    }
+
+    new_mm_outputs = {}
+    for mm_key in cache.mm_cache_keys:
+        new_mm_outputs[mm_key] = torch.rand(
+            (num_new_toks_req1 + num_new_toks_req2, feature_dims[mm_key]),
+            dtype=DTYPE,
+        )
+    total_scheduled_tokens = num_new_toks_req1 + num_new_toks_req2
+    assert passthrough_shape[0] == total_scheduled_tokens
+    passthrough_tensor = torch.arange(int(torch.prod(torch.tensor(passthrough_shape))), dtype=DTYPE).reshape(
+        passthrough_shape
+    )
+    new_mm_outputs["passthrough_tensor"] = passthrough_tensor
+
+    input_batch = MockInputBatch(num_computed_tokens_cpu=torch.Tensor([orig_num_tokens_unpadded, 0]))
+
+    mocker.patch(
+        "vllm_omni.core.prefix_cache.OmniTensorPrefixCache._get_cached_block_ids",
+        new=fake_get_cached_block_ids,
+    )
+    merged_mm_outputs = cache.get_merged_multimodal_states(
+        query_start_loc=[0, num_new_toks_req1],
+        input_batch=input_batch,
+        multimodal_outputs=new_mm_outputs,
+        num_scheduled_tokens=num_scheduled_tokens,
+    )
+
+    assert "passthrough_tensor" not in cache.mm_cache_keys
+
+    passthrough_out = merged_mm_outputs["passthrough_tensor"]
+    # req1 (cache hit) gets tokens [0:3), req2 (cache miss) gets tokens [3:5) --
+    # a passthrough tensor is sliced by scheduled position, not merged with
+    # any cached history (unlike the mm_cache_keys tensors above).
+    assert torch.equal(passthrough_out["req1"], passthrough_tensor[0:num_new_toks_req1])
+    assert torch.equal(passthrough_out["req2"], passthrough_tensor[num_new_toks_req1:total_scheduled_tokens])
+
+
+class MergeHarness(NamedTuple):
+    """A primed cache plus the arguments get_merged_multimodal_states needs."""
+
+    cache: OmniTensorPrefixCache
+    input_batch: MockInputBatch
+    num_scheduled_tokens: dict[str, int]
+    query_start_loc: list[int]
+    total_scheduled_tokens: int
+    new_mm_outputs: dict[str, object]
+
+
+def build_mixed_hit_miss_harness(mocker, num_new_toks_req1: int = 3, num_new_toks_req2: int = 2) -> MergeHarness:
+    """Two requests over one cache key: req1 hits 8 cached tokens, req2 misses,
+    and their scheduled lengths differ -- the same batch shape as
+    test_get_merged_multimodal_outputs, which is what makes a per-request
+    boundary error observable. Callers add their passthrough keys to
+    ``new_mm_outputs`` before calling get_merged_multimodal_states.
+    """
+    cache = get_omni_pcache_with_mm_tensors({"foo": 100}, seq_len=DEFAULT_SEQ_LEN)
+
+    orig_num_tokens_unpadded = 8
+    feature_dims = {key: val.shape[-1] for key, val in cache.mm_outputs_cache.items()}
+    cache.update_omni_tensor_prefix_cache(
+        hidden_states=None,
+        multimodal_outputs={
+            key: torch.rand((orig_num_tokens_unpadded, feature_dims[key]), dtype=DTYPE) for key in cache.mm_cache_keys
+        },
+        num_tokens_unpadded=orig_num_tokens_unpadded,
+        slot_mapping=torch.arange(8, 8 + orig_num_tokens_unpadded),
+        num_tokens_padded=None,
+    )
+    cache.add_prefix_cached_new_req_id("req1")
+
+    total_scheduled_tokens = num_new_toks_req1 + num_new_toks_req2
+    mocker.patch(
+        "vllm_omni.core.prefix_cache.OmniTensorPrefixCache._get_cached_block_ids",
+        new=fake_get_cached_block_ids,
+    )
+    return MergeHarness(
+        cache=cache,
+        input_batch=MockInputBatch(num_computed_tokens_cpu=torch.Tensor([orig_num_tokens_unpadded, 0])),
+        num_scheduled_tokens={"req1": num_new_toks_req1, "req2": num_new_toks_req2},
+        query_start_loc=[0, num_new_toks_req1],
+        total_scheduled_tokens=total_scheduled_tokens,
+        new_mm_outputs={
+            key: torch.rand((total_scheduled_tokens, feature_dims[key]), dtype=DTYPE) for key in cache.mm_cache_keys
+        },
+    )
+
+
+def test_get_merged_multimodal_outputs_passes_per_request_tensor_list_through(mocker):
+    """The passthrough shape qwen3-tts actually emits: a nested ``codes``/``meta``
+    dict whose ``codes.ref`` holds one tensor per request
+    (qwen3_tts_talker.make_omni_output), carried through to_payload_element's
+    pass_lists_through branch beside 1D int metadata that IS sliced per request.
+
+    Each request must receive the list *whole*. Per-request indexing happens
+    later, in the runner's _unwrap_lists; a W2 that split the list here would
+    make that second index select a different request's reference codes. The
+    existing list coverage passes a list of strings, which cannot catch a
+    tensor-specific regression, and no other test covers an int dtype.
+    """
+    harness = build_mixed_hit_miss_harness(mocker)
+
+    ref_codes = [torch.tensor([11, 12], dtype=torch.long), torch.tensor([21], dtype=torch.long)]
+    ref_code_len = torch.arange(harness.total_scheduled_tokens, dtype=torch.int32)
+    harness.new_mm_outputs["codes"] = {"ref": ref_codes}
+    harness.new_mm_outputs["meta"] = {"ref_code_len": ref_code_len}
+
+    merged = harness.cache.get_merged_multimodal_states(
+        query_start_loc=harness.query_start_loc,
+        input_batch=harness.input_batch,
+        multimodal_outputs=harness.new_mm_outputs,
+        num_scheduled_tokens=harness.num_scheduled_tokens,
+    )
+
+    assert "codes" not in harness.cache.mm_cache_keys
+    for req_id in ("req1", "req2"):
+        req_refs = merged["codes"][req_id]["ref"]
+        assert len(req_refs) == len(ref_codes)
+        for got, want in zip(req_refs, ref_codes):
+            assert torch.equal(got, want)
+            assert got.dtype == torch.long
+
+    # Per-request clones: the branch exists so one request's downstream mutation
+    # cannot reach another request's payload or the source list.
+    merged["codes"]["req1"]["ref"][0] += 100
+    assert torch.equal(merged["codes"]["req2"]["ref"][0], torch.tensor([11, 12], dtype=torch.long))
+    assert torch.equal(ref_codes[0], torch.tensor([11, 12], dtype=torch.long))
+
+    # Token-aligned 1D metadata is still split per request, unlike the list.
+    req1_len = harness.num_scheduled_tokens["req1"]
+    assert torch.equal(merged["meta"]["req1"]["ref_code_len"], ref_code_len[:req1_len])
+    assert torch.equal(merged["meta"]["req2"]["ref_code_len"], ref_code_len[req1_len:])
+    assert merged["meta"]["req1"]["ref_code_len"].dtype == torch.int32
+
+
+@pytest.mark.parametrize(
+    "passthrough_shape",
+    [(3, 4), (1, 4), (7,)],
+    ids=["shorter_2d", "single_row_2d", "longer_1d"],
+)
+def test_get_merged_multimodal_outputs_broadcasts_non_token_aligned_passthrough(passthrough_shape, mocker):
+    """A passthrough tensor whose first dimension is not the scheduled-token
+    count is request-invariant: to_payload_element hands every request the whole
+    tensor, cloned, rather than slicing it.
+
+    W2 batching most plausibly breaks by assuming every passthrough tensor is
+    token-aligned; that assumption would slice these and give each request a
+    truncated view, or an empty one once start exceeds the first dimension.
+    """
+    harness = build_mixed_hit_miss_harness(mocker)
+    assert passthrough_shape[0] != harness.total_scheduled_tokens
+
+    passthrough = torch.arange(math.prod(passthrough_shape), dtype=DTYPE).reshape(passthrough_shape)
+    expected = passthrough.clone()
+    harness.new_mm_outputs["speaker_embedding"] = passthrough
+
+    merged = harness.cache.get_merged_multimodal_states(
+        query_start_loc=harness.query_start_loc,
+        input_batch=harness.input_batch,
+        multimodal_outputs=harness.new_mm_outputs,
+        num_scheduled_tokens=harness.num_scheduled_tokens,
+    )
+
+    assert "speaker_embedding" not in harness.cache.mm_cache_keys
+    out = merged["speaker_embedding"]
+    for req_id in ("req1", "req2"):
+        assert out[req_id].shape == passthrough_shape
+        assert torch.equal(out[req_id], expected)
+
+    out["req1"] += 100
+    assert torch.equal(out["req2"], expected)
+    assert torch.equal(passthrough, expected)
@@ -0,0 +1,119 @@
+import pytest
+import torch
+
+from vllm_omni.core.prefix_cache import OmniTensorPrefixCache
+
+# `cpu` is the lane selector, not a hardware claim: the only Buildkite steps that
+# reach tests/ select `core_model and cpu`, and they run on GPU hardware (l4_1).
+# Marking this `cuda` collects it nowhere. Real hardware need is the skipif below.
+pytestmark = [
+    pytest.mark.core_model,
+    pytest.mark.cpu,
+    pytest.mark.skipif(not torch.cuda.is_available(), reason="requires a CUDA device"),
+]
+
+NUM_BLOCKS = 10
+BLOCK_SIZE = 4
+HIDDEN_SIZE = 2
+DTYPE = torch.float32
+
+
+def get_omni_pcache() -> OmniTensorPrefixCache:
+    """Build an OmniTensorPrefixCache, but don't init mm tensors."""
+    return OmniTensorPrefixCache(
+        num_blocks=NUM_BLOCKS,
+        block_size=BLOCK_SIZE,
+        hidden_size=HIDDEN_SIZE,
+        hs_dtype=DTYPE,
+    )
+
+
+def test_drain_ready_async_writes_is_a_noop_before_the_event_is_ready(monkeypatch):
+    """drain_ready_async_writes() must not touch the cache while the
+    scheduled D2H's event has not completed, and must consume it exactly
+    once after it has.
+
+    Real GPU timing (e.g. via torch.cuda._sleep to stall the copy stream)
+    was tried and found unreliable on this box: cross-stream wait_stream/
+    wait_event did not reliably delay event.query() from reporting ready
+    immediately, likely a virtualization/driver quirk of this cloud GPU.
+    Forcing the event's query() result directly tests the branching logic
+    in drain_ready_async_writes/_consume_pending_write deterministically,
+    independent of real CUDA scheduling behavior.
+    """
+    cache = get_omni_pcache()
+    mm_key = "codes.audio"
+    cache.maybe_init_missing_mm_cache_keys({mm_key: torch.zeros(4, HIDDEN_SIZE)}, seq_len=4)
+
+    num_tokens = 4
+    slot_mapping_gpu = torch.arange(8, 8 + num_tokens, dtype=torch.int64, device="cuda")
+    mm_gpu = {
+        mm_key: torch.arange(num_tokens * HIDDEN_SIZE, dtype=torch.float32, device="cuda").reshape(
+            num_tokens, HIDDEN_SIZE
+        )
+    }
+
+    cache.schedule_async_write(
+        hidden_states_gpu=None,
+        multimodal_outputs_gpu=mm_gpu,
+        slot_mapping_gpu=slot_mapping_gpu,
+        num_tokens_unpadded=num_tokens,
+        num_tokens_padded=num_tokens,
+    )
+
+    pending_event = cache._pending_write.event
+    rows = cache.mm_outputs_cache[mm_key].view(-1, HIDDEN_SIZE)
+
+    monkeypatch.setattr(pending_event, "query", lambda: False)
+    assert cache.drain_ready_async_writes() == 0
+    assert torch.all(rows[8 : 8 + num_tokens] == 0)
+    assert cache._pending_write is not None
+
+    monkeypatch.setattr(pending_event, "query", lambda: True)
+    assert cache.drain_ready_async_writes() == 1
+    assert torch.equal(rows[8 : 8 + num_tokens], mm_gpu[mm_key].cpu())
+    assert cache._pending_write is None
+
+    # Nothing pending anymore: a second drain is a no-op, not a re-scatter.
+    assert cache.drain_ready_async_writes() == 0
+
+
+def test_schedule_async_write_consumes_previous_pending_write_first():
+    """Scheduling step N+1's write must first consume (scatter) step N's
+    pending write, even if the caller never explicitly called
+    drain_ready_async_writes() in between -- otherwise step N's data would
+    be silently dropped when overwritten by step N+1's pending write."""
+    cache = get_omni_pcache()
+    mm_key = "codes.audio"
+    cache.maybe_init_missing_mm_cache_keys({mm_key: torch.zeros(4, HIDDEN_SIZE)}, seq_len=4)
+
+    num_tokens = 4
+    first_slots = torch.arange(8, 8 + num_tokens, dtype=torch.int64, device="cuda")
+    first_gpu = {mm_key: torch.full((num_tokens, HIDDEN_SIZE), 1.0, device="cuda")}
+    second_slots = torch.arange(12, 12 + num_tokens, dtype=torch.int64, device="cuda")
+    second_gpu = {mm_key: torch.full((num_tokens, HIDDEN_SIZE), 2.0, device="cuda")}
+
+    cache.schedule_async_write(
+        hidden_states_gpu=None,
+        multimodal_outputs_gpu=first_gpu,
+        slot_mapping_gpu=first_slots,
+        num_tokens_unpadded=num_tokens,
+        num_tokens_padded=num_tokens,
+    )
+    torch.accelerator.synchronize()
+
+    # This call's internal _consume_pending_write() must flush the FIRST
+    # write before the second one is scheduled; the caller never drained.
+    cache.schedule_async_write(
+        hidden_states_gpu=None,
+        multimodal_outputs_gpu=second_gpu,
+        slot_mapping_gpu=second_slots,
+        num_tokens_unpadded=num_tokens,
+        num_tokens_padded=num_tokens,
+    )
+    torch.accelerator.synchronize()
+    cache.drain_ready_async_writes()
+
+    rows = cache.mm_outputs_cache[mm_key].view(-1, HIDDEN_SIZE)
+    assert torch.all(rows[8:12] == 1.0)
+    assert torch.all(rows[12:16] == 2.0)
@@ -3,15 +3,13 @@ import os
 import re
 import subprocess
 import threading
-import time
 from datetime import datetime
 from pathlib import Path
 from typing import Any

 import pytest

-from tests.dfx.reliability.helpers import list_remote_process_pids_by_pattern, post_chat_completions_raw
-from tests.helpers.mark import hardware_marks
+from tests.helpers.mark import get_hardware_mark_list, hardware_marks
 from tests.helpers.runtime import OmniServerParams
 from tests.helpers.stage_config import modify_stage_config
 from vllm_omni.platforms import current_omni_platform
@@ -119,32 +117,6 @@ def create_unique_server_pytest_params(
    ]


-def create_benchmark_pytest_params(
-    benchmark_configs: list[dict[str, Any]],
-    server_to_benchmark_mapping: dict[str, dict],
-) -> list[Any]:
-    """Like :func:`create_benchmark_indices`, but wrap each index in ``pytest.param`` with JSON marks."""
-    marks_by_name = _marks_by_test_name(benchmark_configs)
-    params: list[Any] = []
-    seen: set[str] = set()
-    for config in benchmark_configs:
-        test_name = config["test_name"]
-        if test_name in seen:
-            continue
-        seen.add(test_name)
-        params_list = get_benchmark_params_for_server(test_name, server_to_benchmark_mapping)
-        id_suffixes = _unique_benchmark_param_id_suffixes(params_list)
-        for idx, id_suffix in enumerate(id_suffixes):
-            params.append(
-                pytest.param(
-                    (test_name, idx),
-                    marks=marks_by_name.get(test_name, []),
-                    id=f"{test_name}-{id_suffix}",
-                )
-            )
-    return params
-
-
 def create_paired_benchmark_pytest_params(
    server_entries: list[tuple[Any, str]],
    params_by_test_name: dict[str, list[Any]],
@@ -357,10 +329,6 @@ def supports_video_generation(model_name: str) -> bool:
    return any(key in lower for key in ("wan", "video", "i2v", "t2v"))


-def supports_chat_generation(model_name: str) -> bool:
-    return not supports_video_generation(model_name)
-
-
 def parse_stage_devices(stage_config_path: str) -> str:
    text = Path(stage_config_path).read_text(encoding="utf-8")
    raw_devices: list[str] = re.findall(r"^\s*devices:\s*\"?([0-9,\s]+)\"?\s*$", text, flags=re.MULTILINE)
@@ -389,48 +357,6 @@ def assert_fault_exception(exc: Exception, error_keywords: tuple[str, ...]) -> N
    assert any(key in text for key in error_keywords), f"unexpected error under fault injection: {exc}"


-def wait_chat_request_ready(host: str, port: int, model: str, timeout_sec: int = 180) -> None:
-    """Poll a minimal chat request until success."""
-    deadline = time.time() + timeout_sec
-    payload = json.dumps(
-        {
-            "model": model,
-            "messages": [{"role": "user", "content": "Say hello in one short sentence."}],
-            "stream": False,
-            "modalities": ["text"],
-        }
-    )
-    last_error: str | None = None
-    while time.time() < deadline:
-        try:
-            status, body = post_chat_completions_raw(host, port, payload)
-            if status == 200:
-                return
-            last_error = f"http={status} body={body[:200]!r}"
-        except Exception as exc:  # noqa: BLE001
-            last_error = str(exc)
-        time.sleep(2)
-    raise TimeoutError(f"runtime-teardown warmup request did not succeed within {timeout_sec}s: {last_error}")
-
-
-def assert_no_extra_worker_processes(
-    baseline_pids: set[int],
-    worker_pattern: str,
-    timeout_sec: int = 60,
-) -> None:
-    """Ensure no extra worker PIDs remain after teardown."""
-    deadline = time.time() + timeout_sec
-    while time.time() < deadline:
-        current = set(list_remote_process_pids_by_pattern(worker_pattern))
-        extra = current - baseline_pids
-        if not extra:
-            return
-        time.sleep(2)
-    current = set(list_remote_process_pids_by_pattern(worker_pattern))
-    extra = sorted(current - baseline_pids)
-    assert not extra, f"orphan worker processes remain after container teardown: {extra}"
-
-
 def create_test_parameter_mapping(configs: list[dict[str, Any]]) -> dict[str, dict]:
    mapping = {}
    for config in configs:
@@ -520,36 +446,16 @@ def _unique_benchmark_param_id_suffixes(params_list: list[dict[str, Any]]) -> li
    return unique


-def extract_mark_resource_label(mark_field: Any) -> str:
-    """Return a filename-safe hardware label from ``mark.hardware_marks.res`` values.
-
-    Example: ``{"cuda": "H100"}`` -> ``"H100"``; multiple platforms join with ``-``.
-
-    Prefer :func:`get_runtime_resource_label` for perf result filenames so labels
-    reflect the machine that actually ran the benchmark.
-    """
-    if isinstance(mark_field, list):
-        for item in mark_field:
-            if isinstance(item, dict) and "hardware_marks" in item:
-                return extract_mark_resource_label(item)
-        return "na"
-    if not isinstance(mark_field, dict):
-        return "na"
-    hw = mark_field.get("hardware_marks")
-    if not isinstance(hw, dict):
-        return "na"
-    res = hw.get("res")
-    if not isinstance(res, dict) or not res:
-        return "na"
-    labels = [_safe_filename_token(value) for value in res.values()]
-    return "-".join(labels) if labels else "na"
-
-
-_KNOWN_RUNTIME_RESOURCE_TOKENS: tuple[str, ...] = (
+# Device marketing-name tokens for ``get_runtime_resource_label`` only.
+# Longer / more specific tokens must appear before shorter substrings
+# (e.g. ``H200`` before ``H20``, ``910B4`` before ``910``).
+_RUNTIME_DEVICE_ALIASES: tuple[str, ...] = (
    "H100",
    "H800",
    "H200",
    "H20",
+    "B200",
+    "GB200",
    "L40S",
    "L40",
    "L4",
@@ -578,7 +484,7 @@ def _normalize_runtime_device_label(raw: str) -> str:
    if not raw or not str(raw).strip():
        return "na"
    upper = str(raw).upper()
-    for token in _KNOWN_RUNTIME_RESOURCE_TOKENS:
+    for token in _RUNTIME_DEVICE_ALIASES:
        if token.upper() in upper:
            return _safe_filename_token(token)
    compact = re.sub(r"[^a-zA-Z0-9]+", "", str(raw))
@@ -631,34 +537,41 @@ def resource_label_for_filename(resource_label: str | None) -> str:
    return token


-def extract_configs_resource_label(configs: list[dict[str, Any]]) -> str:
-    """Return runtime hardware label for perf result filenames."""
-    del configs
-    return get_runtime_resource_label()
+def is_hardware_nested_baseline(baseline: dict[str, Any]) -> bool:
+    """True for ``{"H100": {"metric": ...}, "A3": {...}}`` baselines.
+
+    Detection is structural + hardware-label allowlist (not metric-name hints):
+
+    - every top-level value is a non-empty dict (per-hardware metric map);
+    - every top-level key is in :func:`tests.helpers.mark.get_hardware_mark_list`
+      (``[hardware-resource]`` markers from ``pyproject.toml``).
+
+    Metric names under each hardware bucket are unconstrained.
+    Runtime device aliases (:data:`_RUNTIME_DEVICE_ALIASES`) are unrelated.
+    """
+    if not baseline:
+        return False
+    if not all(isinstance(value, dict) and bool(value) for value in baseline.values()):
+        return False
+    allowed = {label.upper() for label in get_hardware_mark_list()}
+    return all((token := str(key).strip()) and token.upper() in allowed for key in baseline.keys())


 def resolve_baseline_value(
    baseline_raw: Any,
    *,
    sweep_index: int | None,
-    max_concurrency: Any = None,
-    request_rate: Any = None,
 ) -> Any:
-    """Pick the baseline threshold for this sweep step."""
+    """Pick the baseline threshold for one metric at this sweep step.
+
+    Accepts a sweep-aligned list/tuple (indexed by ``sweep_index``) or a scalar.
+    """
    if baseline_raw is None:
-        return 100000
+        return None
    if isinstance(baseline_raw, dict):
-        if max_concurrency is not None:
-            for key in (max_concurrency, str(max_concurrency)):
-                if key in baseline_raw:
-                    return baseline_raw[key]
-        if request_rate is not None:
-            for key in (request_rate, str(request_rate)):
-                if key in baseline_raw:
-                    return baseline_raw[key]
-        raise KeyError(
-            f"baseline dict has no key for max_concurrency={max_concurrency!r} "
-            f"or request_rate={request_rate!r}; keys={list(baseline_raw.keys())!r}"
+        raise TypeError(
+            "per-metric baseline dict keyed by concurrency/request-rate is not supported; "
+            "use a sweep-aligned list or a scalar under each hardware bucket"
        )
    if isinstance(baseline_raw, (list, tuple)):
        if sweep_index is None:
@@ -669,22 +582,62 @@ def resolve_baseline_value(
    return baseline_raw


-def _baseline_thresholds_for_step(
-    baseline_data: dict[str, Any],
+def resolve_baseline_for_sweep(
+    baseline_config: dict[str, Any] | None,
    *,
    sweep_index: int | None = None,
-    max_concurrency: Any = None,
-    request_rate: Any = None,
 ) -> dict[str, Any]:
-    """Resolve baseline config to one threshold per metric for this iteration."""
-    return {
-        metric_name: resolve_baseline_value(
-            baseline_raw,
-            sweep_index=sweep_index,
-            max_concurrency=max_concurrency,
-            request_rate=request_rate,
+    """Resolve sweep-dependent baselines while keeping every hardware bucket.
+
+    Expects hardware-nested input only::
+
+        {"H100": {"throughput_qps": [a, b, c]}, "A3": {"throughput_qps": [x, y, z]}}
+
+    with ``sweep_index=2`` (e.g. concurrency 32) becomes::
+
+        {"H100": {"throughput_qps": c}, "A3": {"throughput_qps": z}}
+
+    Per-metric values may be a sweep-aligned list or a scalar.
+    """
+    if not baseline_config:
+        return {}
+    if not is_hardware_nested_baseline(baseline_config):
+        allowed = sorted(get_hardware_mark_list())
+        unknown = [
+            str(key).strip()
+            for key in baseline_config.keys()
+            if str(key).strip().upper() not in {label.upper() for label in allowed}
+        ]
+        nested_shape = all(isinstance(value, dict) and bool(value) for value in baseline_config.values())
+        if nested_shape and unknown:
+            raise ValueError(
+                "baseline top-level keys must be [hardware-resource] pytest markers from "
+                "pyproject.toml (tool.pytest.ini_options.markers). "
+                f"Unknown hardware label(s): {unknown!r}. "
+                f"Known markers: {allowed!r}. "
+                "Add a marker such as '\"A100: [hardware-resource] Tests that require A100 GPU\"' "
+                "to pyproject.toml, then retry."
+            )
+        raise ValueError(
+            "baseline must be hardware-nested with [hardware-resource] marker names as "
+            "top-level keys, e.g. "
+            '{"H100": {"throughput_qps": [...]}, "A3": {"custom_metric": 1.0}}. '
+            f"got top-level keys={list(baseline_config.keys())!r}; "
+            f"known markers={allowed!r}. "
+            "If you need a new hardware bucket, add it to pyproject.toml markers with "
+            "the [hardware-resource] tag."
        )
-        for metric_name, baseline_raw in baseline_data.items()
+
+    return {
+        hardware: {
+            metric_name: resolve_baseline_value(
+                baseline_raw,
+                sweep_index=sweep_index,
+            )
+            for metric_name, baseline_raw in metrics.items()
+        }
+        for hardware, metrics in baseline_config.items()
+        if isinstance(metrics, dict)
    }


@@ -697,17 +650,17 @@ def run_benchmark(
    *,
    baseline_config: dict[str, Any] | None = None,
    sweep_index: int | None = None,
-    request_rate: Any | None = None,
-    max_concurrency: Any | None = None,
    random_input_len: Any | None = None,
    random_output_len: Any | None = None,
    resource_label: str | None = None,
 ) -> dict[str, Any]:
    """Run one ``vllm bench serve --omni`` iteration and return parsed metrics.

-    After ``vllm bench`` writes the JSON, ``result["baseline"]`` holds the resolved per-metric thresholds
-    (when ``baseline_config`` is provided). If the benchmark exits without writing a result file,
-    ``result_omni_template.json`` is used as a fallback.
+    After ``vllm bench`` writes the JSON, ``result["baseline"]`` stores the
+    sweep-resolved baseline (when ``baseline_config`` is provided). Hardware-nested
+    maps keep every hardware bucket, but each metric is reduced to the value for
+    this concurrency / request-rate step. If the benchmark exits without writing a
+    result file, ``result_omni_template.json`` is used as a fallback.
    """
    current_dt = datetime.now().strftime("%Y%m%d-%H%M%S")
    ri = _safe_filename_token(random_input_len)
@@ -774,11 +727,10 @@ def run_benchmark(
            result = json.load(f)

    if baseline_config:
-        result["baseline"] = _baseline_thresholds_for_step(
+        # Keep every hardware bucket; resolve list metrics to this sweep step.
+        result["baseline"] = resolve_baseline_for_sweep(
            baseline_config,
            sweep_index=sweep_index,
-            request_rate=request_rate,
-            max_concurrency=max_concurrency,
        )
    else:
        result["baseline"] = {}
@@ -800,12 +752,3 @@ def pytest_addoption(parser: pytest.Parser) -> None:
        default=None,
        help=("Path to benchmark config JSON. Example: --test-config-file tests/dfx/perf/tests/test_tts.json"),
    )
-    parser.addoption(
-        "--assert-baseline",
-        action="store_true",
-        default=False,
-        help=(
-            "When set, omni/diffusion perf runners compare metrics against the baseline block in the JSON config "
-            "(default: off)."
-        ),
-    )
@@ -21,10 +21,12 @@
      "num-input-images": 0,
      "enable-negative-prompt": false,
      "baseline": {
-        "throughput_qps": 0,
-        "latency_mean": 0,
-        "peak_memory_mb_max": 0,
-        "peak_memory_mb_mean": 0
+        "H100": {
+          "throughput_qps": 0,
+          "latency_mean": 0,
+          "peak_memory_mb_max": 0,
+          "peak_memory_mb_mean": 0
+        }
      }
    },
    "result": {
@@ -46,9 +46,11 @@
  "median_audio_duration_s": 0,
  "p99_audio_duration_s": 0,
  "baseline": {
-    "mean_ttft_ms": 0,
-    "mean_audio_ttfp_ms": 0,
-    "mean_audio_rtf": 0
+    "H100": {
+      "mean_ttft_ms": 0,
+      "mean_audio_ttfp_ms": 0,
+      "mean_audio_rtf": 0
+    }
  },
  "random_input_len": 0,
  "random_output_len": 0
@@ -15,15 +15,10 @@ from tests.dfx.conftest import (
    get_runtime_resource_label,
    is_diffusion_perf_config,
    load_benchmark_configs,
-    resolve_baseline_value,
    run_benchmark,
 )
 from tests.helpers.runtime import OmniServer

-# Compare metrics to each test JSON ``baseline`` block only when pytest is run with ``--assert-baseline``
-# (registered in ``tests/dfx/conftest.py``; default: off). ``run_benchmark`` and ``_resolve_baseline_value`` are
-# defined in the same module.
-#
 # Optional JSON field ``mark`` is applied as pytest marks via
 # ``create_paired_omni_benchmark_pytest_params`` (e.g. ``"mark": [{"hardware_marks":
 # {"res": {"cuda": "H100"}, "num_cards": 2}}, "full_model", "omni"]``).
@@ -162,36 +157,8 @@ def benchmark_params(request):
    }


-def assert_result(
-    result,
-    params,
-    num_prompt,
-    *,
-    assert_baseline: bool,
-    sweep_index: int | None = None,
-    max_concurrency: Any | None = None,
-    request_rate: Any | None = None,
-) -> None:
+def assert_result(result, num_prompt) -> None:
    assert result["completed"] == num_prompt, "Request failures exist"
-    if not assert_baseline:
-        return
-    baseline_data = params.get("baseline", {})
-    for metric_name, baseline_raw in baseline_data.items():
-        current_value = result[metric_name]
-        baseline_value = resolve_baseline_value(
-            baseline_raw,
-            sweep_index=sweep_index,
-            max_concurrency=max_concurrency,
-            request_rate=request_rate,
-        )
-        if "throughput" in metric_name:
-            if current_value <= baseline_value:
-                print(
-                    f"ERROR: Throughput test results were below baseline: {metric_name}: {current_value} > {baseline_value}"
-                )
-        else:
-            if current_value >= baseline_value:
-                print(f"ERROR: Test results exceeded baseline: {metric_name}: {current_value} < {baseline_value}")


@pytest.mark.benchmark
@@ -200,7 +167,7 @@ def assert_result(
    paired_benchmark_params,
    indirect=["omni_server", "benchmark_params"],
 )
-def test_performance_benchmark(omni_server, benchmark_params, request):
+def test_performance_benchmark(omni_server, benchmark_params):
    test_name = benchmark_params["test_name"]
    params = benchmark_params["params"]
    dataset_name = params.get("dataset_name", "")
@@ -212,7 +179,6 @@ def test_performance_benchmark(omni_server, benchmark_params, request):
    print(f"Running benchmark for model: {model}")
    print(f"Benchmark parameters: {benchmark_params}")

-    assert_baseline = request.config.getoption("--assert-baseline", default=False)
    resource_label = get_runtime_resource_label()

    def to_list(value, default=None):
@@ -271,7 +237,7 @@ def test_performance_benchmark(omni_server, benchmark_params, request):
        break

    # QPS / request-rate sweep
-    for i, (qps, num_prompt) in enumerate(zip(qps_list, num_prompt_list)):
+    for sweep_index, (qps, num_prompt) in enumerate(zip(qps_list, num_prompt_list)):
        args = args + ["--request-rate", str(qps), "--num-prompts", str(num_prompt)]
        result = run_benchmark(
            args=args,
@@ -280,24 +246,15 @@ def test_performance_benchmark(omni_server, benchmark_params, request):
            dataset_name=dataset_name,
            num_prompt=num_prompt,
            baseline_config=params.get("baseline"),
-            sweep_index=i,
-            request_rate=qps,
-            max_concurrency=None,
+            sweep_index=sweep_index,
            random_input_len=params.get("random_input_len"),
            random_output_len=params.get("random_output_len"),
            resource_label=resource_label,
        )
-        assert_result(
-            result,
-            params,
-            num_prompt,
-            assert_baseline=assert_baseline,
-            sweep_index=i,
-            request_rate=qps,
-        )
+        assert_result(result, num_prompt)

    # concurrency test
-    for i, (concurrency, num_prompt) in enumerate(zip(max_concurrency_list, num_prompt_list)):
+    for sweep_index, (concurrency, num_prompt) in enumerate(zip(max_concurrency_list, num_prompt_list)):
        args = args + ["--max-concurrency", str(concurrency), "--num-prompts", str(num_prompt), "--request-rate", "inf"]
        result = run_benchmark(
            args=args,
@@ -306,18 +263,9 @@ def test_performance_benchmark(omni_server, benchmark_params, request):
            dataset_name=dataset_name,
            num_prompt=num_prompt,
            baseline_config=params.get("baseline"),
-            sweep_index=i,
-            request_rate=None,
-            max_concurrency=concurrency,
+            sweep_index=sweep_index,
            random_input_len=params.get("random_input_len"),
            random_output_len=params.get("random_output_len"),
            resource_label=resource_label,
        )
-        assert_result(
-            result,
-            params,
-            num_prompt,
-            assert_baseline=assert_baseline,
-            sweep_index=i,
-            max_concurrency=concurrency,
-        )
+        assert_result(result, num_prompt)
@@ -13,8 +13,6 @@ A config JSON file may be passed via --test-config-file. If omitted, every ``*.j
  pytest run_diffusion_benchmark.py -m "diffusion"
  pytest run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json

-Optional: ``--assert-baseline`` compares metrics to the ``baseline`` block in each benchmark entry (default: off).
-
 Optional JSON field ``mark`` is applied as pytest marks on that case via
 ``pytest.param`` (e.g. ``"mark": [{"hardware_marks": {"res": {"cuda": "H100"}, "num_cards": 1}}, "full_model", "diffusion"]``).

@@ -46,6 +44,7 @@ from tests.dfx.conftest import (
    get_runtime_resource_label,
    hardware_json_value,
    is_diffusion_perf_config,
+    resolve_baseline_for_sweep,
    resolve_pytest_marks,
    resource_label_for_filename,
 )
@@ -274,11 +273,6 @@ def _write_result_record(record: dict[str, Any]) -> Path:
    return target


-def _append_to_aggregated_file(record: dict[str, Any]) -> None:
-    """Backward-compatible wrapper; prefer :func:`_write_result_record`."""
-    _write_result_record(record)
-
-
 _server_lock = threading.Lock()

 # ---------------------------------------------------------------------------
@@ -356,10 +350,49 @@ def _kill_process_tree(pid: int) -> None:


 def _resolve_offline_model(model: str) -> str:
-    """Under HF_HUB_OFFLINE, resolve a HF repo id to its local snapshot dir."""
+    """Resolve a model string to a local directory.
+
+    Local paths pass through unchanged. HF repo ids are resolved to their local
+    snapshot dir when ``HF_HUB_OFFLINE`` is set. ``repo_id/subfolder`` (e.g.
+    ``MiniMaxAI/MiniMax-H3/FL2VA``) is resolved to the matching partition
+    subfolder of the snapshot, downloading only that subfolder when online.
+    """
    import huggingface_hub

-    if not model or os.path.isdir(model) or not huggingface_hub.constants.HF_HUB_OFFLINE:
+    if not model or os.path.isdir(model):
+        return model
+
+    # MiniMax-H3 custom code uses relative imports across files, which breaks
+    # when the HF cache snapshot is symlinked (``get_class_from_dynamic_module``
+    # resolves ``realpath`` into the blobs dir). Mirror the accuracy test's env
+    # overrides so CI / local runs can point at a materialized partition. The
+    # repo-root key (used with ``--task-type fl2va|ref2va``) expects a
+    # materialized root containing the FL2VA/Ref2VA subfolders.
+    model_env_overrides = {
+        "MiniMaxAI/MiniMax-H3": "VLLM_TEST_MINIMAX_H3_MODEL",
+        "MiniMaxAI/MiniMax-H3/FL2VA": "VLLM_TEST_MINIMAX_H3_FL2VA_MODEL",
+        "MiniMaxAI/MiniMax-H3/Ref2VA": "VLLM_TEST_MINIMAX_H3_REF2VA_MODEL",
+    }
+    env_name = model_env_overrides.get(model)
+    if env_name:
+        env_model = os.environ.get(env_name)
+        if env_model:
+            return env_model
+
+    parts = model.split("/")
+    if len(parts) >= 3:
+        repo_id = "/".join(parts[:2])
+        subfolder = "/".join(parts[2:])
+        from huggingface_hub import snapshot_download
+
+        snapshot_root = snapshot_download(
+            repo_id,
+            allow_patterns=[f"{subfolder}/**"],
+            local_files_only=huggingface_hub.constants.HF_HUB_OFFLINE,
+        )
+        return str(Path(snapshot_root) / subfolder)
+
+    if not huggingface_hub.constants.HF_HUB_OFFLINE:
        return model
    from huggingface_hub import snapshot_download

@@ -393,6 +426,11 @@ class DiffusionServer:
    def _start_server(self) -> None:
        env = os.environ.copy()
        env["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
+        # Give each run its own writable video-storage dir. The server default
+        # (/tmp/storage) may be missing or not writable on shared hosts / CI.
+        storage_path = Path(os.environ.get("VLLM_OMNI_STORAGE_PATH", str(BENCHMARK_RESULT_DIR / "storage")))
+        storage_path.mkdir(parents=True, exist_ok=True)
+        env["VLLM_OMNI_STORAGE_PATH"] = str(storage_path)

        cmd = [
            sys.executable,
@@ -678,7 +716,7 @@ def run_benchmark(
    with tempfile.NamedTemporaryFile(mode="w", suffix=".json", prefix="diffusion_bench_tmp_", delete=False) as tmp:
        tmp_result_file = Path(tmp.name)

-    exclude_keys = {"baseline", "dataset", "task", "name", "skip-performance-assertion"}
+    exclude_keys = {"baseline", "dataset", "task", "name"}

    cmd = [
        sys.executable,
@@ -772,12 +810,17 @@ def run_benchmark(
    completed = metrics.get("completed_requests", metrics.get("completed", 0))
    failed = metrics.get("failed_requests", metrics.get("failed", 0))

+    # Persist sweep-resolved baseline from params (already narrowed in _build_run_params).
+    baseline = params.get("baseline") or {}
+    metrics["baseline"] = baseline
+
    record: dict[str, Any] = {
        "test_name": test_name,
        "endpoint": endpoint,
        "timestamp": timestamp,
        "server_params": server_cfg.get("server_params"),
        "benchmark_params": params,
+        "baseline": baseline,
        "result": metrics,
        "log_file": str(log_file),
        "Model": model,
@@ -827,70 +870,14 @@ def run_benchmark(
 # ---------------------------------------------------------------------------


-def _resolve_baseline_value(
-    baseline_raw: Any,
-    *,
-    sweep_index: int | None,
-    max_concurrency: Any = None,
-    request_rate: Any = None,
-) -> Any:
-    """Pick the baseline threshold for this sweep step."""
-    if baseline_raw is None:
-        return 100000
-    if isinstance(baseline_raw, dict):
-        if max_concurrency is not None:
-            for key in (max_concurrency, str(max_concurrency)):
-                if key in baseline_raw:
-                    return baseline_raw[key]
-        if request_rate is not None:
-            for key in (request_rate, str(request_rate)):
-                if key in baseline_raw:
-                    return baseline_raw[key]
-        raise KeyError(
-            f"baseline dict has no key for max_concurrency={max_concurrency!r} "
-            f"or request_rate={request_rate!r}; keys={list(baseline_raw.keys())!r}"
-        )
-    if isinstance(baseline_raw, (list, tuple)):
-        if sweep_index is None:
-            raise ValueError("sweep_index is required when baseline is a list or tuple")
-        return baseline_raw[sweep_index]
-    return baseline_raw
-
-
 def assert_result(
    result: dict[str, Any],
-    params: dict[str, Any],
    num_prompts: int,
-    *,
-    sweep_index: int | None = None,
-    max_concurrency: Any = None,
-    request_rate: Any = None,
-    assert_baseline: bool = True,
 ) -> None:
-    """Assert that benchmark metrics satisfy the configured baselines."""
+    """Assert that the expected number of requests completed."""
    completed = result.get("completed_requests", result.get("completed", 0))
    assert completed == num_prompts, f"Expected {num_prompts} completed requests, got {completed}"

-    if not assert_baseline:
-        return
-    if params.get("skip-performance-assertion", False):
-        print("Skipping performance assertions.")
-        return
-
-    for metric, baseline_raw in params.get("baseline", {}).items():
-        current = result.get(metric)
-        assert current is not None, f"Metric '{metric}' not found in result: {list(result.keys())}"
-        threshold = _resolve_baseline_value(
-            baseline_raw,
-            sweep_index=sweep_index,
-            max_concurrency=max_concurrency,
-            request_rate=request_rate,
-        )
-        if "throughput" in metric:
-            assert current >= threshold, f"{metric}: {current:.4f} < baseline {threshold}"
-        else:
-            assert current <= threshold, f"{metric}: {current:.4f} > baseline {threshold}"
-

 def _default_benchmark_endpoint_for_task(task: str) -> str:
    """Return the default client-side benchmark endpoint for a diffusion task."""
@@ -932,15 +919,11 @@ def _build_run_params(
    if max_concurrency is not None:
        run_params["max-concurrency"] = max_concurrency
    if "baseline" in params:
-        run_params["baseline"] = {
-            metric: _resolve_baseline_value(
-                baseline_raw,
-                sweep_index=sweep_index,
-                max_concurrency=max_concurrency,
-                request_rate=request_rate,
-            )
-            for metric, baseline_raw in params["baseline"].items()
-        }
+        # Keep all hardware buckets; pick the metric value for this sweep step only.
+        run_params["baseline"] = resolve_baseline_for_sweep(
+            params.get("baseline"),
+            sweep_index=sweep_index,
+        )
    return run_params


@@ -963,36 +946,30 @@ def _iter_sweep_runs(params: dict[str, Any]) -> list[dict[str, Any]]:

    sweep_runs: list[dict[str, Any]] = []

-    for i, (request_rate, num_prompts) in enumerate(zip(request_rate_list, num_prompt_list)):
+    for sweep_index, (request_rate, num_prompts) in enumerate(zip(request_rate_list, num_prompt_list)):
        sweep_runs.append(
            {
                "params": _build_run_params(
                    params,
                    request_rate=request_rate,
                    num_prompts=num_prompts,
-                    sweep_index=i,
+                    sweep_index=sweep_index,
                ),
                "num_prompts": num_prompts,
-                "sweep_index": i,
-                "request_rate": request_rate,
-                "max_concurrency": None,
            }
        )

-    for i, (max_concurrency, num_prompts) in enumerate(zip(max_concurrency_list, num_prompt_list)):
+    for sweep_index, (max_concurrency, num_prompts) in enumerate(zip(max_concurrency_list, num_prompt_list)):
        sweep_runs.append(
            {
                "params": _build_run_params(
                    params,
                    max_concurrency=max_concurrency,
                    num_prompts=num_prompts,
-                    sweep_index=i,
                    request_rate="inf",
+                    sweep_index=sweep_index,
                ),
                "num_prompts": num_prompts,
-                "sweep_index": i,
-                "request_rate": None,
-                "max_concurrency": max_concurrency,
            }
        )

@@ -1003,11 +980,9 @@ def _iter_sweep_runs(params: dict[str, Any]) -> list[dict[str, Any]]:
                "params": _build_run_params(
                    params,
                    num_prompts=default_num_prompts,
+                    sweep_index=0,
                ),
                "num_prompts": default_num_prompts,
-                "sweep_index": None,
-                "request_rate": None,
-                "max_concurrency": None,
            }
        )

@@ -1023,14 +998,12 @@ def _iter_sweep_runs(params: dict[str, Any]) -> list[dict[str, Any]]:
    paired_benchmark_params,
    indirect=["diffusion_server", "benchmark_params"],
 )
-def test_diffusion_performance_benchmark(diffusion_server, benchmark_params, request):
+def test_diffusion_performance_benchmark(diffusion_server, benchmark_params):
    """Run the diffusion performance benchmark and verify request completion.

    One server is started per unique parallel configuration (module scope).
    For each server, all benchmark parameter sets defined in the config JSON
    are executed sequentially; metrics are recorded to the aggregated result file.
-
-    Pass ``--assert-baseline`` to compare ``throughput_qps``, ``latency_mean``, etc. to the JSON ``baseline`` block.
    """
    test_name = benchmark_params["test_name"]
    params = benchmark_params["params"]
@@ -1073,14 +1046,4 @@ def test_diffusion_performance_benchmark(diffusion_server, benchmark_params, req
            print(f"\n  Aggregated results: {_aggregated_result_file_for_source(str(source))}")
        print("=" * 60)

-        assert_baseline = request.config.getoption("--assert-baseline", default=False)
-
-        assert_result(
-            result,
-            params,
-            sweep_run["num_prompts"],
-            sweep_index=sweep_run["sweep_index"],
-            max_concurrency=sweep_run["max_concurrency"],
-            request_rate=sweep_run["request_rate"],
-            assert_baseline=assert_baseline,
-        )
+        assert_result(result, sweep_run["num_prompts"])
@@ -40,10 +40,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.495,
-          "latency_mean": 2.0238,
-          "peak_memory_mb_max": 29393.4286,
-          "peak_memory_mb_mean": 29392.7143
+          "H100": {
+            "throughput_qps": 0.495,
+            "latency_mean": 2.0238,
+            "peak_memory_mb_max": 29393.4286,
+            "peak_memory_mb_mean": 29392.7143
+          }
        }
      }
    ]
@@ -90,10 +92,12 @@
        "enable-negative-prompt": true,
        "num-input-images": 1,
        "baseline": {
-          "throughput_qps": 0.3451,
-          "latency_mean": 2.9031,
-          "peak_memory_mb_max": 30480.2857,
-          "peak_memory_mb_mean": 30479.2857
+          "H100": {
+            "throughput_qps": 0.3451,
+            "latency_mean": 2.9031,
+            "peak_memory_mb_max": 30480.2857,
+            "peak_memory_mb_mean": 30479.2857
+          }
        }
      }
    ]
@@ -139,10 +143,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.5048,
-          "latency_mean": 1.987,
-          "peak_memory_mb_max": 29380.0,
-          "peak_memory_mb_mean": 29380.0
+          "H100": {
+            "throughput_qps": 0.5048,
+            "latency_mean": 1.987,
+            "peak_memory_mb_max": 29380.0,
+            "peak_memory_mb_mean": 29380.0
+          }
        }
      }
    ]
@@ -189,10 +195,12 @@
        "enable-negative-prompt": true,
        "num-input-images": 1,
        "baseline": {
-          "throughput_qps": 0.149,
-          "latency_mean": 6.7157,
-          "peak_memory_mb_max": 30574.0,
-          "peak_memory_mb_mean": 30574.0
+          "H100": {
+            "throughput_qps": 0.149,
+            "latency_mean": 6.7157,
+            "peak_memory_mb_max": 30574.0,
+            "peak_memory_mb_mean": 30574.0
+          }
        }
      }
    ]
@@ -24,10 +24,12 @@
                "warmup-num-inference-steps": 28,
                "enable-negative-prompt": true,
                "baseline": {
-                    "throughput_qps": {"1": 0.0303, "2": 0.0302, "4": 0.0302},
-                    "latency_mean": {"1": 32.6, "2": 62.2, "4": 111.7},
-                    "peak_memory_mb_max": 40500,
-                    "peak_memory_mb_mean": 40500
+                    "H100": {
+                        "throughput_qps": [0.0303, 0.0302, 0.0302],
+                        "latency_mean": [32.6, 62.2, 111.7],
+                        "peak_memory_mb_max": 40500,
+                        "peak_memory_mb_mean": 40500
+                    }
                }
            }
        ]
@@ -24,10 +24,12 @@
                "warmup-num-inference-steps": 28,
                "enable-negative-prompt": true,
                "baseline": {
-                    "throughput_qps": {"1": 0.0549, "2": 0.0547, "4": 0.0549},
-                    "latency_mean": {"1": 18.0, "2": 34.3, "4": 61.4},
-                    "peak_memory_mb_max": 40500,
-                    "peak_memory_mb_mean": 40500
+                    "H100": {
+                        "throughput_qps": [0.0549, 0.0547, 0.0549],
+                        "latency_mean": [18.0, 34.3, 61.4],
+                        "peak_memory_mb_max": 40500,
+                        "peak_memory_mb_mean": 40500
+                    }
                }
            }
        ]
@@ -48,9 +48,11 @@
          "guardrails": false
        },
        "baseline": {
-          "throughput_qps": 1.107,
-          "latency_mean": 0.8929,
-          "peak_memory_mb_mean": 80000.0
+          "H100": {
+            "throughput_qps": 1.107,
+            "latency_mean": 0.8929,
+            "peak_memory_mb_mean": 80000.0
+          }
        }
      }
    ]
@@ -115,9 +117,11 @@
          "use_duration_template": false
        },
        "baseline": {
-          "throughput_qps": 0.027,
-          "latency_mean": 36.6824,
-          "peak_memory_mb_mean": 30155.4
+          "H100": {
+            "throughput_qps": 0.027,
+            "latency_mean": 36.6824,
+            "peak_memory_mb_mean": 30155.4
+          }
        }
      }
    ]
@@ -181,9 +185,11 @@
          "guardrails": false
        },
        "baseline": {
-          "throughput_qps": 0.027,
-          "latency_mean": 36.968,
-          "peak_memory_mb_mean": 30443.6
+          "H100": {
+            "throughput_qps": 0.027,
+            "latency_mean": 36.968,
+            "peak_memory_mb_mean": 30443.6
+          }
        }
      }
    ]
@@ -251,9 +257,11 @@
          "condition_video_keep": "first"
        },
        "baseline": {
-          "throughput_qps": 0.027,
-          "latency_mean": 37.2117,
-          "peak_memory_mb_mean": 30135.6
+          "H100": {
+            "throughput_qps": 0.027,
+            "latency_mean": 37.2117,
+            "peak_memory_mb_mean": 30135.6
+          }
        }
      }
    ]
@@ -46,12 +46,14 @@
        "seed_tts_locale": "en",
        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "median_audio_ttfp_ms": [
-            223.3584
-          ],
-          "median_audio_rtf": [
-            0.4393
-          ]
+          "H100": {
+            "median_audio_ttfp_ms": [
+              223.3584
+            ],
+            "median_audio_rtf": [
+              0.4393
+            ]
+          }
        }
      },
      {
@@ -70,15 +72,17 @@
        "seed_tts_locale": "en",
        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "median_audio_ttfp_ms": [
-            276.575
-          ],
-          "median_audio_rtf": [
-            0.5073
-          ],
-          "audio_throughput": [
-            27.3728
-          ]
+          "H100": {
+            "median_audio_ttfp_ms": [
+              276.575
+            ],
+            "median_audio_rtf": [
+              0.5073
+            ],
+            "audio_throughput": [
+              27.3728
+            ]
+          }
        }
      }
    ]
@@ -133,13 +133,14 @@
        "bot-task": "think_recaption",
        "max-concurrency": 16,
        "num-prompts": 96,
-        "skip-performance-assertion": true,
        "baseline": {
-          "throughput_qps": 0.1874,
-          "latency_p99": 101.852,
-          "latency_mean": 79.0385,
-          "peak_memory_mb_max": 70000,
-          "peak_memory_mb_mean": 70000
+          "H200": {
+            "throughput_qps": 0.1874,
+            "latency_p99": 101.852,
+            "latency_mean": 79.0385,
+            "peak_memory_mb_max": 70000,
+            "peak_memory_mb_mean": 70000
+          }
        }
      }
    ]
@@ -37,12 +37,13 @@
        "num-inference-steps": 8,
        "num-prompts": 10,
        "max-concurrency": 1,
-        "skip-performance-assertion": true,
        "baseline": {
-          "throughput_qps": 0.21,
-          "latency_p99": 4.7469,
-          "peak_memory_mb_max": 101100,
-          "peak_memory_mb_mean": 101100
+          "H200": {
+            "throughput_qps": 0.21,
+            "latency_p99": 4.7469,
+            "peak_memory_mb_max": 101100,
+            "peak_memory_mb_mean": 101100
+          }
        }
      }
    ]
@@ -37,12 +37,13 @@
        "num-inference-steps": 8,
        "num-prompts": 10,
        "max-concurrency": 1,
-        "skip-performance-assertion": true,
        "baseline": {
-          "throughput_qps": 0.20,
-          "latency_p99": 5.1025,
-          "peak_memory_mb_max": 97402,
-          "peak_memory_mb_mean": 97402
+          "H200": {
+            "throughput_qps": 0.20,
+            "latency_p99": 5.1025,
+            "peak_memory_mb_max": 97402,
+            "peak_memory_mb_mean": 97402
+          }
        }
      }
    ]
@@ -36,13 +36,14 @@
        "num-inference-steps": 8,
        "num-prompts": 10,
        "max-concurrency": 1,
-        "skip-performance-assertion": true,
        "baseline": {
-          "throughput_qps": 0.213,
-          "latency_mean": 4.6953,
-          "latency_p99": 4.8601,
-          "peak_memory_mb_max": 56912.0,
-          "peak_memory_mb_mean": 56912.0
+          "H200": {
+            "throughput_qps": 0.213,
+            "latency_mean": 4.6953,
+            "latency_p99": 4.8601,
+            "peak_memory_mb_max": 56912.0,
+            "peak_memory_mb_mean": 56912.0
+          }
        }
      }
    ]
@@ -26,8 +26,7 @@
                "warmup-requests": 1,
                "warmup-concurrency": 1,
                "warmup-num-inference-steps": 2,
-                "enable-negative-prompt": true,
-                "skip-performance-assertion": true
+                "enable-negative-prompt": true
            }
        ]
    }
@@ -0,0 +1,295 @@
+[
+  {
+    "test_name": "test_minimax_h3_t2v_4gpu",
+    "mark": [
+      {
+        "hardware_marks": {
+          "res": {
+            "cuda": "H100"
+          },
+          "num_cards": 4
+        }
+      },
+      "full_model",
+      "diffusion"
+    ],
+    "description": "MiniMax-H3 T2VA text-to-video+audio on 4 GPUs (USP4 + HSDP4, encoder-TP4, VAE-PP4 tile)",
+    "server_type": "vllm-omni",
+    "benchmark_endpoint": "/v1/videos",
+    "server_params": {
+      "model": "MiniMaxAI/MiniMax-H3",
+      "serve_args": {
+        "trust-remote-code": true,
+        "task-type": "fl2va",
+        "num-gpus": 4,
+        "usp": 4,
+        "ring": 1,
+        "use-hsdp": true,
+        "hsdp-shard-size": 4,
+        "text-encoder-tp-size": 4,
+        "vae-patch-parallel-size": 4,
+        "vae-parallel-mode": "tile",
+        "vae-use-tiling": true,
+        "diffusion-attention-backend": "FLASH_ATTN",
+        "stage-init-timeout": 1800,
+        "init-timeout": 1800,
+        "enable-diffusion-pipeline-profiler": true
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "1344x768_frames209_steps8",
+        "dataset": "random",
+        "task": "t2v",
+        "num-prompts": 3,
+        "max-concurrency": 1,
+        "warmup-requests": 1,
+        "warmup-concurrency": 1,
+        "seed": 42,
+        "random-request-config": [
+          {
+            "width": 1344,
+            "height": 768,
+            "prompt": "Cinematic drone shot pushing through misty mountain ridges at sunrise, golden light catching the clouds, slow and smooth.",
+            "num_inference_steps": 8,
+            "num_frames": 209,
+            "fps": 24,
+            "weight": 1
+          }
+        ],
+        "extra-body": {
+          "flow_shift": 12.0,
+          "extra_params": "{\"task\":\"t2va\",\"duration\":8.7,\"aspect_ratio\":\"16:9\",\"audio_flow_shift\":3.0}"
+        },
+        "baseline": {
+          "H100": {
+            "throughput_qps": 0.0211,
+            "latency_mean": 38.3252,
+            "peak_memory_mb_mean": 54219.6
+          }
+        }
+      }
+    ]
+  },
+  {
+    "test_name": "test_minimax_h3_ti2v_4gpu",
+    "mark": [
+      {
+        "hardware_marks": {
+          "res": {
+            "cuda": "H100"
+          },
+          "num_cards": 4
+        }
+      },
+      "full_model",
+      "diffusion"
+    ],
+    "description": "MiniMax-H3 FL2VA first-frame image-to-video+audio on 4 GPUs (USP4 + HSDP4, encoder-TP4, VAE-PP4 tile)",
+    "server_type": "vllm-omni",
+    "benchmark_endpoint": "/v1/videos",
+    "server_params": {
+      "model": "MiniMaxAI/MiniMax-H3",
+      "serve_args": {
+        "trust-remote-code": true,
+        "task-type": "fl2va",
+        "num-gpus": 4,
+        "usp": 4,
+        "ring": 1,
+        "use-hsdp": true,
+        "hsdp-shard-size": 4,
+        "text-encoder-tp-size": 4,
+        "vae-patch-parallel-size": 4,
+        "vae-parallel-mode": "tile",
+        "vae-use-tiling": true,
+        "diffusion-attention-backend": "FLASH_ATTN",
+        "stage-init-timeout": 1800,
+        "init-timeout": 1800,
+        "enable-diffusion-pipeline-profiler": true
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "1344x768_frames209_steps8",
+        "dataset": "random",
+        "task": "ti2v",
+        "num-prompts": 3,
+        "max-concurrency": 1,
+        "warmup-requests": 1,
+        "warmup-concurrency": 1,
+        "seed": 42,
+        "random-request-config": [
+          {
+            "width": 1344,
+            "height": 768,
+            "prompt": "Continue the scene: the camera slowly pushes in on the subject as the background city lights drift and blur, with the actor singing softly and emotionally.",
+            "num_inference_steps": 8,
+            "num_frames": 209,
+            "fps": 24,
+            "weight": 1
+          }
+        ],
+        "extra-body": {
+          "flow_shift": 12.0,
+          "extra_params": "{\"task\":\"fl2va\",\"duration\":8.7,\"audio_flow_shift\":3.0}"
+        },
+        "num-input-images": 1,
+        "baseline": {
+          "H100": {
+            "throughput_qps": 0.0212,
+            "latency_mean": 38.1385,
+            "peak_memory_mb_mean": 54622.8
+          }
+        }
+      }
+    ]
+  },
+  {
+    "test_name": "test_minimax_h3_v2v_4gpu",
+    "mark": [
+      {
+        "hardware_marks": {
+          "res": {
+            "cuda": "H100"
+          },
+          "num_cards": 4
+        }
+      },
+      "full_model",
+      "diffusion"
+    ],
+    "description": "MiniMax-H3 Ref2VA video-reference-to-video+audio on 4 GPUs (USP4 + HSDP4, encoder-TP4, VAE-PP4 tile)",
+    "server_type": "vllm-omni",
+    "benchmark_endpoint": "/v1/videos",
+    "server_params": {
+      "model": "MiniMaxAI/MiniMax-H3",
+      "serve_args": {
+        "trust-remote-code": true,
+        "task-type": "ref2va",
+        "num-gpus": 4,
+        "usp": 4,
+        "ring": 1,
+        "use-hsdp": true,
+        "hsdp-shard-size": 4,
+        "text-encoder-tp-size": 4,
+        "vae-patch-parallel-size": 4,
+        "vae-parallel-mode": "tile",
+        "vae-use-tiling": true,
+        "diffusion-attention-backend": "FLASH_ATTN",
+        "stage-init-timeout": 1800,
+        "init-timeout": 1800,
+        "enable-diffusion-pipeline-profiler": true
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "1344x768_frames209_steps8",
+        "dataset": "random",
+        "task": "v2v",
+        "num-prompts": 3,
+        "max-concurrency": 1,
+        "warmup-requests": 1,
+        "warmup-concurrency": 1,
+        "seed": 42,
+        "random-request-config": [
+          {
+            "width": 1344,
+            "height": 768,
+            "prompt": "Restyle the reference footage into a cinematic night scene with neon reflections, keeping the same camera motion and subject.",
+            "num_inference_steps": 8,
+            "num_frames": 209,
+            "fps": 24,
+            "weight": 1
+          }
+        ],
+        "extra-body": {
+          "flow_shift": 12.0,
+          "extra_params": "{\"task\":\"ref2va\",\"duration\":8.7,\"aspect_ratio\":\"16:9\",\"audio_flow_shift\":3.0}"
+        },
+        "baseline": {
+          "H100": {
+            "throughput_qps": 0.0077,
+            "latency_mean": 105.7508,
+            "peak_memory_mb_mean": 55180.8
+          }
+        }
+      }
+    ]
+  },
+  {
+    "test_name": "test_minimax_h3_t2v_dlo_4gpu",
+    "mark": [
+      {
+        "hardware_marks": {
+          "res": {
+            "cuda": "H100"
+          },
+          "num_cards": 4
+        }
+      },
+      "full_model",
+      "diffusion"
+    ],
+    "description": "MiniMax-H3 T2VA with 4-card distributed layerwise offload (DLO), 8 denoise steps (USP4 + HSDP4, encoder-TP4, VAE-PP4 tile)",
+    "server_type": "vllm-omni",
+    "benchmark_endpoint": "/v1/videos",
+    "server_params": {
+      "model": "MiniMaxAI/MiniMax-H3",
+      "serve_args": {
+        "trust-remote-code": true,
+        "task-type": "fl2va",
+        "num-gpus": 4,
+        "usp": 4,
+        "ring": 1,
+        "use-hsdp": true,
+        "hsdp-shard-size": 4,
+        "text-encoder-tp-size": 4,
+        "vae-patch-parallel-size": 4,
+        "vae-parallel-mode": "tile",
+        "vae-use-tiling": true,
+        "diffusion-attention-backend": "FLASH_ATTN",
+        "stage-init-timeout": 1800,
+        "init-timeout": 1800,
+        "enable-diffusion-pipeline-profiler": true,
+        "enable-distributed-layerwise-offload": true,
+        "dlo-no-use-allgather": true,
+        "dlo-resident-layers": 8,
+        "enforce-eager": true
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "1344x768_frames209_steps8",
+        "dataset": "random",
+        "task": "t2v",
+        "num-prompts": 3,
+        "max-concurrency": 1,
+        "warmup-requests": 1,
+        "warmup-concurrency": 1,
+        "seed": 42,
+        "random-request-config": [
+          {
+            "width": 1344,
+            "height": 768,
+            "prompt": "Cinematic drone shot pushing through misty mountain ridges at sunrise, golden light catching the clouds, slow and smooth.",
+            "num_inference_steps": 8,
+            "num_frames": 209,
+            "fps": 24,
+            "weight": 1
+          }
+        ],
+        "extra-body": {
+          "flow_shift": 12.0,
+          "extra_params": "{\"task\":\"t2va\",\"duration\":8.7,\"aspect_ratio\":\"16:9\",\"audio_flow_shift\":3.0}"
+        },
+        "baseline": {
+          "H100": {
+            "throughput_qps": 0.0212,
+            "latency_mean": 38.2089,
+            "peak_memory_mb_mean": 36218.4
+          }
+        }
+      }
+    ]
+  }
+]
@@ -44,27 +44,29 @@
        "ignore_eos": true,
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            96.4061,
-            140.8147,
-            271.9025,
-            362.2916,
-            507.776
-          ],
-          "mean_e2el_ms": [
-            18507.4582,
-            28365.4956,
-            31907.9613,
-            48161.5829,
-            72630.8601
-          ],
-          "mean_audio_rtf": [
-            0.1175,
-            0.18,
-            0.2359,
-            0.3278,
-            0.6296
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              96.4061,
+              140.8147,
+              271.9025,
+              362.2916,
+              507.776
+            ],
+            "mean_e2el_ms": [
+              18507.4582,
+              28365.4956,
+              31907.9613,
+              48161.5829,
+              72630.8601
+            ],
+            "mean_audio_rtf": [
+              0.1175,
+              0.18,
+              0.2359,
+              0.3278,
+              0.6296
+            ]
+          }
        }
      },
      {
@@ -91,15 +93,17 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            354.0951
-          ],
-          "mean_e2el_ms": [
-            3952.5615
-          ],
-          "mean_audio_rtf": [
-            0.117
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              354.0951
+            ],
+            "mean_e2el_ms": [
+              3952.5615
+            ],
+            "mean_audio_rtf": [
+              0.117
+            ]
+          }
        }
      },
      {
@@ -128,15 +132,17 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            259.6317
-          ],
-          "mean_e2el_ms": [
-            4253.3009
-          ],
-          "mean_audio_rtf": [
-            0.1448
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              259.6317
+            ],
+            "mean_e2el_ms": [
+              4253.3009
+            ],
+            "mean_audio_rtf": [
+              0.1448
+            ]
+          }
        }
      },
      {
@@ -167,15 +173,17 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            457.4453
-          ],
-          "mean_e2el_ms": [
-            5912.5623
-          ],
-          "mean_audio_rtf": [
-            0.1967
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              457.4453
+            ],
+            "mean_e2el_ms": [
+              5912.5623
+            ],
+            "mean_audio_rtf": [
+              0.1967
+            ]
+          }
        }
      },
      {
@@ -206,20 +214,22 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            90.8033,
-            202.1779,
-            326.5864,
-            544.8914,
-            961.3929
-          ],
-          "mean_e2el_ms": [
-            4517.3411,
-            7857.9711,
-            9861.1687,
-            13283.3363,
-            19541.0736
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              90.8033,
+              202.1779,
+              326.5864,
+              544.8914,
+              961.3929
+            ],
+            "mean_e2el_ms": [
+              4517.3411,
+              7857.9711,
+              9861.1687,
+              13283.3363,
+              19541.0736
+            ]
+          }
        }
      },
      {
@@ -251,12 +261,14 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            318.048
-          ],
-          "mean_e2el_ms": [
-            807.6114
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              318.048
+            ],
+            "mean_e2el_ms": [
+              807.6114
+            ]
+          }
        }
      },
      {
@@ -290,12 +302,14 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            186.168
-          ],
-          "mean_e2el_ms": [
-            726.1137
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              186.168
+            ],
+            "mean_e2el_ms": [
+              726.1137
+            ]
+          }
        }
      },
      {
@@ -331,12 +345,14 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            375.5282
-          ],
-          "mean_e2el_ms": [
-            1006.1663
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              375.5282
+            ],
+            "mean_e2el_ms": [
+              1006.1663
+            ]
+          }
        }
      }
    ]
@@ -35,10 +35,12 @@
        "ignore_eos": true,
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-            "mean_ttft_ms": [130.5751, 278.6469, 324.7249, 911.6085],
-            "mean_tpot_ms": [7.9854, 11.1209, 16.1507, 20.7485],
-            "mean_audio_ttfp_ms": [486.486, 954.1791, 1289.6514, 2196.8275],
-            "mean_audio_rtf": [0.1896, 0.3962, 0.4098, 0.7322]
+            "H100": {
+                "mean_ttft_ms": [130.5751, 278.6469, 324.7249, 911.6085],
+                "mean_tpot_ms": [7.9854, 11.1209, 16.1507, 20.7485],
+                "mean_audio_ttfp_ms": [486.486, 954.1791, 1289.6514, 2196.8275],
+                "mean_audio_rtf": [0.1896, 0.3962, 0.4098, 0.7322]
+            }
        }
      },
      {
@@ -61,9 +63,11 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-            "mean_ttft_ms": [340.5122],
-            "mean_audio_ttfp_ms": [463.0752],
-            "mean_audio_rtf": [0.1153]
+            "H100": {
+                "mean_ttft_ms": [340.5122],
+                "mean_audio_ttfp_ms": [463.0752],
+                "mean_audio_rtf": [0.1153]
+            }
        }
      },
      {
@@ -88,9 +92,11 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-            "mean_ttft_ms": [236.926],
-            "mean_audio_ttfp_ms": [395.2182],
-            "mean_audio_rtf": [0.1259]
+            "H100": {
+                "mean_ttft_ms": [236.926],
+                "mean_audio_ttfp_ms": [395.2182],
+                "mean_audio_rtf": [0.1259]
+            }
        }
      },
      {
@@ -117,9 +123,11 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-            "mean_ttft_ms": [470.6126],
-            "mean_audio_ttfp_ms": [665.5474],
-            "mean_audio_rtf": [0.166]
+            "H100": {
+                "mean_ttft_ms": [470.6126],
+                "mean_audio_ttfp_ms": [665.5474],
+                "mean_audio_rtf": [0.166]
+            }
        }
      }
    ]
@@ -44,27 +44,29 @@
        "ignore_eos": true,
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            115.8348,
-            173.1823,
-            446.672,
-            464.5985,
-            3420.0634
-          ],
-          "mean_e2el_ms": [
-            25070.9523,
-            31992.0646,
-            37619.7669,
-            50393.8257,
-            136619.2743
-          ],
-          "mean_audio_rtf": [
-            0.169,
-            0.2111,
-            0.2475,
-            0.3551,
-            0.9266
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              115.8348,
+              173.1823,
+              446.672,
+              464.5985,
+              3420.0634
+            ],
+            "mean_e2el_ms": [
+              25070.9523,
+              31992.0646,
+              37619.7669,
+              50393.8257,
+              136619.2743
+            ],
+            "mean_audio_rtf": [
+              0.169,
+              0.2111,
+              0.2475,
+              0.3551,
+              0.9266
+            ]
+          }
        }
      },
      {
@@ -91,15 +93,17 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            343.1377
-          ],
-          "mean_e2el_ms": [
-            4811.1123
-          ],
-          "mean_audio_rtf": [
-            0.1463
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              343.1377
+            ],
+            "mean_e2el_ms": [
+              4811.1123
+            ],
+            "mean_audio_rtf": [
+              0.1463
+            ]
+          }
        }
      },
      {
@@ -128,15 +132,17 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            245.5446
-          ],
-          "mean_e2el_ms": [
-            5019.1256
-          ],
-          "mean_audio_rtf": [
-            0.1713
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              245.5446
+            ],
+            "mean_e2el_ms": [
+              5019.1256
+            ],
+            "mean_audio_rtf": [
+              0.1713
+            ]
+          }
        }
      },
      {
@@ -167,15 +173,17 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
        "baseline": {
-          "mean_ttft_ms": [
-            506.1407
-          ],
-          "mean_e2el_ms": [
-            6814.9656
-          ],
-          "mean_audio_rtf": [
-            0.2213
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              506.1407
+            ],
+            "mean_e2el_ms": [
+              6814.9656
+            ],
+            "mean_audio_rtf": [
+              0.2213
+            ]
+          }
        }
      }
    ]
@@ -51,20 +51,22 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            69.8081,
-            173.0651,
-            271.8108,
-            413.4079,
-            497.2546
-          ],
-          "mean_e2el_ms": [
-            4720.8225,
-            7937.2532,
-            9962.3566,
-            13724.5479,
-            18324.091
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              69.8081,
+              173.0651,
+              271.8108,
+              413.4079,
+              497.2546
+            ],
+            "mean_e2el_ms": [
+              4720.8225,
+              7937.2532,
+              9962.3566,
+              13724.5479,
+              18324.091
+            ]
+          }
        }
      },
      {
@@ -97,12 +99,14 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            25.9477
-          ],
-          "mean_e2el_ms": [
-            517.3155
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              25.9477
+            ],
+            "mean_e2el_ms": [
+              517.3155
+            ]
+          }
        }
      },
      {
@@ -137,12 +141,14 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            26.7056
-          ],
-          "mean_e2el_ms": [
-            551.308
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              26.7056
+            ],
+            "mean_e2el_ms": [
+              551.308
+            ]
+          }
        }
      },
      {
@@ -179,12 +185,14 @@
        },
        "percentile-metrics": "ttft,tpot,itl,e2el",
        "baseline": {
-          "mean_ttft_ms": [
-            27.6906
-          ],
-          "mean_e2el_ms": [
-            596.5284
-          ]
+          "H100": {
+            "mean_ttft_ms": [
+              27.6906
+            ],
+            "mean_e2el_ms": [
+              596.5284
+            ]
+          }
        }
      }
    ]
@@ -34,10 +34,12 @@
        "num-input-images": 2,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0721,
-          "latency_mean": 13.8398,
-          "peak_memory_mb_max": 57610.0,
-          "peak_memory_mb_mean": 57610.0
+          "H100": {
+            "throughput_qps": 0.0721,
+            "latency_mean": 13.8398,
+            "peak_memory_mb_max": 57610.0,
+            "peak_memory_mb_mean": 57610.0
+          }
        }
      },
      {
@@ -52,10 +54,12 @@
        "num-input-images": 2,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0186,
-          "latency_mean": 53.6942,
-          "peak_memory_mb_max": 67790.0,
-          "peak_memory_mb_mean": 67790.0
+          "H100": {
+            "throughput_qps": 0.0186,
+            "latency_mean": 53.6942,
+            "peak_memory_mb_max": 67790.0,
+            "peak_memory_mb_mean": 67790.0
+          }
        }
      }
    ]
@@ -100,10 +104,12 @@
        "num-input-images": 2,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0538,
-          "latency_mean": 18.5591,
-          "peak_memory_mb_max": 56449.4286,
-          "peak_memory_mb_mean": 56449.4286
+          "H100": {
+            "throughput_qps": 0.0538,
+            "latency_mean": 18.5591,
+            "peak_memory_mb_max": 56449.4286,
+            "peak_memory_mb_mean": 56449.4286
+          }
        }
      }
    ]
@@ -157,10 +163,12 @@
        "num-input-images": 2,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.2776,
-          "latency_mean": 3.5938,
-          "peak_memory_mb_max": 58113.7143,
-          "peak_memory_mb_mean": 58075.1429
+          "H100": {
+            "throughput_qps": 0.2776,
+            "latency_mean": 3.5938,
+            "peak_memory_mb_max": 58113.7143,
+            "peak_memory_mb_mean": 58075.1429
+          }
        }
      },
      {
@@ -175,10 +183,12 @@
        "num-input-images": 2,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.1051,
-          "latency_mean": 9.4918,
-          "peak_memory_mb_max": 68107.4286,
-          "peak_memory_mb_mean": 68107.4286
+          "H100": {
+            "throughput_qps": 0.1051,
+            "latency_mean": 9.4918,
+            "peak_memory_mb_max": 68107.4286,
+            "peak_memory_mb_mean": 68107.4286
+          }
        }
      }
    ]
@@ -33,10 +33,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.1129,
-          "latency_mean": 8.8352,
-          "peak_memory_mb_max": 58572.0,
-          "peak_memory_mb_mean": 58572.0
+          "H100": {
+            "throughput_qps": 0.1129,
+            "latency_mean": 8.8352,
+            "peak_memory_mb_max": 58572.0,
+            "peak_memory_mb_mean": 58572.0
+          }
        }
      },
      {
@@ -50,10 +52,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.024,
-          "latency_mean": 41.5686,
-          "peak_memory_mb_max": 66368.0,
-          "peak_memory_mb_mean": 66368.0
+          "H100": {
+            "throughput_qps": 0.024,
+            "latency_mean": 41.5686,
+            "peak_memory_mb_max": 66368.0,
+            "peak_memory_mb_mean": 66368.0
+          }
        }
      }
    ]
@@ -96,10 +100,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0672,
-          "latency_mean": 14.8517,
-          "peak_memory_mb_max": 57076.0,
-          "peak_memory_mb_mean": 57076.0
+          "H100": {
+            "throughput_qps": 0.0672,
+            "latency_mean": 14.8517,
+            "peak_memory_mb_max": 57076.0,
+            "peak_memory_mb_mean": 57076.0
+          }
        }
      }
    ]
@@ -152,10 +158,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.2879,
-          "latency_mean": 3.4701,
-          "peak_memory_mb_max": 58758.0,
-          "peak_memory_mb_mean": 58758.0
+          "H100": {
+            "throughput_qps": 0.2879,
+            "latency_mean": 3.4701,
+            "peak_memory_mb_max": 58758.0,
+            "peak_memory_mb_mean": 58758.0
+          }
        }
      },
      {
@@ -169,10 +177,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.1113,
-          "latency_mean": 8.9656,
-          "peak_memory_mb_max": 67424.0,
-          "peak_memory_mb_mean": 67424.0
+          "H100": {
+            "throughput_qps": 0.1113,
+            "latency_mean": 8.9656,
+            "peak_memory_mb_max": 67424.0,
+            "peak_memory_mb_mean": 67424.0
+          }
        }
      }
    ]
@@ -33,10 +33,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0703,
-          "latency_mean": 14.208,
-          "peak_memory_mb_max": 63225.1429,
-          "peak_memory_mb_mean": 63225.1429
+          "H100": {
+            "throughput_qps": 0.0703,
+            "latency_mean": 14.208,
+            "peak_memory_mb_max": 63225.1429,
+            "peak_memory_mb_mean": 63225.1429
+          }
        }
      },
      {
@@ -50,10 +52,12 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0417,
-          "latency_mean": 23.914,
-          "peak_memory_mb_max": 63225.1429,
-          "peak_memory_mb_mean": 63225.1429
+          "H100": {
+            "throughput_qps": 0.0417,
+            "latency_mean": 23.914,
+            "peak_memory_mb_max": 63225.1429,
+            "peak_memory_mb_mean": 63225.1429
+          }
        }
      }
    ]
@@ -33,9 +33,11 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.4153,
-          "latency_mean": 2.4065,
-          "peak_memory_mb_mean": 55121.4286
+          "H100": {
+            "throughput_qps": 0.4153,
+            "latency_mean": 2.4065,
+            "peak_memory_mb_mean": 55121.4286
+          }
        }
      },
      {
@@ -49,9 +51,11 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0423,
-          "latency_mean": 23.5991,
-          "peak_memory_mb_mean": 65118.0
+          "H100": {
+            "throughput_qps": 0.0423,
+            "latency_mean": 23.5991,
+            "peak_memory_mb_mean": 65118.0
+          }
        }
      }
    ]
@@ -91,9 +95,11 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.4175,
-          "latency_mean": 2.3953,
-          "peak_memory_mb_mean": 55088.5714
+          "H100": {
+            "throughput_qps": 0.4175,
+            "latency_mean": 2.3953,
+            "peak_memory_mb_mean": 55088.5714
+          }
        }
      },
      {
@@ -107,9 +113,11 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.0423,
-          "latency_mean": 23.5768,
-          "peak_memory_mb_mean": 66012.2857
+          "H100": {
+            "throughput_qps": 0.0423,
+            "latency_mean": 23.5768,
+            "peak_memory_mb_mean": 66012.2857
+          }
        }
      }
    ]
@@ -152,9 +160,11 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.1214,
-          "latency_mean": 8.2211,
-          "peak_memory_mb_mean": 54501.1429
+          "H100": {
+            "throughput_qps": 0.1214,
+            "latency_mean": 8.2211,
+            "peak_memory_mb_mean": 54501.1429
+          }
        }
      }
    ]
@@ -207,9 +217,11 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.559,
-          "latency_mean": 1.794,
-          "peak_memory_mb_mean": 55290.5714
+          "H100": {
+            "throughput_qps": 0.559,
+            "latency_mean": 1.794,
+            "peak_memory_mb_mean": 55290.5714
+          }
        }
      },
      {
@@ -223,9 +235,11 @@
        "max-concurrency": 1,
        "enable-negative-prompt": true,
        "baseline": {
-          "throughput_qps": 0.1918,
-          "latency_mean": 5.2022,
-          "peak_memory_mb_mean": 64838.8571
+          "H100": {
+            "throughput_qps": 0.1918,
+            "latency_mean": 5.2022,
+            "peak_memory_mb_mean": 64838.8571
+          }
        }
      }
    ]
@@ -269,9 +283,11 @@
        "warmup-num-inference-steps": 20,
        "enable-negative-prompt": true,
        "baseline": {
-            "throughput_qps": [0.4125, 0.6714, 0.6674, 0.8159],
-            "latency_mean": [2.4215, 2.992, 5.9293, 9.656],
-            "peak_memory_mb_mean": [55147.4286, 55148.0, 55150.4643, 55152.3821]
+            "H100": {
+                "throughput_qps": [0.4125, 0.6714, 0.6674, 0.8159],
+                "latency_mean": [2.4215, 2.992, 5.9293, 9.656],
+                "peak_memory_mb_mean": [55147.4286, 55148.0, 55150.4643, 55152.3821]
+            }
        }
      }
    ]
Author	SHA1	Message	Date
Hongsheng Liu	01e7d09119	docs: define module ownership contracts Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-08-07 11:45:59 +08:00
Wang fuyin	3d7fc3b9ba	[Config] Read engine args from VllmOmniConfig (#5678 ) Signed-off-by: Acerak01-fy <wfy2003324@163.com> Signed-off-by: Wang fuyin <wfy2003324@163.com>	2026-08-07 11:17:33 +08:00
Fyrgo 道路自信 amy-why-3459	6ec9033065	[Model] MiniCPM-o 4.5: reuse FlashAttention unpadding metadata (#5165 ) Signed-off-by: 道路自信 <182356120+daoluzixin@users.noreply.github.com> Co-authored-by: 道路自信 <182356120+daoluzixin@users.noreply.github.com> Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>	2026-08-07 10:25:04 +08:00
WeiQing Chen	a874b8e09b	[Perf][CI] Add MiniMax-H3 4xH100 diffusion perf config (#5836 ) Signed-off-by: David Chen <530634352@qq.com>	2026-08-07 00:12:07 +08:00
Hongsheng Liu	62bf8d630d	docs: document distributed layerwise offload compatibility (#5839 ) Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-08-06 23:21:43 +08:00
wangyuandamy-why-3459	4b625d1175	[CI/Build] Resolve hardware-nested perf baselines per concurrency sweep (#5845 ) Signed-off-by: wangyu <410167048@qq.com> Signed-off-by: [Your Name] <your.email@example.com> Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>	2026-08-06 21:20:04 +08:00
Isotr0py lishunyang12 SYLAR David Chen	48298030d4	[Diffusion] Add Minimax-H3 modular pipeline support (#5720 ) Signed-off-by: Isotr0py <Isotr0py@outlook.com> Signed-off-by: lishunyang12 <lishunyang12@163.com> Signed-off-by: David Chen <530634352@qq.com> Co-authored-by: lishunyang12 <lishunyang12@163.com> Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com> Co-authored-by: David Chen <530634352@qq.com>	2026-08-06 11:43:23 +00:00
Bo Li lishunyang12 lishunyang12	d219d93bbb	[Attention] Refine TRTLLM attention support for MiniMax H3 (#5779 ) Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com> Signed-off-by: lishunyang12 <lishunyang03@gmail.com> Signed-off-by: lishunyang12 <lishunyang12@163.com> Co-authored-by: lishunyang12 <lishunyang03@gmail.com> Co-authored-by: lishunyang12 <lishunyang12@163.com>	2026-08-06 09:38:17 +00:00
Xunzhuo	a9ba47d8ce	[Perf] Bound memory during video frame conversion (#5732 ) Signed-off-by: Xunzhuo Liu <xunzhuo.liu@amd.com>	2026-08-06 14:41:59 +08:00
Zijun HuangandWeiQing Chen	93457f29ff	[NPU][Quantization] Add RainFusion attention and INT8 online quantization for MiniMax H3 (#5706 ) Signed-off-by: Zijun Huang <zijunhua@usc.edu> Co-authored-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com>	2026-08-06 14:15:21 +08:00
Hongsheng Liu	bc95e7415f	docs: reorganize vLLM-Omni design navigation (#5833 ) Signed-off-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-08-06 12:32:56 +08:00
ooooooye	f879ad4983	fix(minimax-h3): pass device_type to fork_rng so VAE condition encode works on NPU devices (#5837 ) Signed-off-by: brandneway <gyuan4892@gmail.com>	2026-08-06 03:56:55 +00:00
fan2956andFan	f2e3d2e25b	[Perf] Minimax-H3 support fused RMSNorm and RoPE opt (#5801 ) Signed-off-by: Fan <fan@FandeMacBook-Air.local> Co-authored-by: Fan <fan@FandeMacBook-Air.local>	2026-08-06 11:39:27 +08:00
Mu GuanLin	240e6655be	[CI][Bugfix] Fix Minimax-H3 FP8 Accuracy Test (#5829 ) Signed-off-by: mglyn <1203789601@qq.com>	2026-08-06 03:27:01 +00:00
hesiruandamy-why-3459	9235b0ae65	Optimize MiniCPM-o 4.5 Whisper chunk-attention mask construction (#5382 ) Signed-off-by: a-kaa <siruhe666@gmail.com> Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>	2026-08-06 09:48:46 +08:00
SYLARandlishunyang12	1c2a81f6d8	feat(minimax-h3): enable 2x RTX 4090/5090 with DLO (#5764 ) Signed-off-by: lishunyang12 <lishunyang03@gmail.com> Signed-off-by: lishunyang12 <lishunyang12@163.com> Co-authored-by: lishunyang12 <lishunyang03@gmail.com>	2026-08-06 09:02:29 +08:00
zhengjiaandCursor	b4581b293b	[Doc] Add Cosmos3-Nano ROCm recipe (1x MI350X) (#5634 ) Signed-off-by: zjli2013 <leezhengjiang@126.com> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-08-05 22:50:15 +08:00
bjf-frz	9f978923f2	[docs] consolidate diffusion execution modes (#5599 ) Signed-off-by: bjf-frz <frz123db@gmail.com>	2026-08-05 22:41:18 +08:00
Nick Cao	589afcef6c	Add @NickCao to CODEOWNERS (#5807 ) Signed-off-by: Nick Cao <ncao@redhat.com>	2026-08-05 21:50:48 +08:00
Zeyu Huang \| 黃澤宇	556c5f88a0	[skip ci][Doc] add comfyui hint for Minimal-H3 (#5785 ) Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>	2026-08-05 11:12:06 +00:00
R0CKSTAR	2eada79b99	[Hardware][MUSA] Make MiniMax H3 conditioned VAE RNG device-aware (#5703 ) Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>	2026-08-05 03:11:46 -07:00
wangyu	7fb62db5c2	[CI]For NPU CI, Nest perf baselines by hardware label (H100/A3) (#5402 ) Signed-off-by: wangyu <410167048@qq.com>	2026-08-05 17:35:41 +08:00
ooooooye	6510de344e	[Bugfix][Diffusion] Fix DLO AllGather size mismatch for models with h… (#5802 ) Signed-off-by: brandneway <gyuan4892@gmail.com>	2026-08-05 09:32:41 +00:00
Wallbreazzz	740cb35a59	[BugFix] Fix MOSS-TTS codec v1/v2 detection and align vendored tokenizer with upstream (#5635 ) Signed-off-by: Wallbreazzz <110282866+Wallbreazzz@users.noreply.github.com>	2026-08-05 08:24:32 +00:00
WeiQing Chen	ae7bee0c43	[Feature] Align MiniMax H3 official input matrix (#5752 ) Signed-off-by: david6666666 <530634352@qq.com>	2026-08-05 07:12:01 +00:00
GXIN	6a604be610	[Bugfix][NPU] Fallback to BSND RoPE when Qwen3-TTS short-seq BNSD li… (#5608 ) Signed-off-by: XIN GAO <1037396230@qq.com>	2026-08-05 13:02:47 +08:00
Yuekai Zhang Claude Fable 5 amy-why-3459	f201b7171f	[Perf][MiniCPM-o] TensorRT acceleration for the Code2Wav vocoder (DiT estimator + campplus) (#5638 ) Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: amy-why-3459 <wuhaiyan17@huawei.com>	2026-08-04 20:58:19 -07:00
wangyu	0a7ac4c67e	[CI/Build] Speed up omni sleep-mode entrypoint tests with shared engines (#5713 ) Signed-off-by: wangyu <410167048@qq.com>	2026-08-05 03:26:23 +00:00
Yueqian LinandSy0307	e788ef6e76	[Feature] PersonaPlex (Moshi-based full-duplex S2S): native vLLM port + duplex serving (#4771 ) Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Signed-off-by: Sy0307 <24841857+Sy0307@users.noreply.github.com> Co-authored-by: Sy0307 <24841857+Sy0307@users.noreply.github.com>	2026-08-04 20:10:33 -07:00
Shenglei FuandClaude	f24a6165ff	[Test] Prefix-cache passthrough test coverage prerequisite for W2 (#4855 ) (#5310 ) Signed-off-by: Shenglei Fu <sfu@confluent.io> Co-authored-by: Claude <noreply@anthropic.com>	2026-08-04 20:03:57 -07:00
Joshna-Medisetty	bfaaa3c2c3	[Bugfix][XPU][Tests] Add tests/e2e/accuracy/__init__.py to fix pytest… (#5780 ) Signed-off-by: Joshna Medisetty <joshna.medisetty@intel.com>	2026-08-05 07:14:22 +08:00
hsliuustc0106	69766960f2	docs: update module design navigation Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>	2026-07-16 14:09:07 +08:00
hsliuustc0106	80fd96c1da	docs: reorganize module design documents Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>	2026-07-16 12:08:45 +08:00