chore: complete v0.21.0 → v0.22.0 version bump across all files

Update remaining Dockerfiles (cuda, rocm, xpu, ci) and docs (rocm.inc.md, cuda.inc.md, quickstart.md, gpu.md) that were missed in the initial pass — pip install lines, git checkout tags, Docker image tags, wheel URLs, and prose references. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
[Feat]upgrade vllm version
2026-06-01 03:39:26 +00:00 · 2026-05-31 15:44:08 +00:00 · 2026-05-31 17:39:25 +08:00 · 2026-05-31 08:06:18 +08:00 · 2026-05-31 07:58:00 +08:00 · 2026-05-31 07:47:36 +08:00
362 changed files with 32533 additions and 5361 deletions
@@ -52,6 +52,7 @@ time timeout -k 30 30m docker run \
    pip install tblib==3.1.0
    cd /workspace/vllm-omni
    pytest -v -s -m "core_model and xpu and B60"
+    pytest -v -s tests/diffusion/quantization/test_mxfp8_config.py
    export VLLM_XPU_USE_SAMPLER_KERNEL=0    # NOTE: Remove this after vLLM v0.21.1 is merged. Fixes Qwen2-5 omni-expansion tests.
    pytest -v -s -m "advanced_model and xpu and B60"
 '
@@ -389,6 +389,31 @@ steps:
              volumes:
                - "/fsx/hf_cache:/fsx/hf_cache"

+      - label: "Omni · Higgs-Audio V2 Test"
+        timeout_in_minutes: 30
+        commands:
+          - |
+            timeout 30m bash -c "
+              export VLLM_LOGGING_LEVEL=DEBUG
+              export VLLM_WORKER_MULTIPROC_METHOD=spawn
+              export VLLM_USE_DEEP_GEMM=0
+              export VLLM_MOE_USE_DEEP_GEMM=0
+              pytest -s -v tests/e2e/online_serving/test_higgs_audio_v2.py tests/e2e/offline_inference/test_higgs_audio_v2.py -m 'advanced_model and cuda' --run-level 'advanced_model'
+            "
+        agents:
+          queue: "gpu_1_queue"
+        plugins:
+          - docker#v5.2.0:
+              image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
+              always-pull: true
+              propagate-environment: true
+              shm-size: "8gb"
+              environment:
+                - "HF_HOME=/fsx/hf_cache"
+                - "HF_TOKEN"
+              volumes:
+                - "/fsx/hf_cache:/fsx/hf_cache"
+
      - label: "Diffusion · Qwen Image Edit Test"
        timeout_in_minutes: 20
        commands:
@@ -305,6 +305,51 @@ steps:
                      path: /mnt/hf-cache
                      type: DirectoryOrCreate

+      - label: ":full_moon: Omni · Perf Test · Multi-Replica"
+        key: nightly-omni-performance-multi-replicas
+        timeout_in_minutes: 180
+        commands:
+          - export BENCHMARK_DIR=tests/dfx/perf/results
+          - |
+            set +e
+            pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json
+            EXIT=$$?
+            buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
+            exit $$EXIT
+        agents:
+          queue: "mithril-h100-pool"
+        plugins:
+          - kubernetes:
+              podSpec:
+                containers:
+                  - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
+                    resources:
+                      limits:
+                        nvidia.com/gpu: 3
+                    volumeMounts:
+                      - name: devshm
+                        mountPath: /dev/shm
+                      - name: hf-cache
+                        mountPath: /root/.cache/huggingface
+                    env:
+                      - name: HF_HOME
+                        value: /root/.cache/huggingface
+                      - name: HF_TOKEN
+                        valueFrom:
+                          secretKeyRef:
+                            name: hf-token-secret
+                            key: token
+                nodeSelector:
+                  node.kubernetes.io/instance-type: gpu-h100-sxm
+                volumes:
+                  - name: devshm
+                    emptyDir:
+                      medium: Memory
+                  - name: hf-cache
+                    hostPath:
+                      path: /mnt/hf-cache
+                      type: DirectoryOrCreate
+

      - label: ":full_moon: Omni · Multi-Replica Startup Test with 4x H100"
        timeout_in_minutes: 45
@@ -414,6 +459,52 @@ steps:
                      path: /mnt/hf-cache
                      type: DirectoryOrCreate

+      - label: ":full_moon: VoxCPM2 · Perf Test"
+        key: nightly-voxcpm2-performance
+        timeout_in_minutes: 180
+        commands:
+          - export BENCHMARK_DIR=tests/dfx/perf/results
+          - export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
+          - |
+            set +e
+            pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_voxcpm2.json
+            EXIT=$$?
+            buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
+            exit $$EXIT
+        agents:
+          queue: "mithril-h100-pool"
+        plugins:
+          - kubernetes:
+              podSpec:
+                containers:
+                  - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
+                    resources:
+                      limits:
+                        nvidia.com/gpu: 1
+                    volumeMounts:
+                      - name: devshm
+                        mountPath: /dev/shm
+                      - name: hf-cache
+                        mountPath: /root/.cache/huggingface
+                    env:
+                      - name: HF_HOME
+                        value: /root/.cache/huggingface
+                      - name: HF_TOKEN
+                        valueFrom:
+                          secretKeyRef:
+                            name: hf-token-secret
+                            key: token
+                nodeSelector:
+                  node.kubernetes.io/instance-type: gpu-h100-sxm
+                volumes:
+                  - name: devshm
+                    emptyDir:
+                      medium: Memory
+                  - name: hf-cache
+                    hostPath:
+                      path: /mnt/hf-cache
+                      type: DirectoryOrCreate
+
  # Diffusion X2I suite: x2i / x2a / x2t and related non-video paths; x2v is only in "Diffusion X2V Model Test" below.
  - group: ":card_index_dividers: Diffusion X2I(&A&T) Model Test"
    key: nightly-diffusion-x2iat-group
@@ -525,6 +616,7 @@ steps:
            tests/e2e/online_serving/test_longcat_image_expansion.py
            tests/e2e/online_serving/test_longcat_image_edit_expansion.py
            tests/e2e/online_serving/test_flux_2_dev_expansion.py
+            tests/e2e/offline_inference/test_glm_image_autoround_w4a16_expansion.py
            -m "full_model and diffusion and H100 and distributed_cuda"
            --run-level "full_model"
        agents:
@@ -578,7 +670,7 @@ steps:
                  - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
                    resources:
                      limits:
-                        nvidia.com/gpu: 3
+                        nvidia.com/gpu: 2
                    volumeMounts:
                      - name: devshm
                        mountPath: /dev/shm
@@ -797,8 +889,8 @@ steps:
                      path: /mnt/hf-cache
                      type: DirectoryOrCreate

-      - label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image-Edit-2509"
-        key: nightly-diffusion-x2iat-performance-qwen-image-edit-2509
+      - label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image-Edit-2511"
+        key: nightly-diffusion-x2iat-performance-qwen-image-edit-2511
        timeout_in_minutes: 180
        commands:
          - export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
@@ -806,7 +898,7 @@ steps:
          - export CACHE_DIT_VERSION=1.3.0
          - |
            set +e
-            pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_edit_2509_vllm_omni.json
+            pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_edit_2511_vllm_omni.json
            EXIT=$$?
            buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
            buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
@@ -893,26 +985,20 @@ steps:
                      path: /mnt/hf-cache
                      type: DirectoryOrCreate

-      - label: ":full_moon: Diffusion X2I(&A&T) · HunyuanImage3 · DiT Perf Test"
-        key: nightly-hunyuan-image3-performance
-        soft_fail: true
-        timeout_in_minutes: 120
-        if: build.env("RUN_HUNYUAN_IMAGE3_PERF") == "1"
+      - label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · BAGEL"
+        key: nightly-diffusion-x2iat-performance-bagel
+        timeout_in_minutes: 180
        commands:
          - export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
          - export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
          - export CACHE_DIT_VERSION=1.3.0
          - |
            set +e
-            pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuan_image_tp4_fp8.json
-            EXIT1=$$?
-            pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuan_image_tp2_fp8_sp2.json
-            EXIT2=$$?
-            pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuan_image_tp2_fp8_cfgp2.json
-            EXIT3=$$?
+            pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_bagel_vllm_omni.json
+            EXIT=$$?
            buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
            buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
-            exit $$((EXIT1 | EXIT2 | EXIT3))
+            exit $$EXIT
        agents:
          queue: "mithril-h100-pool"
        plugins:
@@ -922,7 +1008,7 @@ steps:
                  - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
                    resources:
                      limits:
-                        nvidia.com/gpu: 4
+                        nvidia.com/gpu: 1
                    volumeMounts:
                      - name: devshm
                        mountPath: /dev/shm
@@ -998,6 +1084,8 @@ steps:
        timeout_in_minutes: 180
        commands:
          - pytest -s -v tests/e2e/accuracy/wan22_i2v/test_wan22_i2v_video_similarity.py -m full_model --run-level full_model
+          - pytest -s -v tests/e2e/accuracy/hunyuanvideo15_t2v/test_hunyuanvideo15_t2v_video_similarity.py -m full_model --run-level full_model
+          - pytest -s -v tests/e2e/accuracy/hunyuanvideo15_i2v/test_hunyuanvideo15_i2v_video_similarity.py -m full_model --run-level full_model
          - pytest -s -v tests/e2e/accuracy/test_ltx2_3_video_similarity.py -m full_model --run-level full_model
          - pytest -s -v tests/e2e/accuracy/test_diffusers_backend_similarity.py -k '2v_matches_diffusers' -m full_model --run-level full_model
        agents:
@@ -1188,11 +1276,13 @@ steps:
      - nightly-omni-performance-no-async-chunk
      - nightly-omni-performance-async-chunk
      - nightly-omni-performance-vllm-text
+      - nightly-omni-performance-multi-replicas
      - nightly-tts-performance
      - nightly-diffusion-x2iat-performance-qwen-image
      - nightly-diffusion-x2iat-performance-qwen-image-edit
-      - nightly-diffusion-x2iat-performance-qwen-image-edit-2509
+      - nightly-diffusion-x2iat-performance-qwen-image-edit-2511
      - nightly-diffusion-x2iat-performance-qwen-image-layered
+      - nightly-diffusion-x2iat-performance-bagel
      - nightly-diffusion-x2v-performance
      - nightly-testcase-statistics
    if: build.env("NIGHTLY") == "1" && build.env("EMAIL_DISTRIBUTION") == "1"
@@ -1204,10 +1294,12 @@ steps:
      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-omni-performance-no-async-chunk
      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-omni-performance-async-chunk
      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-omni-performance-vllm-text
+      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-omni-performance-multi-replicas
      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image
      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image-edit
-      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image-edit-2509
+      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image-edit-2511
      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image-layered
+      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-bagel
      - buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2v-performance
      - buildkite-agent artifact download "tests/dfx/perf/results/*.html" . --step nightly-testcase-statistics
      - python tools/nightly/generate_nightly_perf_excel.py
@@ -21,6 +21,9 @@ Before proposing changes, ask for the optimization scene if it is not already kn
 - Current enabled strategies: USP/SP, CFG parallel, HSDP/FSDP, VAE patch parallel, torch.compile, profiler options.
 - Optimization target: latency, throughput, memory, cost, or quality-preserving speed.
 - Precision/quality tolerance: bf16/fp8/quantization/sparsity/approximate attention allowed or not.
+- Quality validation method: exact-output comparison when feasible, or metrics
+  such as SSIM, PSNR, LPIPS, cosine similarity, MAE, MSE, temporal flicker
+  checks, baseline self-run variance, and reviewable output artifacts.

 ## Workflow

@@ -79,7 +82,19 @@ Before proposing changes, ask for the optimization scene if it is not already kn
   - Record latency, stage timings, memory, output quality, and logs.
   - Report comparison tables in ms. Include at least end-to-end client time, server `inference_time`, server stage generation time if available, `vae.encode`, `diffuse`, `vae.decode`, and peak memory.

-6. **Collect diagnostic trace only after narrowing hypotheses.**
+6. **Enforce the quality and precision gate for every optimization.**
+   - Do not mark an optimization ready until quality is checked on the same
+     workload: model, prompt/media, seed, shape, steps, scheduler, dtype,
+     backend, parallelism, and output encoding.
+   - For math-preserving changes, require exact/near-exact agreement or show
+     that differences are within baseline self-run variance.
+   - For precision, quantization, approximate attention, sparse/custom kernels,
+     or backend changes, include reviewable artifacts and metrics such as SSIM,
+     PSNR, LPIPS, cosine similarity, MAE, MSE, and temporal flicker checks.
+   - If quality tolerance is not stated, default to quality-preserving behavior.
+     Failed or inconclusive quality validation blocks a ready-to-merge claim.
+
+7. **Collect diagnostic trace only after narrowing hypotheses.**
   - Use torch profiler for a small number of requests.
   - Run two separate diagnostic traces instead of mixing concerns:
     - **Operator/shape trace**: enable `torch_profiler_record_shapes=True` and keep stack collection disabled. Use this to rank CUDA kernels, NCCL collectives, attention/MLP/norm/RoPE work, and shape-specific hot operators.
@@ -91,15 +106,16 @@ Before proposing changes, ask for the optimization scene if it is not already kn
   - Diagnostic reports must be written to disk and preserve: server command, profiler config, warmup/request/polling commands, trace artifact paths, rank analyzed, analyzer output or summary, and the decision about whether additional ranks are necessary.
   - Analyze both rank-level balance and device-level free bubbles when additional ranks are opened.

-7. **Analyze host, communication, and operators.**
+8. **Analyze host, communication, and operators.**
   - Find GPU idle/free intervals and map each large gap to the enclosing CPU/Python code.
   - Separate real GPU idle from profiler overhead such as CUPTI `Command Buffer Full`.
   - Compare NCCL kernel time to user annotations; annotations can overcount nested intervals.
   - Rank operator work by total CUDA time and by repeated small-kernel launch count.

-8. **Produce an optimization plan.**
+9. **Produce an optimization plan.**
   - Classify candidates as P0/P1/P2.
   - For each candidate, state necessity, expected benefit, implementation path, validation plan, and quality risk.
+   - Include a concrete quality/precision validation gate for each candidate.
   - Do not implement high-risk operator rewrites before proving the operator is a bottleneck for the target shapes.
   - End the plan with a user-facing candidate selection table. The assistant
     should not automatically choose a risky optimization just because it is
@@ -124,7 +140,13 @@ Before proposing changes, ask for the optimization scene if it is not already kn
 - **P1:** meaningful code changes with contained risk. Examples: cross-attention KV caching, VAE gather/broadcast reduction, AdaLayerNorm/RMSNorm/RoPE fusion after trace evidence.
 - **P2:** high implementation or quality risk. Examples: FA to LA replacement, custom Triton/CUDA fused kernels, FP8/quantization, sparsity/Rainfusion-style acceleration.

-Every implemented optimization needs A/B validation and quality regression. A/B means same workload and hardware before/after. Quality regression means checking generated image/video stability, artifacts, temporal flicker, and seed behavior when precision or approximate kernels change.
+Every implemented optimization needs A/B validation and a passing
+quality/precision gate, including math-preserving changes such as padding trim,
+layout cleanup, cache reuse, or host/runtime cleanup. Use objective metrics
+when available, keep reviewable artifacts, and compare against baseline
+self-run variance for precision, quantization, approximate kernels, or backend
+changes. If quality validation fails or is inconclusive, do not present the
+optimization as ready to merge.

 ## Optimization Layers

@@ -10,6 +10,11 @@ build:
  jobs:
    post_checkout:
      - git fetch --unshallow || true
+    pre_install:
+      # Pre-install CPU torch so the subsequent `pip install .[docs]` does not
+      # pull ~2 GB of CUDA 13 libs (cuBLAS/cuDNN/NCCL/...) that exceed RTD's
+      # 1 GB memory cap and time out the mkdocs build.
+      - python -m pip install --index-url https://download.pytorch.org/whl/cpu torch

 mkdocs:
  configuration: mkdocs.yml
@@ -1,44 +1,60 @@
-# Benchmarks Overview and Architecture
+# Benchmarks

-This document explains the benchmark architecture across all benchmark assets in this repo. It describes what we measure, and where to find or plug in new scenarios. Per-task details remain in subfolder READMEs (e.g., `benchmarks/<model>/README.md`).
+This directory contains benchmark suites for evaluating different model families and infrastructure components in vLLM-Omni. Each subfolder targets a different benchmark family with its own scripts, configs, and metrics. See the per-subfolder READMEs for detailed usage.

-## Scope and goals
- Establish repeatable latency/throughput measurements for multimodal LLM pipelines.
- Provide both HF Transformers (offline) and vLLM-Omni (multi-stage/pipeline) baselines.
- Make it easy to plug in new datasets and models with minimal changes to the runner scripts.
+## Benchmark families

-## Dataset and inputs
- Default example: SeedTTS top-100 prompts (`benchmarks/build_dataset/top100.txt`) via `benchmarks/build_dataset/`.
- Extensible: drop in new prompt files or modality-aligned payloads; keep the expected format for the consuming scripts (e.g., one prompt per line).
- If you add a new dataset, document it under `benchmarks/<model>/README.md` and point scripts to your data path.
+### [TTS](tts/README.md) — Text-to-Speech

-## Directory layout
- `benchmarks/build_dataset/` — dataset prep utilities (e.g., SeedTTS top100).
- `benchmarks/<model>/vllm_omni/` — vLLM-Omni pipeline benchmarks, logs, outputs.
- `benchmarks/accuracy/` — accuracy benchmark integrations that adapt external
-  benchmark suites to vLLM-Omni serving and evaluation flows.
- Add new tasks under `benchmarks/<model>/...` with the same pattern: `transformers/`, `vllm_omni/`, task-specific README, and (optionally) dataset prep notes.
+Model-agnostic serving benchmarks for TTS models, including Qwen3-TTS and VoxCPM2.

-## Reference workflows
- **HF Transformers (offline, single process)**  
-  Script (example): `benchmarks/<model>/transformers/eval_qwen3_moe_omni_transformers.sh`  
-  Outputs: `benchmark_results/perf_stats.json`, `benchmark_results/results.json`, `benchmark_results/audio/` (if audio is produced).
+- **Layout**: `tts/bench_tts.py` (serving benchmark driver), `tts/model_configs.yaml` (model registry), `tts/plot_results.py` (result plotting)
+- **Dataset**: Seed-TTS full or text-only datasets, plus bundled smoke/design prompts under `build_dataset/`
+- **Key metrics**: TTFP (time to first audio packet), E2E latency, RTF (real-time factor), throughput (audio seconds / wall-clock second)

- **vLLM-Omni end-to-end pipeline**  
-  Script (example): `benchmarks/<model>/vllm_omni/eval_qwen3_moe_omni.sh`  
-  Outputs: `vllm_omni/logs/*.stats.jsonl` (per-stage/overall latency & TPS), `vllm_omni/logs/stage*.log`, `vllm_omni/outputs/` (text/audio artifacts).
+### [Diffusion](diffusion/README.md) — Image and Video Generation

- **Adding a new task/model**  
-  1) Create `benchmarks/<model>/transformers/` and/or `benchmarks/<model>/vllm_omni/` with scripts referencing your model and dataset.  
-  2) Add a task README describing dataset, configs, and expected outputs.  
-  3) Keep the output/log structure similar for easy comparison (perf_stats/results/audio or text outputs; stats.jsonl/logs for pipeline).
+Online-serving benchmark for diffusion image/video models, sending requests to the configured vLLM serving endpoint (`/v1/chat/completions`, `/v1/images/generations`, or `/v1/videos`, depending on backend/task).

-## Metrics to watch
- **Throughput**: `overall_tps`, `*_tps_avg` per stage.
- **Latency distribution**: look for long tails in `*.stats.jsonl`.
- **Quality/completeness**: missing outputs or errors in stage logs indicate pipeline failures or misconfigurations.
+- **Tasks**: text-to-image, text-to-video, image-to-image, image-to-video, text+image-to-image, text+image-to-video
+- **Datasets**: `vbench`, `trace`, `random`
+- **Key metrics**: request throughput, latency percentiles, optional SLO attainment

-## Troubleshooting
- Verify GPU/driver/FlashAttention2 requirements for your chosen model/config.
- Ensure network access for dataset/model downloads (Google Drive, Hugging Face, etc.).
- If outputs are missing or slow, inspect per-stage logs and `*.stats.jsonl` for errors, stragglers, or contention.
+### [GLM-Image](glm_image/README.md) — Text-to-Image and Image-to-Image
+
+Benchmarks for GLM-Image performance across HuggingFace baseline, vLLM-Omni offline inference, and vLLM-Omni online serving.
+
+- **Layout**: `glm_image/huggingface/` (HF baseline), `glm_image/vllm-omni/` (offline inference), `glm_image/benchmark_glm_image.py` (online serving)
+- **Tasks**: text-to-image and image-to-image
+- **Key metrics**: request/image throughput, latency percentiles, optional per-stage pipeline timings
+
+### [Distributed](distributed/omni_connectors/README.md) — RDMA Connector Testing
+
+RDMA environment setup and transfer tests for `MooncakeTransferEngineConnector`, including pytest-based single-node checks and manual cross-node benchmarks.
+
+- **Transfer modes**: `copy`, `zerocopy`, `gpu` (GPUDirect)
+- **Supports**: single-node pytest suites and manual multi-node/cross-node transfer testing
+
+### [Accuracy](accuracy/README.md) — Image Generation and Editing Quality
+
+Accuracy benchmarks for image generation/editing models, adapting external suites to vLLM-Omni serving and local judge-evaluation flows.
+
+- **Layout**: `accuracy/text_to_image/` (GEBench), `accuracy/image_to_image/` (GEdit-Bench)
+- **Method**: generation and judge scoring both run through local `vllm-omni serve` endpoints
+
+### Common serving metrics framework
+
+`vllm_omni/benchmarks/` extends `vllm bench serve --omni` with Omni-specific datasets, backends, and multimodal metrics. Key metrics include:
+
+- **Text output**: TTFT (time to first token), TPOT (time per output token), ITL (inter-token latency)
+- **Audio output**: TTFP (time to first audio packet), E2E latency, RTF (real-time factor)
+- **Throughput**: request throughput, output token throughput, total token throughput, audio throughput
+
+See `vllm_omni/benchmarks/serve.py` for the `vllm bench serve --omni` runner wrapper and `vllm_omni/benchmarks/metrics/` for Omni metric definitions.
+
+## Adding a new benchmark
+
+1. Create a subfolder under `benchmarks/<name>/` with scripts, configs if needed, and a `README.md`.
+2. If comparing against another runtime, use clear backend subfolders where applicable, such as `huggingface/` and `vllm-omni/`, or follow the shared TTS serving benchmark pattern in `tts/`.
+3. Add reusable dataset or prompt-building utilities to `build_dataset/` if applicable.
+4. Update this README with a link to the new benchmark family.
@@ -0,0 +1,328 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Standalone diagnostic for the diffusion attention backends added in #3079.
+
+Exercises the same synthetic attention shape through each backend and each
+torch SDPA sub-kernel, with and without an attn_mask, so we can pinpoint why
+CUDNN_ATTN underperforms on some SKUs. Two typical causes:
+
+  (a) cuDNN lacks a tuned kernel for this (SM, head_dim, seq) combination
+      and silently falls back to MATH.
+  (b) attn_mask is non-None and the cuDNN SDPA dispatch rejects it,
+      walking CUDNN -> FLASH -> MATH.
+
+Run:
+    python benchmarks/diffusion/bench_attention_backends.py --preset hv15
+    python benchmarks/diffusion/bench_attention_backends.py --preset wan22
+    python benchmarks/diffusion/bench_attention_backends.py \
+        --batch 1 --heads 24 --seq 14336 --head-dim 128
+
+Optional extras:
+    pip install --pre flash-attn-4        # FA4 is currently pre-release only
+    pip install -U flashinfer              # latest FlashInfer (0.6.9)
+
+The table at the end is the data we want on the PR — surface the row where a
+backend is >1.5x the SDPA baseline, that's the one to gate off the auto-route.
+"""
+
+from __future__ import annotations
+
+import argparse
+import time
+
+import torch
+from torch.nn.attention import SDPBackend, sdpa_kernel
+
+# Presets mirror the DiT hot-path attention call (single-stream block) at the
+# shapes we run in the PR's validation configs.
+_PRESETS = {
+    # HunyuanVideo-1.5 480p/33f: (30 * 52 * 9) latent + ~256 text tokens.
+    "hv15": {"batch": 1, "heads": 24, "seq": 14336, "head_dim": 128},
+    # Wan 2.2 480p/33f rough estimate; adjust when we measure real shapes.
+    "wan22": {"batch": 1, "heads": 40, "seq": 16384, "head_dim": 128},
+    # Smaller image-gen shape for quick smoke tests.
+    "flux": {"batch": 1, "heads": 24, "seq": 4096, "head_dim": 128},
+}
+
+_SDPA_BACKENDS = [
+    ("CUDNN_ATTENTION", [SDPBackend.CUDNN_ATTENTION]),
+    ("FLASH_ATTENTION", [SDPBackend.FLASH_ATTENTION]),
+    ("EFFICIENT_ATTENTION", [SDPBackend.EFFICIENT_ATTENTION]),
+    ("MATH", [SDPBackend.MATH]),
+    # The PR's CUDNN_ATTN impl uses this priority chain; keep it here to show
+    # what torch actually picks when multiple are allowed.
+    ("CUDNN_ATTN_CHAIN", [SDPBackend.CUDNN_ATTENTION, SDPBackend.FLASH_ATTENTION, SDPBackend.MATH]),
+]
+
+
+def _env_report() -> None:
+    if not torch.cuda.is_available():
+        raise SystemExit("CUDA not available; this diagnostic needs a GPU.")
+    cc = torch.cuda.get_device_capability()
+    print("=" * 72)
+    print(f"GPU           : {torch.cuda.get_device_name()}")
+    print(f"SM capability : sm_{cc[0]}{cc[1]}")
+    print(f"torch         : {torch.__version__}")
+    print(f"cuDNN version : {torch.backends.cudnn.version()}")
+    print("flashinfer    : ", end="")
+    try:
+        import flashinfer
+
+        print(getattr(flashinfer, "__version__", "present"))
+    except Exception as e:
+        print(f"not installed ({type(e).__name__})")
+    print("=" * 72)
+
+
+def _make_qkv(batch: int, heads: int, seq: int, head_dim: int, device: str, dtype: torch.dtype):
+    # Layout: (B, S, H, D) — what the PR's backends accept as input.
+    q = torch.randn(batch, seq, heads, head_dim, device=device, dtype=dtype)
+    k = torch.randn(batch, seq, heads, head_dim, device=device, dtype=dtype)
+    v = torch.randn(batch, seq, heads, head_dim, device=device, dtype=dtype)
+    return q, k, v
+
+
+def _make_mask(batch: int, seq: int, device: str, dtype: torch.dtype, pad_tokens: int = 128) -> torch.Tensor:
+    # Mirrors a text-encoder padding mask: last `pad_tokens` positions masked.
+    mask = torch.zeros(batch, 1, seq, seq, device=device, dtype=dtype)
+    if pad_tokens > 0:
+        mask[..., -pad_tokens:] = float("-inf")
+    return mask
+
+
+def _time_call(fn, *args, warmup: int = 3, iters: int = 10) -> tuple[float, str]:
+    """Return (median ms, error string). Skips on any exception so that a
+    backend that rejects our args (wrong dtype, missing JIT module, unsupported
+    kwarg value) doesn't abort the whole sweep."""
+    try:
+        for _ in range(warmup):
+            fn(*args)
+        torch.accelerator.synchronize()
+        times = []
+        for _ in range(iters):
+            torch.accelerator.synchronize()
+            t0 = time.perf_counter()
+            fn(*args)
+            torch.accelerator.synchronize()
+            times.append((time.perf_counter() - t0) * 1000.0)
+        times.sort()
+        return times[len(times) // 2], ""
+    except Exception as e:  # noqa: BLE001 — probe script, keep the whole table going
+        msg = str(e).split("\n", 1)[0][:60]
+        return float("nan"), f"{type(e).__name__}: {msg}" if msg else type(e).__name__
+
+
+def _run_sdpa_variants(q, k, v, attn_mask, scale: float) -> list[tuple[str, float, str]]:
+    # q/k/v come in (B, S, H, D); F.sdpa wants (B, H, S, D).
+    q_t = q.permute(0, 2, 1, 3)
+    k_t = k.permute(0, 2, 1, 3)
+    v_t = v.permute(0, 2, 1, 3)
+    rows: list[tuple[str, float, str]] = []
+    for name, backends in _SDPA_BACKENDS:
+
+        def _call(mask=attn_mask):
+            with sdpa_kernel(backends):
+                return torch.nn.functional.scaled_dot_product_attention(
+                    q_t, k_t, v_t, attn_mask=mask, dropout_p=0.0, is_causal=False, scale=scale
+                )
+
+        ms, err = _time_call(_call)
+        rows.append((name, ms, err))
+    return rows
+
+
+def _run_flashinfer(q, k, v, scale: float, backend: str | None = None, attn_mask=None) -> tuple[float, str]:
+    """Call FlashInfer's dense single-prefill.
+
+    ``backend`` hints at cutlass/fa3/trtllm if the installed version exposes
+    that kwarg. ``attn_mask`` is the same additive 4D mask we pass to SDPA —
+    we convert it to the 2D boolean ``custom_mask`` form FlashInfer accepts.
+    Per FlashInfer docs, ``custom_mask`` only applies when ``causal=False``.
+    """
+    try:
+        import inspect
+
+        from flashinfer.prefill import single_prefill_with_kv_cache
+    except Exception as e:
+        return float("nan"), f"import-{type(e).__name__}"
+
+    kwargs: dict = {"sm_scale": scale, "causal": False, "return_lse": False}
+    if backend is not None:
+        sig = inspect.signature(single_prefill_with_kv_cache)
+        if "backend" not in sig.parameters:
+            return float("nan"), "no-backend-kwarg"
+        kwargs["backend"] = backend
+
+    if attn_mask is not None:
+        # Collapse (B, 1, S, S) additive float mask to (S, S) boolean.
+        # FlashInfer expects True = keep, False = masked out.
+        mask_2d = attn_mask[0, 0]
+        kwargs["custom_mask"] = mask_2d != float("-inf")
+
+    def _call():
+        out = single_prefill_with_kv_cache(q[0], k[0], v[0], **kwargs)
+        return out.unsqueeze(0)
+
+    return _time_call(_call)
+
+
+def _run_fa4(q, k, v, scale: float) -> tuple[float, str]:
+    """Call FlashAttention-4 directly (``pip install flash-attn-4``). FA4
+    ships a Blackwell-native kernel via CuTe-DSL; on sm_120 it should beat
+    cuDNN by ~20%. API lives under ``flash_attn.cute``; the older
+    ``flash_attn.flash_attn_func`` path is FA2/FA3 only."""
+    try:
+        from flash_attn.cute import flash_attn_func
+    except Exception as e:
+        return float("nan"), f"import-{type(e).__name__}"
+
+    def _call():
+        # FA4 accepts (B, S, H, D) directly — same layout the PR's backends use.
+        return flash_attn_func(q, k, v, softmax_scale=scale, causal=False)
+
+    return _time_call(_call)
+
+
+def _run_flashinfer_cudnn_batch(q, k, v, scale: float) -> tuple[float, str]:
+    """Call FlashInfer's direct cuDNN wrapper. Bypasses PyTorch SDPA dispatch
+    (which has a few hundred ns of overhead per call) and talks to cuDNN FMHA
+    straight. Useful as a ceiling for 'pure cuDNN, no SDPA' on Blackwell."""
+    try:
+        from flashinfer.prefill import cudnn_batch_prefill_with_kv_cache
+    except Exception as e:
+        return float("nan"), f"import-{type(e).__name__}"
+
+    b, s, h, d = q.shape
+    qo_indptr = torch.tensor([0, s], dtype=torch.int32, device=q.device)
+    kv_indptr = torch.tensor([0, s], dtype=torch.int32, device=q.device)
+
+    def _call():
+        return cudnn_batch_prefill_with_kv_cache(
+            q.reshape(b * s, h, d),
+            k.reshape(b * s, h, d),
+            v.reshape(b * s, h, d),
+            qo_indptr=qo_indptr,
+            kv_indptr=kv_indptr,
+            max_qo_len=s,
+            max_kv_len=s,
+            sm_scale=scale,
+            causal=False,
+        )
+
+    return _time_call(_call)
+
+
+def _print_table(title: str, rows: list[tuple[str, float, str]], baseline_name: str | None = None) -> None:
+    print(f"\n{title}")
+    print("-" * 88)
+    header = f"{'backend':<24} {'median (ms)':>14}    {'vs baseline':>12}    status"
+    print(header)
+    print("-" * 88)
+    baseline_ms = None
+    if baseline_name is not None:
+        for name, ms, _ in rows:
+            if name == baseline_name and ms == ms:  # not NaN
+                baseline_ms = ms
+                break
+    for name, ms, err in rows:
+        ms_str = f"{ms:>14.3f}" if ms == ms else f"{'n/a':>14}"
+        if baseline_ms is not None and ms == ms:
+            ratio = baseline_ms / ms
+            ratio_str = f"{ratio:>11.2f}x"
+        else:
+            ratio_str = f"{'—':>12}"
+        status = "ok" if not err else f"FAILED ({err})"
+        print(f"{name:<24} {ms_str}    {ratio_str}    {status}")
+
+
+def _pick_winner(rows: list[tuple[str, float, str]]) -> tuple[str, float] | None:
+    """Return (backend, ms) of the fastest non-failing row, or None."""
+    ok_rows = [(n, m) for n, m, err in rows if not err and m == m]
+    if not ok_rows:
+        return None
+    return min(ok_rows, key=lambda x: x[1])
+
+
+def _bench_one_shape(shape: dict, dtype: torch.dtype, device: str) -> tuple[list, list]:
+    torch.manual_seed(0)
+    q, k, v = _make_qkv(**shape, device=device, dtype=dtype)
+    scale = 1.0 / (shape["head_dim"] ** 0.5)
+
+    rows_nomask = _run_sdpa_variants(q, k, v, attn_mask=None, scale=scale)
+    rows_nomask.append(("FLASHINFER (default)", *_run_flashinfer(q, k, v, scale)))
+    # `trtllm-gen` has no sm_120 cubins (NVIDIA/TensorRT-LLM#11799) — skipped.
+    for fi_backend in ("fa2", "fa3", "cutlass", "auto"):
+        rows_nomask.append((f"FLASHINFER ({fi_backend})", *_run_flashinfer(q, k, v, scale, backend=fi_backend)))
+    rows_nomask.append(("FLASHINFER (cudnn-batch)", *_run_flashinfer_cudnn_batch(q, k, v, scale)))
+    rows_nomask.append(("FA4 (direct)", *_run_fa4(q, k, v, scale)))
+
+    mask = _make_mask(shape["batch"], shape["seq"], device=device, dtype=dtype, pad_tokens=256)
+    rows_mask = _run_sdpa_variants(q, k, v, attn_mask=mask, scale=scale)
+    rows_mask.append(("FLASHINFER (dense)", *_run_flashinfer(q, k, v, scale, attn_mask=mask)))
+    rows_mask.append(("FA4 (direct)", float("nan"), "mask-not-supported"))
+
+    return rows_nomask, rows_mask
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--preset", choices=list(_PRESETS.keys()), default="hv15")
+    parser.add_argument("--sweep", action="store_true", help="Run all presets and print a ranking")
+    parser.add_argument("--batch", type=int, default=None)
+    parser.add_argument("--heads", type=int, default=None)
+    parser.add_argument("--seq", type=int, default=None)
+    parser.add_argument("--head-dim", type=int, default=None)
+    parser.add_argument("--dtype", choices=["bf16", "fp16"], default="bf16")
+    parser.add_argument("--device", default="cuda:0")
+    args = parser.parse_args()
+
+    dtype = torch.bfloat16 if args.dtype == "bf16" else torch.float16
+    _env_report()
+    print(f"dtype         : {args.dtype}")
+
+    presets = list(_PRESETS.keys()) if args.sweep else [args.preset]
+    summary: dict[str, tuple[str, float] | None] = {}
+
+    for preset_name in presets:
+        shape = dict(_PRESETS[preset_name])
+        if not args.sweep:
+            for k in ("batch", "heads", "seq", "head_dim"):
+                v = getattr(args, k if k != "head_dim" else "head_dim")
+                if v is not None:
+                    shape[k] = v
+
+        print("\n" + "=" * 88)
+        print(
+            f"Preset: {preset_name}  |  batch={shape['batch']} heads={shape['heads']} "
+            f"seq={shape['seq']} head_dim={shape['head_dim']}"
+        )
+        print("=" * 88)
+
+        rows_nomask, rows_mask = _bench_one_shape(shape, dtype, args.device)
+        _print_table(f"[{preset_name}] No attention mask", rows_nomask, baseline_name="CUDNN_ATTENTION")
+        _print_table(
+            f"[{preset_name}] With attention mask (pad 256 tokens)", rows_mask, baseline_name="CUDNN_ATTENTION"
+        )
+
+        summary[preset_name] = _pick_winner(rows_nomask)
+
+    if args.sweep or len(presets) > 1:
+        print("\n" + "=" * 88)
+        print("Winners per preset (no-mask path)")
+        print("=" * 88)
+        for preset_name, winner in summary.items():
+            if winner is None:
+                print(f"  {preset_name:<10} — no successful backend")
+            else:
+                name, ms = winner
+                print(f"  {preset_name:<10} {name:<24} {ms:>8.3f} ms")
+
+    print("\nNotes:")
+    print("  * Ratios are relative to CUDNN_ATTENTION. >1.0x means faster than cuDNN.")
+    print("  * Mask-path winner inherits CUDNN_ATTN's fallback in the PR's backends.")
+    print("  * `trtllm-gen` and FA4 4.0.0b10 are known-broken on sm_120 as of Apr 2026.")
+    print("  * For e2e timings, run text_to_video.py / text_to_image.py while varying DIFFUSION_ATTENTION_BACKEND.")
+
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,108 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+#
+# 2x2 timing matrix (backend x compile mode) for SAGE_ATTN_3 vs TORCH_SDPA,
+# on HunyuanVideo-1.5 480p T2V and Wan 2.2 TI2V-5B. Run from the repo root:
+#
+#     bash benchmarks/diffusion/bench_sage_attn3_matrix.sh
+#
+# Requires: sageattn3 installed (for SAGE_ATTN_3 rows) and a Blackwell GPU.
+
+set -u
+
+STEPS=${STEPS:-50}
+SEED=${SEED:-42}
+PROMPT=${PROMPT:-"A dog running across a field of golden wheat."}
+OUT_DIR=${OUT_DIR:-/tmp/bench_sage3}
+mkdir -p "$OUT_DIR"
+
+declare -A TOTAL PER_STEP TEXT_ENC TRANSFORMER VAE
+
+run() {
+    local name="$1"; shift
+    local model="$1"; shift
+    local h="$1"; shift
+    local w="$1"; shift
+    local f="$1"; shift
+    local g="$1"; shift
+    local log="$OUT_DIR/${name}.log"
+    echo "[$(date +%T)] Running $name ..."
+    env "$@" python examples/offline_inference/text_to_video/text_to_video.py \
+        --model "$model" \
+        --prompt "$PROMPT" \
+        --height "$h" --width "$w" --num-frames "$f" \
+        --num-inference-steps "$STEPS" --seed "$SEED" --guidance-scale "$g" \
+        --enable-diffusion-pipeline-profiler \
+        --output "$OUT_DIR/${name}.mp4" > "$log" 2>&1 || {
+            echo "  FAILED — see $log"
+            TOTAL[$name]="FAIL"; PER_STEP[$name]="FAIL"
+            TEXT_ENC[$name]="-"; TRANSFORMER[$name]="-"; VAE[$name]="-"
+            return
+        }
+
+    local wait_ms
+    wait_ms=$(grep "add_req_and_wait=" "$log" | tail -1 | sed -nE 's/.*add_req_and_wait=([0-9.]+) ms.*/\1/p')
+    local total_s
+    total_s=$(grep -oE "Total generation time: [0-9.]+ seconds" "$log" | tail -1 | sed -nE 's/.*: ([0-9.]+) seconds.*/\1/p')
+    local text_enc_s
+    text_enc_s=$(grep -oE "text_encoder.forward took [0-9.]+s" "$log" | awk '{sub(/s$/,"",$NF); sum+=$NF} END {printf "%.3f", sum+0}')
+    local vae_s
+    vae_s=$(grep -oE "vae.decode took [0-9.]+s" "$log" | awk '{sub(/s$/,"",$NF); sum+=$NF} END {printf "%.3f", sum+0}')
+    local pipeline_s
+    pipeline_s=$(grep -oE "Pipeline\.forward took [0-9.]+s" "$log" | tail -1 | sed -nE 's/.*took ([0-9.]+)s.*/\1/p')
+    local transformer_s
+    transformer_s=$(awk "BEGIN {printf \"%.3f\", ${pipeline_s:-0} - ${vae_s:-0} - ${text_enc_s:-0}}")
+
+    TOTAL[$name]="${total_s:-0}"
+    PER_STEP[$name]=$(awk "BEGIN {printf \"%.3f\", ${wait_ms:-0} / 1000 / $STEPS}")
+    TEXT_ENC[$name]="$text_enc_s"
+    TRANSFORMER[$name]="$transformer_s"
+    VAE[$name]="$vae_s"
+
+    echo "  total=${TOTAL[$name]}s  per-step=${PER_STEP[$name]}s/it  transformer=${TRANSFORMER[$name]}s  vae=${VAE[$name]}s"
+}
+
+# ========== HunyuanVideo-1.5 480p T2V (480x832x33, CFG=6.0) ==========
+HV_MODEL=hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v
+run hv15_sdpa_eager     "$HV_MODEL" 480 832 33 6.0 TORCH_COMPILE_DISABLE=1
+run hv15_sdpa_compiled  "$HV_MODEL" 480 832 33 6.0
+run hv15_sage3_eager    "$HV_MODEL" 480 832 33 6.0 TORCH_COMPILE_DISABLE=1 DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN_3
+run hv15_sage3_compiled "$HV_MODEL" 480 832 33 6.0                         DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN_3
+
+# ========== Wan 2.2 TI2V-5B (704x1280x33, CFG=5.0) ==========
+WAN_MODEL=Wan-AI/Wan2.2-TI2V-5B-Diffusers
+run wan22_sdpa_eager     "$WAN_MODEL" 704 1280 33 5.0 TORCH_COMPILE_DISABLE=1
+run wan22_sdpa_compiled  "$WAN_MODEL" 704 1280 33 5.0
+run wan22_sage3_eager    "$WAN_MODEL" 704 1280 33 5.0 TORCH_COMPILE_DISABLE=1 DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN_3
+run wan22_sage3_compiled "$WAN_MODEL" 704 1280 33 5.0                         DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN_3
+
+print_table() {
+    local title="$1"; shift
+    echo
+    echo "$title (${STEPS} steps, seed=${SEED})"
+    echo
+    printf "| %-23s | %-9s | %-15s | %-12s | %-15s | %-14s |\n" \
+        "Config" "Total (s)" "Per-step (s/it)" "Text Enc (s)" "Transformer (s)" "VAE Decode (s)"
+    printf "| %-23s | %-9s | %-15s | %-12s | %-15s | %-14s |\n" \
+        "-----------------------" "---------" "---------------" "------------" "---------------" "--------------"
+    for row in "$@"; do
+        case "$row" in
+            *_sdpa_eager)     label="SDPA + Eager" ;;
+            *_sdpa_compiled)  label="SDPA + Compiled" ;;
+            *_sage3_eager)    label="SAGE_ATTN_3 + Eager" ;;
+            *_sage3_compiled) label="SAGE_ATTN_3 + Compiled" ;;
+        esac
+        printf "| %-23s | %-9s | %-15s | %-12s | %-15s | %-14s |\n" \
+            "$label" "${TOTAL[$row]}" "${PER_STEP[$row]}" "${TEXT_ENC[$row]}" "${TRANSFORMER[$row]}" "${VAE[$row]}"
+    done
+}
+
+print_table "HunyuanVideo-1.5 480p T2V, 832x480x33" \
+    hv15_sdpa_eager hv15_sdpa_compiled hv15_sage3_eager hv15_sage3_compiled
+
+print_table "Wan 2.2 TI2V-5B, 1280x704x33" \
+    wan22_sdpa_eager wan22_sdpa_compiled wan22_sage3_eager wan22_sage3_compiled
+
+echo
+echo "Logs: $OUT_DIR/*.log"
@@ -47,6 +47,13 @@ Usage:
        --endpoint /v1/images/edits --dataset random --task ti2i --num-prompts 10 \
        --bot-task think

+    python benchmarks/diffusion/diffusion_benchmark_serving.py \
+      --endpoint /v1/images/edits \
+      --dataset custom \
+      --dataset-path custom_requests.jsonl \
+      --task ti2i \
+      --bot-task think
+
    i2i (chat-based models such as Qwen-Image-Edit):
    python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
        --endpoint /v1/chat/completions --dataset vbench --task i2i --num-prompts 10
@@ -566,6 +573,135 @@ class TraceDataset(BaseDataset):
        return [self[i] for i in range(len(self))]


+class CustomDataset(BaseDataset):
+    """
+    Custom dataset that loads requests from a JSONL file.
+
+    Each line in the JSONL file should be a JSON object with the following fields:
+    - prompt (required): The text prompt for the request
+    - width (optional): Image/video width
+    - height (optional): Image height
+    - num_inference_steps (optional): Number of diffusion steps
+    - seed (optional): Random seed
+    - image_paths (optional): List of input image paths for i2i/i2v/ti2i tasks
+    - image_urls (optional): List of input image URLs (alternative to image_paths)
+
+    Example JSONL for ti2i:
+    {"prompt": "Add sunset lighting", "width": 1024, "height": 1024, "image_urls": ["https://example.com/image.jpg"]}
+    """
+
+    def __init__(self, args, api_url: str, model: str):
+        super().__init__(args, api_url, model)
+        self.dataset_path = args.dataset_path
+        if not self.dataset_path:
+            raise ValueError("--dataset-path must be provided when using 'custom' dataset")
+        self.load_data()
+
+    def load_data(self) -> None:
+        """Load data from JSONL file."""
+        import pandas as pd
+
+        if not self.dataset_path.endswith(".jsonl"):
+            raise ValueError("Custom dataset must be a JSONL file")
+
+        try:
+            df = pd.read_json(path_or_buf=self.dataset_path, lines=True)
+        except Exception as e:
+            raise ValueError(f"Failed to load JSONL file {self.dataset_path}: {e}")
+
+        if "prompt" not in df.columns:
+            raise ValueError("JSONL file must contain a 'prompt' column")
+
+        self.data = df.to_dict("records")
+        print(f"Loaded {len(self.data)} requests from {self.dataset_path}")
+
+    def _resolve_image_paths(self, item: dict) -> list[str] | None:
+        # Handle local file paths
+        if "image_paths" in item and item["image_paths"]:
+            paths = item["image_paths"]
+            if isinstance(paths, str):
+                paths = [paths]
+
+            # Verify all paths exist
+            valid_paths = []
+            for path in paths:
+                if os.path.exists(path):
+                    valid_paths.append(path)
+                else:
+                    raise ValueError(f"Image file not found: {path}")
+
+            return valid_paths if valid_paths else None
+
+        # Handle URLs - download to temp files
+        if "image_urls" in item and item["image_urls"]:
+            urls = item["image_urls"]
+            if isinstance(urls, str):
+                urls = [urls]
+
+            downloaded_paths = []
+            for url in urls:
+                try:
+                    response = requests.get(url, timeout=30)
+                    response.raise_for_status()
+
+                    # Create temp file with appropriate extension
+                    suffix = os.path.splitext(url)[1] or ".png"
+                    with tempfile.NamedTemporaryFile(delete=False, suffix=suffix) as tmp:
+                        tmp.write(response.content)
+                        downloaded_paths.append(tmp.name)
+                except Exception as e:
+                    raise ValueError(f"Failed to download image from {url}: {e}")
+
+            return downloaded_paths if downloaded_paths else None
+
+        return None
+
+    def __len__(self) -> int:
+        return len(self.data)
+
+    def __getitem__(self, idx: int) -> RequestFuncInput:
+        item = self.data[idx]
+
+        # Extract fields with fallback to args defaults
+        prompt = item["prompt"]
+        width = item.get("width", self.args.width)
+        height = item.get("height", self.args.height)
+        num_inference_steps = item.get("num_inference_steps", self.args.num_inference_steps)
+        seed = item.get("seed", self.args.seed)
+        reserved_keys = {"prompt", "width", "height", "num_inference_steps", "seed", "image_paths", "image_urls"}
+        extra_body = {k: v for k, v in item.items() if k not in reserved_keys}
+
+        # Handle image paths/URLs
+        image_paths = self._resolve_image_paths(item)
+
+        return RequestFuncInput(
+            prompt=prompt,
+            api_url=self.api_url,
+            model=self.model,
+            seed=seed,
+            image_paths=image_paths,
+            extra_body=extra_body,
+            width=width,
+            height=height,
+            num_inference_steps=num_inference_steps,
+        )
+
+    def get_requests(self) -> list[RequestFuncInput]:
+        """Get all requests, cycling through data if num_prompts > dataset size."""
+        num_requests = getattr(self.args, "num_prompts", len(self.data))
+
+        if num_requests <= len(self.data):
+            return [self[i] for i in range(num_requests)]
+
+        # Cycle through the dataset to reach num_prompts
+        requests = []
+        for i in range(num_requests):
+            idx = i % len(self.data)
+            requests.append(self[idx])
+
+        return requests
+
+
 class RandomDataset(BaseDataset):
    def __init__(self, args, api_url: str, model: str, enable_negative_prompt: bool = False):
        super().__init__(args, api_url, model)
@@ -950,6 +1086,8 @@ async def benchmark(args):
        dataset = TraceDataset(args, api_url, args.model)
    elif args.dataset == "random":
        dataset = RandomDataset(args, api_url, args.model, args.enable_negative_prompt)
+    elif args.dataset == "custom":
+        dataset = CustomDataset(args, api_url, args.model)
    else:
        raise ValueError(f"Unknown dataset: {args.dataset}")

@@ -1097,7 +1235,7 @@ if __name__ == "__main__":
        "--dataset",
        type=str,
        default="vbench",
-        choices=["vbench", "trace", "random"],
+        choices=["vbench", "trace", "random", "custom"],
        help="Dataset to use.",
    )
    parser.add_argument(
@@ -130,6 +130,9 @@ def build_bench_args(
    if task_extra_body:
        cmd += ["--extra-body", json.dumps(task_extra_body, separators=(",", ":"))]

+    if model_cfg.get("trust_remote_code"):
+        cmd.append("--trust-remote-code")
+
    if concurrency is not None:
        cmd += ["--max-concurrency", str(concurrency), "--request-rate", "inf"]

@@ -32,8 +32,11 @@ models:
        task_type: Base

  openbmb/VoxCPM2:
-    supported_tasks: [voice_clone]
+    supported_tasks: [voice_clone, default_voice]
+    trust_remote_code: true
    backend: openai-audio-speech
    endpoint: /v1/audio/speech
    task_extra_body:
      voice_clone: {}
+      default_voice:
+        voice: default
@@ -1,5 +1,5 @@
 ARG VLLM_BASE_IMAGE=vllm/vllm-openai
-ARG VLLM_BASE_TAG=v0.21.0
+ARG VLLM_BASE_TAG=v0.22.0
 FROM ${VLLM_BASE_IMAGE}:${VLLM_BASE_TAG}
 ARG APP_DIR=/workspace/vllm-omni
 WORKDIR ${APP_DIR}
@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=vllm/vllm-openai:v0.20.0
+ARG BASE_IMAGE=vllm/vllm-openai:v0.22.0
 FROM ${BASE_IMAGE}

 ARG COMMON_WORKDIR=/app
@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=vllm/vllm-openai-rocm:v0.21.0
+ARG BASE_IMAGE=vllm/vllm-openai-rocm:v0.22.0
 FROM ${BASE_IMAGE} AS base

 # Declare a variable to know if we want to use the nightly build or the stable build.
@@ -88,7 +88,7 @@ ENV UV_EXTRA_INDEX_URL=${PIP_EXTRA_INDEX_URL}
 ENV UV_INDEX_STRATEGY="unsafe-best-match"
 ENV UV_LINK_MODE="copy"

-ARG VLLM_VERSION=v0.21.0
+ARG VLLM_VERSION=v0.22.0
 RUN git clone -b ${VLLM_VERSION} https://github.com/vllm-project/vllm
 WORKDIR /workspace/vllm

@@ -169,15 +169,13 @@ class DiffusionRequestStatus(enum.IntEnum):

@dataclass
 class DiffusionRequestState:
-    sched_req_id: str
+    request_id: str
    req: OmniDiffusionRequest
    status: DiffusionRequestStatus = DiffusionRequestStatus.WAITING
 ```

 **Design Features**:

- **Scheduler-owned ID**: Each `OmniDiffusionRequest` is tracked by an internal `sched_req_id`, separated from public `request_id` values.
-
 - **Explicit lifecycle**: Requests move through waiting, running, optional preemption, and terminal states.

 - **Centralized error handling**: Completion, abort, and error states are all normalized in the scheduler layer.
@@ -188,7 +186,6 @@ class DiffusionRequestState:
 class _BaseScheduler(SchedulerInterface):
    def __init__(self) -> None:
        self._request_states = {}
-        self._request_id_to_sched_req_id = {}
        self._waiting = deque()
        self._running = []
        self._finished_req_ids = set()
@@ -199,7 +196,8 @@ class _BaseScheduler(SchedulerInterface):

 - **Common state storage**: Shared request maps and waiting/running sets live in the base class.

- **Shared cleanup logic**: Request-id registration, finish handling, and state removal are centralized instead of duplicated in each policy.
+- **Shared cleanup logic**: Duplicate-request checks, finish handling, and state removal are centralized instead of
+  duplicated in each policy.

 - **Current constraint boundary**: `_BaseScheduler` derives `max_num_running_reqs` from `max_num_seqs`, but request-mode diffusion is still clamped back to `1` by the engine. The step-wise path can keep this above `1` for compatible-request batching.

@@ -219,12 +217,12 @@ class RequestScheduler(_BaseScheduler):

 - **Single-request admission**: `RequestScheduler` still admits one active request at a time because request-mode execution completes a whole request per dispatch.

- **Executor result feedback**: `update_from_output()` converts executor output into `FINISHED_COMPLETED` or `FINISHED_ERROR` and returns finished scheduler ids.
+- **Executor result feedback**: `update_from_output()` converts executor output into `FINISHED_COMPLETED` or `FINISHED_ERROR` and returns finished request ids.

 #### 2.5 Engine-Driven Execution Loop

 ```python
-sched_req_id = scheduler.add_request(request)
+request_id = scheduler.add_request(request)
 while True:
    sched_output = scheduler.schedule()
    output = executor.add_req(req)
@@ -34,7 +34,7 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr

 ### Pre-built wheels

-Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, and 0.21.0. If you need a newer unreleased revision, please [build from source](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/#build-wheel-from-source).
+Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, 0.21.0, and 0.22.0. If you need a newer unreleased revision, please [build from source](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/#build-wheel-from-source).

 === "NVIDIA CUDA"

@@ -20,7 +20,7 @@ Therefore, it is recommended to install vLLM and vLLM-Omni with a **fresh new**

 vLLM-Omni is built based on vLLM. Please install it with command below.
 ```bash
-uv pip install vllm==0.21.0 --torch-backend=auto
+uv pip install vllm==0.22.0 --torch-backend=auto
 ```

 #### Installation of vLLM-Omni
@@ -39,13 +39,13 @@ uv pip install 'vllm-omni[demo]'
 # --8<-- [start:build-wheel-from-source]

 #### Installation of vLLM
-If you do not need to modify source code of vLLM, you can directly install the stable 0.21.0 release version of the library
+If you do not need to modify source code of vLLM, you can directly install the stable 0.22.0 release version of the library

 ```bash
-uv pip install vllm==0.21.0 --torch-backend=auto
+uv pip install vllm==0.22.0 --torch-backend=auto
 ```

-The 0.21.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
+The 0.22.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.

 #### Installation of vLLM-Omni
 Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -66,12 +66,12 @@ If you want to check, modify or debug with source code of vLLM, install the libr
 ```bash
 git clone https://github.com/vllm-project/vllm.git
 cd vllm
-git checkout v0.21.0
+git checkout v0.22.0
 ```
 Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set `VLLM_PRECOMPILED_WHEEL_LOCATION` as your local absolute path of whl file.
 ```bash
-#For CUDA 13.0 (the default for v0.21.0; the wheel filename has no `+cu130` suffix)
-export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.21.0/vllm-0.21.0-cp38-abi3-manylinux_2_35_x86_64.whl
+#For CUDA 13.0 (the default for v0.22.0; the wheel filename has no `+cu130` suffix)
+export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.22.0/vllm-0.22.0-cp38-abi3-manylinux_2_35_x86_64.whl
 ```
 Install vllm with command below (If you have no existing PyTorch).
 ```bash
@@ -102,7 +102,7 @@ docker run --runtime nvidia --gpus 2 \
    --env "HF_TOKEN=$HF_TOKEN" \
    -p 8091:8091 \
    --ipc=host \
-    vllm/vllm-omni:v0.21.0 \
+    vllm/vllm-omni:v0.22.0 \
    vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
 ```

@@ -13,7 +13,7 @@ vLLM-Omni current recommends the steps in under setup through Docker Images.

 vLLM-Omni is built based on vLLM. Please install it with command below.
 ```bash
-uv pip install vllm==0.21.0+rocm722 --extra-index-url https://wheels.vllm.ai/rocm/0.21.0/rocm722
+uv pip install vllm==0.22.0+rocm722 --extra-index-url https://wheels.vllm.ai/rocm/0.22.0/rocm722
 ```

 #### Installation of vLLM-Omni
@@ -34,13 +34,13 @@ uv pip install onnxruntime-rocm
 # --8<-- [start:build-wheel-from-source]

 #### Installation of vLLM
-If you do not need to modify source code of vLLM, you can directly install the stable 0.21.0 release version of the library
+If you do not need to modify source code of vLLM, you can directly install the stable 0.22.0 release version of the library

 ```bash
-uv pip install vllm==0.21.0+rocm722 --extra-index-url https://wheels.vllm.ai/rocm/0.21.0/rocm722
+uv pip install vllm==0.22.0+rocm722 --extra-index-url https://wheels.vllm.ai/rocm/0.22.0/rocm722
 ```

-The pre-built 0.21.0 vLLM wheel targets ROCm 7.2.2. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
+The pre-built 0.22.0 vLLM wheel targets ROCm 7.2.2. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.

 #### Installation of vLLM-Omni
 Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -58,7 +58,7 @@ If you want to check, modify or debug with source code of vLLM, install the libr
 ```bash
 git clone https://github.com/vllm-project/vllm.git
 cd vllm
-git checkout v0.21.0
+git checkout v0.22.0
 python3 -m pip install -r requirements/rocm.txt
 python3 setup.py develop
 ```
@@ -130,7 +130,7 @@ docker run --rm \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  -p 8091:8091 \
-  vllm/vllm-omni-rocm:v0.21.0 \
+  vllm/vllm-omni-rocm:v0.22.0 \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
 ```

@@ -149,7 +149,7 @@ docker run --rm -it \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=$HF_TOKEN" \
  --entrypoint bash \
-  vllm/vllm-omni-rocm:v0.21.0
+  vllm/vllm-omni-rocm:v0.22.0
 ```

 # --8<-- [end:pre-built-images]
@@ -19,10 +19,10 @@ uv venv --python 3.12 --seed
 source .venv/bin/activate

 # On CUDA
-uv pip install vllm==0.21.0 --torch-backend=auto
+uv pip install vllm==0.22.0 --torch-backend=auto

 # On ROCm
-uv pip install vllm==0.21.0+rocm721 --extra-index-url https://wheels.vllm.ai/rocm/0.21.0/rocm721
+uv pip install vllm==0.22.0+rocm721 --extra-index-url https://wheels.vllm.ai/rocm/0.22.0/rocm721

 git clone https://github.com/vllm-project/vllm-omni.git
 cd vllm-omni
@@ -35,7 +35,7 @@ For additional installation methods — please see the [installation guide](inst
 !!! note
    It is important to install the same major & minor version of vLLM and vLLM Omni, otherwise things may not work as expected. If the versions are misaligned, you will see a warning when you import vLLM Omni.

-    If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.21.0` and vLLM Omni `0.21.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.
+    If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.22.0` and vLLM Omni `0.22.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.

 ## Offline Inference

@@ -16,6 +16,7 @@ ROOT_DIR_RELATIVE = "../../../../.."
 EXAMPLE_DIR = ROOT_DIR / "examples"
 EXAMPLE_DOC_DIR = ROOT_DIR / "docs/user_guide/examples"
 NAV_FILE = ROOT_DIR / "docs/.nav.yml"
+MAX_INLINE_MATERIAL_SIZE = 8 * 1024


 def fix_case(text: str) -> str:
@@ -191,18 +192,21 @@ class Example:

        return re.sub(link_pattern, replace_link, content)

+    def github_url(self, path: Path) -> str:
+        url = "https://github.com/vllm-project/vllm-omni/"
+        url += "tree/main" if path.is_dir() else "blob/main"
+        return f"{url}/{path.relative_to(ROOT_DIR).as_posix()}"
+
    def generate(self) -> str:
        content = f"# {self.title}\n\n"
-        url = "https://github.com/vllm-project/vllm-omni/"
-        url += "tree/main" if self.path.is_dir() else "blob/main"
-        content += f"Source <{url}/{self.path.relative_to(ROOT_DIR)}>.\n\n"
+        content += f"Source <{self.github_url(self.path)}>.\n\n"

        # Use long code fence to avoid issues with
        # included files containing code fences too
        code_fence = "``````"

        if self.is_code:
-            main_file_rel = self.main_file.relative_to(ROOT_DIR)
+            main_file_rel = self.main_file.relative_to(ROOT_DIR).as_posix()
            content += f'{code_fence}{self.main_file.suffix[1:]}\n--8<-- "{main_file_rel}"\n{code_fence}\n'
        else:
            with open(self.main_file, encoding="utf-8") as f:
@@ -216,10 +220,15 @@ class Example:

        content += "## Example materials\n\n"
        for file in sorted(self.other_files):
-            content += f'??? abstract "{file.relative_to(self.path)}"\n'
+            content += f'??? abstract "{file.relative_to(self.path).as_posix()}"\n'
+            if file.stat().st_size > MAX_INLINE_MATERIAL_SIZE:
+                content += (
+                    f"    Large file omitted from the rendered docs. View it on GitHub: <{self.github_url(file)}>.\n\n"
+                )
+                continue
            if file.suffix != ".md":
                content += f"    {code_fence}{file.suffix[1:]}\n"
-            content += f'    --8<-- "{file.relative_to(ROOT_DIR)}"\n'
+            content += f'    --8<-- "{file.relative_to(ROOT_DIR).as_posix()}"\n'
            if file.suffix != ".md":
                content += f"    {code_fence}\n"

@@ -291,7 +300,7 @@ def update_nav_file(examples: list[Example]):
        for example in category_examples:
            doc_path = EXAMPLE_DOC_DIR / example.category / f"{example.path.stem}.md"
            rel_path = doc_path.relative_to(ROOT_DIR / "docs")
-            category_items.append({example.title: str(rel_path)})
+            category_items.append({example.title: rel_path.as_posix()})

        if category_items:
            # Format category name (e.g., "offline_inference" -> "Offline Inference")
@@ -0,0 +1,29 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Reduce browser search-index memory for generated API reference pages."""
+
+from material.plugins.search.plugin import SearchIndex
+
+_add_entry_from_context = SearchIndex.add_entry_from_context
+
+
+def add_entry_from_context(self: SearchIndex, page) -> None:
+    search = page.meta.get("search") or {}
+    if search.get("exclude"):
+        return
+
+    if str(page.url).startswith("api/"):
+        entry = {
+            "location": page.url,
+            "title": str(page.meta.get("title", page.title)),
+            "text": "",
+        }
+        if "boost" in search:
+            entry["boost"] = search["boost"]
+        self.entries.append(entry)
+        return
+
+    _add_entry_from_context(self, page)
+
+
+SearchIndex.add_entry_from_context = add_entry_from_context
@@ -1,20 +1,44 @@
-// Enables MathJax rendering
+// Enables MathJax rendering only on pages that contain math.
 window.MathJax = {
-    tex: {
-      inlineMath: [["\\(", "\\)"]],
-      displayMath: [["\\[", "\\]"]],
-      processEscapes: true,
-      processEnvironments: true
-    },
-    options: {
-      ignoreHtmlClass: ".*|",
-      processHtmlClass: "arithmatex"
-    }
-  };
+  tex: {
+    inlineMath: [["\\(", "\\)"]],
+    displayMath: [["\\[", "\\]"]],
+    processEscapes: true,
+    processEnvironments: true,
+  },
+  options: {
+    ignoreHtmlClass: ".*|",
+    processHtmlClass: "arithmatex",
+  },
+};

-  document$.subscribe(() => {
-    MathJax.startup.output.clearCache()
-    MathJax.typesetClear()
-    MathJax.texReset()
-    MathJax.typesetPromise()
-  })
+let mathJaxLoading;
+
+function loadMathJax() {
+  if (window.MathJax.typesetPromise) {
+    return Promise.resolve();
+  }
+  if (!mathJaxLoading) {
+    mathJaxLoading = new Promise((resolve, reject) => {
+      const script = document.createElement("script");
+      script.src = "https://unpkg.com/mathjax@3.2.2/es5/tex-mml-chtml.js";
+      script.async = true;
+      script.onload = resolve;
+      script.onerror = reject;
+      document.head.appendChild(script);
+    });
+  }
+  return mathJaxLoading;
+}
+
+document$.subscribe(() => {
+  if (!document.querySelector(".arithmatex")) {
+    return;
+  }
+  loadMathJax().then(() => {
+    MathJax.startup.output.clearCache();
+    MathJax.typesetClear();
+    MathJax.texReset();
+    MathJax.typesetPromise();
+  });
+});
@@ -0,0 +1,2 @@
+// Opt out of bfcache so large generated API pages are released after navigation.
+window.addEventListener("unload", () => {});
@@ -1,21 +1,42 @@
-// Initialize Mermaid for diagram rendering
-mermaid.initialize({
-  startOnLoad: false,
-  theme: 'default',
-  securityLevel: 'loose',
-  flowchart: {
-    useMaxWidth: true,
-    htmlLabels: true
-  }
-});
+// Load Mermaid only on pages that contain diagrams.
+let mermaidLoading;

-// Render Mermaid diagrams when page content is ready
-document$.subscribe(() => {
-  const mermaidElements = document.querySelectorAll('.mermaid');
-  if (mermaidElements.length > 0) {
-    mermaid.run({
-      querySelector: '.mermaid',
-      nodes: mermaidElements
+function loadMermaid() {
+  if (window.mermaid) {
+    return Promise.resolve();
+  }
+  if (!mermaidLoading) {
+    mermaidLoading = new Promise((resolve, reject) => {
+      const script = document.createElement("script");
+      script.src = "https://unpkg.com/mermaid@10/dist/mermaid.min.js";
+      script.async = true;
+      script.onload = () => {
+        mermaid.initialize({
+          startOnLoad: false,
+          theme: "default",
+          securityLevel: "loose",
+          flowchart: {
+            useMaxWidth: true,
+            htmlLabels: true,
+          },
+        });
+        resolve();
+      };
+      script.onerror = reject;
+      document.head.appendChild(script);
    });
  }
+  return mermaidLoading;
+}
+
+document$.subscribe(() => {
+  if (!document.querySelector(".mermaid")) {
+    return;
+  }
+  loadMermaid().then(() => {
+    mermaid.run({
+      querySelector: ".mermaid",
+      nodes: document.querySelectorAll(".mermaid"),
+    });
+  });
 });
@@ -19,6 +19,7 @@ th {
 |--------------|--------|-------------------|------------|---------|-----|-----------|
 | `Qwen3OmniMoeForConditionalGeneration` | Qwen3-Omni | `Qwen/Qwen3-Omni-30B-A3B-Instruct` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `Qwen2_5OmniForConditionalGeneration` | Qwen2.5-Omni | `Qwen/Qwen2.5-Omni-7B`, `Qwen/Qwen2.5-Omni-3B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
+| `MingFlashOmniForConditionalGeneration` + `MingImagePipeline` | Ming-flash-omni-2.0 (omni-speech + imagegen<sup>1</sup>) | `Jonathan1909/Ming-flash-omni-2.0` | ✅︎ |   |   |   |
 | `BagelForConditionalGeneration` | BAGEL (DiT-only) | `ByteDance-Seed/BAGEL-7B-MoT` | ✅︎ | ✅︎ | | ✅︎ |
 | `InternVLAA1Pipeline` | InternVLA-A1 | `InternRobotics/InternVLA-A1-3B` | ✅︎ | ✅︎ | | |
 | `HunyuanImage3ForCausalMM` | HunyuanImage3.0 (DiT-only) | `tencent/HunyuanImage-3.0`, `tencent/HunyuanImage-3.0-Instruct` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
@@ -46,7 +47,7 @@ th {
 | `LongcatImagePipeline` | LongCat-Image | `meituan-longcat/LongCat-Image` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `LongCatImageEditPipeline` | LongCat-Image-Edit | `meituan-longcat/LongCat-Image-Edit` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ |
-| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | |
+| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ |
 | `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | |
 | `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `FluxKontextPipeline` | FLUX.1-Kontext-dev | `black-forest-labs/FLUX.1-Kontext-dev` | ✅︎ | ✅︎ | | |
@@ -13,7 +13,7 @@ For example...
 ```bash
 # Qwen-Image
 vllm serve Qwen/Qwen-Image-Edit-2511 --omni --port 8000
-
+```

 ### Generate Images

@@ -53,7 +53,7 @@ result = client.images.edit(
    output_format='jpeg',
    # url格式
    extra_body={
-        "url": [input_image_url1,input_image_url],
+        "url": [input_image_url,input_image_url],
        "num_inference_steps": 50,
        "guidance_scale": 1,
        "seed": 777,
@@ -92,7 +92,8 @@ Content-Type: multipart/form-data
 | `user` | string | null | User identifier for tracking |
 | `output_format` | string | "png" | The format in which the generated images are returned. Must be one of "png", "jpg", "jpeg", "webp". |
 | `output_compression` | integer | 100 | The compression level (0-100%) for the generated images. |
-| `background` | string or null | "auto" | Allows to set transparency for the background of the generated image(s).
+| `background` | string or null | "auto" | Allows to set transparency for the background of the generated image(s). |
+| `stream` | boolean | false | Return Server-Sent Events instead of a single JSON response. Currently supported only by multi-stage image edit pipelines that produce AR text before the final image, such as HunyuanImage3 IT2I AR+DiT. |

 #### vllm-omni Extension Parameters

@@ -109,6 +110,8 @@ Content-Type: multipart/form-data

 ### Response Format

+When `stream=false` or omitted, the endpoint returns the standard image edit response:
+
 ```json
 {
  "created": 1701234567,
@@ -120,10 +123,56 @@ Content-Type: multipart/form-data
    }
  ],
  "output_format": null,
-  "size": null,
+  "size": null
 }
 ```

+### Streaming Response Format
+
+Set `stream=true` when you want to receive the HunyuanImage3 IT2I AR
+recaption text before the final edited image is ready. Streaming is only
+available for multi-stage image edit pipelines where stage 0 produces AR text
+and a later diffusion stage produces the image. Single-stage diffusion image
+edit pipelines reject `stream=true` with HTTP 400.
+
+The response uses Server-Sent Events with `Content-Type: text/event-stream`.
+Each event is sent as one `data:` line. The stream order is:
+
+1. One or more AR text delta chunks.
+2. One final image chunk.
+3. `data: [DONE]`.
+
+AR text chunks expose only the generated text delta and the AR completion
+`index`. They do not expose token ids, ratio tokens, KV cache data, or internal
+prompt token ids.
+
+Example AR delta event:
+
+```text
+data: {"object":"image.edit.chunk","type":"ar_delta","delta":"A close-up product photo...","index":0,"created":1701234567,"model":"tencent/HunyuanImage-3.0-Instruct"}
+```
+
+Example final image event:
+
+```text
+data: {"object":"image.edit.chunk","type":"image","data":[{"b64_json":"<base64-encoded PNG>","url":null,"revised_prompt":null}],"output_format":"png","size":"1024x1024","created":1701234567,"model":"tencent/HunyuanImage-3.0-Instruct"}
+```
+
+Terminal event:
+
+```text
+data: [DONE]
+```
+
+If the engine fails after the SSE response has started, the stream emits an
+error chunk followed by `[DONE]`:
+
+```text
+data: {"object":"error","created":1701234567,"model":"tencent/HunyuanImage-3.0-Instruct","error":{"message":"<error message>","type":"server_error","code":500}}
+
+data: [DONE]
+```
+
 ## Examples

 ### Multiple Images input
@@ -134,12 +183,28 @@ curl -s -D >(grep -i x-request-id >&2) \
  -X POST "http://localhost:8000/v1/images/edits" \
  -F "model=xxx" \
  -F "image=@xx.png" \
-  -F "image=@xx.png"
+  -F "image=@xx.png" \
  -F "prompt='this bear is wearing sportwear. holding a basketball, and bending one leg.'" \
  -F "size=1024x1024" \
  -F "output_format=png"
 ```

+### Streaming HunyuanImage3 IT2I AR Text
+
+```bash
+curl -N -X POST "http://localhost:8000/v1/images/edits" \
+  -F "model=tencent/HunyuanImage-3.0-Instruct" \
+  -F "image=@./input.png" \
+  -F "prompt=Turn the product into a clean studio advertisement." \
+  -F "size=1024x1024" \
+  -F "output_format=png" \
+  -F "stream=true"
+```
+
+Use the `ar_delta` events for progressive display of the AR-generated
+recaption. Decode the `b64_json` field from the final `image` event to get the
+edited image.
+

 ## Parameter Handling

@@ -1,28 +1,26 @@
 # Diffusion Attention Backends

-This document describes the diffusion attention backends available in vLLM-Omni, how to select them globally and per-role, and how to use SageAttention.
+This document describes the diffusion attention backends available in vLLM-Omni, how to select them globally and per-role, the per-platform defaults, and how to use SageAttention.

 ## Overview

 Diffusion attention backend selection is resolved in `vllm_omni.diffusion.attention.selector`. It looks up the backend from a structured `AttentionConfig` carried on `OmniDiffusionConfig` and falls back to the platform default when nothing is configured.

-This backend is used by diffusion attention layers such as the DiT attention in video and image generation models.
+This backend is used by diffusion attention layers such as the DiT attention in video and image generation models. It does **not** affect autoregressive (LLM) attention paths — those go through vLLM's own attention backend selector.

-On CUDA, the practical choices today are:
-
- `FLASH_ATTN`: FlashAttention backend. This is the default on supported CUDA systems when FlashAttention is installed.
- `TORCH_SDPA`: PyTorch `scaled_dot_product_attention`.
- `SAGE_ATTN`: SageAttention backend, if `sageattention` is installed.
-
-If no attention backend is configured, vLLM-Omni asks the current platform to choose the default. On CUDA, that normally means `FLASH_ATTN` when available, otherwise `TORCH_SDPA`.
+The full set of backends and their platform defaults is in the **Backend Options** and **Platform Defaults** sections below. If no attention backend is configured, vLLM-Omni asks the current platform to choose the default.

 ## Backend Options

 | Value | Notes |
 |---|---|
-| `FLASH_ATTN` | Default on CUDA when FlashAttention is available. Good default for most diffusion workloads. |
-| `TORCH_SDPA` | Most conservative fallback. Useful for debugging or compatibility. |
-| `SAGE_ATTN` | Requires `sageattention`. Can improve performance on some workloads, but output quality must be validated model-by-model. |
+| `FLASH_ATTN` | Wraps FlashAttention 2. Default on Hopper / Ada / Ampere when `flash-attn` is installed. |
+| `CUDNN_ATTN` | Pins `sdpa_kernel([CUDNN_ATTENTION])`. Default on Blackwell (sm_10x / sm_12x) with cuDNN ≥ 9.5. Wins on mask-heavy DiTs (HunyuanVideo-1.5: 2× e2e vs SDPA). |
+| `FLASHINFER_ATTN` | Calls FlashInfer's dense `single_prefill_with_kv_cache` directly with `custom_mask` for non-causal masked attention. Used as Blackwell fallback when cuDNN is unavailable. Requires `flashinfer`. |
+| `TORCH_SDPA` | PyTorch `scaled_dot_product_attention` with the default backend dispatcher. Most conservative; always available. |
+| `SAGE_ATTN` | SageAttention 2.2 — INT8-quantized attention with FP16 accumulation. Lossy but typically visually indistinguishable on diffusion outputs. Requires `sageattention`. |
+| `SAGE_ATTN_3` | Requires `sageattn3` from `SageAttention/sageattention3_blackwell`. CUDA only, intended for Blackwell GPUs, with GQA/MQA requests falling back to PyTorch SDPA. |
+

 ## Configuration

@@ -95,6 +93,80 @@ config = OmniDiffusionConfig(

 A plain dict is also accepted and normalized to `AttentionConfig`.

+## Platform Defaults
+
+### Blackwell (sm_100 / sm_103 / sm_120 / sm_121)
+
+Auto-route preference, in order:
+
+1. `CUDNN_ATTN` — when cuDNN ≥ 9.5 is available (ships in PyTorch 2.5+ wheels)
+2. `FLASHINFER_ATTN` — when `flashinfer` is installed but cuDNN < 9.5
+3. `FLASH_ATTN` — when `flash-attn` is installed with the Blackwell CUTE kernel
+4. `TORCH_SDPA` — last resort
+
+The startup log line `Defaulting to diffusion attention backend CUDNN_ATTN (Blackwell sm_120, cuDNN 91002)` confirms the route.
+
+**Why CUDNN_ATTN by default**: on mask-heavy diffusion models (HunyuanVideo-1.5, Qwen-Image), cuDNN's pinned FMHA kernel sidesteps a PyTorch SDPA dispatch quirk where the unpinned dispatcher picks `EFFICIENT_ATTENTION` (~25 ms) for masked calls instead of cuDNN (~11 ms). The pin gives 2× e2e on HV-1.5 with no regression on lighter models.
+
+### Hopper (sm_90) / Ada (sm_89) / Ampere (sm_80–sm_86)
+
+Auto-route preference:
+
+1. `FLASH_ATTN` — when `flash-attn` is installed
+2. `TORCH_SDPA` — fallback
+
+`CUDNN_ATTN` and `FLASHINFER_ATTN` are still selectable via env var on these GPUs but are not in the auto-route — FlashAttention 2 is the well-tuned path on pre-Blackwell hardware.
+
+## End-to-End Benchmark (BF16, sm_120 RTX Pro 6000 Blackwell)
+
+Same prompt and seed across runs. `Total generation time` from `text_to_video.py` / `text_to_image.py`.
+
+| Model | Shape | TORCH_SDPA | CUDNN_ATTN | FLASHINFER_ATTN |
+|---|---|---|---|---|
+| HunyuanVideo-1.5 (T2V) | 480p / 33f / 50 steps | 147.05 s | **73.02 s** | 127.84 s |
+| Wan 2.2 14B (T2V) | 480p / 33f / 40 steps | 117.75 s | 117.17 s | **115.07 s** |
+| Qwen-Image (T2I) | 1024² / 50 steps | 17.41 s | **15.14 s** | 16.02 s |
+| FLUX.2-dev (T2I) | 1024² / 50 steps, TP=2 | 53.62 s | **53.30 s** | 54.94 s |
+
+Pattern: mask-heavy DiTs (HV-1.5, Qwen-Image) favor `CUDNN_ATTN`; lighter-mask DiTs and TP-saturated configs (Wan 2.2, FLUX.2 TP=2) tie within noise.
+
+## Known Limitations
+
+### LTX-2.0: `CUDNN_ATTN` crashes under torch.compile
+
+LTX-2's audio attention has a symbolic head_dim under torch.compile tracing. cuDNN's SDPA backend selector rejects symbolic dims and Dynamo aborts compilation. Tracked in [#3121](https://github.com/vllm-project/vllm-omni/issues/3121).
+
+**Workaround**: explicitly select `FLASHINFER_ATTN` or `TORCH_SDPA` for LTX-2.0:
+
+```bash
+DIFFUSION_ATTENTION_BACKEND=FLASHINFER_ATTN python examples/offline_inference/text_to_video/text_to_video.py \
+    --model Lightricks/LTX-2 ...
+```
+
+### FA4 not yet integrated
+
+FlashAttention-4 (released March 2026) targets Blackwell natively and reportedly beats cuDNN by ~20% on B200. As of this writing the `flash-attn-4 4.0.0b10` wheel crashes with `AttributeError: 'NoneType' object has no attribute '_trait'` during JIT on sm_120. Not yet wired into vLLM-Omni; revisit when stable lands.
+
+## Choosing a Backend Manually
+
+### When to override the default
+
+- **Quality validation**: compare a new backend against `TORCH_SDPA` as the reference, since SDPA's default dispatcher is the most extensively tested.
+- **Lossy speedup hunting**: try `SAGE_ATTN` (INT8 quantized) on diffusion outputs — typically indistinguishable visually but always validate.
+- **Workaround for known issues**: see Known Limitations above.
+
+### Verifying which backend is in use
+
+The startup log prints one of:
+
+```
+Using diffusion attention backend 'CUDNN_ATTN'           # explicit override
+Defaulting to diffusion attention backend CUDNN_ATTN ... # auto-route
+Defaulting to diffusion attention backend SDPA           # nothing else available
+```
+
+If you don't see one of these, the model didn't reach diffusion stage init — check earlier logs for failures.
+
 ## SageAttention Installation

 vLLM-Omni expects SageAttention to be installed into the same Python environment as vLLM-Omni.
@@ -115,11 +187,56 @@ Quick check:
 python -c "import sageattention; print(sageattention.__file__)"
 ```

-## Usage
+## SageAttention3 Installation

-### Enable SageAttention
+vLLM-Omni expects SageAttention3 to be installed into the same Python environment as vLLM-Omni.

-Example: HunyuanVideo-1.5 text-to-video
+Build from source:
+
+```bash
+git clone https://github.com/thu-ml/SageAttention.git
+cd SageAttention/sageattention3_blackwell
+python setup.py install
+```
+
+Quick check:
+
+```bash
+python -c "import sageattn3; print(sageattn3.__file__)"
+```
+
+Notes:
+
+- `SAGE_ATTN_3` is only selected on CUDA when `sageattn3` is importable and the GPU is Blackwell-class.
+- SageAttention3's Blackwell kernel assumes `Hq == Hkv`. In vLLM-Omni, GQA/MQA diffusion requests fall back to PyTorch SDPA for correctness.
+
+## Usage Examples
+
+### Default (auto-route)
+
+```bash
+python examples/offline_inference/text_to_video/text_to_video.py \
+    --model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v \
+    --prompt "A dog running across a field of golden wheat." \
+    --height 480 --width 832 --num-frames 33 \
+    --num-inference-steps 50 --seed 42 --guidance-scale 6.0 \
+    --output hv15.mp4
+```
+
+On Blackwell this picks `CUDNN_ATTN` automatically. Check the log for the `Defaulting to ...` line.
+
+### Explicit backend selection
+
+```bash
+DIFFUSION_ATTENTION_BACKEND=FLASHINFER_ATTN python examples/offline_inference/text_to_video/text_to_video.py \
+    --model Lightricks/LTX-2 \
+    --prompt "A dog running across a field of golden wheat." \
+    --height 480 --width 832 --num-frames 33 \
+    --num-inference-steps 40 --seed 42 --guidance-scale 4.0 \
+    --output ltx2.mp4
+```
+
+### SageAttention (lossy)

 ```bash
 DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN python examples/offline_inference/text_to_video/text_to_video.py \
@@ -128,7 +245,7 @@ DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN python examples/offline_inference/text_to_
    --height 480 --width 832 --num-frames 33 \
    --num-inference-steps 30 --seed 42 --guidance-scale 6.0 \
    --tensor-parallel-size 2 \
-    --output ../tmp/hv15_modelopt_sage.mp4
+    --output hv15_sage.mp4
 ```

 Example: Wan2.2 TI2V 5B
@@ -143,6 +260,20 @@ DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN python examples/offline_inference/text_to_
    --output outputs/wan22_sage.mp4
 ```

+### Enable SageAttention3
+
+Example:
+
+```bash
+DIFFUSION_ATTENTION_BACKEND=SAGE_ATTN_3 python examples/offline_inference/text_to_video/text_to_video.py \
+    --model hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v \
+    --prompt "A dog running across a field of golden wheat." \
+    --height 480 --width 832 --num-frames 33 \
+    --num-inference-steps 30 --seed 42 --guidance-scale 6.0 \
+    --tensor-parallel-size 2 \
+    --output outputs/hv15_sage3.mp4
+```
+
 ### Mixed backends across roles

 Use `FLASH_ATTN` for self-attention and `TORCH_SDPA` for cross-attention:
@@ -173,13 +304,13 @@ python examples/offline_inference/text_to_video/text_to_video.py \

 ## Validation Guidance

-Do not assume that a faster attention backend is numerically interchangeable with `FLASH_ATTN`.
+Don't assume a faster attention backend is numerically interchangeable with `TORCH_SDPA`.

 Always compare:

 - End-to-end runtime
- DiT / diffusion stage runtime
- Output quality against a known-good baseline
+- Diffusion-stage runtime (`add_req_and_wait` line in DiffusionEngine.step breakdown)
+- Output quality against a known-good baseline (CLIP similarity, frame-level diff, or visual review)

 At minimum, keep the same:

@@ -187,6 +318,19 @@ At minimum, keep the same:
 - prompt
 - seed
 - resolution
- frame count
- inference steps
- parallel config
+- frame count / step count
+- parallel config (TP / CFG-parallel / Ulysses degrees)
+
+## Reproducing the Benchmark Table
+
+The end-to-end numbers above were collected by running `text_to_video.py` /
+`text_to_image.py` with the same prompt and seed while varying
+`DIFFUSION_ATTENTION_BACKEND`. For a quick kernel-level comparison of the
+backends without loading a model:
+
+```bash
+python benchmarks/diffusion/bench_attention_backends.py --preset hv15
+```
+
+It runs all three BF16 backends on representative DiT attention shapes and
+prints a ranking table at the end.
@@ -113,7 +113,7 @@ The following tables show which models support each feature:
 | **FLUX.1-schnell**       |     ❌     |     ✅      |           ❌           |       ✅        |         ✅         |          ❌          |   ✅    |             ❌             |          ❌           |       ✅        |        ❌         |
 | **FLUX.2-klein**         |     ✅     |     ✅      |           ✅           |       ✅        |         ✅         |          ❌          |   ✅    |             ❌             |          ❌           |       ✅        |        ❌         |
 | **FLUX.1-Kontext-dev**   |     ❌     |     ❌      |           ❌           |       ❌        |         ✅         |          ❌          |   ✅    |             ❌             |          ❌           |       ❌        |        ❌         |
-| **FLUX.2-dev**           |     ✅     |     ✅      |           ❌           |       ✅        |         ✅         |          ❌          |   ✅    |             ❌             |          ❌           |       ❌        |        ❌         |
+| **FLUX.2-dev**           |     ✅     |     ✅      |           ✅           |       ✅        |         ✅         |          ❌          |   ✅    |             ❌             |          ❌           |       ❌        |        ❌         |
 | **GLM-Image**            |     ❌     |     ❌      |           ❌           |       ✅        |         ✅         |          ❌          |   ✅    |             ❌             |          ❌           |       ❌        |        ❌         |
 | **Hidream-I1-Full**        |     ❌     |     ❌      |           ❌           |       ❌        |         ✅         |          ❌          |   ❌    |             ❌             |          ❌           |       ❌        |        ❌         |
 | **HunyuanImage3**        |     ❌     |     ✅      |           ❌           |       ❌        |         ✅         |          ❌          |   ❌    |             ❌             |          ❌           |       ✅        |        ❌         |
@@ -129,7 +129,7 @@ The following tables show which models support each feature:
 | **Qwen-Image-Edit**      |     ✅     |     ✅      |           ✅           |       ✅        |         ✅         |          ❌          |   ✅    |             ✅             |      ✅ (decode)      |       ❌        |        ❌         |
 | **Qwen-Image-Edit-2509** |     ✅     |     ✅      |           ✅           |       ✅        |         ✅         |          ❌          |   ✅    |        ✅ (decode)         |          ✅           |       ❌        |        ❌         |
 | **Qwen-Image-Layered**   |     ✅     |     ✅      |           ✅           |       ✅        |         ✅         |          ❌          |   ✅    |             ✅             |      ✅ (decode)      |       ❌        |        ❌         |
-| **SenseNova-U1**         |     ❌     |     ❌      |           ❌           |       ❌        |         ✅         |          ❌          |   ❌    |             ✅             |          ❌           |       ❌        |        ❌         |
+| **SenseNova-U1**         |     ❌     |     ✅      |           ❌           |       ❌        |         ✅         |          ❌          |   ❌    |             ✅             |          ❌           |       ❌        |        ❌         |
 | **Stable-Diffusion3.5**  |     ❌     |     ✅      |           ❌           |       ✅        |         ✅         |          ❌          |   ❌    |             ✅             |      ✅ (decode)      |       ❌        |        ❌         |
 | **Z-Image**              |     ✅     |     ✅      |           ✅           |       ❓        |   ✅ (TP=2 only)   |          ❌          |   ✅    |             ❌             |      ✅ (decode)      |       ✅        |        ❌         |
 | **ERNIE-Image**          |     ❌     |     ✅      |           ✅           |       ❓        |         ✅         |          ❌          |   ✅    |             ✅             |          ❌           |       ❌        |        ❌         |
@@ -145,9 +145,9 @@ The following tables show which models support each feature:
 | **Wan2.2**                   |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ✅         |   ✅    |             ✅             |  ✅ (encode/decode)   |       ❌        |        ❌         |
 | **Wan2.1-VACE**              |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ❌         |   ✅    |             ✅             |      ✅ (decode)      |       ❌        |        ❌         |
 | **LTX-2**                    |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ❌         |   ✅    |             ✅             |          ❌           |       ❌        |        ❌         |
-| **LTX-2.3**                  |     ❌     |     ✅      |           ✅           |       ❌        |         ✅         |         ❌         |   ❌    |             ❌             |          ❌           |       ❌        |        ❌         |
+| **LTX-2.3**                  |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ❌         |   ❌    |             ❌             |          ❌           |       ❌        |        ❌         |
 | **Helios**                   |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ❌         |   ✅    |             ✅             |          ❌           |       ❌        |        ❌         |
-| **HunyuanVideo-1.5 T2V I2V** |     ❌     |     ✅      |           ❌           |       ✅        |         ✅         |         ❌         |   ✅    |             ✅             |      ✅ (decode)      |       ✅        |        ❌         |
+| **HunyuanVideo-1.5 T2V I2V** |     ❌     |     ✅      |           ✅           |       ✅        |         ✅         |         ❌         |   ✅    |             ✅             |  ✅ (encode/decode)   |       ✅        |        ❌         |
 | **DreamID-Omni**             |     ❌     |     ❌      |           ❌           |       ✅        |         ❌         |         ❌         |   ✅    |             ✅             |          ❌           |       ❌        |        ❌         |

 **Frame Interpolation Support**
@@ -23,6 +23,7 @@ list of supported architectures across all modalities, see
 | OmniVoice | `k2-fsa/OmniVoice` | 2 (gen + dec) | ✓ | — | voice design (`--instruct`), language (`--lang`) | 24 kHz |
 | Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | 2 (talker + code2wav) | ✓ (Base) | ✓ | 3 task variants (`--query-type`) | 24 kHz |
 | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | varies | ✓ | ✓ | voice presets (`--voice`) | 24 kHz |
+| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` (+ `k2-fsa/OmniVoice/audio_tokenizer/` codec) | 2 (talker + code2wav, DualFFN) | ✓ (`--ref-audio` + `--ref-text`) | — | — | 24 kHz |

 ## Common Quick Start

@@ -333,6 +334,46 @@ Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-T
 - `--concurrency M` requires `--streaming` and must evenly divide `--num-prompts`.
 - Run `--help` for the full argument surface.

+---
+
+## higgs-audio v2
+
+2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) over the shared-memory connector. Stage 1 builds on the HiggsAudio decoder kernel at `vllm_omni/model_executor/models/higgs_audio_v2/higgs_audio_decoder.py`.
+
+### Prerequisites
+
+Voice clone uses HF's `HiggsAudioV2TokenizerModel`, instantiated from `k2-fsa/OmniVoice/audio_tokenizer/` — the boson-ai standalone tokenizer Hub repo's `model.safetensors` is actually the 3B talker LM, so we point HF at k2's repackaged codec weights instead. Only the `audio_tokenizer/` subdir (~806 MB) is downloaded.
+
+```bash
+pip install -U "transformers>=5.3.0"
+```
+
+### Quick start (plain TTS)
+
+```bash
+python examples/offline_inference/text_to_speech/higgs_audio_v2/end2end.py \
+    --texts "Hello world." "The quick brown fox jumps over the lazy dog." \
+    --output-dir results/wavs
+```
+
+### Voice cloning
+
+Pass both `--ref-audio` and `--ref-text` together:
+
+```bash
+python examples/offline_inference/text_to_speech/higgs_audio_v2/end2end.py \
+    --texts "Hello, this is a cloned voice." \
+    --ref-audio /path/to/reference.wav \
+    --ref-text  "Exact transcript spoken in reference.wav." \
+    --output-dir results/wavs
+```
+
+### Notes
+
+- Deploy config auto-loads from `vllm_omni/deploy/higgs_audio_v2.yaml`.
+- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades cloned-voice quality.
+- Out of scope (rejected with 4xx by the request validator): multi-speaker `[SPEAKERn]` dialogue, `profile:` text-only speaker descriptions, the `ref_audio_in_system_message` system-block variant, chunked long-form generation, and per-request `voice` / `instructions` / `task_type` / `language` / `speed != 1.0` / `x_vector_only_mode` / `speaker_embedding`.
+
 ## Example materials

 ??? abstract "cosyvoice3/end2end.py"
@@ -343,6 +384,9 @@ Available voice presets are listed on the HF model card (`mistralai/Voxtral-4B-T
    ``````py
    --8<-- "examples/offline_inference/text_to_speech/fish_speech/end2end.py"
    ``````
+??? abstract "higgs_audio_v2/end2end.py"
+    ``````py
+    --8<-- "examples/offline_inference/text_to_speech/higgs_audio_v2/end2end.py"
 ??? abstract "glm_tts/end2end.py"
    ``````py
    --8<-- "examples/offline_inference/text_to_speech/glm_tts/end2end.py"
@@ -19,6 +19,7 @@ For the full list of supported architectures across all modalities, see
 |---|---|---|---|---|---|
 | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | — |
 | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
+| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✓ (`ref_audio`+`ref_text`) | ✓ (codec_streaming) | — | — |
 | GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
 | OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
 | Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) |
@@ -193,6 +194,45 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091  # if server

 ---

+## higgs-audio v2
+
+2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.
+
+### Prerequisites
+
+Voice clone uses HF's `HiggsAudioV2TokenizerModel` loaded from `k2-fsa/OmniVoice/audio_tokenizer/` (~806 MB subdir; the boson-ai standalone tokenizer Hub repo's `model.safetensors` is the 3B talker LM, not the codec):
+
+```bash
+pip install -U "transformers>=5.3.0"
+```
+
+### Launch
+```bash
+GPUS=6,7 PORT=8094 bash examples/online_serving/text_to_speech/higgs_audio_v2/run_server.sh
+```
+Deploy config auto-loads from `vllm_omni/deploy/higgs_audio_v2.yaml`.
+
+### Sending requests
+```bash
+# Plain TTS
+python higgs_audio_v2/batch_speech_client.py \
+    --base-url http://localhost:8094 \
+    --output-dir /tmp/higgs_out \
+    --prompts "Hello world." "The quick brown fox jumps over the lazy dog."
+
+# Voice cloning — pass a reference clip and its transcript together
+python higgs_audio_v2/batch_speech_client.py \
+    --base-url http://localhost:8094 \
+    --output-dir /tmp/higgs_clone \
+    --ref-audio /path/to/reference.wav \
+    --ref-text  "Exact transcript spoken in reference.wav." \
+    --prompts "Hello, this is a cloned voice."
+```
+
+### Notes
+- Output: 24 kHz mono.
+- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades cloned-voice quality.
+- Out of scope (rejected with explicit 4xx): multi-speaker `[SPEAKERn]` tags inside `input`, `profile:` text-only speaker descriptions, the `ref_audio_in_system_message` system-block variant, chunked long-form generation, and per-request `voice` / `instructions` / `task_type` / `language` / `speed != 1.0` / `x_vector_only_mode` / `speaker_embedding`.
 ## GLM-TTS

 2-stage TTS (AR + DiT flow-matching) at 24 kHz. Every request requires `ref_audio` + `ref_text`.
@@ -289,6 +329,16 @@ Single-GPU serves now default to the uniproc executor (lower IPC overhead, the B

 To opt out of chunked streaming, pass `--no-async-chunk` — the pipeline auto-dispatches to the end-to-end codec processor.

+### Tuning stage 1 `max_num_seqs` per task type
+The bundled `qwen3_tts.yaml` ships stage 1 (Code2Wav) at `max_num_seqs: 10`, tuned for Base voice cloning: stage-1 lifetimes are long (~3 s/req), so admitting up to 10 concurrent codec sequences lets requests progress in parallel in the scheduler — ~2× TTFA p95 at c=4 / c=8 (1× H100, 1.7B-Base, seed-tts) at an 8–12 % audio-throughput cost.
+
+CustomVoice / VoiceDesign have much shorter stage-1 lifetimes (~50–200 ms) and are TTFA-optimal at `max_num_seqs: 1`. Override the default when serving those task types:
+
+```bash
+vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni \
+    --stage-overrides '{"1": {"max_num_seqs": 1}}'
+```
+
 ### Sending requests
 ```bash
 # CustomVoice with a predefined speaker
@@ -459,6 +509,13 @@ The demo handles voice-preset selection and reference-audio upload. `voxtral_tts
    ``````py
    --8<-- "examples/online_serving/text_to_speech/fish_speech/speech_client.py"
    ``````
+??? abstract "higgs_audio_v2/batch_speech_client.py"
+    ``````py
+    --8<-- "examples/online_serving/text_to_speech/higgs_audio_v2/batch_speech_client.py"
+    ``````
+??? abstract "higgs_audio_v2/run_server.sh"
+    ``````sh
+    --8<-- "examples/online_serving/text_to_speech/higgs_audio_v2/run_server.sh"
 ??? abstract "glm_tts/gradio_demo.py"
    ``````py
    --8<-- "examples/online_serving/text_to_speech/glm_tts/gradio_demo.py"
@@ -52,8 +52,8 @@ vLLM-Omni's runtime module names.

 | Model | Scope | Status | Notes |
 |-------|-------|--------|-------|
+| GLM-Image | Diffusion transformer  | ✅ | `Intel/GLM-Image-int4-AutoRound` |
 | BAGEL | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint |
-| GLM-Image | Checkpoint-defined diffusion or transformer stage | Not validated | Requires a compatible AutoRound checkpoint |

 ## Configuration

@@ -7,26 +7,28 @@ using the OCP MX format: groups of 32 K-dimension elements share a single
 `float8_e8m0fnu` exponent scale. This gives better accuracy than channel-wise
 FP8 while keeping the same 8-bit weight footprint.

-This method supports two modes:
+This method supports three modes:

 | Mode | Description |
 |------|-------------|
 | **Online** | BF16 weights are quantized to MXFP8 at load time — no pre-processing needed |
-| **Offline** | msModelSlim-exported MXFP8 weights converted to diffusers format via `merge_mxfp8_checkpoint.py` — weights and scales are loaded directly from the preprocessed checkpoint |
+| **Offline (Native)** | msModelSlim-exported MXFP8 weights converted to diffusers format via `merge_mxfp8_checkpoint.py` — weights and scales are loaded directly from the preprocessed checkpoint |
+| **Offline (AutoRound)** | AutoRound MXFP8 checkpoints with `data_type="mx_fp"` — auto-detected from `config.json` |

 ## Hardware Support

-| Device | Support |
-|--------|---------|
-| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
-| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ |
-| NVIDIA Ampere GPU (SM 80+) | ⭕ |
-| AMD ROCm | ⭕ |
-| Intel XPU | ⭕ |
-| Ascend NPU (Atlas 950 A5) | ✅ |
+| Device | Online | Offline (Native) | Offline (AutoRound) |
+|--------|--------|------------------|---------------------|
+| NVIDIA Blackwell GPU (SM 100+) | ⭕ | ⭕ | ⭕ |
+| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ | ⭕ | ⭕ |
+| NVIDIA Ampere GPU (SM 80+) | ⭕ | ⭕ | ⭕ |
+| AMD ROCm | ⭕ | ⭕ | ⭕ |
+| Intel XPU | ✅ | ❌ | ✅ |
+| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ⭕ |

-Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
-guide.
+Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
+
+**Note**: Intel XPU only supports AutoRound MXFP8 for offline mode. Use AutoRound quantized checkpoints or online mode for XPU.

 ## Model Type Support

@@ -82,9 +84,9 @@ python text_to_video.py --model <your-model> --quantization mxfp8
 vllm serve <your-model> --omni --quantization mxfp8
 ```

-### Offline Mode
+### Offline Mode (Native)

-Offline mode loads a pre-quantized checkpoint from msModelSlim. A preprocessing
+Native offline mode loads a pre-quantized checkpoint from msModelSlim. A preprocessing
 step converts the raw quantized output to the diffusers format expected by
 vLLM-Omni and injects the quantization config into `transformer/config.json` so
 that vLLM-Omni auto-detects the offline path without a `--quantization` flag.
@@ -159,10 +161,41 @@ omni = Omni(model="/path/to/Wan2.2-TI2V-5B-MXFP8")
 ```

 !!! note
-    No `--quantization` flag is needed for offline mode. The preprocessing
+    No `--quantization` flag is needed for native offline mode. The preprocessing
    script injects `quantization_config` into each `transformer/config.json`,
    which vLLM-Omni reads automatically to activate the offline MXFP8 method.

+### Offline Mode (AutoRound)
+
+AutoRound MXFP8 checkpoints declare `quant_method="auto-round"` with `data_type="mx_fp"` in their `config.json`. These are automatically detected and use the `IncMxfp8OfflineLinearMethod` backend.
+
+To use an AutoRound MXFP8 checkpoint:
+
+```bash
+python text_to_video.py --model <autoround-mxfp8-model>
+
+# Online serving
+vllm serve <autoround-mxfp8-model> --omni
+```
+
+Python API:
+
+```python
+omni = Omni(model="<autoround-mxfp8-model>")
+```
+
+!!! note
+    AutoRound MXFP8 checkpoints are auto-detected from `config.json` and do not require a `--quantization` flag. The config must include:
+    ```json
+    {
+      "quantization_config": {
+        "quant_method": "auto-round",
+        "data_type": "mx_fp",
+        ...
+      }
+    }
+    ```
+
 ## Parameters

 | Parameter | Type | Default | Description |
@@ -21,7 +21,7 @@ by msModelSlim and the merge tools.
 | NVIDIA Ada/Hopper GPU (SM 89+) | ✅ | ✅ | ⭕ | ⭕ |
 | NVIDIA Ampere GPU (SM 80+) | ✅ | ✅ | ⭕ | ⭕ |
 | AMD ROCm | ⭕ | ⭕ | ⭕ | ⭕ |
-| Intel XPU | ⭕ | ⭕ | ⭕ | ⭕ |
+| Intel XPU | ⭕ | ⭕ | ✅ | ⭕ |
 | Ascend NPU | ❌ | ✅ | ✅ | ✅ |

 Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this
@@ -36,7 +36,7 @@ MXFP4 are documented for the Ascend NPU path.
 |--------|-------|----------------|--------|
 | FP8 W8A8 | [FP8](fp8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image family and other DiT models |
 | Int8 W8A8 | [Int8](int8.md) | Qwen-Image; Wan2.2 is not validated | Validated for Qwen-Image and Z-Image |
-| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Ascend NPU only |
+| MXFP8 W8A8 | [MXFP8](mxfp8.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, Wan2.2-TI2V-5B | Validated on Ascend NPU and Intel XPU |
 | MXFP4 W4A4 | [MXFP4](mxfp4.md) | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B | Ascend NPU only; TI2V-5B is not supported |

 ### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
@@ -17,7 +17,7 @@ from vllm_omni.entrypoints.omni import Omni
 from vllm_omni.inputs.data import OmniPromptType

 _REPO_ROOT = Path(__file__).resolve().parents[3]
-_DEFAULT_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3.yaml")
+_DEFAULT_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image_3_moe.yaml")
 _DEFAULT_AR_DEPLOY_CONFIG = str(_REPO_ROOT / "vllm_omni" / "deploy" / "hunyuan_image3_ar.yaml")

 _MODALITY_TASK_MAP: dict[str, tuple[str, str | None]] = {
@@ -61,8 +61,8 @@ def parse_args():
    parser.add_argument("--steps", type=int, default=50, help="Number of inference steps.")
    parser.add_argument("--guidance-scale", type=float, default=5.0, help="Classifier-free guidance scale.")
    parser.add_argument("--seed", type=int, default=42, help="Random seed.")
-    parser.add_argument("--height", type=int, default=1024, help="Output image height.")
-    parser.add_argument("--width", type=int, default=1024, help="Output image width.")
+    parser.add_argument("--height", type=int, default=None, help="Output image height.")
+    parser.add_argument("--width", type=int, default=None, help="Output image width.")
    parser.add_argument("--vae-use-tiling", action="store_true", help="Enable VAE tiling.")
    parser.add_argument(
        "--bot-task",
@@ -229,7 +229,21 @@ def main():

    from vllm_omni.inputs.data import OmniDiffusionSamplingParams

-    ar_stop_token_ids = resolve_stop_token_ids(task=task, bot_task=bot_task, tokenizer=tokenizer)
+    if (args.height is None) != (args.width is None):
+        raise ValueError("--height and --width must both be specified or both omitted.")
+    user_specified_size = args.height is not None and args.width is not None
+    if args.modality in ("img2text", "text2text"):
+        ar_image_size = "auto"
+    elif user_specified_size:
+        ar_image_size = f"{args.width}x{args.height}"
+    else:
+        ar_image_size = None
+    ar_stop_token_ids = resolve_stop_token_ids(
+        task=task, bot_task=bot_task, tokenizer=tokenizer, image_size=ar_image_size
+    )
+    print(
+        f"[AR Config] task={task}, bot_task={bot_task}, image_size={ar_image_size}, stop_token_ids={ar_stop_token_ids}"
+    )
    for sp in params_list:
        if isinstance(sp, OmniDiffusionSamplingParams):
            sp.num_inference_steps = args.steps
@@ -127,7 +127,12 @@ def run_one_path(
            ),
            args.device,
        )
-        pred = run_pipeline_forward(pipeline, batch_inputs, noise)
+        pred = run_pipeline_forward(
+            pipeline,
+            batch_inputs,
+            noise,
+            request_id=f"internvla-a1-sample-{index}",
+        )
        pred = pred[:, :, : dataset.physical_action_dim].to(torch.float32).cpu()
        results.append(
            {
@@ -150,6 +155,7 @@ def run_pipeline_forward(
    pipeline,
    batch_inputs: dict[str, torch.Tensor],
    noise: torch.Tensor,
+    request_id: str,
 ) -> torch.Tensor:
    output = pipeline.forward(
        OmniDiffusionRequest(
@@ -161,6 +167,7 @@ def run_pipeline_forward(
                    "decode_image": False,
                }
            ),
+            request_id=request_id,
        )
    )
    if output.error:
@@ -212,23 +219,38 @@ def benchmark_forward(

    _synchronize(args.device)
    cold_start_begin = time.perf_counter()
-    pred = run_pipeline_forward(pipeline, batch_inputs, noise)
+    pred = run_pipeline_forward(
+        pipeline,
+        batch_inputs,
+        noise,
+        request_id=f"internvla-a1-benchmark-{index}-cold",
+    )
    _synchronize(args.device)
    cold_start_ms = (time.perf_counter() - cold_start_begin) * 1000.0

    warmup_ms: list[float] = []
-    for _ in range(args.warmup_iters):
+    for iter_idx in range(args.warmup_iters):
        _synchronize(args.device)
        begin = time.perf_counter()
-        _ = run_pipeline_forward(pipeline, batch_inputs, noise)
+        _ = run_pipeline_forward(
+            pipeline,
+            batch_inputs,
+            noise,
+            request_id=f"internvla-a1-benchmark-{index}-warmup-{iter_idx}",
+        )
        _synchronize(args.device)
        warmup_ms.append((time.perf_counter() - begin) * 1000.0)

    benchmark_ms: list[float] = []
-    for _ in range(args.benchmark_iters):
+    for iter_idx in range(args.benchmark_iters):
        _synchronize(args.device)
        begin = time.perf_counter()
-        _ = run_pipeline_forward(pipeline, batch_inputs, noise)
+        _ = run_pipeline_forward(
+            pipeline,
+            batch_inputs,
+            noise,
+            request_id=f"internvla-a1-benchmark-{index}-iter-{iter_idx}",
+        )
        _synchronize(args.device)
        benchmark_ms.append((time.perf_counter() - begin) * 1000.0)

@@ -295,7 +317,12 @@ def main() -> None:
            dataset=dataset,
            train_meta=train_meta,
            collate_samples=collate_open_loop_samples,
-            run_sample_actions=lambda policy, batch_inputs, noise: run_pipeline_forward(policy, batch_inputs, noise),
+            run_sample_actions=lambda policy, batch_inputs, noise: run_pipeline_forward(
+                policy,
+                batch_inputs,
+                noise,
+                request_id="internvla-a1-open-loop",
+            ),
            num_episodes=args.num_episodes,
            seed=args.seed,
            device=args.device,
@@ -2,12 +2,13 @@

 [Ming-flash-omni-2.0](https://github.com/inclusionAI/Ming) is an omni-modal model supporting text, image, video, and audio understanding, with text and speech outputs.

-vLLM-Omni supports two deployment modes:
+vLLM-Omni supports three deployment modes:

 | Mode | Deploy config | Output |
 |------|--------------|--------|
 | Thinker + Talker (omni-speech, default) | `vllm_omni/deploy/ming_flash_omni.yaml` | Text + Audio |
 | Thinker only (multimodal understanding) | `vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
+| Thinker + Imagegen (text-to-image / img2img) | `vllm_omni/deploy/ming_flash_omni_image.yaml` | Image (online-serving only at the moment) |

 For standalone TTS (talker only), see the [Ming-flash-omni-TTS section in the Text-To-Speech hub](../text_to_speech/README.md#ming-flash-omni-tts).

@@ -87,6 +88,17 @@ The default deploy YAML allocates thinker on GPUs 0–3 and talker on GPU 3 for

 Pass `--deploy-config /path/to/your_deploy.yaml` to any of the commands above to override the bundled deploy config.

+### Image generation (text-to-image / img2img)
+
+The diffusion-stage image-generation path is currently only wired through
+the online OpenAI-compatible chat endpoint (`/v1/chat/completions` with
+`"modalities": ["image"]`). For payloads, optional knobs
+(`steps`/`cfg`/`seed`/`byte5_text`/`negative_prompt`), and the img2img
+reference-image flow, see the
+[image-gen section in the online-serving README](../../online_serving/ming_flash_omni/README.md#image-generation-text-to-image).
+`end2end.py` does not yet exercise the imagegen stage; that is tracked as
+follow-up work.
+
 ## Online serving

 For online serving via the OpenAI-compatible API, see [examples/online_serving/ming_flash_omni/README.md](../../online_serving/ming_flash_omni/README.md).
@@ -37,6 +37,10 @@ python end2end.py --modality text2text \
 # Custom resolution (image generation)
 python end2end.py --prompt "A futuristic cityscape at sunset" \
                  --width 2048 --height 1024 --think
+
+# Text-to-image with Cache-DiT acceleration
+python end2end.py --prompt "A cute cat" \
+                  --cache-backend cache_dit \
 ```

 > **Note**: Default configuration works on a single **NVIDIA A100 (80GB)** or **H100** GPU.
@@ -135,6 +139,8 @@ python end2end.py \
 | `--tensor-parallel-size` | 1 | Number of GPUs for tensor parallelism |
 | `--enforce-eager` | False | Disable torch.compile |
 | `--enable-cpu-offload` | False | Enable module-wise (sequential) CPU offload to reduce peak VRAM |
+| `--cache-backend` | None | Set to `cache_dit` for Cache-DiT acceleration |
+| `--enable-cache-dit-summary` | False | Print Cache-DiT cache statistics after generation |

 ## Reducing GPU Memory Usage

@@ -155,6 +155,18 @@ def parse_args():
            "of extra CPU<->GPU transfers."
        ),
    )
+    parser.add_argument(
+        "--cache-backend",
+        type=str,
+        default=None,
+        choices=["cache_dit"],
+        help="Cache backend to use for image generation acceleration.",
+    )
+    parser.add_argument(
+        "--enable-cache-dit-summary",
+        action="store_true",
+        help="Enable cache-dit summary logging after diffusion forward passes.",
+    )

    nullify_stage_engine_defaults(parser)
    return parser.parse_args()
@@ -182,6 +194,8 @@ def main():
        tensor_parallel_size=args.tensor_parallel_size,
        enforce_eager=args.enforce_eager,
        enable_cpu_offload=args.enable_cpu_offload,
+        cache_backend=args.cache_backend,
+        enable_cache_dit_summary=args.enable_cache_dit_summary,
    )

    extra_args = {
@@ -225,6 +239,7 @@ def main():
    print(f"  Seed           : {args.seed}")
    print(f"  Think mode     : {args.think}")
    print(f"  TP size        : {args.tensor_parallel_size}")
+    print(f"  Cache backend  : {args.cache_backend or 'none'}")
    print(f"{'=' * 60}\n")

    # Build prompt dict
@@ -271,6 +271,14 @@ python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
    --lang zh
 ```

+### Seed for Reproducibility
+```bash
+python examples/offline_inference/text_to_speech/omnivoice/end2end.py \
+    --model k2-fsa/OmniVoice \
+    --text "Hello, this is a test." \
+    --seed 42
+```
+
 ### Notes
 - Stage 0 (Generator): Qwen3-0.6B with 32-step iterative unmasking.
 - Stage 1 (Decoder): HiggsAudioV2 RVQ + DAC at 24 kHz.
@@ -0,0 +1,38 @@
+# higgs-audio v2 — offline example
+
+Drives Stage 0 (DualFFN talker) + Stage 1 (HiggsAudio codec) for `bosonai/higgs-audio-v2-generation-3B-base` end-to-end through the vLLM-Omni engine and writes a 24 kHz mono WAV per prompt.
+
+## Prerequisites
+
+Voice clone needs `transformers>=5.3.0` — vllm-omni loads the audio codec via HF's `HiggsAudioV2TokenizerModel`, instantiated from the `k2-fsa/OmniVoice/audio_tokenizer/` subdirectory (only that ~806 MB subdir is downloaded). The boson-ai standalone tokenizer repo's `model.safetensors` is actually a copy of the 3B talker LM, so HF can't load it directly; the k2 bundle ships the same codec weights repackaged with HF-compatible key naming.
+
+```bash
+pip install -U "transformers>=5.3.0"
+```
+
+## Quick start
+
+Plain TTS:
+```bash
+python examples/offline_inference/text_to_speech/higgs_audio_v2/end2end.py \
+    --texts "Hello world." "The quick brown fox jumps over the lazy dog." \
+    --output-dir results/wavs
+```
+
+## Voice cloning
+
+Pass both `--ref-audio` and `--ref-text` together:
+```bash
+python examples/offline_inference/text_to_speech/higgs_audio_v2/end2end.py \
+    --texts "Hello, this is a cloned voice." \
+    --ref-audio /path/to/reference.wav \
+    --ref-text  "Exact transcript spoken in reference.wav." \
+    --output-dir results/wavs
+```
+
+## Notes
+
+- Output: 24 kHz mono WAV.
+- Deploy config: `vllm_omni/deploy/higgs_audio_v2.yaml` (auto-loaded by HF `model_type`).
+- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades cloned-voice quality.
+- For online serving, see [`examples/online_serving/text_to_speech/higgs_audio_v2/`](../../../online_serving/text_to_speech/higgs_audio_v2/).
@@ -0,0 +1,205 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Offline higgs-audio v2 inference example.
+
+Runs Stage 0 (DualFFN talker) + Stage 1 (HiggsAudio codec) end-to-end
+through the vLLM-Omni engine without going through the HTTP server, and
+saves a 24 kHz mono WAV per prompt.
+
+Example:
+
+    python examples/offline_inference/text_to_speech/higgs_audio_v2/end2end.py \\
+        --texts "Hello world." \\
+                "The quick brown fox jumps over the lazy dog." \\
+        --output-dir results/wavs
+"""
+
+from __future__ import annotations
+
+import os
+
+# DeepGEMM FP8 kernels require an optional backend that may not be installed.
+# Disable the warmup before importing vLLM so engine startup falls back to the
+# regular gemm path. Users with deep_gemm installed can override these.
+os.environ.setdefault("VLLM_USE_DEEP_GEMM", "0")
+os.environ.setdefault("VLLM_MOE_USE_DEEP_GEMM", "0")
+
+import time
+from pathlib import Path
+
+import numpy as np
+import soundfile as sf
+import torch
+from vllm.utils.argparse_utils import FlexibleArgumentParser
+
+from vllm_omni import Omni
+from vllm_omni.engine.arg_utils import nullify_stage_engine_defaults
+
+SAMPLE_RATE = 24_000
+DEFAULT_TEXTS = (
+    "Hello world.",
+    "The quick brown fox jumps over the lazy dog.",
+)
+
+
+def parse_args():
+    parser = FlexibleArgumentParser(description="Offline higgs-audio v2 inference")
+    parser.add_argument(
+        "--model",
+        type=str,
+        default="bosonai/higgs-audio-v2-generation-3B-base",
+        help="Stage-0 talker model id or path.",
+    )
+    parser.add_argument(
+        "--texts",
+        type=str,
+        nargs="+",
+        default=list(DEFAULT_TEXTS),
+        help="One or more plain-text prompts to synthesize.",
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=str,
+        default="results/wavs",
+        help="Directory to write per-prompt WAV files.",
+    )
+    parser.add_argument(
+        "--deploy-config",
+        type=str,
+        default=None,
+        help="Override the deploy config path. Auto-loads "
+        "vllm_omni/deploy/higgs_audio_v2.yaml from the HF model_type by default.",
+    )
+    parser.add_argument(
+        "--max-new-tokens",
+        type=int,
+        default=500,
+        help="Cap on Stage-0 codec frames per request.",
+    )
+    parser.add_argument(
+        "--ref-audio",
+        type=str,
+        default=None,
+        help="Reference clip for voice clone (path to a WAV file). Paired with --ref-text.",
+    )
+    parser.add_argument(
+        "--ref-text",
+        type=str,
+        default=None,
+        help="Transcript of the reference clip. Required when --ref-audio is set.",
+    )
+    nullify_stage_engine_defaults(parser)
+    return parser.parse_args()
+
+
+def _slugify(text: str) -> str:
+    import re
+
+    slug = re.sub(r"\s+", "_", text.strip().lower())
+    slug = re.sub(r"[^a-z0-9_]+", "", slug)
+    return slug[:48] or "prompt"
+
+
+def _extract_pcm(multimodal_output: dict) -> torch.Tensor:
+    """Pull the final concatenated PCM tensor out of a request's multimodal_output."""
+    audio = multimodal_output.get("model_outputs")
+    if audio is None:
+        audio = multimodal_output.get("audio")
+    if audio is None:
+        raise ValueError(f"no audio key in multimodal_output: {list(multimodal_output.keys())}")
+    if isinstance(audio, list):
+        valid = [torch.as_tensor(a).float().cpu().reshape(-1) for a in audio if a is not None]
+        if not valid:
+            raise ValueError("audio list is empty")
+        return torch.cat(valid, dim=0) if len(valid) > 1 else valid[0]
+    return torch.as_tensor(audio).float().cpu().reshape(-1)
+
+
+def _pcm_to_int16(pcm: torch.Tensor) -> np.ndarray:
+    arr = pcm.numpy()
+    if arr.dtype.kind == "f":
+        arr = np.clip(arr, -1.0, 1.0)
+        arr = (arr * 32767.0).astype(np.int16)
+    else:
+        arr = arr.astype(np.int16)
+    return arr
+
+
+def main():
+    args = parse_args()
+    if (args.ref_audio is None) != (args.ref_text is None):
+        raise SystemExit("--ref-audio and --ref-text must be supplied together")
+    output_dir = Path(args.output_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    engine = Omni(model=args.model, deploy_config=args.deploy_config)
+
+    # Build prompt_token_ids using the same path serving_speech.py takes online.
+    from transformers import AutoProcessor
+
+    from vllm_omni.model_executor.models.higgs_audio_v2.higgs_audio_v2_tokenizer import (
+        build_plain_text_prompt,
+        build_voice_clone_prompt,
+        input_ids_to_python_list,
+    )
+
+    processor = AutoProcessor.from_pretrained(args.model, trust_remote_code=True)
+
+    # Voice-clone path: load the reference clip once. The HF processor will
+    # encode it via the bundled HiggsAudioV2TokenizerModel each time we build
+    # a prompt below. This is cheap on CPU for a few-second clip.
+    ref_wav: np.ndarray | None = None
+    ref_sr: int | None = None
+    if args.ref_audio is not None:
+        ref_wav, ref_sr = sf.read(args.ref_audio, always_2d=False)
+        if ref_wav.ndim == 2:
+            ref_wav = ref_wav.mean(axis=1)
+
+    print(f"Model       : {args.model}")
+    print(f"Prompts     : {len(args.texts)}")
+    print(f"Output dir  : {output_dir}")
+    print(f"Voice clone : {'yes' if ref_wav is not None else 'no'}")
+
+    # Run one prompt at a time. The Stage-0 talker's per-slot audio state is
+    # request-scoped; submitting multiple prompts in the same engine.generate()
+    # call would batch them in the AR runner and exercise a code path that is
+    # not validated for this model yet.
+    total_elapsed = 0.0
+    total_dur = 0.0
+    for text in args.texts:
+        if ref_wav is not None:
+            out = build_voice_clone_prompt(processor, text, ref_wav, int(ref_sr), args.ref_text)
+            prompt = {
+                "prompt_token_ids": out["prompt_token_ids"],
+                # Bare tensors (NOT list-wrapped): the msgspec serializer in
+                # vllm_omni.data_entry_keys routes torch.Tensor → tensor_data
+                # and list[Tensor] → list_data (which silently strips tensors).
+                "additional_information": {
+                    "audio_input_ids": out["audio_input_ids"],
+                    "audio_input_ids_mask": out["audio_input_ids_mask"],
+                },
+            }
+        else:
+            inputs = build_plain_text_prompt(processor, text)
+            prompt = {"prompt_token_ids": input_ids_to_python_list(inputs)}
+        t_start = time.perf_counter()
+        outputs = engine.generate([prompt])
+        elapsed = time.perf_counter() - t_start
+        total_elapsed += elapsed
+
+        mm = outputs[0].outputs[0].multimodal_output
+        pcm = _extract_pcm(mm)
+        slug = _slugify(text)
+        out_path = output_dir / f"{slug}.wav"
+        sf.write(str(out_path), _pcm_to_int16(pcm), SAMPLE_RATE, format="WAV", subtype="PCM_16")
+        dur = pcm.numel() / SAMPLE_RATE
+        total_dur += dur
+        print(f"  {slug:<50} dur={dur:5.2f}s  -> {out_path}")
+
+    rtf = total_elapsed / total_dur if total_dur > 0 else float("inf")
+    print(f"Total infer : {total_elapsed:.2f}s  total audio: {total_dur:.2f}s  RTF: {rtf:.3f}")
+
+
+if __name__ == "__main__":
+    os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")
+    main()
@@ -80,6 +80,12 @@ def run_e2e():
        default=600,
        help="Stage initialization timeout in seconds",
    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        default=None,
+        help="Random seed for generation",
+    )
    nullify_stage_engine_defaults(parser)
    args = parser.parse_args()

@@ -122,7 +128,7 @@ def run_e2e():
    if mm_processor_kwargs:
        prompts["mm_processor_kwargs"] = mm_processor_kwargs

-    sampling_params_list = [OmniDiffusionSamplingParams()]
+    sampling_params_list = [OmniDiffusionSamplingParams(extra_args={"seed": args.seed})]

    print(f"Generating speech for: {args.text}")

@@ -43,8 +43,8 @@ def _estimate_prompt_len(
    """
    try:
        from vllm_omni.model_executor.models.qwen3_tts.configuration_qwen3_tts import Qwen3TTSConfig
-        from vllm_omni.model_executor.models.qwen3_tts.qwen3_tts_talker import (
-            Qwen3TTSTalkerForConditionalGeneration,
+        from vllm_omni.model_executor.models.qwen3_tts.prompt_embeds_builder import (
+            Qwen3TTSPromptEmbedsBuilder,
        )

        if model_name not in _cache:
@@ -124,7 +124,7 @@ def _estimate_prompt_len(
            except Exception:
                return None

-        return Qwen3TTSTalkerForConditionalGeneration.estimate_prompt_len_from_additional_information(
+        return Qwen3TTSPromptEmbedsBuilder.estimate_prompt_len_from_additional_information(
            additional_information=additional_information,
            task_type=task_type,
            tokenize_prompt=lambda t: tok(t, padding=False)["input_ids"],
@@ -1,24 +1,105 @@
 # VACE Video Generation

-[VACE](https://github.com/ali-vilab/VACE) (Video All-in-one Creation Engine) supports multiple video tasks through a single model.
+Generate videos from text prompts, images, or video conditions using vLLM-Omni's VACE diffusion pipeline.

-| Model | Architecture | Model Weights (bf16) | HuggingFace |
-|-------|-------------|----------------------|-------------|
-| Wan2.1-VACE (1.3B) | Wan2.1 | ~10 GB | [Wan-AI/Wan2.1-VACE-1.3B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B-diffusers) |
-| Wan2.1-VACE (14B) | Wan2.1 | ~38 GB | [Wan-AI/Wan2.1-VACE-14B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers) |
+- `vace_video_generation.py`: command-line script for multi-mode video generation with advanced options.

-## Text-to-Video (T2V)
+## Table of Contents
+
+- [Overview](#overview)
+- [Quick Start](#quick-start)
+- [Key Arguments](#key-arguments)
+- [More CLI Examples](#more-cli-examples)
+
+## Overview
+
+[VACE](https://github.com/ali-vilab/VACE) (Video All-in-one Creation Engine) supports multiple video tasks through a single unified model, including text-to-video, image-to-video, first-last-frame interpolation, inpainting, and reference image-guided generation.
+
+### Supported Models
+
+| Model | Architecture | Peak VRAM (GiB) * | Model Weights (GiB) | HuggingFace |
+|-------|-------------|-------------------|---------------------|-------------|
+| Wan2.1-VACE (1.3B) | Wan2.1 | TBD | ~10 | [Wan-AI/Wan2.1-VACE-1.3B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B-diffusers) |
+| Wan2.1-VACE (14B) | Wan2.1 | TBD | ~38 | [Wan-AI/Wan2.1-VACE-14B-diffusers](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers) |
+
+!!! info
+*Peak VRAM: based on basic single-card usage, 480x832 resolution, 81 frames, 30 inference steps, without any acceleration/optimization features.
+
+Default model: `Wan-AI/Wan2.1-VACE-14B-diffusers`
+
+## Quick Start
+
+### Python API
+
+Text-to-video generation:
+
+```python
+import torch
+
+from vllm_omni.entrypoints.omni import Omni
+from vllm_omni.inputs.data import OmniDiffusionSamplingParams
+
+if __name__ == "__main__":
+    omni = Omni(model="Wan-AI/Wan2.1-VACE-1.3B-diffusers")
+    prompt = "A sleek robot stands in a vast warehouse filled with boxes"
+    outputs = omni.generate(
+        prompt,
+        OmniDiffusionSamplingParams(
+            height=480,
+            width=832,
+            num_frames=81,
+            num_inference_steps=30,
+            guidance_scale=5.0,
+        ),
+    )
+    video = outputs[0].images
+
+    from diffusers.utils import export_to_video
+    export_to_video(list(video[0]), "t2v_output.mp4", fps=16)
+    omni.close()
+```
+
+### Local CLI Usage

 ```bash
 python vace_video_generation.py \
  --mode t2v \
  --prompt "A sleek robot stands in a vast warehouse filled with boxes" \
  --height 480 --width 832 --num-frames 81 \
-  --num-inference-steps 30 --guidance-scale 5.0 --flow-shift 5.0 \
  --output t2v_output.mp4
 ```

-## Image-to-Video (I2V)
+## Key Arguments
+
+| Argument | Type | Default | Description |
+| -------- | ---- | ------- | ----------- |
+| `--mode` | str | `"t2v"` | VACE task mode: `t2v`, `i2v`, `v2lf`, `flf2v`, `inpaint`, `r2v` |
+| `--model` | str | `"Wan-AI/Wan2.1-VACE-14B-diffusers"` | Model ID or local path |
+| `--prompt` | str | `"A cat walking in a garden"` | Text description of desired video |
+| `--negative-prompt` | str | `""` | Negative prompt for classifier-free guidance |
+| `--image` | str | `None` | Input image path (for I2V, R2V, FLF2V, inpaint modes) |
+| `--last-image` | str | `None` | Last frame image path (for FLF2V mode) |
+| `--height` | int | `480` | Output video height in pixels (should be a multiple of 16) |
+| `--width` | int | `832` | Output video width in pixels (should be a multiple of 16) |
+| `--num-frames` | int | `81` | Number of video frames to generate |
+| `--num-inference-steps` | int | `30` | Number of denoising steps (more steps = higher quality, slower) |
+| `--guidance-scale` | float | `5.0` | Classifier-free guidance scale |
+| `--flow-shift` | float | `5.0` | Scheduler flow shift parameter |
+| `--seed` | int | `42` | Random seed for deterministic sampling |
+| `--fps` | int | `16` | Frames per second for the saved MP4 |
+| `--output` | str | `"vace_output.mp4"` | Path to save the generated video |
+| `--vae-use-tiling` | flag | on | Enable VAE tiling for memory optimization |
+| `--cfg-parallel-size` | int | `1` | Set to `2` to enable CFG Parallel |
+| `--ulysses-degree` | int | `1` | Ulysses sequence parallel degree for multi-GPU inference |
+| `--ring-degree` | int | `1` | Ring sequence parallel degree for hybrid Ulysses + Ring inference |
+| `--tensor-parallel-size` | int | — | Tensor parallel size |
+| `--enforce-eager` | flag | off | Disable torch.compile |
+
+> If you encounter OOM errors, try `--vae-use-tiling` or multi-GPU parallelism options (`--ulysses-degree`, `--cfg-parallel-size`).
+
+## More CLI Examples
+
+### Image-to-Video (I2V)

 First frame is kept, remaining frames are generated:

@@ -31,7 +112,7 @@ python vace_video_generation.py \
  --output i2v_output.mp4
 ```

-## First-Last-Frame Interpolation (FLF2V)
+### First-Last-Frame Interpolation (FLF2V)

 ```bash
 python vace_video_generation.py \
@@ -42,7 +123,7 @@ python vace_video_generation.py \
  --output flf2v_output.mp4
 ```

-## Inpainting
+### Inpainting

 Center vertical stripe is masked and regenerated:

@@ -55,7 +136,7 @@ python vace_video_generation.py \
  --output inpaint_output.mp4
 ```

-## Reference Image-guided (R2V)
+### Reference Image-guided (R2V)

 ```bash
 python vace_video_generation.py \
@@ -65,24 +146,3 @@ python vace_video_generation.py \
  --height 480 --width 832 --num-frames 81 \
  --output r2v_output.mp4
 ```
-
-## Key Arguments
-
- `--mode`: VACE task mode (`t2v`, `i2v`, `flf2v`, `inpaint`, `r2v`).
- `--model`: Model ID (default: `Wan-AI/Wan2.1-VACE-1.3B-diffusers`).
- `--image`: Input image for I2V, inpainting, and R2V modes.
- `--last-image`: Last frame image for FLF2V mode.
- `--prompt`: Text description of desired video.
- `--height/--width`: Output resolution (default 480x832). Dimensions should be multiples of 16.
- `--num-frames`: Number of frames (default 81).
- `--guidance-scale`: CFG scale (default 5.0).
- `--flow-shift`: Scheduler flow shift (default 5.0).
- `--num-inference-steps`: Number of denoising steps (default 30).
- `--fps`: Frames per second for the saved MP4 (default 16).
- `--output`: Path to save the generated video.
- `--vae-use-tiling`: Enable VAE tiling for memory optimization.
- `--ulysses-degree`: Ulysses sequence parallelism degree for multi-GPU.
- `--cfg-parallel-size`: CFG parallel size for multi-GPU.
- `--tensor-parallel-size`: Tensor parallel size.
-
-> If you encounter OOM errors, try `--vae-use-tiling` or multi-GPU parallelism options.
@@ -102,3 +102,5 @@ Key arguments:
 - `--audio-negative-prompt`: negative prompt for audio generation.
 - `--enable-cpu-offload`: enable CPU offload (defaults False).
 - `--cache-backend`: enable `cache_dit` for acceleration.
+- `--quantization`: online (dynamic) quantization method — `fp8` or `int8`.
+  (VAEs, the T5 text encoder, norms and modulation stay bf16.)
@@ -4,6 +4,7 @@
 import argparse
 import re
 import time
+from typing import Any

 import numpy as np
 from PIL import Image
@@ -41,6 +42,12 @@ def parse_args() -> argparse.Namespace:
        choices=[1, 2, 3, 4],
        help="Number of GPUs used for classifier free guidance parallel size (max 4 branches).",
    )
+    parser.add_argument(
+        "--tensor-parallel-size",
+        type=int,
+        default=1,
+        help=("Number of GPUs used for tensor parallelism (TP) inside the DiT."),
+    )
    parser.add_argument(
        "--video-negative-prompt",
        default="jitter, bad hands, blur, distortion",
@@ -52,6 +59,13 @@ def parse_args() -> argparse.Namespace:
        help="Negative prompt for audio.",
    )
    parser.add_argument("--output", default="dreamid_output.mp4", help="Output video path.")
+    parser.add_argument(
+        "--quantization",
+        type=str,
+        default=None,
+        choices=["fp8", "int8"],
+        help=("Online (dynamic) quantization method for the DreamID-Omni transformer. "),
+    )
    parser.add_argument(
        "--enable-cpu-offload",
        action="store_true",
@@ -100,6 +114,27 @@ def load_image_and_audio(image_paths, audio_paths):
    return image, audio


+def _extract_peak_memory_mb(result: Any) -> float:
+    """Pull worker-reported peak VRAM (MiB) generation result.
+
+    Mirrors vllm_omni/entrypoints/openai/serving_video.py:_extract_peak_memory_mb.
+    """
+    if isinstance(result, list):
+        result = result[0] if result else None
+    if result is None:
+        return 0.0
+    val = getattr(result, "peak_memory_mb", 0.0)
+    if not val:
+        inner = getattr(result, "request_output", None)
+        if isinstance(inner, list):
+            inner = inner[0] if inner else None
+        val = getattr(inner, "peak_memory_mb", 0.0)
+    try:
+        return float(val or 0.0)
+    except (TypeError, ValueError):
+        return 0.0
+
+
 def main() -> None:
    args = parse_args()
    if args.prompt is None and args.prompt_file is None:
@@ -142,6 +177,7 @@ def main() -> None:

    parallel_config = DiffusionParallelConfig(
        cfg_parallel_size=args.cfg_parallel_size,
+        tensor_parallel_size=args.tensor_parallel_size,
        use_hsdp=args.use_hsdp,
        hsdp_shard_size=args.hsdp_shard_size,
        hsdp_replicate_size=args.hsdp_replicate_size,
@@ -162,7 +198,7 @@ def main() -> None:
            "scm_steps_policy": "dynamic",
        }

-    omni = Omni(
+    omni_kwargs = dict(
        model=args.model,
        parallel_config=parallel_config,
        model_type=args.model_type,
@@ -171,6 +207,9 @@ def main() -> None:
        cache_backend=args.cache_backend,
        cache_config=cache_config,
    )
+    if args.quantization is not None:
+        omni_kwargs["quantization"] = args.quantization
+    omni = Omni(**omni_kwargs)
    start = time.perf_counter()
    outputs = omni.generate(prompt, sampling_params)
    elapsed = time.perf_counter() - start
@@ -208,6 +247,9 @@ def main() -> None:
        f.write(video_bytes)
    print(f"Saved generated video to {output_path}")
    print(f"Total time: {elapsed:.2f}s")
+    peak_mb = _extract_peak_memory_mb(outputs)
+    if peak_mb:
+        print(f"Worker peak GPU memory (reserved): {peak_mb:.2f} MiB ({peak_mb / 1024:.2f} GiB)")


 if __name__ == "__main__":
@@ -10,6 +10,7 @@ Please refer to [README.md](../../../README.md)
 |------|---------------|--------|
 | Thinker + Talker (omni-speech, default) | `vllm serve ... --omni` | Text + Audio |
 | Thinker only (multimodal understanding) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_thinker_only.yaml` | Text |
+| Thinker + Imagegen (text-to-image / img2img) | `vllm serve ... --omni --deploy-config vllm_omni/deploy/ming_flash_omni_image.yaml` | Image |

 For standalone TTS (talker only), see the [Ming-flash-omni-TTS section in the Text-To-Speech hub](../text_to_speech/README.md#ming-flash-omni-tts).

@@ -56,8 +57,91 @@ bash run_curl_multimodal_generation.sh use_image
 bash run_curl_multimodal_generation.sh use_audio
 bash run_curl_multimodal_generation.sh use_video
 bash run_curl_multimodal_generation.sh use_mixed_modalities
+bash run_curl_multimodal_generation.sh use_image_gen
 ```

+## Image generation (text-to-image)
+
+Ming-flash-omni-2.0 also exposes an image-generation (diffusion) stage.
+Launch with the image deploy YAML, which adds an image-gen stage behind
+the thinker:
+
+```bash
+vllm serve Jonathan1909/Ming-flash-omni-2.0 --omni \
+    --deploy-config vllm_omni/deploy/ming_flash_omni_image.yaml \
+    --stage-init-timeout 1800 \
+    --init-timeout 1800 \
+    --port 8091
+```
+
+Then request image output by passing `"modalities": ["image"]`:
+
+```bash
+curl -s http://127.0.0.1:8091/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+      "model": "Jonathan1909/Ming-flash-omni-2.0",
+      "messages": [{"role": "user", "content": "Please draw a cute cat."}],
+      "modalities": ["image"]
+    }' | jq -r '.choices[0].message.content[0].image_url.url | split(",")[1]' | base64 -d > ming_imagegen.png
+```
+
+### Optional knobs
+
+Pass image-gen overrides via the diffusion-stage `sampling_params_list[1].extra_args.image_gen`:
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `height` / `width` | from config (1024) | Output resolution (multiples of `vae_scale_factor * 2`, currently 16). |
+| `steps` | 30 | Number of FlowMatchEuler denoise steps. |
+| `cfg` | 2.0 | Classifier-free guidance scale. |
+| `seed` | 42 | Per-request RNG seed (deterministic when ByT5 is also seed-stable). |
+| `byte5_text` | (auto) | Override the glyph text for ByT5 enhancement; raw strings are auto-wrapped to Ming's `Text "...". ` format. |
+| `negative_prompt` | empty | Real CFG negative conditioning (must be set on **stage-0 thinker** `sampling_params` so `expand_cfg_prompts` spawns the companion). |
+
+Example with all the knobs:
+
+```bash
+curl http://127.0.0.1:8091/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+      "model": "Jonathan1909/Ming-flash-omni-2.0",
+      "modalities": ["image"],
+      "sampling_params_list": [
+        {"temperature":0.4,"top_p":0.9,"top_k":1,"max_tokens":1,"seed":42,
+         "extra_args":{"image_gen":{"negative_prompt":"ugly, blurry, distorted"}}},
+        {"seed":42,"extra_args":{"image_gen":{
+            "steps":6,"cfg":1.5,"height":512,"width":512,"seed":123,
+            "byte5_text":["理解与生成统一"]}}}
+      ],
+      "messages": [{"role": "user", "content": "Draw a poster."}]
+    }' | jq -r '.choices[0].message.content[0].image_url.url | split(",")[1]' | base64 -d > ming_imagegen_knobs.png
+```
+
+### img2img (reference image)
+
+Add an `image_url` content part to the user message; the parser routes it
+into the diffusion stage as `extra[reference_image]`:
+
+```jsonc
+"messages": [{
+  "role": "user",
+  "content": [
+    {"type": "text", "text": "Change the background to a sandy beach at sunset."},
+    {"type": "image_url", "image_url": {"url": "data:image/png;base64,<base64>"}}
+  ]
+}]
+```
+
+### GPU layout
+
+The shipped `ming_flash_omni_image.yaml` allocates the thinker on GPUs 0–3
+(TP=4) and the diffusion stage on GPU 4 (TP=1). Copy the YAML and edit
+`devices` per stage to relocate; with fewer GPUs available, drop
+the thinker TP to 2 and run the diffusion stage on a free card. Image-gen
+warmup takes roughly an extra 30–60 s on top of the thinker — set
+`--stage-init-timeout 1800` if the default 300 s is too tight.
+
 ## Modality control

 | `modalities` | Server config | Output |
@@ -65,6 +149,7 @@ bash run_curl_multimodal_generation.sh use_mixed_modalities
 | `["text"]` or omitted | Thinker only | Text |
 | `["audio"]` | Thinker + Talker | Audio (speech) |
 | `["text", "audio"]` | Thinker + Talker | Text + Audio |
+| `["image"]` | Thinker + Imagegen (image deploy YAML) | Image (PNG, base64 in `choices[0].message.content`) |

 For ready-to-copy curl examples (text / audio / multimodal input, SSE
 streaming, reasoning mode), see the recipe at
@@ -7,14 +7,15 @@ PORT="${PORT:-8091}"
 QUERY_TYPE="${1:-text}"

 # Validate query type
-if [[ ! "$QUERY_TYPE" =~ ^(text|use_audio|use_image|use_video|use_mixed_modalities)$ ]]; then
+if [[ ! "$QUERY_TYPE" =~ ^(text|use_audio|use_image|use_video|use_mixed_modalities|use_image_gen)$ ]]; then
    echo "Error: Invalid query type '$QUERY_TYPE'"
-    echo "Usage: $0 [text|use_audio|use_image|use_video|use_mixed_modalities]"
+    echo "Usage: $0 [text|use_audio|use_image|use_video|use_mixed_modalities|use_image_gen]"
    echo "  text: Text-only query"
    echo "  use_audio: Audio + Text query"
    echo "  use_image: Image + Text query"
    echo "  use_video: Video + Text query"
    echo "  use_mixed_modalities: Audio + Image + Video + Text query"
+    echo "  use_image_gen: Text-to-image (diffusion stage) — saves the generated PNG locally"
    exit 1
 fi

@@ -100,6 +101,43 @@ esac
 echo "Running query type: $QUERY_TYPE"
 echo ""

+if [[ "$QUERY_TYPE" == "use_image_gen" ]]; then
+    # Image-gen branch: dual-stage diffusion endpoint, modalities=["image"].
+    # The shipped ming_flash_omni_image.yaml runs the thinker on cards 0-3
+    # (TP=4) and the diffusion stage on card 4 (TP=1). See the image-gen
+    # section in README.md for knobs and img2img.
+    out_path="${OUT_PATH:-/tmp/ming_imagegen.png}"
+    request_body=$(cat <<'EOF'
+{
+  "model": "Jonathan1909/Ming-flash-omni-2.0",
+  "modalities": ["image"],
+  "messages": [
+    {"role": "user", "content": "Draw a beautiful girl with short black hair and red dress."}
+  ]
+}
+EOF
+)
+    output=$(curl -sS --retry 3 --retry-delay 3 --retry-connrefused \
+        -X POST "http://localhost:${PORT}/v1/chat/completions" \
+        -H "Content-Type: application/json" \
+        -d "$request_body")
+    # Server returns base64-encoded PNG either as a data URL in
+    # ``choices[0].message.content[0].image_url.url`` or as a raw string.
+    b64=$(echo "$output" | jq -r '.choices[0].message.content
+        | if type == "array"
+            then (map(select(.image_url? != null)) | .[0].image_url.url // "")
+            elif type == "string" then . else "" end')
+    if [[ -z "$b64" ]]; then
+        echo "Error: no image returned. Raw response:"
+        echo "$output" | jq '.'
+        exit 1
+    fi
+    if [[ "$b64" == data:image* ]]; then b64="${b64#*,}"; fi
+    printf '%s' "$b64" | base64 --decode > "$out_path"
+    echo "Saved generated image: $out_path ($(stat -c%s "$out_path" 2>/dev/null || stat -f%z "$out_path") bytes)"
+    exit 0
+fi
+
 request_body=$(cat <<EOF
 {
  "model": "Jonathan1909/Ming-flash-omni-2.0",
@@ -18,10 +18,6 @@ Asynchronous chunk streaming operates as **enabled by default** within this bund
 Additionally, NPU, ROCm, and XPU per-platform configuration deltas are deterministically merged from the
 `platforms`: section of the corresponding YAML.

-**Note:** The OpenAI-style **`/v1/realtime`** WebSocket interface (facilitating streaming PCM audio input alongside audio and transcription output)
-is currently **unsupported** while the `async_chunk` configuration attribute is enabled.
-It is requisite to instantiate the default omni architecture or utilize a deployment configuration specifying `async_chunk: false` to facilitate real-time streaming sessions.
-
 To explicitly utilize a custom deployment YAML, mandate the configuration path accordingly:
 ```bash
 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
@@ -289,7 +285,7 @@ python openai_realtime_client.py \
 | `--num-requests` | `1` | Number of sequential sessions (see `--concurrency`) |
 | `--concurrency` | `1` | Max concurrent WebSocket sessions when `--num-requests` > 1 |

-Ensure the server is running **without** `async_chunk` if you use `/v1/realtime`, for example:
+Ensure the server is running, for example:

 ```bash
 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
@@ -6,11 +6,21 @@
 vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091
 ```

+With Cache-DiT acceleration:
+
+```bash
+vllm serve SenseNova/SenseNova-U1-8B-MoT --omni --port 8091 \
+    --cache-backend cache_dit
+```
+
 Or use the convenience script:

 ```bash
 cd examples/online_serving/sensenova_u1
 bash run_server.sh
+
+# Cache-DiT
+CACHE_BACKEND=cache_dit bash run_server.sh
 ```

 ### Tensor Parallelism (TP)
@@ -3,10 +3,18 @@

 MODEL="${MODEL:-SenseNova/SenseNova-U1-8B-MoT}"
 PORT="${PORT:-8091}"
+CACHE_BACKEND="${CACHE_BACKEND:-none}"

 echo "Starting SenseNova-U1 server..."
 echo "Model: $MODEL"
 echo "Port: $PORT"
+echo "Cache backend: $CACHE_BACKEND"
+
+EXTRA_ARGS=()
+if [ "$CACHE_BACKEND" != "none" ]; then
+    EXTRA_ARGS+=(--cache-backend "$CACHE_BACKEND")
+fi

 vllm serve "$MODEL" --omni \
-    --port "$PORT"
+    --port "$PORT" \
+    "${EXTRA_ARGS[@]}"
@@ -18,7 +18,7 @@ For the full list of supported architectures across all modalities, see
 | GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
 | Ming-flash-omni-TTS | `Jonathan1909/Ming-flash-omni-2.0` | — (caption-controlled) | — | caption fields (`instructions`) | — |
 | MOSS-TTS-Nano | `OpenMOSS-Team/MOSS-TTS-Nano` | ✓ (`ref_audio` required) | ✓ (PCM stream) | — | ✓ |
-| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
+| OmniVoice | `k2-fsa/OmniVoice` | ✓ | — | — | — |
 | Qwen3-TTS | `Qwen/Qwen3-TTS-12Hz-1.7B-{CustomVoice,VoiceDesign,Base}` | ✓ (Base) | ✓ (PCM + WebSocket) | ✓ (presets + `/v1/audio/voices` upload) | ✓ (standard + FastRTC) |
 | VoxCPM2 | `openbmb/VoxCPM2` | ✓ | ✓ (AudioWorklet via gradio) | — | ✓ |
 | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✓ (gated upstream) | ✓ | ✓ (presets) | ✓ |
@@ -143,6 +143,26 @@ bash examples/online_serving/text_to_speech/glm_tts/run_gradio_demo.sh
 pip install fish-speech
 ```

+### Kvcache attention fast path
+
+Fish Speech S2 Pro uses a Triton decode-only kvcache attention fast path by
+default on CUDA builds. Set `VLLM_OMNI_FISH_KVCACHE_ATTN=0` to disable it, or
+`VLLM_OMNI_FISH_KVCACHE_ATTN=required` to fail fast if the fast path cannot be
+installed.
+
+```bash
+# Verify fast path availability.
+python - <<'PY'
+from vllm_omni.attention import fish_kvcache_attn
+
+print(fish_kvcache_attn.is_available())
+print(fish_kvcache_attn.load_error())
+PY
+
+# Optional: disable the runtime fast path.
+export VLLM_OMNI_FISH_KVCACHE_ATTN=0
+```
+
 ### Launch
 ```bash
 vllm serve fishaudio/s2-pro --omni --port 8091
@@ -301,16 +321,29 @@ vllm serve k2-fsa/OmniVoice --omni --port 8091 --trust-remote-code
 ### CLI client
 ```bash
 cd examples/online_serving/text_to_speech/omnivoice
+# Text-only (auto voice)
 python speech_client.py --text "Hello, how are you?"
+
+# Language hint
 python speech_client.py --text "Bonjour, comment allez-vous?" --language French
+# Voice cloning (reference audio + optional ref_text)
+python speech_client.py \
+--text "Bonjour, comment allez-vous?" \
+--ref-audio /path/to/ref_audio.wav \
+--ref-text "Bonjour, comment allez-vous?"
+
+# Style instruction (voice design-style control)
+python speech_client.py \
+--text "Bonjour, comment allez-vous?" \
+--language French \
+--instructions "loud voice"
+
+# Deterministic output with seed parameter
+python speech_client.py --text "Hello, how are you?" --seed 42
 ```

-The client supports `--api-base`, `--model`, `--text`, `--response-format`, `--language`, `--output`.
+The client supports `--api-base`, `--model`, `--text`, `--response-format`, `--language`, `--voice`, `--ref-audio`, `--ref-text`, `--instructions`, `--seed`, and `--output`.

-### Notes
- Voice cloning and voice design require offline inference; see the [offline OmniVoice section](../../offline_inference/text_to_speech/README.md#omnivoice).
-
---

 ## Qwen3-TTS

@@ -0,0 +1,63 @@
+# higgs-audio v2 online example
+
+This directory contains the online-serving entry points for boson-ai's higgs-audio v2 as integrated by vllm-omni: a 2-stage TTS pipeline (Llama-3.2-3B talker with DualFFN audio expert + HiggsAudio codec decoder) emitting 24 kHz mono speech.
+
+## Prerequisites
+
+Voice clone uses HF's `HiggsAudioV2TokenizerModel` loaded from `k2-fsa/OmniVoice/audio_tokenizer/` (the boson-ai standalone tokenizer Hub repo's `model.safetensors` is the 3B talker LM, not the codec). Only that ~806 MB subdir is downloaded.
+
+```bash
+pip install -U "transformers>=5.3.0"
+```
+
+## Files
+
+- `run_server.sh` — launch the vllm-omni server with the bundled `vllm_omni/deploy/higgs_audio_v2.yaml` deploy config.
+- `batch_speech_client.py` — send a list of prompts to `/v1/audio/speech` and save the returned WAV / PCM bytes to a directory; optionally passes `--ref-audio` + `--ref-text` for shallow voice clone.
+
+## Launching the server
+
+```bash
+GPUS=6,7 PORT=8094 bash examples/online_serving/text_to_speech/higgs_audio_v2/run_server.sh
+```
+
+Environment overrides:
+
+- `MODEL` — HF id of the talker (default `bosonai/higgs-audio-v2-generation-3B-base`).
+- `PORT` — server port (default `8094`).
+- `GPUS` — `CUDA_VISIBLE_DEVICES` value (default `6,7`).
+- `GPU_UTIL` — `--gpu-memory-utilization` (default `0.4`).
+
+The script also exports `VLLM_USE_DEEP_GEMM=0` / `VLLM_MOE_USE_DEEP_GEMM=0` so the example works on images without the optional `deep_gemm` backend.
+
+The deploy YAML ships with `async_chunk: false` and `codec_streaming: true`, i.e. Stage 0 finishes its codec frames before Stage 1 starts decoding, and Stage 1 streams WAV/PCM bytes to the client chunk-by-chunk.
+
+## Driving the server
+
+Plain TTS:
+
+```bash
+python examples/online_serving/text_to_speech/higgs_audio_v2/batch_speech_client.py \
+    --base-url http://localhost:8094 \
+    --model bosonai/higgs-audio-v2-generation-3B-base \
+    --output-dir /tmp/higgs_audio_v2_batch \
+    --prompts "Hello world." \
+              "The quick brown fox jumps over the lazy dog."
+```
+
+Voice clone — pass a reference clip and its transcript (both required together):
+
+```bash
+python examples/online_serving/text_to_speech/higgs_audio_v2/batch_speech_client.py \
+    --base-url http://localhost:8094 \
+    --model bosonai/higgs-audio-v2-generation-3B-base \
+    --output-dir /tmp/higgs_audio_v2_clone \
+    --ref-audio /path/to/reference.wav \
+    --ref-text  "Exact transcript spoken in reference.wav." \
+    --prompts "Hello, this is a cloned voice."
+```
+
+## Notes
+
+- `--ref-text` must be the real transcript of `--ref-audio`; mismatched text degrades cloned-voice quality.
+- Out of scope (rejected with explicit 4xx by the request validator): multi-speaker `[SPEAKERn]` tags inside `input`, `profile:` text-only speaker descriptions, the `ref_audio_in_system_message` system-block variant, chunked long-form generation, and per-request `voice` / `instructions` / `task_type` / `language` / `speed != 1.0` / `x_vector_only_mode` / `speaker_embedding`.
@@ -0,0 +1,122 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Batch client for the higgs-audio v2 online server.
+
+Sends a fixed list of prompts to ``/v1/audio/speech`` and saves the returned
+WAV files (or raw PCM bytes when ``--format pcm``) into ``--output-dir``.
+
+Usage (plain text -> speech):
+
+  python examples/online_serving/text_to_speech/higgs_audio_v2/batch_speech_client.py \
+      --base-url http://localhost:8094 \
+      --output-dir /tmp/higgs_audio_v2_batch \
+      --prompts "Hello world." "The quick brown fox jumps over the lazy dog."
+
+Usage (shallow voice clone — pass a reference clip + its transcript):
+
+  python examples/online_serving/text_to_speech/higgs_audio_v2/batch_speech_client.py \
+      --base-url http://localhost:8094 \
+      --output-dir /tmp/higgs_audio_v2_clone \
+      --ref-audio path/to/reference.wav \
+      --ref-text "the transcript of the reference clip" \
+      --prompts "Hello world."
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import sys
+from pathlib import Path
+
+DEFAULT_PROMPTS = (
+    "Hello world.",
+    "The quick brown fox jumps over the lazy dog.",
+    "It was the night before my birthday.",
+    "Innovation distinguishes between a leader and a follower.",
+)
+
+
+def _slug(text: str) -> str:
+    import re
+
+    s = re.sub(r"\s+", "_", text.strip().lower())
+    return re.sub(r"[^a-z0-9_]+", "", s)[:32] or "prompt"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--base-url", default="http://localhost:8094")
+    parser.add_argument("--model", default="higgs_audio_v2")
+    parser.add_argument("--prompts", nargs="+", default=list(DEFAULT_PROMPTS))
+    parser.add_argument("--output-dir", type=Path, default=Path("/tmp/higgs_audio_v2_batch"))
+    parser.add_argument("--format", choices=("wav", "pcm"), default="wav")
+    parser.add_argument("--max-new-tokens", type=int, default=300)
+    parser.add_argument("--seed", type=int, default=42)
+    parser.add_argument("--timeout-s", type=float, default=120.0)
+    parser.add_argument(
+        "--ref-audio",
+        type=Path,
+        default=None,
+        help="Reference clip for voice clone (path to a WAV file). Must be paired with --ref-text.",
+    )
+    parser.add_argument(
+        "--ref-text",
+        type=str,
+        default=None,
+        help="Transcript of the reference clip. Required when --ref-audio is set.",
+    )
+    args = parser.parse_args()
+
+    if (args.ref_audio is None) != (args.ref_text is None):
+        print("--ref-audio and --ref-text must be supplied together", file=sys.stderr)
+        return 2
+
+    ref_audio_data_url: str | None = None
+    if args.ref_audio is not None:
+        if not args.ref_audio.exists():
+            print(f"ref-audio file not found: {args.ref_audio}", file=sys.stderr)
+            return 2
+        mime = "audio/wav" if args.ref_audio.suffix.lower() == ".wav" else "audio/mpeg"
+        ref_b64 = base64.b64encode(args.ref_audio.read_bytes()).decode("ascii")
+        ref_audio_data_url = f"data:{mime};base64,{ref_b64}"
+
+    try:
+        import httpx
+    except ImportError:
+        print(
+            "this client needs `httpx`. Install with `pip install httpx`.",
+            file=sys.stderr,
+        )
+        return 2
+
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+    url = args.base_url.rstrip("/") + "/v1/audio/speech"
+    failures = 0
+    with httpx.Client(timeout=args.timeout_s) as client:
+        for prompt in args.prompts:
+            payload = {
+                "model": args.model,
+                "input": prompt,
+                "response_format": args.format,
+                "max_new_tokens": args.max_new_tokens,
+                "seed": args.seed,
+            }
+            if ref_audio_data_url is not None:
+                payload["ref_audio"] = ref_audio_data_url
+                payload["ref_text"] = args.ref_text
+            resp = client.post(url, json=payload)
+            if resp.status_code != 200:
+                print(f"[FAIL] {prompt!r} -> {resp.status_code}: {resp.text[:200]}", file=sys.stderr)
+                failures += 1
+                continue
+            suffix = ".wav" if args.format == "wav" else ".pcm"
+            out = args.output_dir / f"{_slug(prompt)}{suffix}"
+            out.write_bytes(resp.content)
+            print(f"[ ok ] {prompt!r} -> {out} ({len(resp.content)} bytes)")
+
+    return 1 if failures else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
@@ -0,0 +1,37 @@
+#!/bin/bash
+# Launch vLLM-Omni server for higgs-audio v2.
+#
+# v1 scope: plain text -> 24 kHz speech only. Voice cloning, multi-speaker,
+# ChatML rich content, and language overrides are rejected by the validator
+# with explicit 4xx (see vllm_omni/entrypoints/openai/serving_speech.py).
+#
+# Usage:
+#   ./run_server.sh                 # default port 8094, GPUs 6 and 7
+#   PORT=8095 GPUS=6,7 ./run_server.sh
+#   MODEL=bosonai/higgs-audio-v2-generation-3B-base ./run_server.sh
+
+set -e
+
+MODEL="${MODEL:-bosonai/higgs-audio-v2-generation-3B-base}"
+PORT="${PORT:-8094}"
+GPUS="${GPUS:-6,7}"
+GPU_UTIL="${GPU_UTIL:-0.4}"
+
+echo "Starting higgs-audio v2 server"
+echo "  MODEL=$MODEL"
+echo "  PORT=$PORT"
+echo "  CUDA_VISIBLE_DEVICES=$GPUS"
+
+# DeepGEMM FP8 kernels are optional and trip warmup on builds without
+# the deep_gemm backend; disable them so the example works out of the box.
+# Users with deep_gemm installed can re-enable via the same env vars.
+CUDA_VISIBLE_DEVICES="$GPUS" \
+VLLM_USE_DEEP_GEMM=0 \
+VLLM_MOE_USE_DEEP_GEMM=0 \
+vllm-omni serve "$MODEL" \
+    --deploy-config vllm_omni/deploy/higgs_audio_v2.yaml \
+    --host 0.0.0.0 \
+    --port "$PORT" \
+    --gpu-memory-utilization "$GPU_UTIL" \
+    --trust-remote-code \
+    --omni
@@ -12,6 +12,8 @@ Examples:
 """

 import argparse
+import base64
+import os

 import httpx

@@ -19,6 +21,24 @@ DEFAULT_API_BASE = "http://localhost:8091"
 DEFAULT_API_KEY = "EMPTY"


+def encode_audio_to_base64(audio_path: str) -> str:
+    """Encode a local audio file to a base64 data URL."""
+    if not os.path.exists(audio_path):
+        raise FileNotFoundError(f"Audio file not found: {audio_path}")
+
+    ext = audio_path.lower().rsplit(".", 1)[-1]
+    mime = {
+        "wav": "audio/wav",
+        "mp3": "audio/mpeg",
+        "flac": "audio/flac",
+        "ogg": "audio/ogg",
+    }.get(ext, "audio/wav")
+
+    with open(audio_path, "rb") as f:
+        b64 = base64.b64encode(f.read()).decode("utf-8")
+    return f"data:{mime};base64,{b64}"
+
+
 def run_tts(args) -> None:
    """Generate speech via /v1/audio/speech API."""
    payload = {
@@ -26,16 +46,36 @@ def run_tts(args) -> None:
        "input": args.text,
        "response_format": args.response_format,
    }
+    if args.seed is not None:
+        payload["extra_params"] = {}
+        payload["extra_params"]["seed"] = args.seed

    if args.voice:
        payload["voice"] = args.voice
    if args.language:
        payload["language"] = args.language

+    if args.ref_audio:
+        ref = args.ref_audio
+        if ref.startswith(("http://", "https://", "data:")):
+            payload["ref_audio"] = ref
+        else:
+            payload["ref_audio"] = encode_audio_to_base64(ref)
+
+    if args.ref_text:
+        payload["ref_text"] = args.ref_text
+
+    if args.instructions:
+        payload["instructions"] = args.instructions
+
    print(f"Model: {args.model}")
    print(f"Text: {args.text}")
+    if args.seed:
+        print(f"Seed: {args.seed}")
+
    if args.voice:
        print(f"Voice: {args.voice}")
+
    if args.language:
        print(f"Language: {args.language}")
    print("Generating audio...")
@@ -45,7 +85,6 @@ def run_tts(args) -> None:
        "Content-Type": "application/json",
        "Authorization": f"Bearer {args.api_key}",
    }
-
    with httpx.Client(timeout=300.0) as client:
        response = client.post(api_url, json=payload, headers=headers)

@@ -86,6 +125,30 @@ def main():
        choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
        help="Audio format (default: wav)",
    )
+    parser.add_argument(
+        "--ref-audio",
+        type=str,
+        default=None,
+        help="Reference audio for voice cloning (local path, URL, or data: URI)",
+    )
+    parser.add_argument(
+        "--ref-text",
+        type=str,
+        default=None,
+        help="Reference text for voice cloning",
+    )
+    parser.add_argument(
+        "--instructions",
+        type=str,
+        default=None,
+        help="Voice style/emotion instructions",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        default=None,
+        help="Random seed for generation, default: None for stochastic output)",
+    )
    parser.add_argument("--output", "-o", default=None, help="Output file path")
    args = parser.parse_args()
    run_tts(args)
@@ -38,13 +38,12 @@ theme:
  features:
    - content.action.edit
    - content.code.copy
-    - navigation.instant
-    - navigation.instant.progress
    - navigation.tracking
    - navigation.tabs
    - navigation.tabs.sticky
    - navigation.sections
    - navigation.indexes
+    - navigation.prune
    - navigation.top
    - search.suggest
    - search.highlight
@@ -56,6 +55,7 @@ theme:
  custom_dir: docs/mkdocs/overrides

 hooks:
+  - docs/mkdocs/hooks/search_index.py
  - docs/mkdocs/hooks/generate_api_readme.py
  - docs/mkdocs/hooks/url_schemes.py
  - docs/mkdocs/hooks/generate_examples.py
@@ -117,6 +117,7 @@ plugins:
            separate_signature: true
            show_overloads: true
            signature_crossrefs: true
+            show_source: false
          inventories:
          - https://docs.python.org/3/objects.inv
          - https://typing-extensions.readthedocs.io/en/latest/objects.inv
@@ -167,9 +168,8 @@ extra_css:
  - mkdocs/stylesheets/extra.css

 extra_javascript:
+  - mkdocs/javascript/memory.js
  - mkdocs/javascript/mathjax.js
-  - https://unpkg.com/mathjax@3.2.2/es5/tex-mml-chtml.js
-  - https://unpkg.com/mermaid@10/dist/mermaid.min.js
  - mkdocs/javascript/mermaid.js
  - mkdocs/javascript/edit_and_feedback.js
  - mkdocs/javascript/slack_and_forum.js
@@ -249,6 +249,7 @@ extend-ignore-identifiers-re = [
    ".*[Oo]no_[Aa]nna.*",
    ".*cann.*",
    ".*[Nn][Dd]3.*",
+    "byt5.*",
 ]
 extend-ignore-re = [
    "\\.\\.\\.n,d->\\.\\.\\.nd",
@@ -261,3 +262,4 @@ Vai = "Vai"
 tockens = "tockens"
 CANN = "CANN"
 ND = "ND"
+byt5 = "byt5"
@@ -0,0 +1,112 @@
+# GLM-Image for text-to-image and image editing
+
+## Summary
+
+- Vendor: Z.ai
+- Model: `GLM/GLM-Image`
+- Task: Text-to-image (T2I) and image-to-image
+- Mode: Online serving with the OpenAI-compatible API
+- Maintainer: Community
+
+## When to use this recipe
+
+Use this recipe when you want a known-good starting point for serving
+`GLM/GLM-Image` with vLLM-Omni on **two 80 GB NVIDIA A800** GPUs (Ampere-class,
+same default layout as the upstream **2×A100 80GB** example: Stage 0 AR on GPU 0,
+Stage 1 diffusion on GPU 1) and validate the deployment with the existing
+`examples/online_serving/glm_image` clients.
+
+## References
+
+- Upstream or canonical docs:
+  [`docs/user_guide/examples/online_serving/glm_image.md`](../../docs/user_guide/examples/online_serving/glm_image.md)
+- Related issue or discussion:
+  [#2888](https://github.com/vllm-project/vllm-omni/pull/2888)
+
+## Hardware Support
+
+This recipe documents **dual-GPU** CUDA layouts on A800 80 GB
+for the same software stack. Add more platforms (for example ROCm / NPU) as
+community validation lands.
+
+## GPU
+
+### 2× A800 80GB
+
+#### Environment
+
+These versions were taken from a working **editable** install: activate `vllm-omni/.venv` (or your equivalent), then align `pip` / Git with the rows below when reproducing this recipe.
+
+- OS: Linux
+- Python: 3.12
+- Driver / runtime: NVIDIA CUDA stack with **two** A800 80 GB GPUs visible (set `CUDA_VISIBLE_DEVICES` on your host if needed)
+- vLLM: **0.19.0**
+- vLLM-Omni: **0.19.0rc2.dev138+g38d5f2d53** (editable install from this repo; Git **`38d5f2d5`**, `git describe` ≈ **`v0.19.0rc1-138-g38d5f2d5`**)
+- Transformers: **5.5.4** (same `.venv` as above; required so `glm_image` configs load for Stage 0)
+
+#### Command
+
+Start the server from the repository root:
+
+```bash
+vllm serve zai-org/GLM-Image --omni --port 8091
+```
+
+To use the bundled stage config explicitly (same default as above):
+
+```bash
+vllm serve zai-org/GLM-Image \
+  --omni \
+  --port 8091 \
+  --deploy-config vllm_omni/deploy/glm_image.yaml
+```
+
+#### Verification
+
+Run one of the existing example clients after the server is ready:
+
+```bash
+curl -s http://172.18.69.133:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "messages": [
+      {"role": "user", "content": "A beautiful landscape painting"}
+    ],
+    "extra_body": {
+      "height": 1920,
+      "width": 1920,
+      "num_inference_steps": 50,
+      "true_cfg_scale": 1.5,
+      "seed": 42
+    }
+  }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > land.png
+```
+After the command finishes, check for the output files:
+
+```bash
+ls output.png
+```
+
+#### Sample end-to-end metrics
+
+One representative **offline** GLM-Image E2E run on this recipe’s **2× A800 80GB**.
+Overall summary from the run’s metrics. Rough wall-time split: **Stage 0 (AR)** ~**25 s**,
+**Stage 1 (diffusion)** ~**34 s** (see `e2e_stage_*_wall_time_ms` below).
+
+| Field | Value |
+| --- | ---: |
+| e2e_requests | 1 |
+| e2e_wall_time_ms | 61,148.679 |
+| e2e_total_tokens | 1,300 |
+| e2e_avg_time_per_request_ms | 61,148.679 |
+| e2e_avg_tokens_per_s | 21.260 |
+| e2e_stage_0_wall_time_ms | 24,708.760 |
+| e2e_stage_1_wall_time_ms | 33,787.442 |
+
+#### Notes
+
+- Memory usage: Roughly **~38 GiB + KV** on Stage 0 (AR) and **~20 GiB** on Stage 1 (DiT+VAE) per the user guide; two 80 GB cards match the default split.
+- Key flags: `--omni` is required; `--stage-configs-path` is optional unless you use a custom YAML (for example single-GPU).
+- Keep **Transformers ≥ 5.5.1** (this recipe used **5.5.4**) so `glm_image` configs resolve; otherwise Stage 0 can fail at `ModelConfig` validation.
+- Known limitations: This starter recipe follows the dual-GPU online path documented under `examples/online_serving/glm_image`. The first request may be slower due to warmup.
+- Generation time: about **61 s** wall time end-to-end for the sample above (50 inference steps, 1024×1024).
@@ -0,0 +1,109 @@
+# Helios for video generation
+
+## Summary
+
+- Vendor: Helios
+- Model: `BestWishYsh/Helios-Base`, `BestWishYsh/Helios-Mid`, `BestWishYsh/Helios-Distilled`
+- Task: Text-to-video, image-to-video, and video-to-video generation
+- Mode: Offline inference with the bundled Helios example
+- Maintainer: Community
+
+## When to use this recipe
+
+Use this recipe when you want a known-good starting point for running Helios
+video generation with vLLM-Omni. The concrete command below focuses on
+`BestWishYsh/Helios-Base` text-to-video generation on one NVIDIA H20 GPU. The
+same example script also supports Helios-Mid, Helios-Distilled, image-to-video,
+and video-to-video; see the linked example docs for those model-specific flags.
+
+## References
+
+- Upstream repository: <https://github.com/PKU-YuanGroup/Helios>
+- Model weights:
+  <https://huggingface.co/BestWishYsh/Helios-Base>
+- Upstream or canonical docs:
+  [`docs/user_guide/examples/offline_inference/helios.md`](../../docs/user_guide/examples/offline_inference/helios.md)
+- Related example under `examples/`:
+  [`examples/offline_inference/helios/README.md`](../../examples/offline_inference/helios/README.md)
+
+## Hardware Support
+
+## GPU
+
+### 1x NVIDIA H20
+
+#### Environment
+
+- OS: Ubuntu Linux x86_64
+- Python: 3.12.12
+- Driver / runtime: NVIDIA driver `580.126.20`, CUDA `13.0`
+- Hardware: 1x NVIDIA H20 GPU from an 8x H20 host
+- vLLM version: `0.19.0`
+- vLLM-Omni version or commit: `a3903810`
+
+#### Command
+
+Run the baseline Helios-Base text-to-video example from the repository root:
+
+```bash
+cd examples/offline_inference/helios
+
+python end2end.py \
+  --model BestWishYsh/Helios-Base \
+  --sample-type t2v \
+  --prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
+  --guidance-scale 5.0 \
+  --output helios_t2v_base.mp4
+```
+
+To use cache-dit acceleration, run on a vLLM-Omni checkout that includes Helios cache-dit support and enable the cache backend:
+
+```bash
+cd examples/offline_inference/helios
+
+python end2end.py \
+  --cache-backend cache_dit \
+  --enable-cache-dit-summary \
+  --model BestWishYsh/Helios-Base \
+  --sample-type t2v \
+  --prompt "A dynamic time-lapse video showing the rapidly moving scenery from the window of a speeding train." \
+  --guidance-scale 5.0 \
+  --output helios_t2v_base.mp4
+```
+
+#### Verification
+
+The script should print the resolved generation configuration, total generation
+time, and output path. A successful run ends with output similar to:
+
+```text
+Total generation time: <seconds> seconds (<milliseconds> ms)
+Saved generated video to helios_t2v_base.mp4
+```
+
+#### Important flags and stage configs
+
+- `--model BestWishYsh/Helios-Base` selects the base Helios checkpoint used by
+  this recipe.
+- `--sample-type t2v` selects text-to-video generation. Use `i2v` with
+  `--image-path` for image-to-video, or `v2v` with `--video-path` for
+  video-to-video.
+- `--guidance-scale 5.0` matches the Helios-Base recommendation. Use
+  `--guidance-scale 1.0` for Helios-Distilled.
+- `--cache-backend cache_dit` enables the cache-dit acceleration path.
+- `--enable-cache-dit-summary` prints cache-dit summary information after
+  diffusion forward passes.
+- No separate stage config is required for this offline recipe; the bundled
+  Helios example configures the pipeline through `end2end.py` arguments.
+- For Helios-Mid, add `--is-enable-stage2`,
+  `--pyramid-num-inference-steps-list 20 20 20`, `--use-cfg-zero-star`,
+  `--use-zero-init`, and `--zero-steps 1`.
+- For Helios-Distilled, add `--is-enable-stage2`,
+  `--pyramid-num-inference-steps-list 2 2 2`, and
+  `--is-amplify-first-chunk`.
+
+#### Known limitations
+
+- Helios generates video in 33-frame chunks. For best performance, set
+  `--num-frames` to a multiple of `33`; non-multiple values are rounded up to
+  the nearest multiple of `33`.
@@ -86,7 +86,6 @@ curl -X POST http://localhost:8000/v1/videos \
 - Duration: Controlled by `num_frames` (frames = duration_seconds * 24 + 1)
 - Known limitations:
  - No image-to-video support yet (LTX23ImageToVideoPipeline is a placeholder)
-  - No CFG-parallel support (single-GPU only)
  - Requires `diffusers >= 0.38.0` (not yet on PyPI, install from git)

 ## Hardware Support
@@ -30,9 +30,11 @@ recipes/
 | [`audiox/AudioX.md`](./audiox/AudioX.md) | Offline + online unified text/video→audio diffusion | 1x L4 24GB |
 | [`Baidu/ERNIE-Image.md`](./Baidu/ERNIE-Image.md) | Text-to-image online serving (ERNIE-Image 8B) | 1x or 2x RTX 4090 24GB |
 | [`fishaudio/Fish-Speech-S2-Pro.md`](./fishaudio/Fish-Speech-S2-Pro.md) | Online serving for TTS | 1x A800 80GB |
+| [`Helios/Helios.md`](./Helios/Helios.md) | Text-to-video, image-to-video, and video-to-video generation | 1x NVIDIA H20 |
 | [`inclusionAI/Ming-flash-omni-2.0.md`](./inclusionAI/Ming-flash-omni-2.0.md) | Online serving for multimodal chat + standalone TTS | 4x H100 / 1x H100 80GB |
 | [`LTX/LTX-2.md`](./LTX/LTX-2.md) | Text-to-video and image-to-video serving | 1x H200 141GB |
 | [`LTX/LTX-2.3.md`](./LTX/LTX-2.3.md) | Text-to-video with audio generation (22B) | 1x GPU (96GB VRAM) |
+| [`mistralai/Voxtral-TTS.md`](./mistralai/Voxtral-TTS.md) | Online serving for TTS | 1x RTX 4090 24GB |
 | [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay and ModelOpt mixed FP8/NVFP4 | 1x A100 80GB / 2x B200 |
 | [`Qwen/Qwen-Image.md`](./Qwen/Qwen-Image.md) | Text-to-image serving with step-wise continuous batching replay | 1x A100 80GB |
 | [`Qwen/Qwen-Image-Edit.md`](./Qwen/Qwen-Image-Edit.md) | Text-guided single-image editing | 1x or 2x H200 141GB |
@@ -44,6 +46,7 @@ recipes/
 | [`Wan-AI/Wan2.2-I2V.md`](./Wan-AI/Wan2.2-I2V.md) | Image-to-video serving (Wan2.2 14B) | 8x Ascend NPU (A2/A3) |
 | [`Wan-AI/Wan2.2-S2V.md`](./Wan-AI/Wan2.2-S2V.md) | Speech-to-video serving (Wan2.2 14B) | 2x A100/H100 80GB |
 | [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB |
+| [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB |

 Within a single recipe file, include different hardware support sections such
 as `GPU`, `ROCm`, and `NPU`, and add concrete tested configurations like
@@ -50,6 +50,26 @@ fi
 pip install fish-speech
 ```

+#### Kvcache attention fast path
+
+Fish Speech S2 Pro uses a Triton decode-only kvcache attention fast path by
+default on CUDA builds. Set `VLLM_OMNI_FISH_KVCACHE_ATTN=0` to disable it, or
+`VLLM_OMNI_FISH_KVCACHE_ATTN=required` to fail fast if the fast path cannot be
+installed.
+
+```bash
+# Verify fast path availability.
+python - <<'PY'
+from vllm_omni.attention import fish_kvcache_attn
+
+print(fish_kvcache_attn.is_available())
+print(fish_kvcache_attn.load_error())
+PY
+
+# Optional: disable the runtime fast path.
+export VLLM_OMNI_FISH_KVCACHE_ATTN=0
+```
+
 #### Command

 ```bash
@@ -0,0 +1,107 @@
+# Voxtral TTS for text-to-speech
+
+## Summary
+
+- Vendor: Mistral AI
+- Model: `mistralai/Voxtral-4B-TTS-2603`
+- Task: Text-to-speech with model-provided voice presets or voice cloning
+- Mode: Online serving with the OpenAI-compatible `/v1/audio/speech` API
+- Maintainer: Community
+
+## When to use this recipe
+
+Use this recipe when you want a known-good starting point for serving Voxtral TTS
+models with vLLM-Omni and validate the deployment with the existing TTS client
+examples in this repository.
+
+## References
+
+- Upstream model card: <https://huggingface.co/mistralai/Voxtral-4B-TTS-2603>
+- Related examples under `examples/`:
+  [`examples/offline_inference/text_to_speech/voxtral_tts/end2end.py`](../../examples/offline_inference/text_to_speech/voxtral_tts/end2end.py)
+  [`examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py`](../../examples/online_serving/text_to_speech/voxtral_tts/gradio_demo.py)
+- Related issue or discussion:
+  [RFC: add recipes folder](https://github.com/vllm-project/vllm-omni/issues/2645)
+
+## Hardware Support
+
+## GPU
+
+### 1x RTX 4090 24GB
+
+#### Environment
+
+- OS: Ubuntu 22.04.5 LTS
+- Python: 3.12.11
+- PyTorch: 2.11.0+cu130
+- Driver / runtime: CUDA 13.0
+- GPU: NVIDIA GeForce RTX 4090, 24 GB
+- vLLM version: 0.20.0
+- vLLM-Omni version or commit: 0.20.0rc2.dev99+g857356d5b
+- Required package: mistral_common >= 1.10.0
+
+#### Command
+
+Online:
+
+```bash
+vllm serve mistralai/Voxtral-4B-TTS-2603 \
+    --omni \
+    --port 8091
+```
+
+The deploy config is auto-loaded from `vllm_omni/deploy/voxtral_tts.yaml`
+
+#### Verification
+
+**Quick smoke test with curl:**
+
+```bash
+curl -X POST http://127.0.0.1:8091/v1/audio/speech \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "mistralai/Voxtral-4B-TTS-2603",
+    "input": "Hello, this is Voxtral TTS running with vLLM-Omni.",
+    "voice": "casual_female",
+    "language": "English",
+    "response_format": "wav"
+  }' \
+  --output voxtral.wav
+```
+
+**Streaming audio:**
+
+```bash
+curl -X POST http://127.0.0.1:8091/v1/audio/speech \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "mistralai/Voxtral-4B-TTS-2603",
+    "input": "Hello, this is Voxtral TTS streaming PCM.",
+    "voice": "casual_female",
+    "language": "English",
+    "stream": true,
+    "response_format": "pcm"
+  }' \
+  --output voxtral_stream.pcm
+```
+Convert the raw PCM output to WAV
+```bash
+ffmpeg -f s16le -ar 24000 -ac 1 -i voxtral_stream.pcm voxtral_stream.wav -y
+ffprobe -hide_banner voxtral_stream.wav
+```
+
+#### Notes
+
+- Memory usage: Stage 0 (`audio_generation`) ~18.95 GiB, Stage 1
+  (`audio_tokenizer`) ~1.55 GiB. The observed server-startup peak was
+  `21006 MiB / 24564 MiB` (`20.51 GiB / 23.99 GiB`, about 85.5%).
+  Both stages share GPU 0 via the deploy config (`gpu_memory_utilization: 0.8`
+  for Stage 0, `0.1` for Stage 1).
+- Key flags: `--omni` is required. `--port 8091` is optional and only selects
+  the HTTP port.
+- Known limitation: Voxtral TTS outputs mono audio at 24 kHz. And with the public
+mistralai/Voxtral-4B-TTS-2603 checkpoint. running the
+voice-cloning e2e test ([`Voice cloning (capability gated upstream)`](../../examples/offline_inference/text_to_speech/README.md#voice-cloning-capability-gated-upstream)
+) or adding a `ref_audio` request parameter will fail
+because the public checkpoint does not provide the encoder weights needed to
+convert reference audio into conditioning features. The expected failure is: `RuntimeError: encode_waveforms requires encoder weights which are not available in the open-source checkpoint.`
@@ -62,6 +62,12 @@ def detect_target_device() -> str:
        else:
            print(f"Warning: Invalid VLLM_OMNI_TARGET_DEVICE '{target_device}', falling back to auto-detection")

+    # Priority 1.5: ReadTheDocs builds use CPU requirements (no GPU, 1 GB RAM cap).
+    # Avoids pulling ~2 GB of CUDA libs that push the mkdocs build into swap.
+    if os.environ.get("READTHEDOCS"):
+        print("Detected READTHEDOCS=1; using cpu requirements for docs build")
+        return "cpu"
+
    # Priority 2: Torch backend detection
    # This is a code path for when user is using
    # --no-build-isolation flag
@@ -28,7 +28,7 @@ class FakeAdapter:
            req = running.pop()
            self.waiting_for_chunk_running_requests.append(req)

-    def restore_queues(self, waiting, running):
+    def restore_queues(self, waiting, running, scheduler_requests=None):
        """Put requests back."""
        self.restore_called = True
        running.extend(self.waiting_for_chunk_running_requests)
@@ -0,0 +1,92 @@
+"""Unit tests for Omni AR streaming-session async placeholder handling."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+
+# Imports must run in this order: vllm_omni applies patches to vllm.v1.request before
+# Request / StreamingUpdate are bound in this module. Ruff isort would reorder them.
+# isort: off
+import vllm_omni  # noqa: F401 - import for side effects (patch vLLM)
+from vllm.sampling_params import SamplingParams
+from vllm.v1.request import Request, RequestStatus, StreamingUpdate
+from vllm_omni.core.sched.omni_ar_scheduler import OmniARScheduler
+
+# isort: on
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+def _make_scheduler(*, stage_id: int = 0) -> OmniARScheduler:
+    sched = OmniARScheduler.__new__(OmniARScheduler)
+    sched._new_prompt_len_snapshot = {}
+    sched.vllm_config = SimpleNamespace(model_config=SimpleNamespace(stage_id=stage_id))
+    sched.num_waiting_for_streaming_input = 0
+    sched.log_stats = False
+    sched.chunk_transfer_adapter = None
+    return sched
+
+
+def _make_request() -> Request:
+    return Request(
+        request_id="req-ar-streaming-test",
+        prompt_token_ids=[1, 2, 3],
+        sampling_params=SamplingParams(max_tokens=8),
+        pooling_params=None,
+        arrival_time=100.0,
+        block_hasher=None,
+    )
+
+
+def _make_update(prompt_token_ids: list[int] | None = None) -> StreamingUpdate:
+    return StreamingUpdate(
+        mm_features=None,
+        prompt_token_ids=[10, 20] if prompt_token_ids is None else prompt_token_ids,
+        max_tokens=32,
+        arrival_time=200.0,
+        sampling_params=SamplingParams(max_tokens=16),
+    )
+
+
+def test_stage0_streaming_update_discards_outstanding_async_placeholder_token() -> None:
+    sched = _make_scheduler(stage_id=0)
+    session = _make_request()
+    session.status = RequestStatus.WAITING_FOR_STREAMING_REQ
+    session.append_output_token_ids([7, 8, 9])
+    session.num_computed_tokens = 6
+    session.num_output_placeholders = 1
+    session.spec_token_ids = [-1]
+
+    sched._update_request_as_session(session, _make_update([10, 20]))
+
+    assert session.discard_latest_async_tokens is True
+    assert session.num_output_placeholders == 0
+    assert session.spec_token_ids == []
+    # The async placeholder makes token 9 unconfirmed, so only 7 and 8 are
+    # carried into the next streaming prompt before the new chunk tokens.
+    assert session.prompt_token_ids == [1, 2, 3, 7, 8, 10, 20]
+    assert list(session._all_token_ids) == [1, 2, 3, 7, 8, 10, 20]
+    assert session._output_token_ids == []
+    assert session.num_prompt_tokens == 7
+    assert sched._new_prompt_len_snapshot[session.request_id] == 2
+
+
+def test_stage0_streaming_update_keeps_all_computed_tokens_without_placeholder() -> None:
+    sched = _make_scheduler(stage_id=0)
+    session = _make_request()
+    session.status = RequestStatus.WAITING_FOR_STREAMING_REQ
+    session.append_output_token_ids([7, 8, 9])
+    session.num_computed_tokens = 6
+    session.num_output_placeholders = 0
+
+    sched._update_request_as_session(session, _make_update([10, 20]))
+
+    assert session.discard_latest_async_tokens is False
+    assert session.num_output_placeholders == 0
+    assert session.prompt_token_ids == [1, 2, 3, 7, 8, 9, 10, 20]
+    assert list(session._all_token_ids) == [1, 2, 3, 7, 8, 9, 10, 20]
+    assert session._output_token_ids == []
+    assert session.num_prompt_tokens == 8
+    assert sched._new_prompt_len_snapshot[session.request_id] == 2
@@ -1,6 +1,6 @@
-"""Unit tests for OmniSchedulerMixin streaming session replacement.
+"""Unit tests for generation streaming session replacement.

-These tests pin the behavior of `_replace_session_with_streaming_update` against
+These tests pin the behavior of `_update_request_as_session` against
 current vLLM `Request` / `StreamingUpdate` (and Omni patches). When upgrading
 vLLM, failures here should highlight incompatible changes to request state or
 update payloads early.
@@ -19,19 +19,34 @@ import vllm_omni  # noqa: F401 - import for side effects (patch vLLM)
 from vllm.sampling_params import SamplingParams
 from vllm.v1.engine import EngineCoreEventType
 from vllm.v1.request import Request, RequestStatus, StreamingUpdate
-from vllm_omni.core.sched.omni_scheduler_mixin import OmniSchedulerMixin
+from vllm_omni.core.sched.omni_generation_scheduler import OmniGenerationScheduler

 # isort: on

 pytestmark = [pytest.mark.core_model, pytest.mark.cpu]


-class _SchedulerStub(OmniSchedulerMixin):
-    """Minimal scheduler surface required by OmniSchedulerMixin."""
+class _SkippedWaitingStub:
+    def __contains__(self, request: Request) -> bool:
+        return False
+
+
+class _ChunkTransferAdapterStub:
+    def __init__(self) -> None:
+        self.segment_finished_requests: set[str] = set()
+
+
+class _SchedulerStub(OmniGenerationScheduler):
+    """Minimal scheduler surface required by OmniGenerationScheduler."""

    def __init__(self, *, log_stats: bool = False) -> None:
        self.num_waiting_for_streaming_input = 0
        self.log_stats = log_stats
+        self.chunk_transfer_adapter = _ChunkTransferAdapterStub()
+        self.skipped_waiting = _SkippedWaitingStub()
+
+    def _enqueue_waiting_request(self, session: Request) -> None:
+        raise AssertionError("unexpected enqueue for skipped_waiting miss")


 def _make_request(**kwargs) -> Request:
@@ -71,7 +86,7 @@ class TestReplaceSessionWithStreamingUpdate:

        update = _make_update(prompt_token_ids=[40, 41, 42])
        sched.num_waiting_for_streaming_input = 3
-        sched._replace_session_with_streaming_update(session, update)
+        sched._update_request_as_session(session, update)

        assert session._output_token_ids == []
        assert list(session._all_token_ids) == [40, 41, 42]
@@ -88,7 +103,7 @@ class TestReplaceSessionWithStreamingUpdate:
        session = _make_request()
        session.status = RequestStatus.RUNNING
        update = _make_update(prompt_token_ids=None)
-        sched._replace_session_with_streaming_update(session, update)
+        sched._update_request_as_session(session, update)

        assert session.prompt_token_ids == ()
        assert list(session._all_token_ids) == []
@@ -108,7 +123,7 @@ class TestReplaceSessionWithStreamingUpdate:
            pytest.skip("StreamingUpdate has no additional_information (Omni patch inactive?)")
        update = replace(base, additional_information=None)

-        sched._replace_session_with_streaming_update(session, update)
+        sched._update_request_as_session(session, update)
        assert session.additional_information is None

    def test_does_not_decrement_waiting_when_not_streaming_status(self) -> None:
@@ -116,14 +131,14 @@ class TestReplaceSessionWithStreamingUpdate:
        session = _make_request()
        session.status = RequestStatus.RUNNING
        sched.num_waiting_for_streaming_input = 5
-        sched._replace_session_with_streaming_update(session, _make_update())
+        sched._update_request_as_session(session, _make_update())
        assert sched.num_waiting_for_streaming_input == 5

    def test_records_queued_event_when_log_stats_enabled(self) -> None:
        sched = _SchedulerStub(log_stats=True)
        session = _make_request()
        session.status = RequestStatus.WAITING_FOR_STREAMING_REQ
-        sched._replace_session_with_streaming_update(session, _make_update())
+        sched._update_request_as_session(session, _make_update())

        assert session.events
        assert session.events[-1].type == EngineCoreEventType.QUEUED
@@ -0,0 +1,104 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Unit coverage for _process_pending_input_timeouts.
+
+Verifies that the mixin correctly *delegates* timed-out requests to the
+base scheduler's ``finish_requests`` API with ``RequestStatus.FINISHED_ERROR``.
+The end-to-end effect (queue removal + status set + per-request cleanup +
+client-facing FINISHED_ERROR emission) is the responsibility of upstream
+vLLM's ``finish_requests`` implementation and is covered by upstream tests;
+this file only asserts the wiring from the mixin to that API.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import pytest
+
+from vllm_omni.core.sched.omni_scheduler_mixin import OmniSchedulerMixin
+
+
+class _FakeCoordinator:
+    def __init__(self, timed_out_ids):
+        self._timed_out_ids = set(timed_out_ids)
+        self.calls = []
+
+    def collect_timed_out_request_ids(self, timeout_s):
+        self.calls.append(timeout_s)
+        return set(self._timed_out_ids)
+
+
+class _FakeScheduler(OmniSchedulerMixin):
+    def __init__(self, requests, coordinator):
+        self.requests = requests
+        self.input_coordinator = coordinator
+        self.finish_calls = []
+
+    def finish_requests(self, req_ids, status):
+        self.finish_calls.append((set(req_ids), status))
+
+
+def test_process_pending_input_timeouts_delegates_to_finish_requests():
+    """Timed-out request present in self.requests is forwarded to finish_requests."""
+    req_id = "stuck-req"
+    requests = {req_id: SimpleNamespace(request_id=req_id)}
+    coord = _FakeCoordinator(timed_out_ids={req_id})
+    scheduler = _FakeScheduler(requests, coord)
+
+    scheduler._process_pending_input_timeouts()
+
+    assert len(coord.calls) == 1, "coordinator should be polled once"
+    assert coord.calls[0] > 0, "timeout must be positive when enabled"
+
+    assert len(scheduler.finish_calls) == 1
+    finished_ids, status = scheduler.finish_calls[0]
+    assert finished_ids == {req_id}
+    # RequestStatus is the upstream enum; the mixin imports it as
+    # RequestStatus.FINISHED_ERROR.  Check by name to avoid hard import here.
+    assert getattr(status, "name", str(status)).endswith("FINISHED_ERROR")
+
+
+def test_process_pending_input_timeouts_skips_already_freed_request():
+    """Timed-out id no longer in self.requests must not be forwarded."""
+    coord = _FakeCoordinator(timed_out_ids={"already-freed"})
+    scheduler = _FakeScheduler(requests={}, coordinator=coord)
+
+    scheduler._process_pending_input_timeouts()
+
+    assert coord.calls == [coord.calls[0]] and coord.calls[0] > 0
+    assert scheduler.finish_calls == []
+
+
+def test_process_pending_input_timeouts_noop_without_coordinator():
+    """No coordinator => no finish_requests call, no crash."""
+
+    class _NoCoord(OmniSchedulerMixin):
+        def __init__(self):
+            self.requests = {}
+            self.input_coordinator = None
+            self.finish_calls = []
+
+        def finish_requests(self, req_ids, status):
+            self.finish_calls.append((set(req_ids), status))
+
+    scheduler = _NoCoord()
+    scheduler._process_pending_input_timeouts()
+    assert scheduler.finish_calls == []
+
+
+def test_process_pending_input_timeouts_disabled_when_timeout_zero(monkeypatch):
+    """Setting DEFAULT_INPUT_WAIT_TIMEOUT_S <= 0 disables the safety net."""
+    from vllm_omni.core.sched import omni_scheduler_mixin
+
+    monkeypatch.setattr(omni_scheduler_mixin, "DEFAULT_INPUT_WAIT_TIMEOUT_S", 0.0)
+
+    coord = _FakeCoordinator(timed_out_ids={"r1"})
+    scheduler = _FakeScheduler(requests={"r1": SimpleNamespace(request_id="r1")}, coordinator=coord)
+    scheduler._process_pending_input_timeouts()
+    assert coord.calls == [], "coordinator must not be polled when timeout is disabled"
+    assert scheduler.finish_calls == []
+
+
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
@@ -16,7 +16,7 @@ import torch
 import vllm_omni.core.sched.omni_scheduling_coordinator as coord_mod
 from vllm_omni.core.sched.omni_scheduling_coordinator import (
    OmniSchedulingCoordinator,
-    uses_qwen3_omni_full_payload_input_coordinator,
+    uses_full_payload_input_coordinator,
 )

 # ------------------------------------------------------------------ #
@@ -92,47 +92,96 @@ class MockQueue:


 class TestFullPayloadCoordinatorSelection(unittest.TestCase):
-    def test_qwen3_omni_talker_and_code2wav_use_full_payload_input_coordinator(self):
-        for model_stage in ("talker", "code2wav"):
+    """Tests for the (model_arch, model_stage) whitelist gate.
+
+    The init_omni_connectors arch allowlist is keyed by ``model_arch`` and
+    is a superset of the stages registered here -- consumer-wait stages
+    must be registered explicitly in ``_FULL_PAYLOAD_INPUT_STAGES``, while
+    the init allowlist covers both producer- and consumer-side runners.
+    These tests pin which ``(arch, stage)`` pairs the gate fires for today.
+    """
+
+    # Expected whitelist (model_arch, model_stage).  Hardcoded to avoid the
+    # tautology of importing _FULL_PAYLOAD_INPUT_STAGES and asserting it
+    # against itself; any drift between this matrix and the whitelist will
+    # fail loudly here.
+    EXPECTED_FULL_PAYLOAD_INPUT_STAGES: frozenset[tuple[str, str]] = frozenset(
+        {
+            ("Qwen3OmniMoeForConditionalGeneration", "talker"),
+            ("Qwen3OmniMoeForConditionalGeneration", "code2wav"),
+            ("Qwen2_5OmniForConditionalGeneration", "talker"),
+            ("Qwen2_5OmniForConditionalGeneration", "code2wav"),
+            ("CovoAudioForConditionalGeneration", "code2wav"),
+            ("MiMoAudioModel", "code2wav"),
+            ("Qwen3TTSCode2Wav", "code2wav"),
+            ("CosyVoice3Model", "cosyvoice3_code2wav"),
+            ("DyninOmniForConditionalGeneration", "token2image"),
+            ("DyninOmniForConditionalGeneration", "token2audio"),
+        }
+    )
+
+    def test_whitelist_matches_expected_matrix(self):
+        """_FULL_PAYLOAD_INPUT_STAGES must equal the hardcoded expected matrix.
+
+        Catches both accidental additions (which would silently enable the
+        consumer-wait gate for a new arch) and accidental removals (which
+        would silently disable an enabled arch).
+        """
+        from vllm_omni.core.sched.omni_scheduling_coordinator import _FULL_PAYLOAD_INPUT_STAGES
+
+        self.assertEqual(
+            frozenset(_FULL_PAYLOAD_INPUT_STAGES),
+            self.EXPECTED_FULL_PAYLOAD_INPUT_STAGES,
+            msg="_FULL_PAYLOAD_INPUT_STAGES drifted from the expected matrix; "
+            "update EXPECTED_FULL_PAYLOAD_INPUT_STAGES if intentional.",
+        )
+
+    def test_all_whitelisted_arch_stage_pairs_fire_gate(self):
+        """Every (arch, stage) pair in the expected matrix must fire
+        the gate when stage_id > 0 and async_chunk=False.
+        """
+        for arch, stage in self.EXPECTED_FULL_PAYLOAD_INPUT_STAGES:
            model_config = SimpleNamespace(
                stage_id=1,
                async_chunk=False,
-                model_arch="Qwen3OmniMoeForConditionalGeneration",
-                model_stage=model_stage,
+                model_arch=arch,
+                model_stage=stage,
+            )
+            self.assertTrue(
+                uses_full_payload_input_coordinator(model_config),
+                msg=f"expected gate to fire for {arch}/{stage}",
            )

-            self.assertTrue(uses_qwen3_omni_full_payload_input_coordinator(model_config))
-
-    def test_async_chunk_and_non_qwen3_omni_do_not_use_full_payload_input_coordinator(self):
+    def test_other_arch_or_stage_or_mode_does_not_fire(self):
        cases = [
            SimpleNamespace(
-                stage_id=1,
-                async_chunk=True,
-                model_arch="Qwen3OmniMoeForConditionalGeneration",
-                model_stage="talker",
+                stage_id=1, async_chunk=True, model_arch="Qwen3OmniMoeForConditionalGeneration", model_stage="talker"
+            ),
+            SimpleNamespace(
+                stage_id=0, async_chunk=False, model_arch="Qwen3OmniMoeForConditionalGeneration", model_stage="thinker"
            ),
            SimpleNamespace(
                stage_id=1,
                async_chunk=False,
-                model_arch="Qwen3TTSForConditionalGeneration",
-                model_stage="code2wav",
-            ),
-            SimpleNamespace(
-                stage_id=1,
-                async_chunk=False,
-                model_arch="Qwen2_5OmniForConditionalGeneration",
-                model_stage="talker",
-            ),
-            SimpleNamespace(
-                stage_id=0,
-                async_chunk=False,
                model_arch="Qwen3OmniMoeForConditionalGeneration",
-                model_stage="thinker",
+                model_stage="some_future_stage",
+            ),
+            SimpleNamespace(
+                stage_id=1, async_chunk=False, model_arch="Qwen3TTSForConditionalGeneration", model_stage="code2wav"
+            ),
+            SimpleNamespace(
+                stage_id=1, async_chunk=False, model_arch="MingFlashOmniForConditionalGeneration", model_stage="talker"
+            ),
+            SimpleNamespace(stage_id=1, async_chunk=False, model_arch=None, model_stage="talker"),
+            SimpleNamespace(
+                stage_id=1, async_chunk=False, model_arch="Qwen3OmniMoeForConditionalGeneration", model_stage=None
            ),
        ]
-
        for model_config in cases:
-            self.assertFalse(uses_qwen3_omni_full_payload_input_coordinator(model_config))
+            self.assertFalse(
+                uses_full_payload_input_coordinator(model_config),
+                msg=f"expected gate OFF for {model_config}",
+            )


 class TestChunkCoordinatorStateTransition(unittest.TestCase):
@@ -13,7 +13,7 @@ DEFAULT_SHAPE = torch.Size([NUM_BLOCKS, BLOCK_SIZE, HIDDEN_SIZE])


 class MockInputBatch:
-    def __init__(self, num_computed_tokens_cpu):
+    def __init__(self, num_computed_tokens_cpu, block_table=None):
        self.req_ids = ["req1", "req2"]
        self.req_id_to_index = {req_id: i for i, req_id in enumerate(self.req_ids)}
        self.num_computed_tokens_cpu = num_computed_tokens_cpu
@@ -21,11 +21,33 @@ class MockInputBatch:
        # Block table is only mocked for validation of length;
        # we don't actually need to add valid values here since
        # we patch the table when testing.
-        class _DummyBlockTable:
-            pass
+        if block_table is None:

-        self.block_table = _DummyBlockTable()
-        self.block_table.block_tables = [None]
+            class _DummyBlockTable:
+                pass
+
+            self.block_table = _DummyBlockTable()
+            self.block_table.block_tables = [None]
+        else:
+
+            class _TensorWrapper:
+                def __init__(self, tensor):
+                    self.cpu = tensor
+
+            class _BlockGroup:
+                def __init__(self, tensor):
+                    self.block_table = _TensorWrapper(tensor)
+
+            class _BlockTable:
+                def __init__(self, tensor):
+                    self._group = _BlockGroup(tensor)
+                    self.block_tables = [self._group.block_table]
+
+                def __getitem__(self, idx):
+                    assert idx == 0
+                    return self._group
+
+            self.block_table = _BlockTable(block_table)


 def get_omni_pcache_with_mm_tensors(feat_dims, seq_len) -> OmniTensorPrefixCache:
@@ -171,6 +193,94 @@ def test_update_with_multimodal_outputs(feat_dims):
            assert torch.all(slot_states == new_output)


+def test_deferred_multimodal_cache_write_commits_on_completion():
+    cache = get_omni_pcache()
+    mm_key = "codes.audio"
+    block_table = torch.tensor(
+        [
+            [2, 3, 4, 5],
+            [6, 7, 8, 9],
+        ],
+        dtype=torch.long,
+    )
+    input_batch = MockInputBatch(
+        num_computed_tokens_cpu=torch.tensor([4, 0], dtype=torch.long),
+        block_table=block_table,
+    )
+
+    first_chunk = torch.arange(8, dtype=torch.long).reshape(4, 2)
+    cache.update_omni_tensor_prefix_cache(
+        hidden_states=None,
+        multimodal_outputs={mm_key: first_chunk},
+        num_tokens_unpadded=4,
+        slot_mapping=torch.arange(8, 12),
+        skip_mm_cache_keys={mm_key},
+    )
+
+    rows = cache.mm_outputs_cache[mm_key].view(-1, 2)
+    assert torch.all(rows[8:12] == 0)
+
+    cache.stage_deferred_mm_outputs(
+        query_start_loc=torch.tensor([0, 4], dtype=torch.long),
+        input_batch=input_batch,
+        multimodal_outputs={mm_key: first_chunk},
+        num_scheduled_tokens={"req1": 4, "req2": 0},
+        deferred_mm_cache_keys={mm_key},
+    )
+
+    second_chunk = torch.arange(8, 16, dtype=torch.long).reshape(4, 2)
+    input_batch.num_computed_tokens_cpu = torch.tensor([8, 0], dtype=torch.long)
+    cache.stage_deferred_mm_outputs(
+        query_start_loc=torch.tensor([0, 4], dtype=torch.long),
+        input_batch=input_batch,
+        multimodal_outputs={mm_key: second_chunk},
+        num_scheduled_tokens={"req1": 4, "req2": 0},
+        deferred_mm_cache_keys={mm_key},
+    )
+
+    cache.commit_deferred_mm_outputs({"req1"}, input_batch)
+
+    expected = torch.cat([first_chunk, second_chunk], dim=0)
+    rows = cache.mm_outputs_cache[mm_key].view(-1, 2)
+    assert torch.equal(rows[8:16], expected)
+
+
+def test_deferred_multimodal_cache_can_be_merged_on_full_block_hit():
+    cache = get_omni_pcache()
+    mm_key = "codes.audio"
+    block_table = torch.tensor(
+        [
+            [2, 3, 4, 5],
+            [6, 7, 8, 9],
+        ],
+        dtype=torch.long,
+    )
+    input_batch = MockInputBatch(
+        num_computed_tokens_cpu=torch.tensor([8, 0], dtype=torch.long),
+        block_table=block_table,
+    )
+    cached_codes = torch.arange(16, dtype=torch.long).reshape(8, 2)
+    cache.stage_deferred_mm_outputs(
+        query_start_loc=torch.tensor([0, 8], dtype=torch.long),
+        input_batch=input_batch,
+        multimodal_outputs={mm_key: cached_codes},
+        num_scheduled_tokens={"req1": 8, "req2": 0},
+        deferred_mm_cache_keys={mm_key},
+    )
+    cache.commit_deferred_mm_outputs({"req1"}, input_batch)
+
+    new_codes = torch.tensor([[99, 100]], dtype=torch.long)
+    cache.add_prefix_cached_new_req_id("req1")
+    merged = cache.get_merged_multimodal_states(
+        query_start_loc=torch.tensor([0, 1], dtype=torch.long),
+        input_batch=input_batch,
+        multimodal_outputs={mm_key: new_codes},
+        num_scheduled_tokens={"req1": 1, "req2": 0},
+    )
+
+    assert torch.equal(merged[mm_key]["req1"], torch.cat([cached_codes, new_codes], dim=0))
+
+
 ### Tests for Merging
 def fake_get_cached_block_ids(self, req_idx, *args, **kwargs):
    """Fake block table lookup.
@@ -191,7 +191,16 @@ def test_performance_benchmark(omni_server, benchmark_params, request):
        raise ValueError("The number of prompts does not match the QPS or max_concurrency")

    args = ["--host", host, "--port", str(port)]
-    exclude_keys = {"request_rate", "baseline", "num_prompts", "max_concurrency", "task", "enabled", "eval_phase"}
+    exclude_keys = {
+        "request_rate",
+        "baseline",
+        "num_prompts",
+        "max_concurrency",
+        "task",
+        "enabled",
+        "eval_phase",
+        "trust_remote_code",
+    }

    for key, value in params.items():
        if key in exclude_keys or value is None:
@@ -207,6 +216,14 @@ def test_performance_benchmark(omni_server, benchmark_params, request):
        elif not isinstance(value, bool):
            args.extend([arg_name, str(value)])

+    for config in BENCHMARK_CONFIGS:
+        if config.get("test_name") != test_name:
+            continue
+        server_params = config.get("server_params") or {}
+        if server_params.get("trust_remote_code") or params.get("trust_remote_code"):
+            args.append("--trust-remote-code")
+        break
+
    # QPS / request-rate sweep
    for i, (qps, num_prompt) in enumerate(zip(qps_list, num_prompt_list)):
        args = args + ["--request-rate", str(qps), "--num-prompts", str(num_prompt)]
@@ -41,6 +41,72 @@ pytestmark = [pytest.mark.diffusion, pytest.mark.full_model]
 os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
 os.environ.setdefault("DIFFUSION_ATTENTION_BACKEND", "FLASH_ATTN")

+
+# ---------------------------------------------------------------------------
+# Inline field processing
+# ---------------------------------------------------------------------------
+def _process_inline_fields(obj: Any, parent_key: str = "") -> None:
+    """Recursively process '*-inline' fields into temp files."""
+
+    if isinstance(obj, list):
+        for item in obj:
+            _process_inline_fields(item, parent_key)
+        return
+
+    if not isinstance(obj, dict):
+        return
+
+    import atexit
+
+    import yaml
+
+    for key in list(obj.keys()):
+        value = obj[key]
+
+        if not key.endswith("-inline"):
+            _process_inline_fields(value, key)
+            continue
+
+        base_key = key[:-7]
+        full_key = f"{parent_key}.{key}" if parent_key else key
+
+        try:
+            if not isinstance(value, dict):
+                raise ValueError("must be a dict")
+
+            file_type = value.get("type")
+            content = value.get("content")
+
+            if file_type not in {"yaml", "jsonl"}:
+                raise ValueError(f"invalid type: {file_type}")
+
+            fd, path = tempfile.mkstemp(
+                suffix=f".{file_type}",
+                prefix=f"{base_key}_",
+            )
+
+            atexit.register(Path(path).unlink, missing_ok=True)
+
+            with os.fdopen(fd, "w", encoding="utf-8") as f:
+                if file_type == "jsonl":
+                    items = content if isinstance(content, list) else [content]
+                    f.writelines(json.dumps(x, ensure_ascii=False) + "\n" for x in items)
+                else:
+                    yaml.dump(
+                        content,
+                        f,
+                        allow_unicode=True,
+                        sort_keys=False,
+                        indent=2,
+                    )
+
+            obj[base_key] = path
+            del obj[key]
+
+        except Exception as e:
+            print(f"Warning: failed processing '{full_key}': {e}")
+
+
 # ---------------------------------------------------------------------------
 # Paths
 # ---------------------------------------------------------------------------
@@ -102,11 +168,14 @@ def _resolve_refs(configs: list[dict[str, Any]], config_dir: Path) -> list[dict[


 def load_configs(config_path: str) -> list[dict[str, Any]]:
+    """Load benchmark configs from JSON file and process inline fields."""
    try:
        abs_path = Path(config_path).resolve()
        with open(abs_path, encoding="utf-8") as f:
            configs = json.load(f)
-        return _resolve_refs(configs, abs_path.parent)
+        configs = _resolve_refs(configs, abs_path.parent)
+        _process_inline_fields(configs)
+        return configs
    except json.JSONDecodeError as e:
        raise ValueError(f"JSON parsing error: {str(e)}")
    except FileNotFoundError:
@@ -654,9 +723,47 @@ def run_benchmark(
 # ---------------------------------------------------------------------------


-def assert_result(result: dict[str, Any], params: dict[str, Any], *, assert_baseline: bool) -> None:
-    """Assert request completion; optionally assert metrics against ``baseline`` in *params*."""
-    num_prompts = params.get("num-prompts", 10)
+def _resolve_baseline_value(
+    baseline_raw: Any,
+    *,
+    sweep_index: int | None,
+    max_concurrency: Any = None,
+    request_rate: Any = None,
+) -> Any:
+    """Pick the baseline threshold for this sweep step."""
+    if baseline_raw is None:
+        return 100000
+    if isinstance(baseline_raw, dict):
+        if max_concurrency is not None:
+            for key in (max_concurrency, str(max_concurrency)):
+                if key in baseline_raw:
+                    return baseline_raw[key]
+        if request_rate is not None:
+            for key in (request_rate, str(request_rate)):
+                if key in baseline_raw:
+                    return baseline_raw[key]
+        raise KeyError(
+            f"baseline dict has no key for max_concurrency={max_concurrency!r} "
+            f"or request_rate={request_rate!r}; keys={list(baseline_raw.keys())!r}"
+        )
+    if isinstance(baseline_raw, (list, tuple)):
+        if sweep_index is None:
+            raise ValueError("sweep_index is required when baseline is a list or tuple")
+        return baseline_raw[sweep_index]
+    return baseline_raw
+
+
+def assert_result(
+    result: dict[str, Any],
+    params: dict[str, Any],
+    num_prompts: int,
+    *,
+    sweep_index: int | None = None,
+    max_concurrency: Any = None,
+    request_rate: Any = None,
+    assert_baseline: bool = True,
+) -> None:
+    """Assert that benchmark metrics satisfy the configured baselines."""
    completed = result.get("completed_requests", result.get("completed", 0))
    assert completed == num_prompts, f"Expected {num_prompts} completed requests, got {completed}"

@@ -666,9 +773,15 @@ def assert_result(result: dict[str, Any], params: dict[str, Any], *, assert_base
        print("Skipping performance assertions.")
        return

-    for metric, threshold in params.get("baseline", {}).items():
+    for metric, baseline_raw in params.get("baseline", {}).items():
        current = result.get(metric)
        assert current is not None, f"Metric '{metric}' not found in result: {list(result.keys())}"
+        threshold = _resolve_baseline_value(
+            baseline_raw,
+            sweep_index=sweep_index,
+            max_concurrency=max_concurrency,
+            request_rate=request_rate,
+        )
        if "throughput" in metric:
            assert current >= threshold, f"{metric}: {current:.4f} < baseline {threshold}"
        else:
@@ -692,6 +805,111 @@ def _resolve_benchmark_endpoint(server_cfg: dict[str, Any], params: dict[str, An
    return _default_benchmark_endpoint_for_task(cast(str, params.get("task", "t2i")))


+def _to_list(value: Any) -> list[Any]:
+    if value is None:
+        return []
+    return [value] if not isinstance(value, (list, tuple)) else list(value)
+
+
+def _build_run_params(
+    params: dict[str, Any],
+    *,
+    num_prompts: int,
+    sweep_index: int | None = None,
+    request_rate: Any | None = None,
+    max_concurrency: Any | None = None,
+) -> dict[str, Any]:
+    run_params = {
+        key: value for key, value in params.items() if key not in {"request-rate", "max-concurrency", "num-prompts"}
+    }
+    run_params["num-prompts"] = num_prompts
+    if request_rate is not None:
+        run_params["request-rate"] = request_rate
+    if max_concurrency is not None:
+        run_params["max-concurrency"] = max_concurrency
+    if "baseline" in params:
+        run_params["baseline"] = {
+            metric: _resolve_baseline_value(
+                baseline_raw,
+                sweep_index=sweep_index,
+                max_concurrency=max_concurrency,
+                request_rate=request_rate,
+            )
+            for metric, baseline_raw in params["baseline"].items()
+        }
+    return run_params
+
+
+def _iter_sweep_runs(params: dict[str, Any]) -> list[dict[str, Any]]:
+    request_rate_list = _to_list(params.get("request-rate"))
+    num_prompt_list = _to_list(params.get("num-prompts", 10))
+    max_concurrency_list = _to_list(params.get("max-concurrency"))
+
+    max_len = max(len(request_rate_list), len(max_concurrency_list))
+    if len(num_prompt_list) == 1 and max_len > 1:
+        num_prompt_list = num_prompt_list * max_len
+    elif max_len == 1 and len(num_prompt_list) > 1:
+        if len(request_rate_list) == 1:
+            request_rate_list = request_rate_list * len(num_prompt_list)
+        if len(max_concurrency_list) == 1:
+            max_concurrency_list = max_concurrency_list * len(num_prompt_list)
+        max_len = max(len(request_rate_list), len(max_concurrency_list))
+    elif len(num_prompt_list) != max_len and max_len > 0:
+        raise ValueError("The number of prompts does not match the request-rate or max-concurrency")
+
+    sweep_runs: list[dict[str, Any]] = []
+
+    for i, (request_rate, num_prompts) in enumerate(zip(request_rate_list, num_prompt_list)):
+        sweep_runs.append(
+            {
+                "params": _build_run_params(
+                    params,
+                    request_rate=request_rate,
+                    num_prompts=num_prompts,
+                    sweep_index=i,
+                ),
+                "num_prompts": num_prompts,
+                "sweep_index": i,
+                "request_rate": request_rate,
+                "max_concurrency": None,
+            }
+        )
+
+    for i, (max_concurrency, num_prompts) in enumerate(zip(max_concurrency_list, num_prompt_list)):
+        sweep_runs.append(
+            {
+                "params": _build_run_params(
+                    params,
+                    max_concurrency=max_concurrency,
+                    num_prompts=num_prompts,
+                    sweep_index=i,
+                    request_rate="inf",
+                ),
+                "num_prompts": num_prompts,
+                "sweep_index": i,
+                "request_rate": None,
+                "max_concurrency": max_concurrency,
+            }
+        )
+
+    if not sweep_runs:
+        default_num_prompts = num_prompt_list[0]
+        sweep_runs.append(
+            {
+                "params": _build_run_params(
+                    params,
+                    num_prompts=default_num_prompts,
+                ),
+                "num_prompts": default_num_prompts,
+                "sweep_index": None,
+                "request_rate": None,
+                "max_concurrency": None,
+            }
+        )
+
+    return sweep_runs
+
+
 # ---------------------------------------------------------------------------
 # Test entry point
 # ---------------------------------------------------------------------------
@@ -715,36 +933,47 @@ def test_diffusion_performance_benchmark(diffusion_server, benchmark_params, req
    test_name = benchmark_params["test_name"]
    params = benchmark_params["params"]
    server_cfg = getattr(diffusion_server, "server_cfg", {})
-    endpoint = _resolve_benchmark_endpoint(server_cfg, params)
+    sweep_runs = _iter_sweep_runs(params)

-    result = run_benchmark(
-        host=diffusion_server.host,
-        port=diffusion_server.port,
-        model=diffusion_server.model,
-        params=params,
-        test_name=test_name,
-        endpoint=endpoint,
-        server_cfg=server_cfg,
-        source_file=cast(str, CONFIG_FILE_PATH),
-    )
+    for sweep_run in sweep_runs:
+        endpoint = _resolve_benchmark_endpoint(server_cfg, sweep_run["params"])
+        result = run_benchmark(
+            host=diffusion_server.host,
+            port=diffusion_server.port,
+            model=diffusion_server.model,
+            params=sweep_run["params"],
+            test_name=test_name,
+            endpoint=endpoint,
+            server_cfg=server_cfg,
+            source_file=cast(str, CONFIG_FILE_PATH),
+        )

-    print(f"\n{'=' * 60}")
-    print(f"Results for {test_name} (server={diffusion_server.server_type}, endpoint={endpoint}):")
-    for key in (
-        "throughput_qps",
-        "latency_mean",
-        "latency_median",
-        "latency_p50",
-        "latency_p99",
-        "peak_memory_mb_max",
-        "peak_memory_mb_mean",
-        "peak_memory_mb_median",
-    ):
-        if key in result:
-            print(f"  {key}: {result[key]:.4f}")
+        print(f"\n{'=' * 60}")
+        print(f"Results for {test_name} (server={diffusion_server.server_type}, endpoint={endpoint}):")
+        for key in (
+            "throughput_qps",
+            "latency_mean",
+            "latency_median",
+            "latency_p50",
+            "latency_p99",
+            "peak_memory_mb_max",
+            "peak_memory_mb_mean",
+            "peak_memory_mb_median",
+        ):
+            if key in result:
+                print(f"  {key}: {result[key]:.4f}")

-    print(f"\n  Aggregated results: {AGGREGATED_RESULT_FILE}")
-    print("=" * 60)
+        print(f"\n  Aggregated results: {AGGREGATED_RESULT_FILE}")
+        print("=" * 60)

-    assert_baseline = request.config.getoption("--assert-baseline", default=False)
-    assert_result(result, params, assert_baseline=assert_baseline)
+        assert_baseline = request.config.getoption("--assert-baseline", default=False)
+
+        assert_result(
+            result,
+            params,
+            sweep_run["num_prompts"],
+            sweep_index=sweep_run["sweep_index"],
+            max_concurrency=sweep_run["max_concurrency"],
+            request_rate=sweep_run["request_rate"],
+            assert_baseline=assert_baseline,
+        )
@@ -0,0 +1,152 @@
+[
+  {
+    "test_name": "test_bagel_single_device_single_stage_t2i",
+    "description": "Single-stage BAGEL (TP=1, CFG=1), t2i Task",
+    "server_type": "vllm-omni",
+    "server_params": {
+      "model": "ByteDance-Seed/BAGEL-7B-MoT",
+      "serve_args": {
+        "enable-diffusion-pipeline-profiler": true,
+        "deploy-config": "../vllm_omni/deploy/bagel_single_stage.yaml",
+        "no-async-chunk": true,
+        "tensor-parallel-size": 1,
+        "cfg-parallel-size": 1,
+        "gpu-memory-utilization": 0.90,
+        "max-model-len": 8192,
+        "max-num-batched-tokens": 9216
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "512x512_t2i_steps20",
+        "dataset": "random",
+        "task": "t2i",
+        "width": 512,
+        "height": 512,
+        "num-inference-steps": 20,
+        "num-prompts": 20,
+        "max-concurrency": 1,
+        "enable-negative-prompt": true,
+        "baseline": {
+          "throughput_qps": 0.3678,
+          "latency_mean": 2.691,
+          "peak_memory_mb_max": 33083.6,
+          "peak_memory_mb_mean": 33082.5
+        }
+      }
+    ]
+  },
+  {
+    "test_name": "test_bagel_single_device_single_stage_i2i",
+    "description": "Single-stage BAGEL (TP=1, CFG=1), i2i Task",
+    "server_type": "vllm-omni",
+    "server_params": {
+      "model": "ByteDance-Seed/BAGEL-7B-MoT",
+      "serve_args": {
+        "enable-diffusion-pipeline-profiler": true,
+        "deploy-config": "../vllm_omni/deploy/bagel_single_stage.yaml",
+        "no-async-chunk": true,
+        "tensor-parallel-size": 1,
+        "cfg-parallel-size": 1,
+        "gpu-memory-utilization": 0.90,
+        "max-model-len": 8192,
+        "max-num-batched-tokens": 9216
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "512x512_i2i_steps20",
+        "dataset": "random",
+        "task": "i2i",
+        "width": 512,
+        "height": 512,
+        "num-inference-steps": 20,
+        "num-prompts": 20,
+        "max-concurrency": 1,
+        "enable-negative-prompt": true,
+        "num-input-images": 1,
+        "baseline": {
+          "throughput_qps": 0.3124,
+          "latency_mean": 3.1686,
+          "peak_memory_mb_max": 33448.8,
+          "peak_memory_mb_mean": 33447.7
+        }
+      }
+    ]
+  },
+  {
+    "test_name": "test_bagel_single_device_multi_stage_t2i",
+    "description": "Two-stage BAGEL (bagel.yaml), TP=1, CFG=1, t2i Task",
+    "server_type": "vllm-omni",
+    "server_params": {
+      "model": "ByteDance-Seed/BAGEL-7B-MoT",
+      "serve_args": {
+        "enable-diffusion-pipeline-profiler": true,
+        "deploy-config": "../vllm_omni/deploy/bagel.yaml",
+        "no-async-chunk": true,
+        "tensor-parallel-size": 1,
+        "cfg-parallel-size": 1,
+        "gpu-memory-utilization": 0.45,
+        "max-model-len": 8192,
+        "max-num-batched-tokens": 9216
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "512x512_t2i_steps20",
+        "dataset": "random",
+        "task": "t2i",
+        "width": 512,
+        "height": 512,
+        "num-inference-steps": 20,
+        "num-prompts": 20,
+        "max-concurrency": 1,
+        "enable-negative-prompt": true,
+        "baseline": {
+          "throughput_qps": 0.3746,
+          "latency_mean": 2.6425,
+          "peak_memory_mb_max": 33077.0,
+          "peak_memory_mb_mean": 33077.0
+        }
+      }
+    ]
+  },
+  {
+    "test_name": "test_bagel_single_device_multi_stage_i2i",
+    "description": "Two-stage BAGEL (bagel.yaml), TP=1, CFG=1, i2i Task",
+    "server_type": "vllm-omni",
+    "server_params": {
+      "model": "ByteDance-Seed/BAGEL-7B-MoT",
+      "serve_args": {
+        "enable-diffusion-pipeline-profiler": true,
+        "deploy-config": "../vllm_omni/deploy/bagel.yaml",
+        "no-async-chunk": true,
+        "tensor-parallel-size": 1,
+        "cfg-parallel-size": 1,
+        "gpu-memory-utilization": 0.45,
+        "max-model-len": 8192,
+        "max-num-batched-tokens": 9216
+      }
+    },
+    "benchmark_params": [
+      {
+        "name": "512x512_i2i_steps20",
+        "dataset": "random",
+        "task": "i2i",
+        "width": 512,
+        "height": 512,
+        "num-inference-steps": 20,
+        "num-prompts": 20,
+        "max-concurrency": 1,
+        "enable-negative-prompt": true,
+        "num-input-images": 1,
+        "baseline": {
+          "throughput_qps": 0.1349,
+          "latency_mean": 7.339,
+          "peak_memory_mb_max": 33321.2,
+          "peak_memory_mb_mean": 33321.2
+        }
+      }
+    ]
+  }
+]
@@ -0,0 +1,131 @@
+[
+    {
+        "test_name": "test_hunyuan_image3_ti2i_tp4_tp4_inline",
+        "description": "AR (TP=4) + DiT (TP=4) - using inline dataset and deploy-config",
+        "server_type": "vllm-omni",
+        "benchmark_endpoint": "/v1/images/edits",
+        "server_params": {
+            "model": "tencent/HunyuanImage-3.0-Instruct",
+            "serve_args": {
+                "deploy-config-inline": {
+                    "type": "yaml",
+                    "content": {
+                        "pipeline": "hunyuan_image_3_moe",
+                        "async_chunk": false,
+                        "trust_remote_code": true,
+                        "connectors": {
+                            "rdma_connector": {
+                                "name": "MooncakeTransferEngineConnector",
+                                "extra": {
+                                    "host": "auto",
+                                    "zmq_port": 50051,
+                                    "protocol": "rdma",
+                                    "device_name": "",
+                                    "memory_pool_size": 4294967296,
+                                    "memory_pool_device": "cpu"
+                                }
+                            }
+                        },
+                        "stages": [
+                            {
+                                "stage_id": 0,
+                                "is_comprehension": true,
+                                "final_output": true,
+                                "final_output_type": "text",
+                                "max_num_seqs": 4,
+                                "gpu_memory_utilization": 0.95,
+                                "enforce_eager": false,
+                                "enable_prefix_caching": false,
+                                "max_num_batched_tokens": 32768,
+                                "devices": "0,1,2,3",
+                                "tensor_parallel_size": 4,
+                                "hf_overrides": {
+                                    "rope_parameters": {
+                                        "mrope_section": [0, 32, 32],
+                                        "rope_type": "default"
+                                    }
+                                },
+                                "omni_kv_config": {
+                                    "need_send_cache": true
+                                },
+                                "output_connectors": {
+                                    "to_stage_1": "rdma_connector"
+                                },
+                                "default_sampling_params": {
+                                    "temperature": 0.0,
+                                    "top_p": 1,
+                                    "top_k": -1,
+                                    "max_tokens": 8192,
+                                    "detokenize": true,
+                                    "skip_special_tokens": false,
+                                    "stop_token_ids": [128025]
+                                }
+                            },
+                            {
+                                "stage_id": 1,
+                                "max_num_seqs": 1,
+                                "enforce_eager": true,
+                                "devices": "4,5,6,7",
+                                "distributed_executor_backend": "mp",
+                                "omni_kv_config": {
+                                    "need_recv_cache": true
+                                },
+                                "parallel_config": {
+                                    "tensor_parallel_size": 4,
+                                    "enable_expert_parallel": true
+                                },
+                                "input_connectors": {
+                                    "from_stage_0": "rdma_connector"
+                                },
+                                "default_sampling_params": {
+                                    "num_inference_steps": 50,
+                                    "guidance_scale": 0
+                                }
+                            }
+                        ],
+                        "edges": [
+                            {
+                                "from": 0,
+                                "to": 1,
+                                "window_size": -1,
+                                "max_inflight": 1
+                            }
+                        ]
+                    }
+                },
+                "enable-diffusion-pipeline-profiler": true
+            }
+        },
+        "benchmark_params": [
+            {
+                "name": "inline_dataset_5_requests",
+                "dataset": "custom",
+                "dataset-path-inline": {
+                    "type": "jsonl",
+                    "content": [
+                        {
+                            "prompt": "新年宠物海报，Q版圆润的可爱标题“新年快乐汪”，副标题“HAPPY NEW YEAR”。 鱼眼镜头，背景是房间门口，近景，上传的主体歪头笑，围着红色围巾，戴着红色毛线帽，高清，绒毛细节，面部特写。 宝丽莱相纸，超现实主义，写实主义，胶片摄影，打印颗粒感肌理。肌理，超写实，复古感。",
+                            "image_urls": ["https://raw.githubusercontent.com/Tencent-Hunyuan/HunyuanImage-3.0/main/assets/demo_instruct_imgs/input_0_0.png"],
+                            "num_inference_steps": 8,
+                            "seed": 42,
+                            "guidance_scale": 1.0
+                        }
+                    ]
+                },
+                "task": "ti2i",
+                "bot-task": "think_recaption",
+                "max-concurrency": 4,
+                "num-prompts": 16,
+                "skip-performance-assertion": true,
+                "baseline": {
+                    "throughput_qps": 0.1,
+                    "latency_p99": 44.46,
+                    "latency_mean": 35.57,
+                    "latency_std_dev": 0.5,
+                    "peak_memory_mb_max": 70000,
+                    "peak_memory_mb_mean": 70000
+                }
+            }
+        ]
+    }
+]
@@ -1,35 +1,33 @@
 [
    {
-        "test_name": "test_hunyuan_image_tp2_fp8_cfgp2",
-        "description": "TP=2 Quantization=fp8 CfgP=2 baseline",
+        "test_name": "test_hunyuan_image_tp2_cfgp2",
+        "description": "TP=2 CfgP=2 baseline",
        "server_type": "vllm-omni",
        "server_params": {
            "model": "tencent/HunyuanImage-3.0-Instruct",
            "serve_args": {
+                "deploy_config": "../vllm_omni/deploy/hunyuan_image3_dit.yaml",
                "tensor-parallel-size": 2,
                "cfg-parallel-size": 2,
-                "quantization": "fp8",
-                "distributed-executor-backend": "mp",
-                "enforce-eager": true,
                "enable-diffusion-pipeline-profiler": true
            }
        },
        "benchmark_params": [
            {
-                "name": "1024x1024_steps50",
+                "name": "1024x1024_steps8",
                "dataset": "random",
                "task": "t2i",
                "width": 1024,
                "height": 1024,
-                "num-inference-steps": 50,
+                "num-inference-steps": 8,
                "num-prompts": 10,
                "max-concurrency": 1,
                "skip-performance-assertion": true,
                "baseline": {
-                    "throughput_qps": 0.1035,
-                    "latency_p99": 9.9057,
-                    "peak_memory_mb_max": 66470,
-                    "peak_memory_mb_mean": 66470
+                    "throughput_qps": 0.21,
+                    "latency_p99": 4.7469,
+                    "peak_memory_mb_max": 101100,
+                    "peak_memory_mb_mean": 101100
                }
            }
        ]
@@ -1,35 +1,33 @@
 [
    {
-        "test_name": "test_hunyuan_image_tp2_fp8_sp2",
-        "description": "TP=2 Quantization=fp8 SP=2 baseline",
+        "test_name": "test_hunyuan_image_tp2_sp2",
+        "description": "TP=2 SP=2 baseline",
        "server_type": "vllm-omni",
        "server_params": {
            "model": "tencent/HunyuanImage-3.0-Instruct",
            "serve_args": {
+                "deploy-config": "../vllm_omni/deploy/hunyuan_image3_dit.yaml",
                "tensor-parallel-size": 2,
-                "usp": 2,
-                "quantization": "fp8",
-                "distributed-executor-backend": "mp",
-                "enforce-eager": true,
+                "ulysses-degree": 2,
                "enable-diffusion-pipeline-profiler": true
            }
        },
        "benchmark_params": [
            {
-                "name": "1024x1024_steps50",
+                "name": "1024x1024_steps8",
                "dataset": "random",
                "task": "t2i",
                "width": 1024,
                "height": 1024,
-                "num-inference-steps": 50,
+                "num-inference-steps": 8,
                "num-prompts": 10,
                "max-concurrency": 1,
                "skip-performance-assertion": true,
                "baseline": {
-                    "throughput_qps": 0.08,
-                    "latency_p99": 12.0731,
-                    "peak_memory_mb_max": 66314,
-                    "peak_memory_mb_mean": 66314
+                    "throughput_qps": 0.20,
+                    "latency_p99": 5.1025,
+                    "peak_memory_mb_max": 97402,
+                    "peak_memory_mb_mean": 97402
                }
            }
        ]
@@ -1,34 +1,32 @@
 [
    {
-        "test_name": "test_hunyuan_image_tp4_fp8",
-        "description": "TP=4 Quantization=fp8 baseline",
+        "test_name": "test_hunyuan_image_tp4",
+        "description": "TP=4 baseline",
        "server_type": "vllm-omni",
        "server_params": {
            "model": "tencent/HunyuanImage-3.0-Instruct",
            "serve_args": {
+                "deploy-config": "../vllm_omni/deploy/hunyuan_image3_dit.yaml",
                "tensor-parallel-size": 4,
-                "quantization": "fp8",
-                "distributed-executor-backend": "mp",
-                "enforce-eager": true,
                "enable-diffusion-pipeline-profiler": true
            }
        },
        "benchmark_params": [
            {
-                "name": "1024x1024_steps50",
+                "name": "1024x1024_steps8",
                "dataset": "random",
                "task": "t2i",
                "width": 1024,
                "height": 1024,
-                "num-inference-steps": 50,
+                "num-inference-steps": 8,
                "num-prompts": 10,
                "max-concurrency": 1,
                "skip-performance-assertion": true,
                "baseline": {
-                    "throughput_qps": 0.08,
-                    "latency_p99": 13.1227,
-                    "peak_memory_mb_max": 46838,
-                    "peak_memory_mb_mean": 46838
+                    "throughput_qps": 0.21,
+                    "latency_p99": 4.9235,
+                    "peak_memory_mb_max": 57576,
+                    "peak_memory_mb_mean": 57576
                }
            }
        ]
@@ -0,0 +1,115 @@
+[
+    {
+        "test_name": "test_qwen3_omni_3gpu_replica2",
+        "server_params": {
+            "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
+            "stage_overrides": {
+                "1": {"num_replicas": 2, "devices": "1,2"},
+                "2": {"num_replicas": 2, "devices": "1,2"}
+            },
+            "extra_cli_args": [
+                "--async-chunk"
+            ]
+        },
+        "benchmark_params": [
+            {
+                "dataset_name": "random",
+                "backend": "openai-chat-omni",
+                "endpoint": "/v1/chat/completions",
+                "num_prompts": [128, 128, 128, 128],
+                "max_concurrency": [8, 16, 24, 32],
+                "random_input_len": 2500,
+                "random_output_len": 900,
+                "ignore_eos": true,
+                "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
+                "baseline": {
+                    "mean_ttft_ms": [1000, 2000, 3000, 4000],
+                    "mean_tpot_ms": [200, 200, 200, 200],
+                    "mean_audio_ttfp_ms": [10000, 15000, 20000, 25000],
+                    "mean_audio_rtf": [1.5, 1.5, 1.5, 1.5]
+                }
+            },
+            {
+                "dataset_name": "random-mm",
+                "backend": "openai-chat-omni",
+                "endpoint": "/v1/chat/completions",
+                "num_prompts": [10],
+                "request_rate": [0.1],
+                "random_input_len": 100,
+                "random_output_len": 100,
+                "random_range_ratio": 0.0,
+                "ignore_eos": true,
+                "random_mm_base_items_per_request": 1,
+                "random_mm_num_mm_items_range_ratio": 0.5,
+                "random_mm_limit_mm_per_prompt": {
+                    "audio": 1
+                },
+                "random_mm_bucket_config": {
+                    "(0, 60, 3)": 1.0
+                },
+                "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
+                "baseline": {
+                    "mean_ttft_ms": [2000],
+                    "mean_audio_ttfp_ms": [2000],
+                    "mean_audio_rtf": [0.25]
+                }
+            },
+            {
+                "dataset_name": "random-mm",
+                "backend": "openai-chat-omni",
+                "endpoint": "/v1/chat/completions",
+                "num_prompts": [40],
+                "request_rate": [0.5],
+                "random_input_len": 100,
+                "random_output_len": 100,
+                "random_range_ratio": 0.0,
+                "ignore_eos": true,
+                "random_mm_base_items_per_request": 2,
+                "random_mm_num_mm_items_range_ratio": 0.5,
+                "random_mm_limit_mm_per_prompt": {
+                    "image": 1,
+                    "video": 1
+                },
+                "random_mm_bucket_config": {
+                    "(256, 256, 1)": 0.5,
+                    "(720, 1280, 2)": 0.5
+                },
+                "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
+                "baseline": {
+                    "mean_ttft_ms": [6000],
+                    "mean_audio_ttfp_ms": [6000],
+                    "mean_audio_rtf": [0.7]
+                }
+            },
+            {
+                "dataset_name": "random-mm",
+                "backend": "openai-chat-omni",
+                "endpoint": "/v1/chat/completions",
+                "num_prompts": [100],
+                "request_rate": [1.0],
+                "random_input_len": 100,
+                "random_output_len": 100,
+                "random_range_ratio": 0.0,
+                "ignore_eos": true,
+                "random_mm_base_items_per_request": 3,
+                "random_mm_num_mm_items_range_ratio": 0.5,
+                "random_mm_limit_mm_per_prompt": {
+                    "image": 1,
+                    "video": 1,
+                    "audio": 1
+                },
+                "random_mm_bucket_config": {
+                    "(256, 256, 1)": 0.34,
+                    "(720, 1280, 2)": 0.33,
+                    "(0, 60, 3)": 0.33
+                },
+                "percentile-metrics": "ttft,tpot,itl,e2el,audio_rtf,audio_ttfp,audio_duration",
+                "baseline": {
+                    "mean_ttft_ms": [12000],
+                    "mean_audio_ttfp_ms": [12000],
+                    "mean_audio_rtf": [1.0]
+                }
+            }
+        ]
+    }
+]
@@ -1,10 +1,10 @@
 [
    {
-        "test_name": "test_qwen_image_edit_2509_single_device",
+        "test_name": "test_qwen_image_edit_2511_single_device",
        "description": "Single-device baseline (two input images)",
        "server_type": "vllm-omni",
        "server_params": {
-            "model": "Qwen/Qwen-Image-Edit-2509",
+            "model": "Qwen/Qwen-Image-Edit-2511",
            "serve_args": {
                "enable-diffusion-pipeline-profiler": true
            }
@@ -49,11 +49,11 @@
        ]
    },
    {
-        "test_name": "test_qwen_image_edit_2509_ulysses2_cfg2_vae_patch4",
+        "test_name": "test_qwen_image_edit_2511_ulysses2_cfg2_vae_patch4",
        "description": "Ulysses SP=2 + CFG=2 + VAE patch parallel=4",
        "server_type": "vllm-omni",
        "server_params": {
-            "model": "Qwen/Qwen-Image-Edit-2509",
+            "model": "Qwen/Qwen-Image-Edit-2511",
            "serve_args": {
                "ulysses-degree": 2,
                "cfg-parallel-size": 2,
@@ -63,24 +63,6 @@
            }
        },
        "benchmark_params": [
-            {
-                "name": "512x512_steps20_i2i_2img",
-                "dataset": "random",
-                "task": "i2i",
-                "width": 512,
-                "height": 512,
-                "num-inference-steps": 20,
-                "num-prompts": 10,
-                "max-concurrency": 1,
-                "num-input-images": 2,
-                "enable-negative-prompt": true,
-                "baseline": {
-                    "throughput_qps": 0.15,
-                    "latency_mean": 10,
-                    "peak_memory_mb_max": 55000,
-                    "peak_memory_mb_mean": 55000
-                }
-            },
            {
                "name": "1536x1536_steps35_i2i_2img",
                "dataset": "random",
@@ -102,11 +84,11 @@
        ]
    },
    {
-        "test_name": "test_qwen_image_edit_2509_ulysses2_cfg2_cache_dit",
+        "test_name": "test_qwen_image_edit_2511_ulysses2_cfg2_cache_dit",
        "description": "Ulysses SP=2 + CFG=2 + CacheDiT",
        "server_type": "vllm-omni",
        "server_params": {
-            "model": "Qwen/Qwen-Image-Edit-2509",
+            "model": "Qwen/Qwen-Image-Edit-2511",
            "serve_args": {
                "ulysses-degree": 2,
                "cfg-parallel-size": 2,
@@ -61,23 +61,6 @@
            }
        },
        "benchmark_params": [
-            {
-                "name": "512x512_steps20_i2i",
-                "dataset": "random",
-                "task": "i2i",
-                "width": 512,
-                "height": 512,
-                "num-inference-steps": 20,
-                "num-prompts": 10,
-                "max-concurrency": 1,
-                "enable-negative-prompt": true,
-                "baseline": {
-                    "throughput_qps": 0.21,
-                    "latency_mean": 10,
-                    "peak_memory_mb_max": 56000,
-                    "peak_memory_mb_mean": 56000
-                }
-            },
            {
                "name": "1536x1536_steps35_i2i",
                "dataset": "random",
@@ -181,5 +181,39 @@
                }
            }
        ]
+    },
+    {
+        "test_name": "test_qwen_image_single_device_step_execution_high_concurrency",
+        "description": "Single-device Qwen-Image step-level execution benchmark at 512x512, 20 steps, concurrency sweep",
+        "server_type": "vllm-omni",
+        "server_params": {
+            "model": "Qwen/Qwen-Image",
+            "serve_args": {
+                "enable-diffusion-pipeline-profiler": true,
+                "step-execution": true,
+                "max-num-seqs": 8
+            }
+        },
+        "benchmark_params": [
+            {
+                "name": "512x512_steps20_high_concurrency",
+                "dataset": "random",
+                "task": "t2i",
+                "width": 512,
+                "height": 512,
+                "num-inference-steps": 20,
+                "num-prompts": [20, 40, 80, 160],
+                "max-concurrency": [1, 2, 4, 8],
+                "warmup-requests": 8,
+                "warmup-concurrency": 8,
+                "warmup-num-inference-steps": 20,
+                "enable-negative-prompt": true,
+                "baseline": {
+                    "throughput_qps": [0.30, 0.50, 0.60, 0.70],
+                    "latency_mean": [3.50, 4.00, 7.00, 12.00],
+                    "peak_memory_mb_mean": [67000, 67000, 67000, 67000]
+                }
+            }
+        ]
    }
 ]
@@ -17,7 +17,16 @@ def test_task_excluded_from_cli_args():
        "percentile-metrics": "audio_rtf,audio_ttfp",
        "baseline": {"mean_audio_rtf": [0.5]},
    }
-    exclude_keys = {"request_rate", "baseline", "num_prompts", "max_concurrency", "task", "enabled", "eval_phase"}
+    exclude_keys = {
+        "request_rate",
+        "baseline",
+        "num_prompts",
+        "max_concurrency",
+        "task",
+        "enabled",
+        "eval_phase",
+        "trust_remote_code",
+    }
    args = []
    for key, value in params.items():
        if key in exclude_keys or value is None:
@@ -0,0 +1,198 @@
+[
+  {
+    "test_name": "test_voxcpm2",
+    "server_params": {
+      "model": "openbmb/VoxCPM2",
+      "trust_remote_code": true
+    },
+    "benchmark_params": [
+      {
+        "task": "voice_clone",
+        "eval_phase": "latency",
+        "dataset_name": "seed-tts",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "linyueqian/seed-tts-eval-subset",
+        "num_prompts": [
+          20
+        ],
+        "max_concurrency": [
+          1
+        ],
+        "seed_tts_locale": "en",
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
+        "baseline": {
+          "median_audio_ttfp_ms": [
+            600
+          ],
+          "median_audio_rtf": [
+            0.2
+          ]
+        }
+      },
+      {
+        "task": "voice_clone",
+        "eval_phase": "throughput",
+        "dataset_name": "seed-tts",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "linyueqian/seed-tts-eval-subset",
+        "num_prompts": [
+          80
+        ],
+        "max_concurrency": [
+          8
+        ],
+        "seed_tts_locale": "en",
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
+        "baseline": {
+          "median_audio_ttfp_ms": [
+            3500
+          ],
+          "median_audio_rtf": [
+            0.85
+          ],
+          "audio_throughput": [
+            8
+          ]
+        }
+      },
+      {
+        "task": "voice_clone",
+        "eval_phase": "quality",
+        "enabled": false,
+        "dataset_name": "seed-tts",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "linyueqian/seed-tts-eval-subset",
+        "num_prompts": [
+          200
+        ],
+        "max_concurrency": [
+          4
+        ],
+        "seed_tts_locale": "en",
+        "seed_tts_wer_eval": true,
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
+        "baseline": {
+          "mean_audio_rtf": [
+            0.45
+          ]
+        }
+      },
+      {
+        "task": "default_voice",
+        "eval_phase": "latency",
+        "dataset_name": "seed-tts-text",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "benchmarks/build_dataset/seed_tts_smoke",
+        "num_prompts": [
+          20
+        ],
+        "max_concurrency": [
+          1
+        ],
+        "seed_tts_locale": "en",
+        "extra_body": {
+          "voice": "default"
+        },
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
+        "baseline": {
+          "median_audio_ttfp_ms": [
+            350
+          ],
+          "median_audio_rtf": [
+            0.15
+          ]
+        }
+      },
+      {
+        "task": "default_voice",
+        "eval_phase": "throughput",
+        "dataset_name": "seed-tts-text",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "benchmarks/build_dataset/seed_tts_smoke",
+        "num_prompts": [
+          80
+        ],
+        "max_concurrency": [
+          8
+        ],
+        "seed_tts_locale": "en",
+        "extra_body": {
+          "voice": "default"
+        },
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
+        "baseline": {
+          "median_audio_ttfp_ms": [
+            3000
+          ],
+          "median_audio_rtf": [
+            0.5
+          ],
+          "audio_throughput": [
+            10
+          ]
+        }
+      },
+      {
+        "task": "default_voice",
+        "eval_phase": "stress",
+        "dataset_name": "seed-tts-text",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "benchmarks/build_dataset/seed_tts_smoke",
+        "num_prompts": [
+          100
+        ],
+        "request_rate": [
+          2.0
+        ],
+        "seed_tts_locale": "en",
+        "extra_body": {
+          "voice": "default"
+        },
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
+        "baseline": {
+          "median_audio_ttfp_ms": [
+            500
+          ],
+          "median_audio_rtf": [
+            0.3
+          ],
+          "audio_throughput": [
+            5
+          ]
+        }
+      },
+      {
+        "task": "default_voice",
+        "eval_phase": "quality",
+        "enabled": false,
+        "dataset_name": "seed-tts-text",
+        "backend": "openai-audio-speech",
+        "endpoint": "/v1/audio/speech",
+        "dataset_path": "benchmarks/build_dataset/seed_tts_smoke",
+        "num_prompts": [
+          200
+        ],
+        "max_concurrency": [
+          4
+        ],
+        "seed_tts_locale": "en",
+        "extra_body": {
+          "voice": "default"
+        },
+        "seed_tts_wer_eval": true,
+        "percentile-metrics": "ttft,e2el,audio_rtf,audio_ttfp,audio_duration",
+        "baseline": {
+          "mean_audio_rtf": [
+            0.35
+          ]
+        }
+      }
+    ]
+  }
+]
@@ -4,10 +4,16 @@ This module keeps fault injection callable from tests directly:
 - GPU OOM (CUDA sidecar memory hog)
 - process kill by pattern and signal
 - post-ready hooks via ``fault_injector`` / ``omni_server_after_fault`` fixtures
+
+Worker PID classification (``worker_pids`` in snapshots) is still used for logging /
+markers; post-fault **process** cleanup assertions use the full captured
+``tree_pids`` (serve root + descendants at fault time) — see
+:func:`assert_no_server_tree_process_residual_and_gpu_release`.
 """

 from __future__ import annotations

+import concurrent.futures
 import http.client
 import json
 import logging
@@ -27,6 +33,16 @@ import pytest

 logger = logging.getLogger(__name__)

+# Substrings matched against ``psutil.Process.name`` + argv text for classifying GPU
+# workers under the test server tree. Extend per-suite via
+# ``server.reliability_worker_markers_extra`` or replace via
+# ``server.reliability_worker_markers`` (see ``_resolve_runtime_worker_markers``).
+DEFAULT_RUNTIME_WORKER_MARKERS: tuple[str, ...] = (
+    "multiprocessing.spawn",
+    # Covers ``VLLM::Worker``, ``VLLM::StageEngineCoreProc_*`` (Omni stage engines), etc.
+    "VLLM::",
+)
+

@dataclass
 class OomHandle:
@@ -452,6 +468,61 @@ def _safe_proc_info(pid: int) -> tuple[str, str]:
        return "<unknown>", "<unavailable>"


+def _normalize_worker_marker_substrings(seq: Sequence[Any], *, context: str) -> tuple[str, ...]:
+    """Return deduped non-empty marker substrings; raises if any entry is blank."""
+    out: list[str] = []
+    for item in seq:
+        s = str(item).strip()
+        if not s:
+            raise ValueError(f"{context}: worker marker must be a non-empty string, got {item!r}")
+        out.append(s)
+    return tuple(dict.fromkeys(out))
+
+
+def _resolve_runtime_worker_markers(
+    server: Any,
+    injector_markers_extra: Sequence[str] | None,
+) -> tuple[str, ...]:
+    """Resolve which substrings classify worker PIDs for fault snapshots.
+
+    If ``server.reliability_worker_markers`` is set to a sequence, **only** those
+    markers are used (no defaults). Otherwise defaults from
+    :data:`DEFAULT_RUNTIME_WORKER_MARKERS` are merged with:
+
+    - ``server.reliability_worker_markers_extra`` when present, and
+    - ``injector_markers_extra`` from :func:`make_server_root_kill_fault_injector` /
+      :func:`make_server_tree_kill_fault_injector`.
+    """
+    replace = getattr(server, "reliability_worker_markers", None)
+    if replace is not None:
+        return _normalize_worker_marker_substrings(replace, context="server.reliability_worker_markers")
+
+    merged: list[str] = []
+    merged.extend(DEFAULT_RUNTIME_WORKER_MARKERS)
+    extra_attr = getattr(server, "reliability_worker_markers_extra", None)
+    if extra_attr is not None:
+        merged.extend(
+            _normalize_worker_marker_substrings(extra_attr, context="server.reliability_worker_markers_extra")
+        )
+    if injector_markers_extra is not None:
+        merged.extend(
+            _normalize_worker_marker_substrings(
+                injector_markers_extra,
+                context="injector worker_markers_extra",
+            )
+        )
+    return tuple(dict.fromkeys(merged))
+
+
+def _pid_looks_like_runtime_worker(pid: int, markers: Sequence[str]) -> bool:
+    """True when process name or argv contains any resolved worker marker substring."""
+    if not markers:
+        return False
+    name, cmdline = _safe_proc_info(pid)
+    text = f"{name} {cmdline}"
+    return any(marker in text for marker in markers)
+
+
 def _list_server_process_tree(server: Any) -> list[int]:
    """Return [root, descendants...] PIDs for the current test server instance."""
    root_proc = getattr(server, "proc", None)
@@ -468,6 +539,29 @@ def _list_server_process_tree(server: Any) -> list[int]:
    return [root_pid, *descendants]


+def _pids_in_server_tree_substring_match(server: Any, pattern: str, *, limit: int) -> list[int]:
+    """Return up to ``limit`` PIDs in the server tree whose name+cmdline contains ``pattern`` literally.
+
+    ``pgrep -f`` matches the process command line used by procps and does not consult the
+    short process name; vLLM-Omni often exposes titles like ``VLLM::Worker`` or
+    ``VLLM::StageEngineCoreProc_*`` via prctl/setproctitle in ``ps``/``name()`` only.
+    This helper keeps kill injection aligned with :func:`_safe_proc_info` / fault snapshots.
+    """
+    if limit <= 0:
+        return []
+    tree = _list_server_process_tree(server)
+    if not tree or not pattern:
+        return []
+    hits: list[int] = []
+    for pid in tree:
+        if len(hits) >= limit:
+            break
+        name, cmdline = _safe_proc_info(int(pid))
+        if pattern in f"{name} {cmdline}":
+            hits.append(int(pid))
+    return hits
+
+
 def _log_server_process_tree(server: Any) -> None:
    """Print server process tree for debugging fault injection targets."""
    pids = _list_server_process_tree(server)
@@ -486,6 +580,315 @@ FaultInjector = Callable[[Any], None]
 """Callable invoked with the live ``OmniServer`` after it is ready (see ``omni_server_after_fault``)."""


+def run_fault_injection_with_rate_load(
+    *,
+    submit_request: Callable[[], Any],
+    inject_fault: Callable[[], None],
+    num_requests: int = 10,
+    request_rate: float = 0.3,
+    submit_interval_sec: float | None = None,
+    completion_timeout_sec: float = 120.0,
+) -> dict[str, Any]:
+    """Submit requests with a fixed rate, then inject fault once an in-flight request is observed."""
+    if num_requests <= 0:
+        raise ValueError("num_requests must be > 0")
+    if request_rate <= 0:
+        raise ValueError("request_rate must be > 0")
+
+    interval_sec = submit_interval_sec if submit_interval_sec is not None else (1.0 / request_rate)
+    if interval_sec < 0:
+        raise ValueError("submit_interval_sec must be >= 0")
+
+    max_workers = min(max(2, num_requests), 8)
+    injected = False
+    futures: list[concurrent.futures.Future[Any]] = []
+    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
+        for idx in range(num_requests):
+            futures.append(executor.submit(submit_request))
+            in_flight = any(not future.done() for future in futures)
+            if in_flight and not injected:
+                inject_fault()
+                injected = True
+            if idx != num_requests - 1 and interval_sec > 0:
+                time.sleep(interval_sec)
+
+        if not injected:
+            in_flight = any(not future.done() for future in futures)
+            if in_flight:
+                inject_fault()
+                injected = True
+
+        if not injected:
+            pytest.skip(
+                "no in-flight request observed before fault injection; "
+                "increase request cost or request rate for this environment"
+            )
+
+        done, pending = concurrent.futures.wait(
+            futures,
+            timeout=completion_timeout_sec,
+            return_when=concurrent.futures.ALL_COMPLETED,
+        )
+        assert not pending, (
+            "some in-flight-load requests did not converge after fault injection: "
+            f"pending={len(pending)} done={len(done)}"
+        )
+    failure_observed = False
+    completed = 0
+    success = 0
+    exceptions = 0
+    for future in futures:
+        completed += 1
+        try:
+            future.result()
+            success += 1
+        except Exception:  # noqa: BLE001
+            exceptions += 1
+            failure_observed = True
+    return {
+        "num_submitted": num_requests,
+        "completed": completed,
+        "success": success,
+        "exceptions": exceptions,
+        "failure_observed": failure_observed,
+        "inflight_observed": True,
+    }
+
+
+def list_alive_pids(pids: Sequence[int]) -> list[int]:
+    """Return PIDs from ``pids`` that still exist in the kernel and are **not** zombies.
+
+    After SIGTERM/SIGKILL the serve child may exit before the test harness calls
+    ``Popen.wait()``; until reaped, ``psutil.pid_exists`` stays true for the defunct
+    slot. Those PIDs must not count as "residual server processes" for leak checks.
+    """
+    out: list[int] = []
+    for pid in pids:
+        pid_i = int(pid)
+        if not psutil.pid_exists(pid_i):
+            continue
+        try:
+            if psutil.Process(pid_i).status() == psutil.STATUS_ZOMBIE:
+                continue
+        except psutil.Error:
+            continue
+        out.append(pid_i)
+    return out
+
+
+def query_gpu_compute_pid_used_memory_mb() -> dict[int, int] | None:
+    """Query NVIDIA compute-process memory map (pid -> used MB).
+
+    Returns ``None`` when ``nvidia-smi`` is unavailable/unreadable on current host.
+    """
+    out = subprocess.run(
+        ["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
+        check=False,
+        capture_output=True,
+        text=True,
+    )
+    if out.returncode != 0:
+        logger.warning("[reliability][gpu] nvidia-smi query failed: %s", out.stderr.strip())
+        return None
+
+    pid_to_mem_mb: dict[int, int] = {}
+    for line in out.stdout.splitlines():
+        row = line.strip()
+        if not row:
+            continue
+        pieces = [part.strip() for part in row.split(",", maxsplit=1)]
+        if len(pieces) != 2:
+            continue
+        pid_raw, mem_raw = pieces
+        if not pid_raw.isdigit():
+            continue
+        mem_digits = "".join(ch for ch in mem_raw if ch.isdigit())
+        if not mem_digits:
+            continue
+        pid_to_mem_mb[int(pid_raw)] = int(mem_digits)
+    return pid_to_mem_mb
+
+
+def worker_residual_timeout_after_kill_signal(signal_name: str) -> float:
+    """Wall-clock budget for fault-snapshot PIDs / GPU to clear after ``signal_name``."""
+    if signal_name == "SIGKILL":
+        return 30.0
+    if signal_name == "SIGINT":
+        return 120.0
+    if signal_name == "SIGTERM":
+        return 90.0
+    return 30.0
+
+
+def assert_no_server_tree_process_residual_and_gpu_release(
+    server: Any,
+    *,
+    scenario: str,
+    timeout_sec: float = 30.0,
+    poll_interval_sec: float = 0.5,
+) -> None:
+    """Assert no live PIDs remain from the fault-time server tree and no GPU use by those PIDs.
+
+    Uses ``reliability_fault_snapshot["tree_pids"]`` (root + all descendants captured at
+    injection time), not the marker-filtered ``worker_pids`` subset — so helpers like
+    ``multiprocessing.resource_tracker`` are included in the residual check.
+
+    **Zombie** PIDs (exited children not yet reaped by the test ``Popen``) are excluded
+    from the "alive" list; ``psutil.pid_exists`` alone would false-positive on them.
+
+    Note: if a child is reparented to PID 1 while staying alive, its PID is unchanged
+    and remains detected; if it **exits and a new unrelated process reuses the same
+    numeric PID**, this check may false-positive (rare on short windows).
+    """
+    snapshot = getattr(server, "reliability_fault_snapshot", None)
+    if not isinstance(snapshot, dict):
+        pytest.fail(f"[{scenario}] missing reliability fault snapshot on server")
+    tree_pids = [int(pid) for pid in snapshot.get("tree_pids", [])]
+    if not tree_pids:
+        pytest.skip(f"[{scenario}] no server process tree PIDs captured for this run")
+
+    tree_pid_set = set(tree_pids)
+    deadline = time.monotonic() + timeout_sec
+    last_alive: list[int] = []
+    last_gpu_leaks: dict[int, int] = {}
+    while time.monotonic() < deadline:
+        last_alive = list_alive_pids(tree_pids)
+        gpu_map = query_gpu_compute_pid_used_memory_mb()
+        if gpu_map is None:
+            pytest.skip(f"[{scenario}] nvidia-smi unavailable; skip GPU release assertion")
+        last_gpu_leaks = {pid: mem_mb for pid, mem_mb in gpu_map.items() if pid in tree_pid_set and mem_mb > 0}
+        if not last_alive and not last_gpu_leaks:
+            return
+        time.sleep(poll_interval_sec)
+
+    assert not last_alive, f"[{scenario}] residual server-tree processes remain alive: {last_alive}"
+    assert not last_gpu_leaks, f"[{scenario}] server-tree PID GPU memory not released: {last_gpu_leaks}"
+
+
+def assert_no_worker_residual_and_gpu_release(
+    server: Any,
+    *,
+    scenario: str,
+    timeout_sec: float = 30.0,
+    poll_interval_sec: float = 0.5,
+) -> None:
+    """Deprecated: use :func:`assert_no_server_tree_process_residual_and_gpu_release`."""
+    assert_no_server_tree_process_residual_and_gpu_release(
+        server,
+        scenario=scenario,
+        timeout_sec=timeout_sec,
+        poll_interval_sec=poll_interval_sec,
+    )
+
+
+def _capture_server_fault_snapshot(
+    server: Any,
+    *,
+    worker_markers_extra: Sequence[str] | None = None,
+) -> dict[str, list[int]]:
+    """Capture current server process snapshot for post-fault assertions.
+
+    ``tree_pids`` is ``[root, ...descendants]`` and is used by
+    :func:`assert_no_server_tree_process_residual_and_gpu_release`.
+    ``worker_pids`` is the marker-filtered subset for debugging only.
+    """
+    markers = _resolve_runtime_worker_markers(server, worker_markers_extra)
+    tree_pids = _list_server_process_tree(server)
+    root_pid = tree_pids[0] if tree_pids else None
+    worker_pids: list[int] = []
+    for pid in tree_pids:
+        if root_pid is not None and int(pid) == int(root_pid):
+            continue
+        if _pid_looks_like_runtime_worker(pid, markers):
+            worker_pids.append(pid)
+    snapshot = {
+        "root_pids": [root_pid] if root_pid is not None else [],
+        "tree_pids": tree_pids,
+        "worker_pids": worker_pids,
+        "worker_markers": list(markers),
+    }
+    setattr(server, "reliability_fault_snapshot", snapshot)
+    return snapshot
+
+
+def make_server_root_kill_fault_injector(
+    *,
+    signal_name: str = "SIGKILL",
+    post_kill_wait_seconds: float = 0.0,
+    worker_markers_extra: Sequence[str] | None = None,
+) -> FaultInjector:
+    """Kill only the current server root process (simulate first strike on PID1-like process).
+
+    ``worker_markers_extra`` is merged into worker PID detection for the fault snapshot
+    (after :data:`DEFAULT_RUNTIME_WORKER_MARKERS` unless
+    ``server.reliability_worker_markers`` replaces the set entirely).
+    """
+
+    def _inject(server: Any) -> None:
+        _log_server_process_tree(server)
+        snapshot = _capture_server_fault_snapshot(server, worker_markers_extra=worker_markers_extra)
+        root_pids = snapshot["root_pids"]
+        if not root_pids:
+            pytest.skip("no server root process found for root-kill injection")
+        root_pid = int(root_pids[0])
+        sig = getattr(signal, signal_name, None)
+        if sig is None:
+            raise ValueError(f"Unsupported signal_name: {signal_name}")
+        name, cmdline = _safe_proc_info(root_pid)
+        print(
+            f"[reliability][process-kill] root-kill pid={root_pid} name={name} signal={signal_name} cmdline={cmdline}",
+            flush=True,
+        )
+        os.kill(root_pid, sig)
+        if post_kill_wait_seconds > 0:
+            time.sleep(post_kill_wait_seconds)
+
+    return _inject
+
+
+def make_server_tree_kill_fault_injector(
+    *,
+    signal_name: str = "SIGKILL",
+    post_kill_wait_seconds: float = 0.0,
+    inter_kill_wait_seconds: float = 0.2,
+    worker_markers_extra: Sequence[str] | None = None,
+) -> FaultInjector:
+    """Kill current server root first, then kill all remaining descendants.
+
+    See :func:`make_server_root_kill_fault_injector` for ``worker_markers_extra``.
+    """
+
+    def _inject(server: Any) -> None:
+        _log_server_process_tree(server)
+        snapshot = _capture_server_fault_snapshot(server, worker_markers_extra=worker_markers_extra)
+        tree_pids = [int(pid) for pid in snapshot["tree_pids"]]
+        if not tree_pids:
+            pytest.skip("no server process tree found for tree-kill injection")
+
+        sig = getattr(signal, signal_name, None)
+        if sig is None:
+            raise ValueError(f"Unsupported signal_name: {signal_name}")
+
+        root_pid = tree_pids[0]
+        kill_order = [root_pid, *[pid for pid in tree_pids if pid != root_pid]]
+        for pid in kill_order:
+            if not psutil.pid_exists(pid):
+                continue
+            name, cmdline = _safe_proc_info(pid)
+            print(
+                f"[reliability][process-kill] tree-kill pid={pid} name={name} signal={signal_name} cmdline={cmdline}",
+                flush=True,
+            )
+            os.kill(pid, sig)
+            if inter_kill_wait_seconds > 0:
+                time.sleep(inter_kill_wait_seconds)
+
+        if post_kill_wait_seconds > 0:
+            time.sleep(post_kill_wait_seconds)
+
+    return _inject
+
+
 def make_process_kill_fault_injector(
    *,
    grep_patterns: str | Sequence[str],
@@ -493,11 +896,18 @@ def make_process_kill_fault_injector(
    limit: int = 1,
    post_kill_wait_seconds: float = 0.0,
 ) -> FaultInjector:
-    """Build a post-ready injector that kills processes matched by ``pgrep -f``.
+    """Build a post-ready injector that kills processes matched by ordered ``grep_patterns``.

-    Tries each pattern in order until at least one PID is killed. If none match,
-    the returned callable issues ``pytest.skip`` (same behavior as the previous
-    inline reliability test).
+    Matching is tried in two phases per pattern (first hit wins):
+
+    1. **Server process tree** (``psutil``): literal substring match against
+       ``Process.name()`` plus argv text. This matches how :func:`_safe_proc_info`
+       logs processes and catches vLLM-style titles (``VLLM::...``) that often do not
+       appear in ``pgrep -f``\'s command-line view.
+    2. **``pgrep -f``** (legacy): scoped to the server PID tree when it is known;
+       uses procps regular-expression rules for the pattern.
+
+    If neither phase finds a target, the returned callable issues ``pytest.skip``.

    Args:
        grep_patterns: One pattern or an ordered list of patterns.
@@ -514,9 +924,45 @@ def make_process_kill_fault_injector(
            logger.warning(
                "[reliability][process-kill] no server process tree found; fallback to global pgrep matching"
            )
+
+        def _kill_and_wait(filtered: list[int], pattern: str, *, source: str) -> None:
+            sig = getattr(signal, signal_name, None)
+            if sig is None:
+                raise ValueError(f"Unsupported signal_name: {signal_name}")
+            for pid in filtered:
+                name, cmdline = _safe_proc_info(pid)
+                print(
+                    f"[reliability][process-kill] killing pid={pid} name={name} signal={signal_name} "
+                    f"source={source} cmdline={cmdline}",
+                    flush=True,
+                )
+                os.kill(pid, sig)
+            print(
+                f"[reliability][process-kill] matched pattern={pattern!r} source={source} "
+                f"killed_pids={filtered} killed_count={len(filtered)}",
+                flush=True,
+            )
+            if post_kill_wait_seconds > 0:
+                print(
+                    f"[reliability][process-kill] waiting {post_kill_wait_seconds:.2f}s after kill",
+                    flush=True,
+                )
+                time.sleep(post_kill_wait_seconds)
+
        for pattern in patterns:
            print(
-                f"[reliability][process-kill] trying pattern={pattern} signal={signal_name} limit={limit}",
+                f"[reliability][process-kill] trying server_tree substring pattern={pattern!r} "
+                f"signal={signal_name} limit={limit}",
+                flush=True,
+            )
+            tree_hits = _pids_in_server_tree_substring_match(server, pattern, limit=limit)
+            if tree_hits:
+                _kill_and_wait(tree_hits, pattern, source="server_tree")
+                return
+
+        for pattern in patterns:
+            print(
+                f"[reliability][process-kill] trying pgrep -f pattern={pattern!r} signal={signal_name} limit={limit}",
                flush=True,
            )
            pids = inject_process_kill(
@@ -535,26 +981,7 @@ def make_process_kill_fault_injector(
                )
                continue
            if filtered:
-                sig = getattr(signal, signal_name, None)
-                if sig is None:
-                    raise ValueError(f"Unsupported signal_name: {signal_name}")
-                for pid in filtered:
-                    name, cmdline = _safe_proc_info(pid)
-                    print(
-                        f"[reliability][process-kill] killing pid={pid} name={name} signal={signal_name} cmdline={cmdline}",
-                        flush=True,
-                    )
-                    os.kill(pid, sig)
-                print(
-                    f"[reliability][process-kill] matched pattern={pattern} killed_pids={filtered} killed_count={len(filtered)}",
-                    flush=True,
-                )
-                if post_kill_wait_seconds > 0:
-                    print(
-                        f"[reliability][process-kill] waiting {post_kill_wait_seconds:.2f}s after kill",
-                        flush=True,
-                    )
-                    time.sleep(post_kill_wait_seconds)
+                _kill_and_wait(filtered, pattern, source="pgrep")
                return
        logger.warning(
            "[reliability][process-kill] no process matched patterns=%s signal=%s limit=%s",
@@ -21,13 +21,19 @@ from tests.dfx.conftest import (
    resolve_oom_device_spec,
 )
 from tests.dfx.reliability.helpers import (
+    FaultInjector,
+    assert_no_server_tree_process_residual_and_gpu_release,
    extract_openai_error_contract_from_bytes,
    get_health_raw,
    inject_gpu_oom,
    make_process_kill_fault_injector,
+    make_server_root_kill_fault_injector,
+    make_server_tree_kill_fault_injector,
    post_chat_completions_raw,
    post_json_raw,
+    run_fault_injection_with_rate_load,
    stop_gpu_oom_hogs,
+    worker_residual_timeout_after_kill_signal,
 )
 from tests.helpers.mark import hardware_test
 from tests.helpers.media import generate_synthetic_audio, generate_synthetic_image, generate_synthetic_video
@@ -97,7 +103,73 @@ FAULT_ERROR_KEYWORDS = (
    "500",
    "503",
 )
-RUNTIME_WORKER_PATTERN = "VLLM::Worker"
+PROCESS_KILL_ERROR_KEYWORDS = (
+    "timeout",
+    "did not complete within",
+    "connection",
+    "engine",
+    "orchestrator",
+    "dead",
+    "internal",
+    "500",
+    "503",
+)
+# Prefix match for vLLM-titled children (``VLLM::Worker``, ``VLLM::StageEngineCoreProc_*``, …).
+# Must be a literal substring of ``name``/``cmdline`` — no trailing whitespace (would break match).
+RUNTIME_WORKER_PATTERN = "VLLM::"
+# RFC#2366 signal x target matrix:
+# - worker: SIGTERM, SIGKILL
+# - serve-root no-load: SIGINT; SIGTERM/SIGKILL skipped (vllm issue#43060)
+# - serve-root with-load: SIGINT; SIGTERM skipped (issue#3683); SIGKILL skipped (issue#43060)
+# - serve-tree no-load: SIGTERM/SIGKILL
+# - serve-tree with-load: SIGKILL; SIGTERM skipped (issue#3683)
+_SERVE_ROOT_NON_SIGINT_SKIP = pytest.mark.skip(reason="vllm issue#43060")
+_WITH_LOAD_SIGTERM_SKIP = pytest.mark.skip(reason="issue#3683")
+SERVE_SIGNAL_PARAMS = [
+    pytest.param("SIGTERM", id="sigterm"),
+    pytest.param("SIGINT", id="sigint"),
+    pytest.param("SIGKILL", id="sigkill", marks=_SERVE_ROOT_NON_SIGINT_SKIP),
+]
+SERVE_WITH_LOAD_SIGNAL_PARAMS = [
+    pytest.param("SIGTERM", id="sigterm", marks=_WITH_LOAD_SIGTERM_SKIP),
+    pytest.param("SIGINT", id="sigint", marks=_WITH_LOAD_SIGTERM_SKIP),
+    pytest.param("SIGKILL", id="sigkill", marks=_WITH_LOAD_SIGTERM_SKIP),
+]
+TREE_SIGNAL_PARAMS = [
+    pytest.param("SIGTERM", id="sigterm"),
+    pytest.param("SIGKILL", id="sigkill"),
+]
+TREE_WITH_LOAD_SIGNAL_PARAMS = [
+    pytest.param("SIGTERM", id="sigterm", marks=_WITH_LOAD_SIGTERM_SKIP),
+    pytest.param("SIGKILL", id="sigkill"),
+]
+WORKER_SIGNAL_FAULT_PARAMS = [
+    pytest.param(
+        make_process_kill_fault_injector(
+            grep_patterns=RUNTIME_WORKER_PATTERN,
+            signal_name="SIGTERM",
+            limit=1,
+            post_kill_wait_seconds=2.0,
+        ),
+        id="runtime_process_chain_sigterm",
+    ),
+    pytest.param(
+        make_process_kill_fault_injector(
+            grep_patterns=RUNTIME_WORKER_PATTERN,
+            signal_name="SIGKILL",
+            limit=1,
+            post_kill_wait_seconds=2.0,
+        ),
+        id="runtime_process_chain_sigkill",
+    ),
+]
+INFLIGHT_INJECTION_REQUEST_RATE = 0.3
+INFLIGHT_INJECTION_REQUEST_COUNT = 10
+
+# Post-fault chat probe: ``MAX_ELAPSED`` aligns with HTTP read timeout; assert allows +1s jitter
+# when the client raises at timeout (e.g. ~30.03s).
+_PROCESS_KILL_WORKER_CHAT_FF_HTTP_TIMEOUT_SEC = 30
+_PROCESS_KILL_WORKER_CHAT_FF_MAX_ELAPSED_SEC = 30


 class _HasServeArgs(Protocol):
@@ -194,6 +266,55 @@ def _looks_like_server_unreachable(exc: BaseException) -> bool:
    return "connection refused" in msg or "actively refused" in msg


+def _chat_request_config(omni_server: _HasServeArgs) -> dict[str, Any]:
+    messages = dummy_messages_from_mix_data(
+        system_prompt=_get_system_prompt(),
+        content_text="What is the capital of China? Answer in one short sentence.",
+    )
+    return {
+        "model": omni_server.model,
+        "messages": messages,
+        "stream": False,
+        "modalities": ["text"],
+    }
+
+
+def _assert_post_fault_chat_fast_fail(host: str, port: int, *, model: str, scenario: str) -> None:
+    payload = {
+        "model": model,
+        "messages": [{"role": "user", "content": "post-fault fast-fail check"}],
+        "stream": False,
+        "modalities": ["text"],
+    }
+    start = time.monotonic()
+    try:
+        status, body = post_json_raw(host, port, "/v1/chat/completions", payload, timeout_sec=20)
+        elapsed = time.monotonic() - start
+        assert elapsed < 15, f"[{scenario} fast_fail] /v1/chat/completions did not fail fast: {elapsed:.2f}s"
+        assert status >= 500, (
+            f"[{scenario} fast_fail] expected server-side error after fault, got status={status}, body={body[:200]!r}"
+        )
+    except Exception:  # noqa: BLE001
+        elapsed = time.monotonic() - start
+        assert elapsed < 15, f"[{scenario} fast_fail] exception was too slow after fault: {elapsed:.2f}s"
+
+
+def _assert_post_fault_health_terminal(host: str, port: int, *, scenario: str) -> None:
+    deadline = time.monotonic() + 20.0
+    last_observation = ""
+    while time.monotonic() < deadline:
+        try:
+            status, body = get_health_raw(host, port, timeout_sec=5)
+            last_observation = f"http={status}, body={body[:200]!r}"
+            if status == 503:
+                return
+        except Exception as exc:  # noqa: BLE001
+            last_observation = f"exception={exc!r}"
+            return
+        time.sleep(0.5)
+    pytest.fail(f"[{scenario} health] no terminal post-fault health observed: {last_observation}")
+
+
 QWEN_PARAMS = create_reliability_omni_server_params(RELIABILITY_SCENARIOS, DEPLOY_CONFIGS_DIR)


@@ -354,17 +475,7 @@ def test_reliability_fault_gpu_oom_concurrent_pressure_failure(omni_server_funct
@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
@pytest.mark.parametrize(
    "fault_injector",
-    [
-        pytest.param(
-            make_process_kill_fault_injector(
-                grep_patterns="VLLM::Worker",
-                signal_name="SIGKILL",
-                limit=1,
-                post_kill_wait_seconds=2.0,
-            ),
-            id="runtime_process_chain",
-        ),
-    ],
+    WORKER_SIGNAL_FAULT_PARAMS,
    indirect=True,
 )
@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
@@ -385,7 +496,7 @@ def test_reliability_fault_process_kill_request_failure(
    try:
        openai_client_function.send_omni_request(request_config, request_num=1)
    except Exception as exc:
-        assert_fault_exception(exc, FAULT_ERROR_KEYWORDS)
+        assert_fault_exception(exc, PROCESS_KILL_ERROR_KEYWORDS)
    else:
        pytest.fail("expected request failure after process-kill injection")

@@ -394,48 +505,22 @@ def test_reliability_fault_process_kill_request_failure(
@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
@pytest.mark.parametrize(
    "fault_injector",
-    [
-        pytest.param(
-            make_process_kill_fault_injector(
-                grep_patterns="VLLM::Worker",
-                signal_name="SIGKILL",
-                limit=1,
-                post_kill_wait_seconds=2.0,
-            ),
-            id="runtime_process_chain",
-        ),
-    ],
+    WORKER_SIGNAL_FAULT_PARAMS,
    indirect=True,
 )
@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
 def test_reliability_fault_process_kill_health_fast_fail_and_concurrent(
    omni_server_after_fault_function,
 ) -> None:
-    """Black-box: after worker SIGKILL, /health→503, chat fails fast, concurrent chat does not hang."""
+    """Black-box: after worker process kill, chat fails fast, concurrent chat does not hang; /health→503 last.
+
+    ``SIGTERM`` on a stage engine often exceeds a 15s client-visible path; ``SIGKILL`` is
+    typically quicker. See module constants ``_PROCESS_KILL_WORKER_CHAT_FF_*``.
+    """
    host = omni_server_after_fault_function.host
    port = omni_server_after_fault_function.port
    model = omni_server_after_fault_function.model

-    deadline = time.monotonic() + 20.0
-    last_observation = ""
-    saw_503 = False
-    health_final_status: int | None = None
-    health_final_body = b""
-    while time.monotonic() < deadline:
-        try:
-            status, body = get_health_raw(host, port, timeout_sec=5)
-            last_observation = f"http={status}, body={body[:200]!r}"
-            health_final_status, health_final_body = status, body
-            if status == 503:
-                saw_503 = True
-                break
-        except Exception as exc:  # noqa: BLE001
-            last_observation = f"exception={exc!r}"
-        time.sleep(0.5)
-    assert saw_503, (
-        f"[process_kill health] expected /health to become 503 after fault injection, got {last_observation}"
-    )
-
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": "Say hello in one short sentence."}],
@@ -444,20 +529,28 @@ def test_reliability_fault_process_kill_health_fast_fail_and_concurrent(
    }
    ff_status: int | None = None
    ff_body = b""
-    ff_exc: BaseException | None = None
    start = time.monotonic()
    try:
-        ff_status, ff_body = post_json_raw(host, port, "/v1/chat/completions", payload, timeout_sec=20)
+        ff_status, ff_body = post_json_raw(
+            host,
+            port,
+            "/v1/chat/completions",
+            payload,
+            timeout_sec=_PROCESS_KILL_WORKER_CHAT_FF_HTTP_TIMEOUT_SEC,
+        )
        elapsed = time.monotonic() - start
-        assert elapsed < 15, f"[process_kill fast_fail] request did not fail fast after fault: {elapsed:.2f}s"
+        assert elapsed <= _PROCESS_KILL_WORKER_CHAT_FF_MAX_ELAPSED_SEC + 1.0, (
+            f"[process_kill fast_fail] request did not fail fast after fault: {elapsed:.2f}s"
+        )
        assert ff_status >= 500, (
            f"[process_kill fast_fail] expected server-side failure after fault, "
            f"got status={ff_status}, body={ff_body[:200]!r}"
        )
-    except Exception as exc:
-        ff_exc = exc
+    except Exception:
        elapsed = time.monotonic() - start
-        assert elapsed < 15, f"[process_kill fast_fail] request exception was too slow after fault: {elapsed:.2f}s"
+        assert elapsed <= _PROCESS_KILL_WORKER_CHAT_FF_MAX_ELAPSED_SEC + 1.0, (
+            f"[process_kill fast_fail] request exception was too slow after fault: {elapsed:.2f}s"
+        )

    payload_json = json.dumps(
        {
@@ -490,32 +583,195 @@ def test_reliability_fault_process_kill_health_fast_fail_and_concurrent(
    assert elapsed < 30, f"[process_kill concurrent] fault-time request convergence is too slow: {elapsed:.2f}s"

    fault_observed = False
-    conc_debug: list[Any] = []
    for future in done:
        try:
            status, body = future.result()
-            conc_debug.append((status, body[:200]))
            if status >= 500:
                fault_observed = True
        except Exception as exc:
-            conc_debug.append(repr(exc))
+            assert_fault_exception(exc, PROCESS_KILL_ERROR_KEYWORDS)
            fault_observed = True
-    # DEBUG: remove before merge
-    print(
-        health_final_status,
-        health_final_body[:200],
-        ff_status,
-        ff_body[:200],
-        ff_exc,
-        conc_debug,
-    )
    assert fault_observed, (
        "[process_kill concurrent] expected at least one request to fail after process-kill fault injection"
    )

+    deadline = time.monotonic() + 20.0
+    last_observation = ""
+    final_health_status: int | None = None
+    while time.monotonic() < deadline:
+        try:
+            status, body = get_health_raw(host, port, timeout_sec=5)
+            last_observation = f"http={status}, body={body[:200]!r}"
+            final_health_status = status
+            if status == 503:
+                break
+        except Exception as exc:  # noqa: BLE001
+            last_observation = f"exception={exc!r}"
+        time.sleep(0.5)
+    if final_health_status != 503:
+        pytest.skip("issue#3050")
+    assert final_health_status == 503, (
+        "[process_kill health] expected /health 503 after fatal fault, "
+        f"got status={final_health_status}, last_observation={last_observation}"
+    )
+

@pytest.mark.slow
-@pytest.mark.skip(reason="issue#2327")
+@pytest.mark.skip(reason="issue#3683")
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize(
+    "fault_injector",
+    WORKER_SIGNAL_FAULT_PARAMS,
+    indirect=True,
+)
+@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_worker_with_load_request_failure(
+    omni_server_function,
+    openai_client_function,
+    fault_injector: FaultInjector,
+) -> None:
+    request_config = _chat_request_config(omni_server_function)
+    scenario = "kill_worker_with_load"
+    load_result = run_fault_injection_with_rate_load(
+        submit_request=lambda: openai_client_function.send_omni_request(request_config, request_num=1),
+        inject_fault=lambda: fault_injector(omni_server_function),
+        num_requests=INFLIGHT_INJECTION_REQUEST_COUNT,
+        request_rate=INFLIGHT_INJECTION_REQUEST_RATE,
+        completion_timeout_sec=120.0,
+    )
+    assert load_result["failure_observed"], (
+        f"[{scenario}] expected at least one load request failure after fault; load_result={load_result}"
+    )
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_chat_fast_fail(host, port, model=omni_server_function.model, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+
+
+@pytest.mark.slow
+@hardware_test(res={"cuda": "H100"}, num_cards=2)
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", SERVE_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_serve_root_no_load_fast_fail_and_cleanup(
+    omni_server_function,
+    signal_name: str,
+) -> None:
+    """Black-box: kill serve root without load; verify fast-fail/health/cleanup."""
+    scenario = f"kill_serve_root_no_load_{signal_name.lower()}"
+    injector = make_server_root_kill_fault_injector(signal_name=signal_name, post_kill_wait_seconds=2.0)
+    injector(omni_server_function)
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_chat_fast_fail(host, port, model=omni_server_function.model, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
+
+
+@pytest.mark.slow
+@hardware_test(res={"cuda": "H100"}, num_cards=2)
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", SERVE_WITH_LOAD_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_serve_root_with_load_fast_fail_and_cleanup(
+    omni_server_function,
+    openai_client_function,
+    signal_name: str,
+) -> None:
+    request_config = _chat_request_config(omni_server_function)
+    scenario = f"kill_serve_root_with_load_{signal_name.lower()}"
+    injector = make_server_root_kill_fault_injector(signal_name=signal_name, post_kill_wait_seconds=2.0)
+    load_result = run_fault_injection_with_rate_load(
+        submit_request=lambda: openai_client_function.send_omni_request(request_config, request_num=1),
+        inject_fault=lambda: injector(omni_server_function),
+        num_requests=INFLIGHT_INJECTION_REQUEST_COUNT,
+        request_rate=INFLIGHT_INJECTION_REQUEST_RATE,
+        completion_timeout_sec=300.0,
+    )
+    assert load_result["failure_observed"], (
+        f"[{scenario}] expected at least one load request failure after fault; load_result={load_result}"
+    )
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_chat_fast_fail(host, port, model=omni_server_function.model, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
+
+
+@pytest.mark.slow
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", TREE_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_tree_no_load_fast_fail_and_cleanup(
+    omni_server_function,
+    signal_name: str,
+) -> None:
+    """Black-box: kill server tree without load; verify fast-fail/health/cleanup."""
+    scenario = f"kill_serve_tree_no_load_{signal_name.lower()}"
+    injector = make_server_tree_kill_fault_injector(
+        signal_name=signal_name,
+        post_kill_wait_seconds=2.0,
+        inter_kill_wait_seconds=0.1,
+    )
+    injector(omni_server_function)
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_chat_fast_fail(host, port, model=omni_server_function.model, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
+
+
+@pytest.mark.slow
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", TREE_WITH_LOAD_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_tree_with_load_fast_fail_and_cleanup(
+    omni_server_function,
+    openai_client_function,
+    signal_name: str,
+) -> None:
+    request_config = _chat_request_config(omni_server_function)
+    scenario = f"kill_serve_tree_with_load_{signal_name.lower()}"
+    injector = make_server_tree_kill_fault_injector(
+        signal_name=signal_name,
+        post_kill_wait_seconds=2.0,
+        inter_kill_wait_seconds=0.1,
+    )
+    load_result = run_fault_injection_with_rate_load(
+        submit_request=lambda: openai_client_function.send_omni_request(request_config, request_num=1),
+        inject_fault=lambda: injector(omni_server_function),
+        num_requests=INFLIGHT_INJECTION_REQUEST_COUNT,
+        request_rate=INFLIGHT_INJECTION_REQUEST_RATE,
+        completion_timeout_sec=120.0,
+    )
+    assert load_result["failure_observed"], (
+        f"[{scenario}] expected at least one load request failure after fault; load_result={load_result}"
+    )
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_chat_fast_fail(host, port, model=omni_server_function.model, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
+
+
+@pytest.mark.slow
+@pytest.mark.skip(reason="OOM injection causes the serving instance to exit")
@hardware_test(res={"cuda": "H100"}, num_cards=2)
@pytest.mark.parametrize("omni_server_function", QWEN_PARAMS, indirect=True)
 def test_reliability_fault_gpu_oom_state_converges_after_fault_removed(
@@ -20,28 +20,25 @@ from tests.dfx.conftest import (
    supports_video_generation,
 )
 from tests.dfx.reliability.helpers import (
+    FaultInjector,
+    assert_no_server_tree_process_residual_and_gpu_release,
    get_health_raw,
    inject_gpu_oom,
    make_process_kill_fault_injector,
+    make_server_root_kill_fault_injector,
+    make_server_tree_kill_fault_injector,
+    run_fault_injection_with_rate_load,
    stop_gpu_oom_hogs,
+    worker_residual_timeout_after_kill_signal,
 )
 from tests.helpers.mark import hardware_test
 from tests.helpers.media import generate_synthetic_image

 RELIABILITY_SCENARIOS: list[dict[str, Any]] = [
    {
-        "test_name": "wan22_t2v_reliability_default",
+        "test_name": "wan22_i2v_reliability_default",
        "server_params": {
-            "model": "Wan-AI/Wan2.2-T2V-A14B-Diffusers",
-            "server_args": [
-                "--num-gpus",
-                "1",
-                "--boundary-ratio",
-                "0.875",
-                "--flow-shift",
-                "5.0",
-                "--disable-log-stats",
-            ],
+            "model": "Wan-AI/Wan2.2-I2V-A14B-Diffusers",
        },
    }
 ]
@@ -73,9 +70,114 @@ PROCESS_KILL_ERROR_KEYWORDS = (
    "500",
    "503",
 )
+# RFC#2366 signal x target matrix (aligned with test_reliability_qwen3_omni.py):
+# - worker: SIGTERM, SIGKILL
+# - serve-root: SIGTERM, SIGINT; SIGKILL skipped (issue#3725)
+# - serve-tree no-load: SIGTERM, SIGKILL
+# - serve-tree with-load: SIGTERM, SIGKILL
+_SERVE_ROOT_SIGKILL_SKIP = pytest.mark.skip(reason="issue#3725")
+SERVE_SIGNAL_PARAMS = [
+    pytest.param("SIGTERM", id="sigterm"),
+    pytest.param("SIGINT", id="sigint"),
+    pytest.param("SIGKILL", id="sigkill", marks=_SERVE_ROOT_SIGKILL_SKIP),
+]
+TREE_SIGNAL_PARAMS = [
+    pytest.param("SIGTERM", id="sigterm"),
+    pytest.param("SIGKILL", id="sigkill"),
+]
+TREE_WITH_LOAD_SIGNAL_PARAMS = [
+    pytest.param("SIGTERM", id="sigterm"),
+    pytest.param("SIGKILL", id="sigkill"),
+]
+WORKER_SIGNAL_FAULT_PARAMS = [
+    pytest.param(
+        make_process_kill_fault_injector(
+            grep_patterns="vLLM-Omni::",
+            signal_name="SIGTERM",
+            limit=1,
+            post_kill_wait_seconds=2.0,
+        ),
+        id="runtime_process_chain_sigterm",
+    ),
+    pytest.param(
+        make_process_kill_fault_injector(
+            grep_patterns="vLLM-Omni::",
+            signal_name="SIGKILL",
+            limit=1,
+            post_kill_wait_seconds=2.0,
+        ),
+        id="runtime_process_chain_sigkill",
+    ),
+]

 WAN_PARAMS = create_reliability_omni_server_params(RELIABILITY_SCENARIOS, E2E_STAGE_CONFIGS_DIR)
 DIFFUSION_VIDEO_PARAMS = [param for param in WAN_PARAMS if supports_video_generation(param.model)]
+INFLIGHT_INJECTION_REQUEST_RATE = 0.3
+INFLIGHT_INJECTION_REQUEST_COUNT = 10
+
+
+def _video_request_config() -> dict[str, Any]:
+    image_data_url = f"data:image/jpeg;base64,{generate_synthetic_image(1280, 720)['base64']}"
+    return {
+        "form_data": {
+            "prompt": "Generate a realistic road-driving video with camera motion.",
+            "width": 512,
+            "height": 512,
+            "fps": 8,
+            "num_frames": 8,
+            "guidance_scale": 1.0,
+            "flow_shift": 5.0,
+            "num_inference_steps": 4,
+            "seed": 42,
+        },
+        "image_reference": image_data_url,
+        "stream": False,
+    }
+
+
+def _assert_post_fault_video_fast_fail(host: str, port: int, *, scenario: str) -> None:
+    url = f"http://{host}:{port}/v1/videos"
+    payload = {
+        "prompt": "post-fault fast-fail check",
+        "width": "512",
+        "height": "512",
+        "fps": "8",
+        "num_frames": "8",
+        "num_inference_steps": "4",
+    }
+    start = time.monotonic()
+    try:
+        response = requests.post(
+            url,
+            data=payload,
+            headers={"Accept": "application/json"},
+            timeout=20,
+        )
+        elapsed = time.monotonic() - start
+        assert elapsed < 15, f"[{scenario} fast_fail] /v1/videos did not fail fast: {elapsed:.2f}s"
+        assert response.status_code >= 500, (
+            f"[{scenario} fast_fail] expected server-side error after fault, "
+            f"got status={response.status_code}, body={response.text[:200]!r}"
+        )
+    except Exception:  # noqa: BLE001
+        elapsed = time.monotonic() - start
+        assert elapsed < 15, f"[{scenario} fast_fail] exception was too slow after fault: {elapsed:.2f}s"
+
+
+def _assert_post_fault_health_terminal(host: str, port: int, *, scenario: str) -> None:
+    deadline = time.monotonic() + 20.0
+    last_observation = ""
+    while time.monotonic() < deadline:
+        try:
+            status, body = get_health_raw(host, port, timeout_sec=5)
+            last_observation = f"http={status}, body={body[:200]!r}"
+            if status == 503:
+                return
+        except Exception as exc:  # noqa: BLE001
+            last_observation = f"exception={exc!r}"
+            return
+        time.sleep(0.5)
+    pytest.fail(f"[{scenario} health] no terminal post-fault health observed: {last_observation}")


@pytest.mark.slow
@@ -122,40 +224,15 @@ def test_reliability_fault_gpu_oom_video_large_request_failure(omni_server_funct
@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
@pytest.mark.parametrize(
    "fault_injector",
-    [
-        pytest.param(
-            make_process_kill_fault_injector(
-                grep_patterns="multiprocessing.spawn",
-                signal_name="SIGKILL",
-                limit=1,
-                post_kill_wait_seconds=2.0,
-            ),
-            id="runtime_process_chain",
-        ),
-    ],
+    WORKER_SIGNAL_FAULT_PARAMS,
    indirect=True,
 )
@pytest.mark.parametrize("omni_server_function", DIFFUSION_VIDEO_PARAMS, indirect=True)
-def test_reliability_fault_process_kill_video_request_failure(
+def test_reliability_fault_process_kill_request_failure(
    omni_server_after_fault_function,
    openai_client_function,
 ) -> None:
-    image_data_url = f"data:image/jpeg;base64,{generate_synthetic_image(1280, 720)['base64']}"
-    request_config = {
-        "form_data": {
-            "prompt": "Generate a realistic road-driving video with camera motion.",
-            "width": 512,
-            "height": 512,
-            "fps": 8,
-            "num_frames": 8,
-            "guidance_scale": 1.0,
-            "flow_shift": 5.0,
-            "num_inference_steps": 4,
-            "seed": 42,
-        },
-        "image_reference": image_data_url,
-        "stream": False,
-    }
+    request_config = _video_request_config()
    try:
        openai_client_function.send_video_diffusion_request(request_config, request_num=1)
    except Exception as exc:
@@ -168,21 +245,11 @@ def test_reliability_fault_process_kill_video_request_failure(
@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
@pytest.mark.parametrize(
    "fault_injector",
-    [
-        pytest.param(
-            make_process_kill_fault_injector(
-                grep_patterns="multiprocessing.spawn",
-                signal_name="SIGKILL",
-                limit=1,
-                post_kill_wait_seconds=2.0,
-            ),
-            id="runtime_process_chain",
-        ),
-    ],
+    WORKER_SIGNAL_FAULT_PARAMS,
    indirect=True,
 )
@pytest.mark.parametrize("omni_server_function", DIFFUSION_VIDEO_PARAMS, indirect=True)
-def test_reliability_fault_process_kill_video_health_fast_fail_and_concurrent(
+def test_reliability_fault_process_kill_health_fast_fail_and_concurrent(
    omni_server_after_fault_function,
    openai_client_function,
 ) -> None:
@@ -290,6 +357,37 @@ def test_reliability_fault_process_kill_video_health_fast_fail_and_concurrent(
    )


+@pytest.mark.slow
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize(
+    "fault_injector",
+    WORKER_SIGNAL_FAULT_PARAMS,
+    indirect=True,
+)
+@pytest.mark.parametrize("omni_server_function", DIFFUSION_VIDEO_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_worker_with_load_request_failure(
+    omni_server_function,
+    openai_client_function,
+    fault_injector: FaultInjector,
+) -> None:
+    request_config = _video_request_config()
+    scenario = "kill_worker_with_load"
+    load_result = run_fault_injection_with_rate_load(
+        submit_request=lambda: openai_client_function.send_video_diffusion_request(request_config, request_num=1),
+        inject_fault=lambda: fault_injector(omni_server_function),
+        num_requests=INFLIGHT_INJECTION_REQUEST_COUNT,
+        request_rate=INFLIGHT_INJECTION_REQUEST_RATE,
+        completion_timeout_sec=120.0,
+    )
+    assert load_result["failure_observed"], (
+        f"[{scenario}] expected at least one load request failure after fault; load_result={load_result}"
+    )
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_video_fast_fail(host, port, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+
+
@pytest.mark.slow
@hardware_test(res={"cuda": "H100"}, num_cards=1)
@pytest.mark.skip(reason="issue#2327")
@@ -409,3 +507,127 @@ def test_reliability_video_oom_recovers_after_fault_removed(
            "unhealthy terminal state should fail fast on /v1/videos, "
            f"got health={terminal_health}, request_status={recovery_status}, body={recovery_text_prefix!r}"
        )
+
+
+@pytest.mark.slow
+@hardware_test(res={"cuda": "H100"}, num_cards=1)
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", SERVE_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", DIFFUSION_VIDEO_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_serve_root_with_load_fast_fail_and_cleanup(
+    omni_server_function,
+    openai_client_function,
+    signal_name: str,
+) -> None:
+    """Black-box: during in-flight video requests, kill serve root and verify post-fault behavior/cleanup."""
+    request_config = _video_request_config()
+    scenario = f"kill_serve_root_with_load_{signal_name.lower()}"
+    injector = make_server_root_kill_fault_injector(signal_name=signal_name, post_kill_wait_seconds=2.0)
+    load_result = run_fault_injection_with_rate_load(
+        submit_request=lambda: openai_client_function.send_video_diffusion_request(request_config, request_num=1),
+        inject_fault=lambda: injector(omni_server_function),
+        num_requests=INFLIGHT_INJECTION_REQUEST_COUNT,
+        request_rate=INFLIGHT_INJECTION_REQUEST_RATE,
+        completion_timeout_sec=120.0,
+    )
+    assert load_result["failure_observed"], (
+        f"[{scenario}] expected at least one load request failure after fault; load_result={load_result}"
+    )
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_video_fast_fail(host, port, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
+
+
+@pytest.mark.slow
+@hardware_test(res={"cuda": "H100"}, num_cards=1)
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", SERVE_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", DIFFUSION_VIDEO_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_serve_root_no_load_fast_fail_and_cleanup(
+    omni_server_function,
+    signal_name: str,
+) -> None:
+    """Black-box: kill serve root without load; verify fast-fail/health/cleanup."""
+    scenario = f"kill_serve_root_no_load_{signal_name.lower()}"
+    injector = make_server_root_kill_fault_injector(signal_name=signal_name, post_kill_wait_seconds=2.0)
+    injector(omni_server_function)
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_video_fast_fail(host, port, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
+
+
+@pytest.mark.slow
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", TREE_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", DIFFUSION_VIDEO_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_tree_no_load_fast_fail_and_cleanup(
+    omni_server_function,
+    signal_name: str,
+) -> None:
+    """Black-box: kill server tree without load; verify fast-fail/health/cleanup."""
+    scenario = f"kill_serve_tree_no_load_{signal_name.lower()}"
+    injector = make_server_tree_kill_fault_injector(
+        signal_name=signal_name,
+        post_kill_wait_seconds=2.0,
+        inter_kill_wait_seconds=0.1,
+    )
+    injector(omni_server_function)
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_video_fast_fail(host, port, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
+
+
+@pytest.mark.slow
+@pytest.mark.skipif(os.name == "nt", reason="process-kill injection helper is POSIX-only")
+@pytest.mark.parametrize("signal_name", TREE_WITH_LOAD_SIGNAL_PARAMS)
+@pytest.mark.parametrize("omni_server_function", DIFFUSION_VIDEO_PARAMS, indirect=True)
+def test_reliability_fault_process_kill_tree_with_load_fast_fail_and_cleanup(
+    omni_server_function,
+    openai_client_function,
+    signal_name: str,
+) -> None:
+    """Black-box: during in-flight video requests, kill server process tree and verify post-fault behavior/cleanup."""
+    request_config = _video_request_config()
+    scenario = f"kill_serve_tree_with_load_{signal_name.lower()}"
+    injector = make_server_tree_kill_fault_injector(
+        signal_name=signal_name,
+        post_kill_wait_seconds=2.0,
+        inter_kill_wait_seconds=0.1,
+    )
+    load_result = run_fault_injection_with_rate_load(
+        submit_request=lambda: openai_client_function.send_video_diffusion_request(request_config, request_num=1),
+        inject_fault=lambda: injector(omni_server_function),
+        num_requests=INFLIGHT_INJECTION_REQUEST_COUNT,
+        request_rate=INFLIGHT_INJECTION_REQUEST_RATE,
+        completion_timeout_sec=120.0,
+    )
+    assert load_result["failure_observed"], (
+        f"[{scenario}] expected at least one load request failure after fault; load_result={load_result}"
+    )
+    host = omni_server_function.host
+    port = omni_server_function.port
+    _assert_post_fault_video_fast_fail(host, port, scenario=scenario)
+    _assert_post_fault_health_terminal(host, port, scenario=scenario)
+    assert_no_server_tree_process_residual_and_gpu_release(
+        omni_server_function,
+        scenario=scenario,
+        timeout_sec=worker_residual_timeout_after_kill_signal(signal_name),
+    )
@@ -0,0 +1,131 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+import importlib
+import sys
+import types
+
+import pytest
+import torch
+from vllm.platforms import current_platform
+
+SAGE_ATTN3_MODULE = "vllm_omni.diffusion.attention.backends.sage_attn3"
+
+
+def load_sage_attn3_module(monkeypatch: pytest.MonkeyPatch, kernel_impl):
+    fake_module = types.ModuleType("sageattn3")
+    fake_module.sageattn3_blackwell = kernel_impl
+    monkeypatch.setitem(sys.modules, "sageattn3", fake_module)
+    sys.modules.pop(SAGE_ATTN3_MODULE, None)
+    return importlib.import_module(SAGE_ATTN3_MODULE)
+
+
+def test_sage_attn3_forward_uses_blackwell_layout(monkeypatch: pytest.MonkeyPatch):
+    calls = {}
+
+    def fake_kernel(query, key, value, is_causal=False):
+        calls["query_shape"] = query.shape
+        calls["is_causal"] = is_causal
+        return query + key + value
+
+    backend_module = load_sage_attn3_module(monkeypatch, fake_kernel)
+    impl = backend_module.SageAttention3Impl(
+        num_heads=4,
+        head_size=64,
+        softmax_scale=1.0 / 8.0,
+        causal=False,
+    )
+
+    query = torch.randn(2, 8, 4, 64)
+    key = torch.randn(2, 8, 4, 64)
+    value = torch.randn(2, 8, 4, 64)
+
+    output = impl.forward_cuda(query, key, value)
+
+    assert calls["query_shape"] == (2, 4, 8, 64)
+    assert calls["is_causal"] is False
+    expected = (query.transpose(1, 2) + key.transpose(1, 2) + value.transpose(1, 2)).transpose(1, 2)
+    assert torch.allclose(output, expected)
+
+
+def test_sage_attn3_falls_back_to_sdpa_for_gqa(monkeypatch: pytest.MonkeyPatch):
+    def fake_kernel(*args, **kwargs):
+        raise AssertionError("sageattn3_blackwell should not be used for GQA")
+
+    backend_module = load_sage_attn3_module(monkeypatch, fake_kernel)
+    sdpa_calls = {}
+
+    def fake_sdpa(query, key, value, **kwargs):
+        sdpa_calls["query_shape"] = query.shape
+        sdpa_calls["key_shape"] = key.shape
+        sdpa_calls["enable_gqa"] = kwargs["enable_gqa"]
+        return query + 1
+
+    monkeypatch.setattr(backend_module.F, "scaled_dot_product_attention", fake_sdpa)
+
+    impl = backend_module.SageAttention3Impl(
+        num_heads=4,
+        head_size=64,
+        softmax_scale=1.0 / 8.0,
+        causal=False,
+    )
+
+    query = torch.randn(2, 8, 4, 64)
+    key = torch.randn(2, 8, 2, 64)
+    value = torch.randn(2, 8, 2, 64)
+
+    output = impl.forward_cuda(query, key, value)
+
+    assert sdpa_calls["query_shape"] == (2, 4, 8, 64)
+    assert sdpa_calls["key_shape"] == (2, 2, 8, 64)
+    assert sdpa_calls["enable_gqa"] is True
+    expected = (query.permute(0, 2, 1, 3) + 1).permute(0, 2, 1, 3)
+    assert torch.allclose(output, expected)
+
+
+@pytest.mark.skipif(not current_platform.is_cuda(), reason="sage_attn3 tests require CUDA platform")
+def test_cuda_platform_selects_sage_attn3_alias(monkeypatch: pytest.MonkeyPatch):
+    from vllm.platforms.interface import DeviceCapability
+
+    from vllm_omni.diffusion.attention.backends.registry import DiffusionAttentionBackendEnum
+    from vllm_omni.diffusion.envs import PACKAGES_CHECKER
+    from vllm_omni.platforms.cuda import platform as cuda_platform_module
+    from vllm_omni.platforms.cuda.platform import CudaOmniPlatform
+
+    original_import_module = importlib.import_module
+
+    monkeypatch.setattr(
+        CudaOmniPlatform,
+        "get_device_capability",
+        classmethod(lambda cls, device_id=0: DeviceCapability(10, 0)),
+    )
+    monkeypatch.setattr(PACKAGES_CHECKER, "get_packages_info", lambda: {"has_flash_attn": False})
+    monkeypatch.setattr(
+        cuda_platform_module.importlib,
+        "import_module",
+        lambda module_name: object() if module_name == "sageattn3" else original_import_module(module_name),
+    )
+
+    backend_path = CudaOmniPlatform.get_diffusion_attn_backend_cls("SAGE_ATTN_3", head_size=64)
+
+    assert backend_path == DiffusionAttentionBackendEnum.SAGE_ATTN_3.get_path()
+
+
+@pytest.mark.skipif(not current_platform.is_cuda(), reason="sage_attn3 tests require CUDA platform")
+def test_cuda_platform_falls_back_when_sage_attn3_gpu_is_unsupported(monkeypatch: pytest.MonkeyPatch):
+    from vllm.platforms.interface import DeviceCapability
+
+    from vllm_omni.diffusion.attention.backends.registry import DiffusionAttentionBackendEnum
+    from vllm_omni.diffusion.envs import PACKAGES_CHECKER
+    from vllm_omni.platforms.cuda.platform import CudaOmniPlatform
+
+    monkeypatch.setattr(
+        CudaOmniPlatform,
+        "get_device_capability",
+        classmethod(lambda cls, device_id=0: DeviceCapability(9, 0)),
+    )
+    monkeypatch.setattr(PACKAGES_CHECKER, "get_packages_info", lambda: {"has_flash_attn": False})
+
+    backend_path = CudaOmniPlatform.get_diffusion_attn_backend_cls("SAGE_ATTN_3", head_size=64)
+
+    assert backend_path == DiffusionAttentionBackendEnum.TORCH_SDPA.get_path()
@@ -19,6 +19,7 @@ from vllm_omni.diffusion.cache.cache_dit_backend import (
    CUSTOM_DIT_ENABLERS,
    CacheDiTBackend,
 )
+from vllm_omni.diffusion.cache.magcache.backend import MagCacheBackend
 from vllm_omni.diffusion.cache.selector import get_cache_backend
 from vllm_omni.diffusion.cache.teacache.backend import TeaCacheBackend
 from vllm_omni.diffusion.data import DiffusionCacheConfig
@@ -313,3 +314,150 @@ class TestCacheSelector:
        """Test getting invalid backend raises error."""
        with pytest.raises(ValueError, match="Unsupported cache backend"):
            get_cache_backend("invalid_backend", {})
+
+
+class TestMagCacheBackend:
+    """Test MagCacheBackend implementation."""
+
+    from vllm_omni.diffusion.cache.magcache.backend import MagCacheBackend
+
+    def test_init(self):
+        """Test initialization."""
+        config = DiffusionCacheConfig(mag_threshold=0.1, mag_max_skip_steps=2, mag_calibrate=True)
+        backend = MagCacheBackend(config)
+        assert backend.config.mag_threshold == 0.1
+        assert backend.config.mag_max_skip_steps == 2
+        assert backend.enabled is False
+
+    @patch("vllm_omni.diffusion.cache.magcache.backend.apply_mag_cache_hook")
+    def test_enable(self, mock_apply_hook):
+        """Test enabling MagCache on pipeline."""
+        mock_pipeline = Mock()
+        mock_pipeline.__class__.__name__ = "FluxPipeline"
+        mock_transformer = Mock()
+        mock_transformer.__class__.__name__ = "FluxTransformer2DModel"
+        mock_pipeline.transformer = mock_transformer
+
+        mock_ratios = [1.0] * 28
+        config = DiffusionCacheConfig(
+            mag_ratios=mock_ratios,
+        )
+        backend = MagCacheBackend(config)
+        backend.enable(mock_pipeline)
+
+        assert backend.enabled is True
+        mock_apply_hook.assert_called_once()
+
+        call_args = mock_apply_hook.call_args
+        assert call_args[0][0] == mock_transformer
+
+    @patch("vllm_omni.diffusion.cache.magcache.backend.apply_mag_cache_hook")
+    def test_enable_with_calibration(self, mock_apply_hook):
+        """Test enabling MagCache in calibration mode."""
+        mock_pipeline = Mock()
+        mock_pipeline.__class__.__name__ = "FluxPipeline"
+        mock_transformer = Mock()
+        mock_transformer.__class__.__name__ = "FluxTransformer2DModel"
+        mock_pipeline.transformer = mock_transformer
+
+        config = DiffusionCacheConfig(
+            mag_calibrate=True,
+        )
+        backend = MagCacheBackend(config)
+        backend.enable(mock_pipeline)
+
+        assert backend.enabled is True
+        mock_apply_hook.assert_called_once()
+
+    def test_refresh(self):
+        """Test refreshing MagCache state calls enable when not registered."""
+        mock_pipeline = Mock()
+        mock_pipeline.__class__.__name__ = "FluxPipeline"
+        mock_transformer = Mock()
+        mock_transformer.__class__.__name__ = "FluxTransformer2DModel"
+        mock_pipeline.transformer = mock_transformer
+
+        mock_transformer.named_children = Mock(return_value=[])
+
+        mock_ratios = [1.0] * 28
+        config = DiffusionCacheConfig(
+            mag_ratios=mock_ratios,
+        )
+        backend = MagCacheBackend(config)
+
+        assert backend._registered is False
+        backend.refresh(mock_pipeline, num_inference_steps=50)
+        assert backend._registered is True
+
+    def test_is_enabled(self):
+        """Test is_enabled method."""
+        mock_ratios = [1.0] * 28
+        config = DiffusionCacheConfig(mag_ratios=mock_ratios)
+        backend = MagCacheBackend(config)
+        assert backend.is_enabled() is False
+
+    def test_get_mag_cache_backend(self):
+        """Test getting MagCache backend via selector."""
+        mock_ratios = [1.0] * 28
+        config_dict = {
+            "mag_ratios": mock_ratios,
+            "num_inference_steps": 28,
+            "threshold": 0.06,
+            "max_skip_steps": 3,
+            "retention_ratio": 0.2,
+        }
+        backend = get_cache_backend("mag_cache", config_dict)
+        assert backend is not None
+        assert isinstance(backend, MagCacheBackend)
+        assert backend.config.threshold == 0.06
+
+    @patch("vllm_omni.diffusion.cache.magcache.backend.apply_mag_cache_hook")
+    def test_enable_single_block(self, mock_apply_hook):
+        """Test enabling MagCache on single transformer block."""
+        mock_pipeline = Mock()
+        mock_pipeline.__class__.__name__ = "FluxPipeline"
+
+        mock_block = Mock()
+        mock_block.__class__.__name__ = "FluxTransformer2DModel"
+        mock_blocks = [mock_block]
+
+        mock_transformer = Mock()
+        mock_transformer.__class__.__name__ = "FluxTransformer2DModel"
+        mock_transformer.blocks = mock_blocks
+        mock_pipeline.transformer = mock_transformer
+
+        mock_ratios = [1.0] * 28
+        config = DiffusionCacheConfig(
+            mag_ratios=mock_ratios,
+        )
+        backend = MagCacheBackend(config)
+        backend.enable(mock_pipeline)
+
+        assert backend.enabled is True
+        mock_apply_hook.assert_called_once()
+
+        call_args = mock_apply_hook.call_args
+        assert call_args[0][0] == mock_transformer
+
+    @patch("vllm_omni.diffusion.cache.magcache.backend.apply_mag_cache_hook")
+    def test_enable_multi_block(self, mock_apply_hook):
+        """Test enabling MagCache on multiple transformer blocks."""
+        mock_pipeline = Mock()
+        mock_pipeline.__class__.__name__ = "FluxPipeline"
+
+        mock_blocks = [Mock() for _ in range(24)]
+
+        mock_transformer = Mock()
+        mock_transformer.__class__.__name__ = "FluxTransformer2DModel"
+        mock_transformer.blocks = mock_blocks
+        mock_pipeline.transformer = mock_transformer
+
+        mock_ratios = [1.0] * 28
+        config = DiffusionCacheConfig(
+            mag_ratios=mock_ratios,
+        )
+        backend = MagCacheBackend(config)
+        backend.enable(mock_pipeline)
+
+        assert backend.enabled is True
+        mock_apply_hook.assert_called_once()
@@ -0,0 +1,529 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+"""
+Unit tests for prompt-embedding cache for diffusion pipelines.
+
+Covers the module :mod:`vllm_omni.diffusion.cache.prompt_embed_cache`:
+
+* :class:`PromptEmbedCache` LRU semantics, stats and thread-safety.
+* :func:`_hashable` safe-input classification (scalars, torch dtype/device,
+  numpy scalars, nested containers, tensors/PIL bypass).
+* :func:`install_prompt_embed_cache` / :func:`uninstall_prompt_embed_cache`
+  behaviour: caching, bypass on tensors, bypass on precomputed embeds,
+  idempotent install, restore on uninstall, detachment of cached tensors.
+* :func:`resolve_prompt_embed_cache_config` env-var overrides.
+"""
+
+from __future__ import annotations
+
+import threading
+
+import pytest
+import torch
+
+from vllm_omni.diffusion.cache.prompt_embed_cache import (
+    _CACHE_MISS,
+    _NOT_HASHABLE,
+    PromptEmbedCache,
+    _build_key,
+    _detach_output,
+    _hashable,
+    install_prompt_embed_cache,
+    resolve_prompt_embed_cache_config,
+    uninstall_prompt_embed_cache,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+# ---------------------------------------------------------------------------
+# _hashable
+# ---------------------------------------------------------------------------
+
+
+class TestHashable:
+    """Tests for the ``_hashable`` helper."""
+
+    @pytest.mark.parametrize(
+        "value",
+        [None, "prompt", "", 0, 1, -5, 3.14, True, False, b"bytes"],
+    )
+    def test_scalars_are_returned_unchanged(self, value):
+        assert _hashable(value) == value
+
+    def test_torch_device_is_normalized(self):
+        out = _hashable(torch.device("cpu"))
+        assert out == ("__torch_device__", "cpu")
+        assert hash(out) is not None  # must actually be hashable
+
+    def test_torch_dtype_is_normalized(self):
+        out = _hashable(torch.float32)
+        assert out[0] == "__torch_dtype__"
+        assert "float32" in out[1]
+        assert hash(out) is not None
+
+    def test_numpy_scalar_is_unwrapped(self):
+        np = pytest.importorskip("numpy")
+        assert _hashable(np.int64(7)) == 7
+        assert _hashable(np.float32(1.5)) == pytest.approx(1.5)
+
+    def test_numpy_ndarray_is_not_hashable(self):
+        np = pytest.importorskip("numpy")
+        assert _hashable(np.array([1, 2, 3])) is _NOT_HASHABLE
+
+    def test_nested_list_and_tuple(self):
+        out = _hashable(["a", ("b", 1), [2, 3]])
+        assert out[0] == "__seq__"
+        # fully hashable
+        assert hash(out) is not None
+
+    def test_dict_is_normalized_by_sorted_keys(self):
+        a = _hashable({"b": 1, "a": 2})
+        b = _hashable({"a": 2, "b": 1})
+        assert a == b
+        assert hash(a) is not None
+
+    def test_tensor_is_not_hashable(self):
+        assert _hashable(torch.zeros(2)) is _NOT_HASHABLE
+
+    def test_list_containing_tensor_is_not_hashable(self):
+        assert _hashable(["a", torch.zeros(1)]) is _NOT_HASHABLE
+
+    def test_dict_containing_tensor_is_not_hashable(self):
+        assert _hashable({"x": torch.zeros(1)}) is _NOT_HASHABLE
+
+    def test_custom_object_is_not_hashable(self):
+        class Obj:
+            pass
+
+        assert _hashable(Obj()) is _NOT_HASHABLE
+
+
+# ---------------------------------------------------------------------------
+# _detach_output
+# ---------------------------------------------------------------------------
+
+
+class TestDetachOutput:
+    def test_detaches_tensor(self):
+        t = torch.zeros(2, requires_grad=True)
+        out = _detach_output(t)
+        assert isinstance(out, torch.Tensor)
+        assert not out.requires_grad
+
+    def test_detaches_inside_tuple(self):
+        t = torch.ones(1, requires_grad=True)
+        out = _detach_output((t, None))
+        assert isinstance(out, tuple)
+        assert not out[0].requires_grad
+        assert out[1] is None
+
+    def test_detaches_inside_list_and_dict(self):
+        t = torch.ones(1, requires_grad=True)
+        out = _detach_output({"a": [t]})
+        assert not out["a"][0].requires_grad
+
+    def test_preserves_namedtuple_type(self):
+        from collections import namedtuple
+
+        NT = namedtuple("NT", ["x", "y"])
+        t = torch.ones(1, requires_grad=True)
+        out = _detach_output(NT(x=t, y=1))
+        assert isinstance(out, NT)
+        assert not out.x.requires_grad
+        assert out.y == 1
+
+    def test_passthrough_for_non_tensor(self):
+        assert _detach_output("hi") == "hi"
+        assert _detach_output(None) is None
+        assert _detach_output(5) == 5
+
+
+# ---------------------------------------------------------------------------
+# PromptEmbedCache
+# ---------------------------------------------------------------------------
+
+
+class TestPromptEmbedCache:
+    def test_rejects_non_positive_max_size(self):
+        with pytest.raises(ValueError):
+            PromptEmbedCache(max_size=0)
+        with pytest.raises(ValueError):
+            PromptEmbedCache(max_size=-1)
+
+    def test_miss_returns_sentinel_and_bumps_miss_counter(self):
+        cache = PromptEmbedCache(max_size=2)
+        assert cache.get("k") is _CACHE_MISS
+        assert cache.stats()["misses"] == 1
+        assert cache.stats()["hits"] == 0
+
+    def test_put_then_get_returns_value_and_bumps_hit_counter(self):
+        cache = PromptEmbedCache(max_size=2)
+        cache.put("k", 42)
+        assert cache.get("k") == 42
+        assert cache.stats()["hits"] == 1
+
+    def test_none_is_a_cacheable_value(self):
+        cache = PromptEmbedCache(max_size=2)
+        cache.put("k", None)
+        # Distinguishable from a miss.
+        assert cache.get("k") is None
+        assert cache.stats()["hits"] == 1
+
+    def test_lru_eviction_keeps_recently_used(self):
+        cache = PromptEmbedCache(max_size=2)
+        cache.put("a", 1)
+        cache.put("b", 2)
+        # Access "a" to mark it MRU.
+        assert cache.get("a") == 1
+        cache.put("c", 3)  # should evict "b"
+        assert cache.get("b") is _CACHE_MISS
+        assert cache.get("a") == 1
+        assert cache.get("c") == 3
+        assert cache.stats()["size"] == 2
+
+    def test_get_refreshes_recency(self):
+        cache = PromptEmbedCache(max_size=2)
+        cache.put("a", 1)
+        cache.put("b", 2)
+        # Hit on "a" → "b" becomes LRU.
+        cache.get("a")
+        cache.put("c", 3)
+        assert cache.get("b") is _CACHE_MISS
+        assert cache.get("a") == 1
+
+    def test_put_overwrites_existing_key(self):
+        cache = PromptEmbedCache(max_size=2)
+        cache.put("k", 1)
+        cache.put("k", 2)
+        assert cache.get("k") == 2
+        assert cache.stats()["size"] == 1
+
+    def test_clear_empties_store_but_keeps_counters(self):
+        cache = PromptEmbedCache(max_size=2)
+        cache.put("k", 1)
+        cache.get("k")
+        cache.clear()
+        assert cache.stats()["size"] == 0
+        # clear only drops entries, counters reflect prior activity.
+        assert cache.stats()["hits"] == 1
+
+    def test_stats_reports_expected_fields(self):
+        cache = PromptEmbedCache(max_size=4)
+        stats = cache.stats()
+        assert set(stats.keys()) == {"size", "max_size", "hits", "misses", "bypassed"}
+        assert stats["max_size"] == 4
+        assert stats["size"] == 0
+
+    def test_thread_safety_under_concurrent_writes(self):
+        cache = PromptEmbedCache(max_size=1024)
+
+        def worker(start: int) -> None:
+            for i in range(start, start + 200):
+                cache.put(f"k{i}", i)
+                cache.get(f"k{i}")
+
+        threads = [threading.Thread(target=worker, args=(i * 200,)) for i in range(4)]
+        for t in threads:
+            t.start()
+        for t in threads:
+            t.join()
+
+        stats = cache.stats()
+        # 4 * 200 = 800 writes, each followed by a successful read.
+        assert stats["hits"] == 800
+        assert stats["size"] == 800
+
+
+# ---------------------------------------------------------------------------
+# _build_key
+# ---------------------------------------------------------------------------
+
+
+def _make_sig(fn):
+    import inspect
+
+    return inspect.signature(fn)
+
+
+class TestBuildKey:
+    def test_positional_and_keyword_collide(self):
+        def encode(prompt, device=None):
+            return None
+
+        sig = _make_sig(encode)
+        k1 = _build_key(sig, "m", ("hello",), {"device": "cpu"})
+        k2 = _build_key(sig, "m", (), {"prompt": "hello", "device": "cpu"})
+        assert k1 == k2
+        assert k1 is not None
+
+    def test_different_prompts_produce_different_keys(self):
+        def encode(prompt):
+            return None
+
+        sig = _make_sig(encode)
+        assert _build_key(sig, "m", ("a",), {}) != _build_key(sig, "m", ("b",), {})
+
+    def test_different_model_tags_produce_different_keys(self):
+        def encode(prompt):
+            return None
+
+        sig = _make_sig(encode)
+        assert _build_key(sig, "m1", ("a",), {}) != _build_key(sig, "m2", ("a",), {})
+
+    def test_tensor_argument_bypasses(self):
+        def encode(prompt, extra=None):
+            return None
+
+        sig = _make_sig(encode)
+        assert _build_key(sig, "m", ("hi",), {"extra": torch.zeros(1)}) is None
+
+    def test_precomputed_prompt_embeds_bypasses(self):
+        def encode(prompt=None, prompt_embeds=None):
+            return None
+
+        sig = _make_sig(encode)
+        # None → not a bypass.
+        assert _build_key(sig, "m", (), {"prompt": "hi"}) is not None
+        # A non-None precomputed embed triggers bypass even if it would
+        # otherwise be "hashable".
+        assert _build_key(sig, "m", (), {"prompt": "hi", "prompt_embeds": "x"}) is None
+
+    def test_invalid_binding_returns_none(self):
+        def encode(prompt):
+            return None
+
+        sig = _make_sig(encode)
+        # Unknown kwarg → TypeError on bind → bypass.
+        assert _build_key(sig, "m", (), {"nope": 1}) is None
+
+
+# ---------------------------------------------------------------------------
+# install_prompt_embed_cache / uninstall_prompt_embed_cache
+# ---------------------------------------------------------------------------
+
+
+class _FakePipeline:
+    """Minimal pipeline with a signature-bearing ``encode_prompt``."""
+
+    def __init__(self):
+        self.call_count = 0
+
+    def encode_prompt(
+        self,
+        prompt: str,
+        device=None,
+        num_images_per_prompt: int = 1,
+        negative_prompt: str | None = None,
+        prompt_embeds=None,
+        negative_prompt_embeds=None,
+    ):
+        self.call_count += 1
+        # Return a fresh tensor so identity changes between real calls.
+        return torch.tensor([float(self.call_count)])
+
+
+class TestInstallAndUninstall:
+    def test_returns_none_when_pipeline_has_no_encode_prompt(self):
+        class Empty:
+            pass
+
+        assert install_prompt_embed_cache(Empty()) is None
+
+    def test_cache_hits_on_identical_args(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe, max_size=4)
+        assert cache is not None
+
+        out1 = pipe.encode_prompt("cat", device="cpu")
+        out2 = pipe.encode_prompt("cat", device="cpu")
+        # Second call must be served from cache (wrapped fn not invoked).
+        assert pipe.call_count == 1
+        # Cache returns the same tensor object.
+        assert torch.equal(out1, out2)
+        stats = cache.stats()
+        assert stats["hits"] == 1
+        assert stats["misses"] == 1
+        assert stats["bypassed"] == 0
+
+    def test_positional_and_keyword_share_cache_slot(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe)
+        pipe.encode_prompt("cat")
+        pipe.encode_prompt(prompt="cat")
+        assert pipe.call_count == 1
+        assert cache.stats()["hits"] == 1
+
+    def test_different_prompts_miss(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe)
+        pipe.encode_prompt("cat")
+        pipe.encode_prompt("dog")
+        assert pipe.call_count == 2
+        assert cache.stats()["misses"] == 2
+
+    def test_tensor_argument_bypasses_and_increments_counter(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe)
+        # Put a tensor into a non-precomputed-embed slot to trigger bypass.
+        pipe.encode_prompt("cat", device=torch.device("cpu"), num_images_per_prompt=1)
+
+        # Now pass a fake tensor via a harmless positional-like call: use
+        # the negative_prompt slot with an unhashable object.
+        class Unhashable:
+            pass
+
+        pipe.encode_prompt("cat", negative_prompt=Unhashable())  # type: ignore[arg-type]
+        pipe.encode_prompt("cat", negative_prompt=Unhashable())  # type: ignore[arg-type]
+        assert cache.stats()["bypassed"] == 2
+        # All three calls actually executed the underlying function.
+        assert pipe.call_count == 3
+
+    def test_precomputed_embeds_bypass_cache(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe)
+        pipe.encode_prompt("cat", prompt_embeds=torch.zeros(1))
+        pipe.encode_prompt("cat", prompt_embeds=torch.zeros(1))
+        assert cache.stats()["bypassed"] == 2
+        assert cache.stats()["hits"] == 0
+        assert pipe.call_count == 2
+
+    def test_install_is_idempotent(self):
+        pipe = _FakePipeline()
+        c1 = install_prompt_embed_cache(pipe, max_size=4)
+        c2 = install_prompt_embed_cache(pipe, max_size=16)
+        assert c1 is c2
+        # The second call must not re-wrap (max_size stays from first install).
+        assert c1.max_size == 4
+
+    def test_disabled_cache_passes_through(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe, enabled=False)
+        pipe.encode_prompt("cat")
+        pipe.encode_prompt("cat")
+        assert pipe.call_count == 2
+        assert cache.stats()["hits"] == 0
+        assert cache.stats()["misses"] == 0
+
+    def test_toggle_enabled_flag_takes_effect(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe, enabled=True)
+        pipe.encode_prompt("cat")
+        pipe.encode_prompt("cat")
+        assert cache.stats()["hits"] == 1
+
+        cache.enabled = False
+        pipe.encode_prompt("cat")
+        pipe.encode_prompt("cat")
+        # Cache-bypassed calls go through; hit count unchanged.
+        assert cache.stats()["hits"] == 1
+
+    def test_cached_tensor_is_detached(self):
+        class GradPipeline:
+            def encode_prompt(self, prompt):
+                return torch.ones(1, requires_grad=True)
+
+        pipe = GradPipeline()
+        install_prompt_embed_cache(pipe)
+        # First call populates the cache.
+        pipe.encode_prompt("cat")
+        # Second call returns the cached tensor which must be detached.
+        cached = pipe.encode_prompt("cat")
+        assert isinstance(cached, torch.Tensor)
+        assert not cached.requires_grad
+
+    def test_lru_eviction_through_wrapper(self):
+        pipe = _FakePipeline()
+        cache = install_prompt_embed_cache(pipe, max_size=2)
+        pipe.encode_prompt("a")
+        pipe.encode_prompt("b")
+        pipe.encode_prompt("c")  # evicts "a"
+        pipe.encode_prompt("a")  # miss again → re-run
+        assert pipe.call_count == 4
+        assert cache.stats()["size"] == 2
+
+    def test_uninstall_restores_original(self):
+        pipe = _FakePipeline()
+        original = pipe.encode_prompt
+        install_prompt_embed_cache(pipe)
+        assert pipe.encode_prompt is not original
+        uninstall_prompt_embed_cache(pipe)
+        # The bound method reference will differ, but identity of underlying
+        # function matches and wrapper attributes are gone.
+        assert not hasattr(pipe, "_prompt_embed_cache") or pipe._prompt_embed_cache is None
+        # After uninstall calls go straight to the real function.
+        pipe.encode_prompt("cat")
+        pipe.encode_prompt("cat")
+        assert pipe.call_count == 2
+
+    def test_uninstall_on_unwrapped_pipeline_is_noop(self):
+        class Empty:
+            def encode_prompt(self, prompt):
+                return None
+
+        pipe = Empty()
+        uninstall_prompt_embed_cache(pipe)  # must not raise
+
+
+# ---------------------------------------------------------------------------
+# resolve_prompt_embed_cache_config
+# ---------------------------------------------------------------------------
+
+
+@pytest.fixture
+def clean_env(monkeypatch):
+    monkeypatch.delenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE", raising=False)
+    monkeypatch.delenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE", raising=False)
+
+
+class TestResolveConfig:
+    def test_defaults_when_nothing_set(self, clean_env):
+        enable, size = resolve_prompt_embed_cache_config()
+        assert enable is False
+        assert size == 32
+
+    def test_explicit_args_used(self, clean_env):
+        enable, size = resolve_prompt_embed_cache_config(enable=True, max_size=8)
+        assert enable is True
+        assert size == 8
+
+    @pytest.mark.parametrize("raw", ["1", "true", "TRUE", "yes", "on"])
+    def test_env_enable_truthy(self, clean_env, monkeypatch, raw):
+        monkeypatch.setenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE", raw)
+        enable, _ = resolve_prompt_embed_cache_config(enable=False)
+        assert enable is True
+
+    @pytest.mark.parametrize("raw", ["0", "false", "FALSE", "no", "off"])
+    def test_env_enable_falsy_overrides_true(self, clean_env, monkeypatch, raw):
+        monkeypatch.setenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE", raw)
+        enable, _ = resolve_prompt_embed_cache_config(enable=True)
+        assert enable is False
+
+    def test_env_enable_invalid_leaves_arg_untouched(self, clean_env, monkeypatch):
+        monkeypatch.setenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE", "garbage")
+        enable, _ = resolve_prompt_embed_cache_config(enable=True)
+        assert enable is True
+
+    def test_env_size_overrides_arg(self, clean_env, monkeypatch):
+        monkeypatch.setenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE", "7")
+        _, size = resolve_prompt_embed_cache_config(max_size=99)
+        assert size == 7
+
+    def test_env_size_zero_or_negative_ignored(self, clean_env, monkeypatch):
+        monkeypatch.setenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE", "0")
+        _, size = resolve_prompt_embed_cache_config(max_size=12)
+        assert size == 12
+
+        monkeypatch.setenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE", "-5")
+        _, size = resolve_prompt_embed_cache_config(max_size=12)
+        assert size == 12
+
+    def test_env_size_non_integer_ignored(self, clean_env, monkeypatch):
+        monkeypatch.setenv("OMNI_DIFFUSION_PROMPT_EMBED_CACHE_SIZE", "notanint")
+        _, size = resolve_prompt_embed_cache_config(max_size=12)
+        assert size == 12
+
+    def test_max_size_none_falls_back_to_default(self, clean_env):
+        _, size = resolve_prompt_embed_cache_config(enable=True, max_size=None)
+        assert size == 32
@@ -60,6 +60,7 @@ def _make_request(**overrides) -> OmniDiffusionRequest:
            output_type="pil",
            generator_device="cpu",
        ),
+        "request_id": "diffusers-backend",
    }
    defaults.update(overrides)
    return OmniDiffusionRequest(**defaults)
@@ -7,6 +7,9 @@ import torch
 from vllm_omni.diffusion.distributed.autoencoders.autoencoder_kl_hunyuan import (
    DistributedAutoencoderKLHunyuan,
 )
+from vllm_omni.diffusion.distributed.autoencoders.autoencoder_kl_hunyuan_video_15 import (
+    DistributedAutoencoderKLHunyuanVideo15,
+)

 pytestmark = [pytest.mark.core_model, pytest.mark.cpu]

@@ -36,6 +39,33 @@ class _DummyDistributedAutoencoderKLHunyuan(DistributedAutoencoderKLHunyuan):
        return b


+class _DummyDistributedAutoencoderKLHunyuanVideo15(DistributedAutoencoderKLHunyuanVideo15):
+    def __init__(self):
+        torch.nn.Module.__init__(self)
+        self.tile_sample_min_height = 8
+        self.tile_sample_min_width = 12
+        self.tile_latent_min_height = 8
+        self.tile_latent_min_width = 12
+        self.tile_overlap_factor = 0.25
+        self.use_tiling = False
+
+    @property
+    def dtype(self) -> torch.dtype:
+        return torch.float32
+
+    def decoder(self, x: torch.Tensor) -> torch.Tensor:
+        return x + 10
+
+    def encoder(self, x: torch.Tensor) -> torch.Tensor:
+        return x + 20
+
+    def blend_v(self, _a: torch.Tensor, b: torch.Tensor, _blend_extent: int) -> torch.Tensor:
+        return b
+
+    def blend_h(self, _a: torch.Tensor, b: torch.Tensor, _blend_extent: int) -> torch.Tensor:
+        return b
+
+
 def test_hunyuan_vae_use_tiling_aliases_spatial_tiling():
    # Verify use_tiling property maps to use_spatial_tiling.
    vae = _DummyDistributedAutoencoderKLHunyuan()
@@ -77,3 +107,43 @@ def test_hunyuan_vae_encode_tiles_round_trip():
    assert grid_spec.tile_spec == {"blend_extent": 2, "row_limit": 6}
    assert len(tile_tasks) == 4
    assert torch.equal(output, x + 20)
+
+
+def test_hunyuan_video15_vae_decode_tiles_round_trip():
+    vae = _DummyDistributedAutoencoderKLHunyuanVideo15()
+    z = torch.arange(192, dtype=torch.float32).reshape(1, 1, 1, 12, 16)
+
+    tile_tasks, grid_spec = vae.tile_split(z)
+    decoded_tiles = {task.grid_coord: vae.tile_exec(task) for task in tile_tasks}
+    output = vae.tile_merge(decoded_tiles, grid_spec)
+
+    assert grid_spec.split_dims == (3, 4)
+    assert grid_spec.grid_shape == (2, 2)
+    assert grid_spec.tile_spec == {
+        "blend_height": 2,
+        "blend_width": 3,
+        "row_limit_height": 6,
+        "row_limit_width": 9,
+    }
+    assert len(tile_tasks) == 4
+    assert torch.equal(output, z + 10)
+
+
+def test_hunyuan_video15_vae_encode_tiles_round_trip():
+    vae = _DummyDistributedAutoencoderKLHunyuanVideo15()
+    x = torch.arange(192, dtype=torch.float32).reshape(1, 1, 1, 12, 16)
+
+    tile_tasks, grid_spec = vae.encode_tile_split(x)
+    encoded_tiles = {task.grid_coord: vae.encode_tile_exec(task) for task in tile_tasks}
+    output = vae.tile_merge(encoded_tiles, grid_spec)
+
+    assert grid_spec.split_dims == (3, 4)
+    assert grid_spec.grid_shape == (2, 2)
+    assert grid_spec.tile_spec == {
+        "blend_height": 2,
+        "blend_width": 3,
+        "row_limit_height": 6,
+        "row_limit_width": 9,
+    }
+    assert len(tile_tasks) == 4
+    assert torch.equal(output, x + 20)
@@ -0,0 +1,169 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Unit tests for Bagel.forward_cache_update_vae packed-sequence assembly."""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+from unittest.mock import MagicMock
+
+import numpy as np
+import pytest
+import torch
+from PIL import Image
+from transformers import PretrainedConfig
+from vllm.transformers_utils.configs.bagel import BagelConfig
+
+from vllm_omni.diffusion.models.bagel.bagel_transformer import (
+    Bagel,
+    BaseNavitOutputWithPast,
+    NaiveCache,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+HIDDEN_SIZE = 32
+NUM_LAYERS = 2
+
+
+def _make_bagel_config() -> BagelConfig:
+    llm_config = PretrainedConfig()
+    llm_config.hidden_size = HIDDEN_SIZE
+    llm_config.layer_module = "Qwen2MoTDecoderLayer"
+    llm_config.num_hidden_layers = NUM_LAYERS
+    llm_config.num_attention_heads = 4
+
+    vae_config = PretrainedConfig()
+    vae_config.downsample = 8
+    vae_config.z_channels = 4
+
+    return BagelConfig(
+        llm_config=llm_config,
+        vae_config=vae_config,
+        visual_gen=True,
+        visual_und=False,
+        latent_patch_size=2,
+        max_latent_size=32,
+    )
+
+
+def _make_tracking_language_model() -> MagicMock:
+    """Return a mock LM that records forward() kwargs."""
+    calls: list[dict] = []
+    hidden_size = HIDDEN_SIZE
+
+    def forward(*_args, **kwargs):
+        calls.append(dict(kwargs))
+        if kwargs.get("return_embeddings_only"):
+            text_ids = kwargs["packed_text_ids"]
+            emb = torch.randn(text_ids.shape[0], hidden_size)
+            return BaseNavitOutputWithPast(packed_query_sequence=emb)
+
+        packed_query_sequence = kwargs.get("packed_query_sequence")
+        if packed_query_sequence is None:
+            text_ids = kwargs["packed_text_ids"]
+            packed_query_sequence = torch.randn(text_ids.shape[0], hidden_size)
+        return BaseNavitOutputWithPast(
+            packed_query_sequence=packed_query_sequence,
+            past_key_values=kwargs.get("past_key_values"),
+        )
+
+    lm = MagicMock()
+    lm.forward = forward
+    lm._calls = calls
+    return lm
+
+
+def _vae_transform(image: Image.Image) -> torch.Tensor:
+    arr = torch.from_numpy(np.array(image.convert("RGB"), dtype=np.float32))
+    arr = arr / 127.5 - 1.0
+    return arr.permute(2, 0, 1)
+
+
+@pytest.fixture
+def bagel_and_vae_input():
+    language_model = _make_tracking_language_model()
+    bagel = Bagel(language_model=language_model, vit_model=None, config=_make_bagel_config())
+
+    image = Image.new("RGB", (64, 64), color=(128, 64, 32))
+    new_token_ids = {"start_of_image": 100, "end_of_image": 101}
+
+    gen_input, _newlens, _new_rope = bagel.prepare_vae_images(
+        curr_kvlens=[0],
+        curr_rope=[0],
+        images=[image],
+        transforms=_vae_transform,
+        new_token_ids=new_token_ids,
+    )
+
+    vae_model = MagicMock()
+    patch_shapes = gen_input["patchified_vae_latent_shapes"]
+    p = bagel.latent_patch_size
+    c = bagel.latent_channel
+
+    def encode(padded_images: torch.Tensor) -> torch.Tensor:
+        # Match AutoEncoder.encode layout: [B, z_channels, h*p, w*p] per image.
+        batch = padded_images.shape[0]
+        assert batch == len(patch_shapes)
+        latents = [torch.randn(c, h * p, w * p) for h, w in patch_shapes]
+        return torch.stack(latents, dim=0)
+
+    vae_model.encode = encode
+
+    expected_seq_len = int(gen_input["packed_seqlens"].sum().item())
+    num_text_tokens = int(gen_input["packed_text_ids"].numel())
+
+    return SimpleNamespace(
+        bagel=bagel,
+        language_model=language_model,
+        vae_model=vae_model,
+        gen_input=gen_input,
+        past_key_values=NaiveCache(NUM_LAYERS),
+        expected_seq_len=expected_seq_len,
+        num_text_tokens=num_text_tokens,
+    )
+
+
+class TestForwardCacheUpdateVae:
+    """Regression tests for single-stage img2img VAE cache prefill."""
+
+    def test_forwards_full_packed_query_sequence(self, bagel_and_vae_input):
+        """Update forward must use packed_query_sequence spanning text + VAE tokens."""
+        ctx = bagel_and_vae_input
+        assert ctx.expected_seq_len > ctx.num_text_tokens, "test setup should include many VAE tokens"
+
+        ctx.bagel.forward_cache_update_vae(
+            ctx.vae_model,
+            ctx.past_key_values,
+            **ctx.gen_input,
+        )
+
+        update_calls = [c for c in ctx.language_model._calls if not c.get("return_embeddings_only")]
+        assert len(update_calls) == 1
+
+        call = update_calls[0]
+        assert "packed_query_sequence" in call
+        assert "packed_text_ids" not in call
+
+        packed = call["packed_query_sequence"]
+        assert packed.shape[0] == ctx.expected_seq_len
+        assert call.get("mode") == "gen"
+
+    def test_packed_sequence_covers_text_and_vae_index_ranges(self, bagel_and_vae_input):
+        """Text and VAE index ranges must lie within packed_query_sequence."""
+        ctx = bagel_and_vae_input
+        ctx.bagel.forward_cache_update_vae(
+            ctx.vae_model,
+            ctx.past_key_values,
+            **ctx.gen_input,
+        )
+
+        update_call = next(c for c in ctx.language_model._calls if not c.get("return_embeddings_only"))
+        packed = update_call["packed_query_sequence"]
+
+        text_idx = ctx.gen_input["packed_text_indexes"]
+        vae_idx = ctx.gen_input["packed_vae_token_indexes"]
+        assert int(text_idx.max()) < packed.shape[0]
+        assert int(vae_idx.max()) < packed.shape[0]
+        assert int(text_idx.min()) >= 0
+        assert int(vae_idx.min()) >= 0
@@ -56,10 +56,7 @@ def _make_generate_args(num_tokens=NUM_TOKENS, hidden_dim=HIDDEN_DIM, cfg=False)
        packed_vae_token_indexes=torch.arange(2, seq_len, dtype=torch.long),
        packed_seqlens=torch.tensor([seq_len], dtype=torch.int),
        packed_position_ids=torch.arange(seq_len, dtype=torch.long),
-        packed_indexes=torch.arange(seq_len, dtype=torch.long),
        past_key_values=NaiveCache(1),
-        key_values_lens=torch.tensor([0], dtype=torch.int),
-        packed_key_value_indexes=torch.zeros(0, dtype=torch.long),
        num_timesteps=NUM_TIMESTEPS,
        timestep_shift=1.0,
        cfg_text_scale=1.0,
@@ -68,11 +65,8 @@ def _make_generate_args(num_tokens=NUM_TOKENS, hidden_dim=HIDDEN_DIM, cfg=False)
    if cfg:
        base |= dict(
            cfg_text_scale=4.0,
-            cfg_text_packed_query_indexes=torch.arange(seq_len, dtype=torch.long),
            cfg_text_packed_position_ids=torch.arange(seq_len, dtype=torch.long),
            cfg_text_past_key_values=NaiveCache(1),
-            cfg_text_key_values_lens=torch.tensor([0], dtype=torch.int),
-            cfg_text_packed_key_value_indexes=torch.zeros(0, dtype=torch.long),
        )
    return base

@@ -48,6 +48,7 @@ def _make_request(prompts=None, **sp_kwargs) -> OmniDiffusionRequest:
    return OmniDiffusionRequest(
        prompts=prompts or [{"prompt": "a cat dancing"}],
        sampling_params=sp,
+        request_id="dmd2-sanitize",
    )


@@ -48,7 +48,7 @@ def _make_pipeline(cls):

 def _make_request(**sp_kwargs) -> OmniDiffusionRequest:
    sp = OmniDiffusionSamplingParams(**sp_kwargs)
-    return OmniDiffusionRequest(prompts=[{"prompt": "a cat"}], sampling_params=sp)
+    return OmniDiffusionRequest(prompts=[{"prompt": "a cat"}], sampling_params=sp, request_id="dmd2-scheduler")


@pytest.fixture(
@@ -73,7 +73,7 @@ def test_should_apply_pe_respects_sampling_extra_args():


 def test_should_apply_pe_disables_dummy_warmup_request():
-    req = SimpleNamespace(request_ids=["dummy_req_id"], sampling_params=SimpleNamespace(extra_args={}))
+    req = SimpleNamespace(is_dummy_run=lambda: True, sampling_params=SimpleNamespace(extra_args={}))

    assert ErnieImagePipeline._should_apply_pe(req) is False
Author	SHA1	Message	Date
tzhouamandClaude Opus 4.8	ea5db9f9e0	chore: complete v0.21.0 → v0.22.0 version bump across all files Update remaining Dockerfiles (cuda, rocm, xpu, ci) and docs (rocm.inc.md, cuda.inc.md, quickstart.md, gpu.md) that were missed in the initial pass — pip install lines, git checkout tags, Docker image tags, wheel URLs, and prose references. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-01 03:39:26 +00:00
rongfu.leng	60c150040e	[Feat]upgrade vllm version Signed-off-by: rongfu.leng <lenronfu@gmail.com>	2026-05-31 15:44:08 +00:00
ChenWenjing	5dfdf584dd	[Feat]audio streaming input for async chunk (#3614 ) Signed-off-by: CHEN <116010019@link.cuhk.edu.cn>	2026-05-31 17:39:25 +08:00
NumberWan	6f4bd3e45c	[BugFix]Qwen-Image performance regression by using omni RMSNorm(RMSNorm backend) (#3933 ) Signed-off-by: NumberWan <wantszkin2003@gmail.com>	2026-05-31 08:06:18 +08:00
Canlin Guo	412bf42a75	[Refactor] Unify _talker_mtp_forward across GPU and NPU model runners (#3476 ) Signed-off-by: gcanlin <canlinguosdu@gmail.com>	2026-05-31 07:58:00 +08:00
zzh	ffd9771c54	Fix hunyuan resolve stop token ids (#3896 ) Signed-off-by: zzh <943967662@qq.com>	2026-05-31 07:47:36 +08:00
DMAN	11c4fcedd3	[Recipe] add mistralai voxtral tts recipe (#3498 ) Signed-off-by: Dmaner <2663515256@qq.com>	2026-05-31 00:18:52 +08:00
wuhang	b0f3672aec	[Bugfix] Fix DiffusionWorker crash on SIGINT: ensure NCCL/ZMQ cleanup on shutdown (#3872 ) Signed-off-by: WU Hang <whlbx@hotmail.com>	2026-05-31 00:13:23 +08:00
WeiQing Chen	2289478e3a	[CI] Temporarily skip failing Bagel connector tests (#4005 ) Signed-off-by: David Chen <530634352@qq.com>	2026-05-30 17:46:29 +08:00
Alex BrooksandHongsheng Liu	8d13b54383	[CI/Bugfix] Async Request ID Aliasing (#3953 ) Signed-off-by: Alex Brooks <albrooks@redhat.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-30 16:59:42 +08:00
wuhangandAlicia	802a6242f0	[Bugfix][Tests] Remove unnecessary device map in tests init (#3958 ) Signed-off-by: WU Hang <whlbx@hotmail.com> Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>	2026-05-30 16:18:48 +08:00
Yuanheng ZhaoandZheng Wengang	ea6d811b58	[Config Refactor] Migrate Ming-flash-omni-2.0 Image-Gen deploy configs (#3975 ) Signed-off-by: yuanheng <jonathan.zhaoyh@gmail.com> Co-authored-by: Zheng Wengang <zwg0606@gmail.com>	2026-05-30 08:13:21 +00:00
WeiQing Chen	5994a0bbf5	docs: update WeChat QR code (#4003 ) Signed-off-by: David Chen <530634352@qq.com>	2026-05-30 16:12:11 +08:00
Viacheslav KlimkovandYueqian Lin	914e92cf80	[Refactor][Qwen3-TTS] Extract reusable prompt-embeds builder and make tts_pad_embed a persistent buffer (#3992 ) Signed-off-by: Viacheslav Klimkov <vklimkov@nvidia.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>	2026-05-30 16:11:00 +08:00
dengyunyang	3b317285c4	[HunyuanImage][Perf] adapt to deploy config changes (#3996 ) Signed-off-by: dengyunyang <584797741@qq.com>	2026-05-30 14:48:50 +08:00
WeiQing Chen	a3456752c5	[Diffusion] Support USP and VAE patch parallel for HunyuanVideo 1.5 (#3979 ) Signed-off-by: david6666666 <530634352@qq.com>	2026-05-30 14:40:42 +08:00
Chendi.Xue	f43bb124bc	[CI][XPU]Fix sage_attn hard-code import for cuda (#3994 ) Signed-off-by: Chendi Xue <chendi.xue@intel.com>	2026-05-30 09:43:14 +08:00
fywc	272e626100	[Feat] support cache-dit for SenseNova-U1 (#3906 ) Signed-off-by: hanzheli <hanzheli@kuaishou.com>	2026-05-29 17:40:54 +00:00
knlnguyen1802andSamit	5377108438	[minor, fix] Allow passing class interface as custom pipeline argument (#2973 ) Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> Signed-off-by: Samit <285365963@qq.com> Co-authored-by: Samit <285365963@qq.com>	2026-05-29 23:39:38 +08:00
nperraud	9fdb3bcb89	[Bugfix][Model] Qwen3-TTS: don't collapse 2D ref_code list when estimating prompt length (#3940 ) Signed-off-by: nperraud <nathanael.perraudin@gmail.com>	2026-05-29 09:18:25 -06:00
Chen ChengandClaude Opus 4.7	f1c2caa3c7	[Perf][Qwen3-TTS] Restore Code2Wav cross-request batching (RFC #3163 P0) (#3322 ) Signed-off-by: ischencheng <cheng21@seas.upenn.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-29 08:48:58 -06:00
Lancer 汪志鹏 Gao Han	b690a10e3a	[Feat] Support MagCache (#1287 ) Signed-off-by: Lancer <maruixiang6688@gmail.com> Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>	2026-05-29 22:26:23 +08:00
Aleksi VesantoandHongsheng Liu	eba5617c95	[ROCm] Enable AITER backend with ring attention (#3511 ) Signed-off-by: Aleksi Vesanto <avesanto@amd.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-29 22:20:29 +08:00
amy-why-3459andHongsheng Liu	2696751003	[Perf] Qwen3-Omni performance optimization (#3878 ) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-29 22:15:23 +08:00
Alicia	8c4a42b16c	Temp skip TEST - Entrypoint Test with H100 (#3989 ) Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>	2026-05-29 22:37:09 +12:00
Huanyu Yang	3457758114	[Bugfix] Fix IndexError in DistributedVaeExecutor when vae_patch_parallel_size < world_size (#3928 ) Signed-off-by: QingZhou-YangHY <3868850350@qq.com>	2026-05-29 03:17:34 -07:00
WeiQing Chen	67bb9b4a12	[CI][Bugfix] Fix LTX audio-video warmup output typing (#3964 ) Signed-off-by: david6666666 <530634352@qq.com>	2026-05-29 07:46:02 +00:00
boatmanandsphinxkkkbc	4d9262a6eb	[Feat]Support voice clone for omnivoice in online serving & add seed parameter for reproducible (#3668 ) Signed-off-by: sphinxkkkbc <binchengkang8@gmail.com> Co-authored-by: sphinxkkkbc <binchengkang8@gmail.com>	2026-05-29 07:42:08 +00:00
knlnguyen1802 Samit Hongsheng Liu	c69ed4037d	[feat] Support prompt embedding caching for diffusion model (#2962 ) Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> Co-authored-by: Samit <285365963@qq.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-29 15:36:03 +08:00
Alex Brooks	68a1d23e4d	[Bugfix/Feature] Remove Hardcoded Flash Attention in Bagel & Support GQA in SDPA Backend (#3728 ) Signed-off-by: Alex Brooks <albrooks@redhat.com>	2026-05-29 10:36:15 +08:00
Chen-Yo Sun	593c4c06aa	[BugFix] Avoid Voxtral TTS loading error msg (#3951 ) Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>	2026-05-29 02:35:01 +00:00
NATURE	4987bec425	[bugfix] Solve Nightly / CI failed - tests/e2e/online_serving/test_bagel_expansion.py #3918 (#3936 ) Signed-off-by: natureofnature <wzliu@connect.hku.hk>	2026-05-29 02:05:07 +00:00
amy-why-3459	5dd1d0a772	[BugFix] Fix prefix-caching issue (#3726 ) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>	2026-05-29 01:51:54 +00:00
ChenWenjingandGao Han	834abdaf8b	[ci][nightly] Voxcpm2 performance benchmark (#3864 ) Signed-off-by: CHEN <116010019@link.cuhk.edu.cn> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>	2026-05-29 09:44:29 +08:00
Jason	5dcf18a984	[Docs] Add recipe for Helios (#3114 ) Signed-off-by: JasonJ2021 <jasonj@zju.edu.cn>	2026-05-29 09:34:37 +08:00
Yuxiang Liang	900b4187a1	[XPU] Add CosyVoice3Model support on Intel XPU (#2325 ) Signed-off-by: Yuxiang Liang <yuliang@habana.ai> Signed-off-by: Yuxiang Liang <yuxiang.liang@intel.com>	2026-05-29 00:59:45 +00:00
wuli666andwuli666	8935439428	[Bugfix] Fix Triton SnakeBeta kernel for bf16/fp16 inputs (#3472 ) Signed-off-by: wuli666 <djjpro975@gmail.com> Co-authored-by: wuli666 <djjpro975@gmail.com>	2026-05-28 17:10:36 +00:00
Dakai An	bb9f21d0db	Fix Qwen3-omni accuracy degradation from deepstack inputs under torch.compile (#3885 ) Signed-off-by: Dakai An <dakaian108@gmail.com>	2026-05-28 23:57:56 +08:00
knlnguyen1802andSamit	5eac50f22c	[bugfix, rl] Fix sleep do not release full memory in custom pipeline (#3818 ) Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com> Co-authored-by: Samit <285365963@qq.com>	2026-05-28 22:27:27 +08:00
WeiQing Chen lishunyang Gao Han	126fb4b4a9	[Blackwell][1/N] Add SageAttention3 diffusion backend on blackwell(GB200/B200/RTX5090/PRO6000/DGX Spark available) (#3015 ) Signed-off-by: lishunyang <lishunyang12@163.com> Signed-off-by: WeiQing Chen <40507679+david6666666@users.noreply.github.com> Co-authored-by: lishunyang <lishunyang12@163.com> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>	2026-05-28 14:08:45 +00:00
zhumingjue138	8d9a4afed9	[Test] Add scenarios for L5 reliability test (#3729 ) Signed-off-by: zhumingjue <zhumingjue@huawei.com>	2026-05-28 21:41:33 +08:00
WeiQing ChenandWeiQing Chen	1acd27c205	[CI][BugFix] Fix and Validate FP8 Z-Image quality gate (#3929 ) Signed-off-by: WeiQing Chen <david6666666@users.noreply.github.com> Co-authored-by: WeiQing Chen <david6666666@users.noreply.github.com>	2026-05-28 12:52:58 +00:00
nainiu258andHongsheng Liu	740ec608f4	[Docs]Add recipe for GLM-Image on 2x A800 GPUs and 1x A800 GPU (#2950 ) Signed-off-by: nainiu258 <cperfect02@163.com> Signed-off-by: nainiu258 <101917677+nainiu258@users.noreply.github.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-28 19:50:37 +08:00
Zeyu Huang \| 黃澤宇	74258b18b7	[CI] Rectify L2~L4 Qwen Image Edit series tests (#3901 )	2026-05-28 18:33:31 +08:00
rein yang	1c876f0cd1	[Perf] Trim Code2Wav CUDA Graph buckets for Qwen3-TTS single-GPU deploy (#3932 ) Signed-off-by: Rein Yang <ruiruyang2@gmail.com>	2026-05-28 17:26:57 +08:00
mglyn	ee212d12b1	[CI][Bugfix] Add request id to LTX2.3 CFG parallel test (#3934 ) Signed-off-by: mglyn <1203789601@qq.com>	2026-05-28 17:26:16 +08:00
Nick CaoandClaude	c20afebce4	[Refactor][Qwen3-TTS] Construct speech tokenizer encoder natively (#3360 ) Signed-off-by: Nick Cao <ncao@redhat.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-05-28 16:20:49 +08:00
WeiQing Chen	0e3bf4de8e	[Doc] Reduce browser memory usage for docs (#3870 ) Signed-off-by: David Chen <530634352@qq.com>	2026-05-28 15:42:09 +08:00
Liang Lv	b72ffb82bb	[AutoRound] Support GLM-Image W4A16 quantization model (#3059 ) Signed-off-by: lvliang-intel <liang1.lv@intel.com>	2026-05-28 07:19:06 +00:00
JackyLiu	168033cac0	[Bugfix] Remove duplicate ffmpeg options in random video generation (#3923 ) Signed-off-by: JLiu4Coding <lzwgre@126.com>	2026-05-28 14:57:59 +08:00
JiangJie Zhang	6fa9dbbe42	[Diffusion] Unify diffusion request identity on request_id (#3744 ) Signed-off-by: jader <yjader@foxmail.com>	2026-05-28 14:56:56 +08:00
blondeandClaude Opus 4.6	beb00fce31	[Doc] Refine vace offline inference example README (#3584 ) Signed-off-by: blondeCS <1580754914@qq.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-05-28 14:45:45 +08:00
NATURE	3a347b10b0	[3/5][core]refactor communication layer: PR 3 of 5, all other models in non async mode (#3719 ) Signed-off-by: natureofnature <wzliu@connect.hku.hk>	2026-05-28 14:06:41 +08:00
Juan Pablo Zuluaga JuanPZuluaga Yueqian Lin	0f3264a9c0	[Perf][Bugfix] cache hot buffers in qwen3_tts talker; fall back on evicted state (#3688 ) Signed-off-by: JuanPZuluaga <juanz9312@gmal.com> Co-authored-by: JuanPZuluaga <juanz9312@gmal.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>	2026-05-28 05:45:54 +00:00
BeatSeat	dab596e468	[Refactor] Unify Snake/SnakeBeta and alias-free activation into common modules (#3886 ) Signed-off-by: BeatSeat <wendavid552@gmail.com>	2026-05-27 23:41:35 -05:00
mglyn	2b999429c4	[Feature] Add cfg-parallel for LTX-2.3 (#3905 ) Signed-off-by: mglyn <1203789601@qq.com>	2026-05-28 04:36:33 +00:00
WeiQing Chen	6031c7d393	[CI] Add HunyuanVideo 1.5 X2V accuracy tests (#3852 ) Signed-off-by: david6666666 <530634352@qq.com>	2026-05-28 11:09:45 +08:00
SYLAR frauttauteffasu Zhang Jian	373d3c7476	[Blackwell] Add CUDNN_ATTN and FLASHINFER_ATTN backends for diffusion (auto-route) (#3079 ) Signed-off-by: lishunyang <lishunyang12@163.com> Signed-off-by: Zhang <jianmusings@gmail.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-authored-by: frauttauteffasu <247219357+frauttauteffasu@users.noreply.github.com> Co-authored-by: Zhang Jian <jianmusings@gmail.com>	2026-05-28 11:09:25 +08:00
Chendi.Xue	1903cee56a	[MXFP8][XPU] enable mxfp8 using vLLM main repo method (#3782 ) Signed-off-by: Chendi Xue <chendi.xue@intel.com>	2026-05-28 02:47:22 +00:00
Yuanheng Zhao	39705f9ee8	[Feat] Support online quantization (fp8/int8) for DreamID-Omni (#3902 ) Signed-off-by: Yuanheng Zhao <jonathan.zhaoyh@gmail.com>	2026-05-28 02:31:58 +00:00
NumberWanandHongsheng Liu	8766c0c59f	[Perf] Bagel Performance Nightly CI test (#2175 ) Signed-off-by: NumberWan <wantszkin2003@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-28 09:41:46 +08:00
rongfu.leng	6ea66f8ed0	fix image edit docs about use error image url (#3873 ) Signed-off-by: rongfu.leng <lenronfu@gmail.com>	2026-05-28 01:29:57 +00:00
Ziming Huangandtzhouam	220ce0b4ea	[Test] Add long text output correctness test for Qwen3-Omni (#3539 ) Signed-off-by: ZeldaHuang <hzm414167@alibaba-inc.com> Signed-off-by: tzhouam <tzhouam@connect.ust.hk> Co-authored-by: tzhouam <tzhouam@connect.ust.hk>	2026-05-28 08:27:29 +08:00
LancerandAlicia	5c3c81920b	[Bugfix] Ensure stage and diffusion subprocesses exit when parent dies unexpectedly (#3751 ) Signed-off-by: Lancer <maruixiang6688@gmail.com> Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>	2026-05-28 08:22:04 +08:00
akshatvishu	1f3a3e3848	Fix Ovis image text encoder dtype (#3876 ) Signed-off-by: akshatvishu <akshatnayak197@gmail.com>	2026-05-28 08:21:37 +08:00
Sy03andYueqian Lin	d37912b071	[TTS][Perf] Optimize Fish Speech S2 Pro high-concurrency serving (#3773 ) Signed-off-by: Sy03 <1370724210@qq.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>	2026-05-27 13:25:49 -05:00
Hacoandzzhuoxin1508	37eebff674	[BugFix] Fix diffusion parallel_config YAML override and add deploy config field allowlist (#3483 ) Signed-off-by: xiaohajiayou <923390377@qq.com> Co-authored-by: zzhuoxin1508 <zzhuoxin1508@users.noreply.github.com>	2026-05-28 01:00:32 +08:00
Zheng Wengang Claude Opus 4.6 Yuanheng Zhao	a698269e47	[Model] Add Ming-flash-omni-2.0 Image Generation (Diffusion) Stage (#2875 ) Signed-off-by: ZhengWG <zwg0606@gmail.com> Signed-off-by: Zheng Wengang <zwg0606@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Yuanheng Zhao <54058983+yuanheng-zhao@users.noreply.github.com>	2026-05-27 23:21:20 +08:00
JohnJan	36dc32ff81	[Feature] support SP for FLUX.2-dev (#3244 )	2026-05-27 20:43:02 +08:00
Aleksi Vesanto	9abf9545df	[ROCm] Add support for AITER GroupNorm (#3419 ) Signed-off-by: Aleksi Vesanto <avesanto@amd.com>	2026-05-27 11:33:17 +00:00
bjf-frzandHongsheng Liu	c33ef34654	[Bugfix]Fix Hunyuan Image3 denoise flow alignment (#3857 ) Signed-off-by: bjf-frz <frz123db@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-27 19:18:01 +08:00
rongfu.lengandPeter Pan	c349f62839	[Feat] opt qwen image model load use ColumnParallelLinear replace ReplicatedLinear (#3875 ) Signed-off-by: rongfu.leng <lenronfu@gmail.com> Co-authored-by: Peter Pan <Peter.Pan@daocloud.io>	2026-05-27 09:02:17 +00:00
WeiQing ChenandSYLAR	41cb6378a2	[Perf] Trim HunyuanVideo encoder padding tokens (#3844 ) Signed-off-by: david6666666 <530634352@qq.com> Co-authored-by: SYLAR <125541396+lishunyang12@users.noreply.github.com>	2026-05-27 14:17:10 +08:00
JasonandZeyu Huang \| 黃澤宇	753508fe02	[Bugfix] Fix Qwen-Image-Edit-2511 TeaCache zero_cond_t handling (#3219 ) Signed-off-by: JasonJ2021 <jasonj@zju.edu.cn> Signed-off-by: Jason <72191212+JasonJ2021@users.noreply.github.com> Co-authored-by: Zeyu Huang \| 黃澤宇 <11222265+fhfuih@users.noreply.github.com>	2026-05-27 11:06:18 +08:00
Dnoob	a3802d1e5c	[Doc][Benchmark] Rewrite benchmarks/README.md as repo-wide index (#1939 ) Signed-off-by: root <Dnoob> Signed-off-by: Dnoob <dxpouo@gmail.com>	2026-05-26 16:13:31 -10:00
TaffyOfficialandTaffyOfficial	3c48257f8d	[Feat][HunyuanImage3] Stream AR text for IT2I image edits (#3723 ) Signed-off-by: TaffyOfficial <2324465096@qq.com> Co-authored-by: TaffyOfficial <2324465096@qq.com>	2026-05-27 12:27:11 +12:00
lyj-jjj Cursor Hongsheng Liu	283873411a	glm-image: fix(npu)per-stage runtime env for HCCL ports + GLM-Image NPU stage config (#3235 ) Signed-off-by: lyj-jjj <liuyingjun5@huawei.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-27 12:18:47 +12:00
Dan	1f1c82ecdd	[BugFix] Fix Qwen3-TTS Code2Wav compatibility with transformers >= 5.9.0 (#3880 ) Signed-off-by: Dan250124 <416947747@qq.com>	2026-05-26 16:24:43 +00:00
Zeng Chuang	e459fdb30a	[bugfix] fix default deploy config in hunyuan_image offline example (#3879 ) Signed-off-by: zengchuang <zengchuang3@huawei.com>	2026-05-26 15:15:21 +00:00
WeiQing Chen	9322cc6176	[Docs] Strengthen diffusion perf optimization quality gate (#3851 ) Signed-off-by: david6666666 <530634352@qq.com>	2026-05-26 14:18:03 +00:00
Hongsheng Liu	abc31731ab	Update CUDA Docker base image to vLLM v0.21.0 (#3859 ) Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>	2026-05-26 21:00:06 +08:00
mglyn	385df0359e	[BugFix] Fix LTX-2.3 audio latent padding for sequence parallelism (#3854 ) Signed-off-by: mglyn <1203789601@qq.com>	2026-05-26 08:45:38 +00:00
dengyunyang	36eb8afba5	[HunyuanImage][End2End Performance CI] Add hunyuan end2end test (#3849 ) Signed-off-by: dengyunyang <584797741@qq.com>	2026-05-26 16:45:05 +08:00
bjf-frz	ec94e83a25	[CI][Bugfix]Fix Wan2.2 I2V reference image upload (#3869 ) Signed-off-by: bjf-frz <frz123db@gmail.com>	2026-05-26 16:23:28 +08:00
Zheng WengangandHongsheng Liu	cad25fa20c	[CI] Add nightly-ci for multi-stage deployment (#3610 ) Signed-off-by: ZhengWG <zwg0606@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>	2026-05-26 11:16:43 +08:00
NumberWan	11a951b962	[BugFix]: CUDA `device-side assert` failures on single-stage BAGEL i2i requests (#3680 ) Signed-off-by: NumberWan <wantszkin2003@gmail.com>	2026-05-26 03:01:10 +00:00
Didan DengandGao Han	410a6a20b0	[Test] Qwen-Image Perf Test with High Concurrency (#2822 ) Signed-off-by: Didan Deng <33117903+wtomin@users.noreply.github.com> Signed-off-by: Gao Han <hgaoaf@connect.ust.hk> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>	2026-05-26 02:56:01 +00:00
Canlin Guo	52020c7af7	[UX] Rename default config to hunyuan_image_3_moe (#3835 ) Signed-off-by: gcanlin <canlinguosdu@gmail.com>	2026-05-26 01:41:15 +08:00
Yuekai Zhang Claude Opus 4.7 Yueqian Lin	8f45e68bcd	[Higgs-Audio] bosonai/higgs-audio-v2-generation-3B-base TTS model support (#3762 ) Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>	2026-05-25 23:16:07 +08:00
rongfu.leng	80dea88007	[Bugfix] fix when diffusion model not set sleeping_stages (#3023 ) Signed-off-by: rongfu.leng <lenronfu@gmail.com> thanks	2026-05-25 23:13:38 +08:00
Yueqian LinandRein Yang	e996723bc9	[Bugfix] Fix Qwen3-TTS Stage 0 prefix-caching correctness (#3665 ) Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Signed-off-by: Rein Yang <ruiruyang2@gmail.com> Co-authored-by: Rein Yang <ruiruyang2@gmail.com> the ci failure case is irrelavent to this PR and occurs rarely	2026-05-25 23:09:38 +08:00