[Rebase] Rebase to vllm 0.28.0 (#6606)

Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

.buildkite/amd/test-amd-ready.yml

@@ -192,7 +192,7 @@ steps:
       grade: Blocking
       commands:
         - |
-          timeout 20m bash -c '
+          timeout 45m bash -c '
             pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py -m "slow" --run-level "core_model"
           '
 

.buildkite/cuda/test-ready.yml

@@ -306,6 +306,24 @@ steps:
             "
         mirror_hardwares: h100_1
 
+      - label: "Diffusion · MiniMax-H3 DLO DP2 T2VA Test"
+        timeout_in_minutes: 90
+        source_file_dependencies:
+          - tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py
+          - tests/helpers/assertions.py
+          - tests/helpers/runtime.py
+          - vllm_omni/diffusion/models/minimax_h3/
+          - vllm_omni/diffusion/offloader/
+          - vllm_omni/diffusion/distributed/
+          - vllm_omni/entrypoints/openai/
+        commands:
+          - |
+            timeout 80m bash -c "
+              export VLLM_IMAGE_FETCH_TIMEOUT=60
+              pytest -s -v tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py -m 'core_model and cuda and diffusion' --run-level 'core_model'
+            "
+        mirror_hardwares: h100_2
+
       - label: "Diffusion · Qwen Image Test"
         source_file_dependencies:
           - tests/e2e/online_serving/test_qwen_image.py

docker/Dockerfile.ci

@@ -1,4 +1,4 @@
-# Target vllm version: v0.27.0 (releases/v0.27.0 branch).
+# Target vllm version: v0.28.0 (tag v0.28.0 @ 2cf0a6915ce5).
 # Keep VLLM_BASE_TAG aligned with the target vLLM release on every rebase.
 #
 # Release-boundary form: there is no precompiled-wheel pin here, because the
@@ -14,11 +14,14 @@
 # core init), registering libnvJitLink.so.13 with ldconfig, and re-pinning numpy
 # under numba's ceiling. All of it was compensation for a base image and a vLLM
 # that disagreed. At a real release boundary they agree, so none of it is
-# needed. If you pin an untagged commit again, restore the reinstall block —
-# agent/tasks/20_update_wheel_url.sh fails the run rather than letting the
-# Dockerfile silently ship the wrong vLLM.
+# needed. The 0.28 rebase repeated the cycle: the v0.28.0rc / final-tag wheels
+# rode on the v0.27.0 base image with the full reinstall block until the
+# official v0.28.0 image shipped, at which point this file dropped back to the
+# form you are reading. If you pin an untagged commit again, restore the
+# reinstall block — agent/tasks/20_update_wheel_url.sh fails the run rather
+# than letting the Dockerfile silently ship the wrong vLLM.
 ARG VLLM_BASE_IMAGE=vllm/vllm-openai
-ARG VLLM_BASE_TAG=v0.27.0
+ARG VLLM_BASE_TAG=v0.28.0
 
 # We use registry-based layer caching for the dependencies,
 # where the tag is determined based on the hash of the contents

docker/Dockerfile.cuda

@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=vllm/vllm-openai:v0.26.0
+ARG BASE_IMAGE=vllm/vllm-openai:v0.28.0
 FROM ${BASE_IMAGE}
 
 ARG COMMON_WORKDIR=/app

docker/Dockerfile.rocm

@@ -1,4 +1,4 @@
-ARG BASE_IMAGE=vllm/vllm-openai-rocm:v0.27.0
+ARG BASE_IMAGE=vllm/vllm-openai-rocm:v0.28.0
 FROM ${BASE_IMAGE} AS base
 
 # Declare a variable to know if we want to use the nightly build or the stable build.
@@ -10,7 +10,7 @@ FROM ${BASE_IMAGE} AS base
 #    we should swap over to use stable release ASAP.
 #    We should avoid relying on custom commits.
 ARG USE_NIGHTLY_BUILD=0
-ARG VLLM_VERSION_OR_COMMIT_HASH=v0.27.0
+ARG VLLM_VERSION_OR_COMMIT_HASH=v0.28.0
 ARG ARG_PYTORCH_ROCM_ARCH
 ENV PYTORCH_ROCM_ARCH=${ARG_PYTORCH_ROCM_ARCH:-${PYTORCH_ROCM_ARCH}}
 

docs/.nav.yml

@@ -53,6 +53,7 @@ nav:
       - BitsAndBytes W4: user_guide/quantization/bitsandbytes.md
       - MXFP8 W8A8: user_guide/quantization/mxfp8.md
       - MXFP4 W4A4: user_guide/quantization/mxfp4.md
+      - SVDQuant W4A4: user_guide/quantization/svdquant.md
       - ModelOpt: user_guide/quantization/modelopt.md
       - AutoRound: user_guide/quantization/autoround.md
       - msModelSlim: user_guide/quantization/msmodelslim.md

docs/configuration/environment_variables.md

@@ -83,6 +83,7 @@ depends on the installed kernels and model path.
 | `SPEAKER_SAMPLES_DIR` | Filesystem path; default `~/.cache/vllm-omni/speakers` | Speech server; read when speaker storage initializes | Environment-only setting. The directory is created; filesystem errors propagate. | Stable |
 | `SPEAKER_MAX_UPLOADED` | Integer; default `1000` | Speech server; read when speaker storage initializes | Environment-only setting. A non-integer logs a warning and uses `1000`; range is not otherwise validated. | Stable |
 | `VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT` | Float seconds; default `600` | Diffusion engine async-output wait in `step_streaming`; resolved per call on the request path, not at import | Environment-only setting. A non-float or `<=0` value warns once and uses the default. | Experimental |
+| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `1`, `true`, `yes` or `on` enables; default `0` (off) | Orchestration loop and the serving-side final-output drain; read once when the `Orchestrator` is constructed | Environment-only setting. Values are stripped and case-normalized; any unrecognized value leaves the legacy poll loop selected. | Experimental |
 | `VLLM_OMNI_INPUT_WAIT_TIMEOUT_S` | Float seconds; default `600`; `<=0` disables | Full-payload input coordinator, not async-chunk transfer; read when the scheduler module imports in each worker | Environment-only setting. A non-float logs a warning and uses `600`. | Stable operational control |
 | `VLLM_OMNI_ORCH_MONITOR_PATH` | Filesystem path; default `<current-working-directory>/vllm_omni_orch_monitor_<timestamp>.json` | Orchestrator monitor enabled by `--enable-orch-monitor`; read when the monitor is created | Environment-only path override. Parent directories are created; write errors are logged. | Diagnostic |
 | `VLLM_OMNI_VIDEO_SYNC_TIMEOUT` | Float seconds; default `600` | Synchronous Videos API; read when the API server module imports | Environment-only setting. A non-float raises `ValueError` during import. | Experimental |

docs/configuration/stage_configs.md

@@ -94,12 +94,14 @@ Each entry under `stages:` accepts any `StageDeployConfig` field directly (no ne
 | `max_num_batched_tokens` | int \| null | optional | `null` | Per-stage prefill/token budget; also contributes to the native maximum in-flight token limit. |
 | `max_model_len` | int \| null | optional | `null` | Per-sequence context or KV length; `-1` enables native cache-capacity auto-fitting, while values above the HF default auto-set `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`. |
 | `async_scheduling` | bool \| null | optional | `null` | Per-stage async scheduling toggle. |
-| `devices` | str \| null | optional | `null` | Device list assigned to this stage. |
+| `devices` | str \| null | optional | `null` | Device list assigned to this stage. The number of device ids must equal this stage's local world size (`tensor_parallel_size` × local data-parallel size × `pipeline_parallel_size`, or `num_replicas` × that product for a replica pool); a mismatch fails early — see the note below. |
 | `output_connectors` | dict \| null | optional | `null` | Keyed by `to_stage_<n>`; values are names registered under top-level `connectors:`. |
 | `input_connectors` | dict \| null | optional | `null` | Keyed by `from_stage_<n>`; values are names registered under top-level `connectors:`. |
 | `default_sampling_params` | dict \| null | optional | `null` | Baseline sampling params. Deep-merged with pipeline `sampling_constraints` (pipeline wins). |
 | `engine_extras` | dict | optional | `{}` | Catch-all for engine fields not listed above; deep-merged across overlays and forwarded to the stage engine. |
 
+**Note:** a stage's `devices` count must equal its local world size (`tensor_parallel_size` × `data_parallel_size_local` × `pipeline_parallel_size`, falling back to global `data_parallel_size` when the local size is unset), or `num_replicas` × that product for a replica pool. A mismatch fails early and names the offending stage. A top-level `--tensor-parallel-size` is broadcast to every stage, so it can make a single-GPU stage violate this contract ([issue #5003](gh-issue:5003)); fix that case with `--stage-overrides` (set `tensor_parallel_size` and `devices` together per stage) or set TP only on the multi-GPU stage.
+
 ### Connector schema
 
 Each entry under top-level `connectors:` follows this shape:

docs/contributing/profiling.md

@@ -322,6 +322,15 @@ Each 1-second window records:
 
 On shutdown the server also logs a short summary (`loop_active_pct`, per-replica queue averages/maxima).
 
+`loop_idle` and `loop_active` count loop iterations, so their absolute scale is
+tied to which orchestration loop is running. Under the default poll loop an idle
+orchestrator records roughly one iteration per millisecond. Under the
+event-driven loop (`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, see
+[Speech API](../serving/speech_api.md#orchestration-loop-experimental)) an idle
+orchestrator wakes only once per 0.5 s reconcile timeout, while a busy one still
+records one iteration per routed output. Window counts and the `loop_active_pct`
+summary are therefore not comparable across the two modes.
+
 ### Relationship to other diagnostics
 
 This monitor is intentionally separate from the existing profiling tools:

docs/design/feature/host_weight_runtime.md

@@ -14,12 +14,12 @@ specified in the [Host Weight Runtime module design](../module/host_weight_runti
 
 ## Status
 
-The first implementation provides contracts and a CPU local-filesystem store.
-The initial diffusion consumer contract additionally defines typed,
+The implementation provides contracts and a CPU local-filesystem store. The
+diffusion consumer additionally defines typed,
 representation-independent final-layout identity/restoration mechanics plus a
-concrete BF16-with-preserved-FP32 policy for MiniMax H3. It remains
-library-only: no loader or DLO path selects, publishes, restores, or transports
-that artifact yet.
+concrete BF16-with-preserved-FP32 policy for MiniMax H3 and
+`black-forest-labs/FLUX.2-klein-4B`. The opt-in no-AllGather DLO integration
+selects, publishes, restores, and transfers these artifacts.
 
 V1 includes:
 
@@ -33,13 +33,12 @@ V1 includes:
 
 V1 does not include:
 
-- a public CLI, loader activation, or default-on model integration;
-- lease handoff to DLO or another transport consumer;
-- FP8, quantized, merged-adaptation, or additional model producers;
-- CUDA registration, pinned staging, H2D scheduling, or GPU kernels;
+- default-on activation or consumers outside no-AllGather DLO;
+- online FP8, quantized, merged-adaptation, or additional model producers;
+- HWR interaction with DLO AllGather;
 - a remote artifact provider or cross-node coordination;
 - automatic eviction; or
-- a change to DLO AllGather or no-AllGather behavior.
+- a change to DLO collective or execution behavior.
 
 ## Motivation and use cases
 
@@ -255,7 +254,8 @@ The contract is intentionally separate from loader activation:
 - `FinalLayoutBF16Producer` accepts only the matching identity context and a
   finalized CPU model. It is `POST_LOAD_ONLY` and `SINGLE_PROCESS` per exact TP
   coordinate. Its BF16 policy preserves model-declared FP32 parameters and
-  buffers and revalidates MiniMax H3 mixed-precision invariants.
+  buffers, revalidates MiniMax H3 mixed-precision invariants, and revalidates
+  FLUX.2-klein's two block stacks, packed QKV mapping, and BF16 base layout.
 
 Other representations reuse source identity, typed parallel identity, tensor
 ownership, and exact restoration only when their policy proves those semantics.

docs/design/feature/offloader/distributed_layerwise_offload.md

@@ -220,9 +220,15 @@ model revision and parallel layout with `required`. TP coordinates have distinct
 identities, while equivalent DP replicas share them.
 
 The current producer/consumer boundary is the model-declared final-layout BF16
-contract (MiniMax H3 today). It supports ordinary-loader final layouts for TP1
-and TP2 rank identities plus SP layout identities; online quantization, HSDP,
-LoRA/adapted weights, and non-default load formats remain ineligible. HWR mode
+contract (MiniMax H3 and `black-forest-labs/FLUX.2-klein-4B`). The FLUX.2-klein
+contract covers both transformer block stacks, constructor-stable packed QKV
+mapping state, BF16 parameters, and any persistent loader-owned `beta`/`eps`
+buffers. The shared HWR machinery supports ordinary-loader final layouts for
+TP1 and TP2 rank identities plus SP layout identities. FLUX.2-klein evidence
+in this change is TP1-only; its TP2/SP layouts rely on the existing
+per-coordinate identity mechanics and remain unmeasured. Online quantization,
+HSDP, LoRA/adapted weights, and non-default load formats remain ineligible.
+HWR mode
 `disabled`, DLO-disabled, and DLO AllGather configurations stop before source
 identity or store construction and retain the existing checkpoint-mmap or
 ordinary-loader path.

docs/design/module/engine_orchestration.md

@@ -66,6 +66,12 @@ the boundary affected by the in-flight stage client/process refactor in
 [#5441](https://github.com/vllm-project/vllm-omni/pull/5441). Names and
 responsibilities proposed only by that PR are not current contracts.
 
+The orchestration loop also has an opt-in event-driven mode
+(`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`, default off) proposed in
+[#5221](https://github.com/vllm-project/vllm-omni/pull/5221). It changes poll
+cadence only: the routing, ordering, and terminal-state contracts below hold
+identically on both loops.
+
 ## Ownership boundary
 
 This document owns `AsyncOmniEngine`, `Orchestrator`, request-state creation,

docs/getting_started/installation/README.md

@@ -1,7 +1,7 @@
 # Installation
 
 !!! important
-    vLLM-Omni is released against the matching upstream vLLM major/minor version. vLLM-Omni 0.26.x requires vLLM 0.26.x; the stable v0.26.0 instructions pin vLLM 0.26.0.
+    vLLM-Omni is released against the matching upstream vLLM major/minor version. vLLM-Omni 0.28.x requires vLLM 0.28.x; the stable v0.28.0 instructions pin vLLM 0.28.0.
 
 vLLM-Omni supports the following hardware platforms:
 

docs/getting_started/installation/gpu.md

@@ -34,7 +34,7 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr
 
 ### Pre-built wheels
 
-Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, and 0.26.0. If you need a newer unreleased revision, please [build from source](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/#build-wheel-from-source).
+Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, and 0.26.0; the 0.28.0 wheels are published together with the v0.28.0 release this branch produces. If you need a newer unreleased revision, please [build from source](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/#build-wheel-from-source).
 
 === "NVIDIA CUDA"
 

docs/getting_started/installation/gpu/cuda.inc.md

@@ -20,7 +20,7 @@ Therefore, it is recommended to install vLLM and vLLM-Omni with a **fresh new**
 
 vLLM-Omni is built based on vLLM. Please install it with command below.
 ```bash
-uv pip install vllm==0.26.0 --torch-backend=auto
+uv pip install vllm==0.28.0 --torch-backend=auto
 ```
 
 #### Installation of vLLM-Omni
@@ -39,13 +39,13 @@ uv pip install 'vllm-omni[demo]'
 # --8<-- [start:build-wheel-from-source]
 
 #### Installation of vLLM
-If you do not need to modify source code of vLLM, you can directly install the stable 0.26.0 release version of the library
+If you do not need to modify source code of vLLM, you can directly install the stable 0.28.0 release version of the library
 
 ```bash
-uv pip install vllm==0.26.0 --torch-backend=auto
+uv pip install vllm==0.28.0 --torch-backend=auto
 ```
 
-The 0.26.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
+The 0.28.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
 
 #### Installation of vLLM-Omni
 Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -66,12 +66,12 @@ If you want to check, modify or debug with source code of vLLM, install the libr
 ```bash
 git clone https://github.com/vllm-project/vllm.git
 cd vllm
-git checkout v0.26.0
+git checkout v0.28.0
 ```
 Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set `VLLM_PRECOMPILED_WHEEL_LOCATION` as your local absolute path of whl file.
 ```bash
-#For CUDA 13.0 (the default for v0.26.0; the wheel filename has no `+cu130` suffix)
-export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.26.0/vllm-0.26.0-cp38-abi3-manylinux_2_28_x86_64.whl
+#For CUDA 13.0 (the default for v0.28.0; the wheel filename has no `+cu130` suffix)
+export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0-cp38-abi3-manylinux_2_28_x86_64.whl
 ```
 Install vllm with command below (If you have no existing PyTorch).
 ```bash
@@ -102,7 +102,7 @@ docker run --runtime nvidia --gpus 2 \
     --env "HF_TOKEN=$HF_TOKEN" \
     -p 8091:8091 \
     --ipc=host \
-    vllm/vllm-omni:v0.26.0 \
+    vllm/vllm-omni:v0.28.0 \
     vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
 ```
 
@@ -124,7 +124,7 @@ If you want to specify the base vLLM version:
 ```bash
 DOCKER_BUILDKIT=1 docker build \
   -f docker/Dockerfile.cuda \
-  --build-arg BASE_IMAGE=vllm/vllm-openai:v0.26.0 \
+  --build-arg BASE_IMAGE=vllm/vllm-openai:v0.28.0 \
   -t vllm-omni-cuda .
 ```
 

docs/getting_started/installation/gpu/rocm.inc.md

@@ -13,7 +13,7 @@ vLLM-Omni current recommends the steps in under setup through Docker Images.
 
 vLLM-Omni is built based on vLLM. Please install it with command below.
 ```bash
-uv pip install vllm==0.26.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.26.0/rocm723
+uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
 ```
 
 #### Installation of vLLM-Omni
@@ -34,13 +34,13 @@ uv pip install onnxruntime-rocm
 # --8<-- [start:build-wheel-from-source]
 
 #### Installation of vLLM
-If you do not need to modify source code of vLLM, you can directly install the stable 0.26.0 release version of the library
+If you do not need to modify source code of vLLM, you can directly install the stable 0.28.0 release version of the library
 
 ```bash
-uv pip install vllm==0.26.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.26.0/rocm723
+uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
 ```
 
-The pre-built 0.26.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
+The pre-built 0.28.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
 
 #### Installation of vLLM-Omni
 Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -58,7 +58,7 @@ If you want to check, modify or debug with source code of vLLM, install the libr
 ```bash
 git clone https://github.com/vllm-project/vllm.git
 cd vllm
-git checkout v0.26.0
+git checkout v0.28.0
 python3 -m pip install -r requirements/rocm.txt
 python3 setup.py develop
 ```
@@ -130,7 +130,7 @@ docker run --rm \
   -v ~/.cache/huggingface:/root/.cache/huggingface \
   --env "HF_TOKEN=$HF_TOKEN" \
   -p 8091:8091 \
-  vllm/vllm-omni-rocm:v0.26.0 \
+  vllm/vllm-omni-rocm:v0.28.0 \
   --model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
 ```
 
@@ -149,7 +149,7 @@ docker run --rm -it \
   -v ~/.cache/huggingface:/root/.cache/huggingface \
   --env "HF_TOKEN=$HF_TOKEN" \
   --entrypoint bash \
-  vllm/vllm-omni-rocm:v0.26.0
+  vllm/vllm-omni-rocm:v0.28.0
 ```
 
 # --8<-- [end:pre-built-images]

docs/getting_started/quickstart.md

@@ -19,10 +19,10 @@ uv venv --python 3.12 --seed
 source .venv/bin/activate
 
 # On CUDA
-uv pip install vllm==0.26.0 --torch-backend=auto
+uv pip install vllm==0.28.0 --torch-backend=auto
 
 # On ROCm
-uv pip install vllm==0.26.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.26.0/rocm723
+uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
 
 git clone https://github.com/vllm-project/vllm-omni.git
 cd vllm-omni
@@ -34,7 +34,7 @@ For additional installation methods — please see the [installation guide](inst
 !!! note
     It is important to install the same major & minor version of vLLM and vLLM Omni, otherwise things may not work as expected. If the versions are misaligned, you will see a warning when you import vLLM Omni.
 
-    If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.26.0` and vLLM Omni `0.26.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.
+    If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.28.0` and vLLM Omni `0.28.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.
 
 ## Offline Inference
 

docs/serving/speech_api.md

@@ -811,6 +811,51 @@ If you encounter OOM errors:
 
 Use `/v1/audio/voices` to list available voices for the loaded model.
 
+## Orchestration Loop (experimental)
+
+Multi-stage omni deployments route stage outputs through a single orchestrator
+loop. By default that loop polls every stage replica on a 1 ms cadence. An
+opt-in event-driven mode replaces the poll with one reader task per live stage
+replica awaiting its client directly, and switches the serving-side
+final-output drain to a condition-variable wakeup at the same time.
+
+**Configuration (environment variables):**
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `VLLM_OMNI_EVENT_DRIVEN_ORCH` | `0` (off) | Switches the orchestration loop and the final-output drain from the legacy 1 ms poll to event-driven wakeups. Enabled by `1`, `true`, `yes`, or `on`, matched case-insensitively after surrounding whitespace is stripped; any other value leaves it off. |
+
+Set it on the process that runs the orchestrator (stage 0 of an omni
+deployment) before starting the server:
+
+```bash
+export VLLM_OMNI_EVENT_DRIVEN_ORCH=1
+vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \
+    --omni \
+    --port 8091
+```
+
+The server logs the selected loop mode and its reader/poller counts once at
+startup, so you can confirm which loop is live.
+
+Routing, output ordering, and terminal-state behavior are identical on both
+loops; only the poll cadence changes. Leaving the variable unset keeps the
+legacy poll loop, which is the supported default.
+
+**Known limitations:**
+
+- The measured serving A/B (idle CPU 2.43% to 0.07%; TTFP p99 -32% at
+  concurrency 8) predates the rebuild on the per-replica fault-isolation work
+  in [#4285](https://github.com/vllm-project/vllm-omni/pull/4285). That work
+  changed dead-replica handling and reader/poller lifecycle rather than the
+  steady-state output path, and the parity suite covers it, but the serving
+  A/B has not been re-run on the current head.
+- The diffusion-poller branch is covered by unit tests only. Deployments whose
+  stages all run as standard engine cores never exercise it, including GLM-TTS,
+  which deploys its DiT without `stage_type: diffusion`.
+- Concurrency 1 and 32 measured at parity with the legacy loop. At 32 the
+  latency is admission-bound, which this mode does not address.
+
 ## Development
 
 Enable debug logging:

docs/user_guide/diffusion/offloader/distributed_layerwise_offload.md

@@ -112,6 +112,12 @@ perform a collective and uses the same rank-local transfer behavior.
 HWR is an opt-in startup optimization for models that declare the final-layout
 BF16 restore contract. Use it only with no-AllGather DLO:
 
+The validated BF16 model contracts currently cover MiniMax H3 and
+`black-forest-labs/FLUX.2-klein-4B`. FLUX.2-klein-9B shares the same model
+class but has not been validated against this contract. FLUX.2-dev, online
+FP8, HSDP, LoRA/adapted weights, and non-default load formats remain outside
+the validated scope.
+
 ```bash
 vllm serve /path/to/model --omni \
   --enable-distributed-layerwise-offload \

docs/user_guide/quantization/svdquant.md

@@ -0,0 +1,50 @@
+# SVDQuant W4A4
+
+## Overview
+
+[SVDQuant](https://arxiv.org/abs/2411.05007) combines four-bit weights
+and activations with a small low-rank branch that corrects part of the
+quantization error. vLLM-Omni consumes an offline-quantized checkpoint;
+it does not calibrate the model while loading.
+
+## Checkpoint contract
+
+Place the following entry in the diffusion transformer's `config.json`:
+
+```json
+{
+  "quantization_config": {
+    "quant_method": "svdquant",
+    "rank": 32,
+    "precision": "nvfp4",
+    "act_unsigned": false,
+    "modules_to_not_convert": []
+  }
+}
+```
+
+For a quantized linear with input size `K`, output size `N`, and correction
+rank `R`, the checkpoint stores:
+
+| Suffix | Shape | dtype |
+| --- | --- | --- |
+| `qweight` | `(N, K / 2)` | `int8` (two packed FP4 values per byte) |
+| `wscales` | `(K / 16, N)` | `float8_e4m3fn` |
+| `proj_down` | `(K, R)` | `bfloat16` |
+| `proj_up` | `(N, R)` | `bfloat16` |
+| `smooth_factor` | `(K,)` | `bfloat16` |
+| `wcscales` | `(N,)` | `bfloat16` |
+| `wtscale` | `(1,)` | `bfloat16` |
+
+`K` must be divisible by 16 on every tensor-parallel rank. Set `wcscales` to
+ones when no per-output correction is needed. Modules listed in
+`modules_to_not_convert` keep their checkpoint precision.
+
+## Runtime support
+
+The compatibility path accepts BF16 inputs and executes an NVFP4 GEMM followed
+by the BF16 rank correction. It supports vLLM's FlashInfer, CUTLASS, and FBGEMM
+NVFP4 tensor layouts; incompatible forced backends fail during model loading.
+SM103 is the currently validated and enabled hardware target. Native fusion of
+the NVFP4 GEMM and rank correction is separate from this checkpoint-loading
+contract.

examples/__init__.py

@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""vLLM-Omni example scripts, importable as a package.
+
+Keep this ``__init__.py``: with the upstream vLLM installed editable (which also
+places an ``examples`` package on ``sys.path``), a regular package at this repo
+root wins over the upstream ``examples`` namespace during ``import examples``.
+"""

recipes/NVIDIA/NemotronLabs-VoiceChat.md

@@ -1,7 +1,7 @@
 # NemotronLabs VoiceChat 11B
 
-> Offline speech-to-speech: a 16 kHz user utterance in, the agent's spoken reply
-> (text + 22.05 kHz WAV) out, on a 3-stage vLLM-Omni pipeline.
+> Speech-to-speech on a 3-stage vLLM-Omni pipeline: offline single-turn
+> inference and experimental model-native, frame-locked Realtime serving.
 
 ## Summary
 
@@ -11,8 +11,8 @@
   timeline: a Conformer + NemotronH hybrid-Mamba thinker emits one text token
   per acoustic frame, a Gemma3-1B EAR-TTS talker turns the text timeline into
   31-quantizer RVQ code stacks, and an RVQ-VAE codec decodes them to audio.
-- Mode: Offline single-turn inference (batch=1). Online/duplex serving,
-  function calling, and batch>1 are documented follow-ups.
+- Mode: Offline single-turn inference (batch=1), plus experimental native
+  duplex Realtime serving at one 80 ms microphone frame per scheduler wake.
 - Maintainer: [`@yuekaizhang`](https://github.com/yuekaizhang)
 
 ## When to use this recipe
@@ -33,6 +33,11 @@ NeMo modules (`nemo_vendored/`), so no `nemo_toolkit` install is needed.
   [`vllm_omni/model_executor/models/nemotron_voicechat/`](../../vllm_omni/model_executor/models/nemotron_voicechat/)
 - Staged pipeline config:
   [`vllm_omni/deploy/nemotron_labs_voicechat.yaml`](../../vllm_omni/deploy/nemotron_labs_voicechat.yaml)
+- Native duplex config and E2E probe:
+  [`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml`](../../vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml),
+  [`tests/e2e/online_serving/nemotron_voicechat_realtime_duplex.py`](../../tests/e2e/online_serving/nemotron_voicechat_realtime_duplex.py)
+- Nightly model-level gate:
+  [`tests/e2e/online_serving/test_nemotron_voicechat_duplex.py`](../../tests/e2e/online_serving/test_nemotron_voicechat_duplex.py)
 - Upstream model card:
   [`nvidia/NVIDIA-NemotronLabs-VoiceChat-11B`](https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B)
 - Reference implementation: NVIDIA-NeMo/Speech, branch `nemotron-labs-voicechat`
@@ -60,7 +65,7 @@ output stayed within one word of the reference in testing.
 
 - OS: Linux
 - Python: 3.12
-- vLLM version: 0.26.0
+- vLLM version: 0.27.0
 - vLLM-Omni version or commit: this PR / current `main`
 
 #### Command
@@ -88,6 +93,55 @@ The reply text should read as a coherent spoken-style answer to the question in
 the input WAV, and the WAV should transcribe to (approximately) the same text
 with any ASR model.
 
+#### Native duplex serving
+
+The checkpoint does not contain the underlying Nemotron text tokenizer. Set
+`NEMOTRON_VOICECHAT_LLM_PATH` to a local snapshot of
+`nvidia/NVIDIA-Nemotron-Nano-9B-v2` before starting an offline deployment.
+
+```bash
+export HF_HUB_OFFLINE=1
+export TRANSFORMERS_OFFLINE=1
+export NEMOTRON_VOICECHAT_LLM_PATH=/path/to/NVIDIA-Nemotron-Nano-9B-v2
+
+vllm-omni serve /path/to/NVIDIA-NemotronLabs-VoiceChat-11B \
+  --omni \
+  --served-model-name nemotron-voicechat \
+  --deploy-config vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml
+```
+
+The single-GPU profile runs three eager fp32 engine processes on one device.
+
+In another shell, stream only the user channel of one of NVIDIA's bundled
+stereo conversation fixtures:
+
+```bash
+python tests/e2e/online_serving/nemotron_voicechat_realtime_duplex.py \
+  --model nemotron-voicechat \
+  --input-wav /path/to/NVIDIA-NemotronLabs-VoiceChat-11B/turn_taking.wav \
+  --input-channel 0 \
+  --output-dir results/nemotron_voicechat_duplex \
+  --timeout-s 600
+```
+
+The probe requires a completed response, non-silent 22.05 kHz output, and the
+advertised native 80 ms append capabilities. Add `--expect-function-call` and
+use `tool_call.wav` to validate the function-call channel. To validate the
+Realtime tool round trip, also provide the expected arguments and return a
+tool result:
+
+```bash
+python tests/e2e/online_serving/nemotron_voicechat_realtime_duplex.py \
+  --model nemotron-voicechat \
+  --input-wav /path/to/NVIDIA-NemotronLabs-VoiceChat-11B/tool_call.wav \
+  --output-dir results/nemotron_voicechat_tool_call \
+  --expect-function-call \
+  --expected-function-name generate_random_number \
+  --expected-function-arguments '{"min":1,"max":50}' \
+  --function-output 20 \
+  --expected-post-tool-text "random number"
+```
+
 #### Notes
 
 - Memory usage: the shipped yaml runs all three stages on one GPU
@@ -109,5 +163,14 @@ with any ASR model.
   instead of pausing.
 - The talker's `max_tokens` is 16383 (its stage prompt is one placeholder
   token, and the stage context is 16384).
-- Known limitations: batch=1 only; offline single turn (no online/duplex
-  serving yet); the function-call channel is decoded but not yet acted on.
+- The 80 ms frame period is a model/protocol contract, not a throughput
+  guarantee. In an eager fp32 H200 profile, Stage 2 emitted 80 ms packets at
+  roughly 0.87--0.88 s intervals (RTF about 10.8--10.9); placing the three
+  stages on separate GPUs did not remove the serial 8-step EAR-TTS bottleneck.
+  The current native duplex path is functionally streaming but not wall-clock
+  realtime.
+- Known limitations: batch=1 only. The native duplex deployment allows one
+  active session and does not support barge-in. Tool execution remains
+  client-owned: the server emits function-call events, accepts a validated
+  `function_call_output`, and resumes the live model with the returned result;
+  it does not execute arbitrary tools itself.

recipes/README.md

@@ -87,7 +87,7 @@ recipes/
 | [`Wan-AI/Wan2.1-VACE.md`](./Wan-AI/Wan2.1-VACE.md) | Unified T2V, I2V, V2LF, FLF2V, inpaint, and R2V | 1x RTX 5090 (1.3B) / 1x L40S 48GB with layerwise offload (14B) |
 | [`XiaomiMiMo/MiMo-Audio.md`](./XiaomiMiMo/MiMo-Audio.md) | Offline + online omni audio (TTS, ASR, dialogue) | 1x RTX 5090 / 5090D 32GB |
 | [`StabilityAI/Stable-Diffusion-3.5.md`](./StabilityAI/Stable-Diffusion-3.5.md) | Text-to-image serving (SD 3.5-medium and SD 3.5-large) | 1x RTX A6000 48GB |
-| [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB |
+| [`zai-org/GLM-TTS.md`](./zai-org/GLM-TTS.md) | Online serving for Chinese/English zero-shot voice-cloned TTS | 1x A40 48GB / 2x non-standard RTX 4090 48GB (49,140 MiB each) |
 | [`GLM/GLM-Image.md`](./GLM/GLM-Image.md) | Online serving for image generation | 1x A800 80GB / 2x A800 80GB |
 | [`JD/JoyAI-VL-Interaction.md`](./JD/JoyAI-VL-Interaction.md) | Real-time streaming video-language interaction (proactive speak/silence/delegate) | 1x GPU 24GB+ |
 

recipes/zai-org/GLM-TTS.md

@@ -1,4 +1,4 @@
-# GLM-TTS for Chinese/English TTS on 1x GPU
+# GLM-TTS for Chinese/English TTS
 
 ## Summary
 
@@ -103,3 +103,137 @@ python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
 - Key flags: `--omni` is required; `--trust-remote-code` is needed for the GLM-TTS phoneme tokenizer; the DiT stage enables bucketed model-internal CUDA graphs with eager fallback.
 - Voice cloning: requires `ref_audio` + `ref_text` together. Reference audio should be 3-10 seconds. Feature extraction (WhisperVQ tokenizer, CampPlus ONNX, mel) runs on the model side.
 - Known limitations: First request may be slow due to lazy model loading (WhisperVQ, CampPlus ONNX). Warm-cache RTF is approximately 0.6-0.7x on A40.
+
+### 2x non-standard RTX 4090 48GB
+
+This configuration was personally validated on two non-standard RTX 4090 cards
+reporting 49,140 MiB each in `nvidia-smi`. Stock 24GB RTX 4090 cards were not
+validated.
+
+#### Environment
+
+- OS: Ubuntu 22.04.1 LTS
+- NVIDIA driver: 580.76.05
+- Python: 3.12.13
+- PyTorch: 2.11.0+cu130
+- CUDA runtime and JIT toolkit: 13.0
+- vLLM: 0.26.0
+- vLLM-Omni: commit `66a9b0c8`, version
+  `0.26.0rc2.dev5+g66a9b0c84`
+- FlashInfer: 0.6.14
+
+The host's default `nvcc` was 11.8. The working CUDA toolkit came from the
+Python environment, so `CUDA_HOME` was set to
+`$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13`.
+
+#### Command
+
+From the repository root, start the async-chunk server with:
+
+```bash
+export CUDA_HOME="$VIRTUAL_ENV/lib/python3.12/site-packages/nvidia/cu13"
+export PATH="$CUDA_HOME/bin:$PATH"
+export LD_LIBRARY_PATH="$CUDA_HOME/lib:${LD_LIBRARY_PATH:-}"
+
+CUDA_VISIBLE_DEVICES=0,1 vllm-omni serve zai-org/GLM-TTS \
+  --omni \
+  --trust-remote-code \
+  --port 8091 \
+  --deploy-config vllm_omni/deploy/glm_tts.yaml \
+  --stage-overrides '{"1":{"devices":"1"}}' \
+  --allowed-local-media-path "${PWD}/tests/assets"
+```
+
+Stage 0 runs on GPU 0, while the override places stage 1 on GPU 1. For the sync
+variant, add `--no-async-chunk` to the same command.
+
+#### Verification
+
+Run the Chinese voice-cloning check:
+
+```bash
+python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
+  --model zai-org/GLM-TTS \
+  --text "今天天气很好，我们一起学习语音模型推理。" \
+  --ref-audio "file://${PWD}/tests/assets/glm_tts/jiayan_zh.wav" \
+  --ref-text "他当时还跟线下其他的站姐吵架，然后，打架进局子了。" \
+  --response-format wav \
+  --output glm_tts_zh.wav
+```
+
+Run the English voice-cloning check with the same Chinese reference:
+
+```bash
+python examples/online_serving/text_to_speech/glm_tts/openai_speech_client.py \
+  --model zai-org/GLM-TTS \
+  --text "Hello, this is a voice cloning test with GLM-TTS." \
+  --ref-audio "file://${PWD}/tests/assets/glm_tts/jiayan_zh.wav" \
+  --ref-text "他当时还跟线下其他的站姐吵架，然后，打架进局子了。" \
+  --response-format wav \
+  --output glm_tts_en.wav
+```
+
+Inspect either output:
+
+```bash
+ffprobe -v error -select_streams a:0 \
+  -show_entries stream=codec_name,sample_rate,channels \
+  -of default=noprint_wrappers=1 glm_tts_zh.wav
+```
+
+Expected output:
+
+```text
+codec_name=pcm_s16le
+sample_rate=24000
+channels=1
+```
+
+The English smoke test produced a valid WAV with an actual duration of 5.020
+seconds.
+
+#### Observed warm-cache results
+
+These observations are from one controlled run, not a hardware or performance
+guarantee. Cold-start and JIT costs were excluded. Values are mean +/- standard
+deviation.
+
+##### Measurement procedure
+
+- For the single-GPU server, use the documented command with
+  `CUDA_VISIBLE_DEVICES=0` and omit `--stage-overrides`; both bundled stages
+  remain on GPU 0. For the dual-GPU server, use the command as shown.
+- Async chunking is the default. Add `--no-async-chunk` for sync mode.
+- For non-streaming measurements, use the exact Chinese client command above.
+  For streaming, add `--stream` to that command and change the output filename
+  to `glm_tts_zh.pcm`; streaming output is raw PCM, not WAV.
+- The fixed Chinese prompt and reference were exactly those shown in
+  Verification and yielded 3.900 seconds of audio. Each case used one warm-up
+  followed by three measured requests at concurrency 1.
+- Wall time starts immediately before the HTTP POST and stops after the complete
+  response body. TTFA runs from immediately before the POST to the first
+  non-empty audio bytes. RTF is wall time divided by audio duration.
+- Peak total device memory used was sampled every 0.2 seconds and reported by
+  NVML via `nvmlDeviceGetMemoryInfo.used`. The readings include background and
+  baseline use and are not baseline-subtracted.
+
+| GPUs | Mode | Response | Wall time (s) | TTFA (s) | RTF |
+|---:|---|---|---:|---:|---:|
+| 1 | Async | Non-streaming | 2.144 +/- 0.076 | -- | 0.550 +/- 0.019 |
+| 2 | Async | Non-streaming | 1.969 +/- 0.130 | -- | 0.505 +/- 0.033 |
+| 1 | Async | Streaming | 2.135 +/- 0.093 | 1.245 +/- 0.178 | 0.548 +/- 0.024 |
+| 2 | Async | Streaming | 1.891 +/- 0.006 | 1.170 +/- 0.042 | 0.485 +/- 0.002 |
+| 1 | Sync | Non-streaming | 1.822 +/- 0.016 | -- | 0.467 +/- 0.004 |
+| 2 | Sync | Non-streaming | 1.792 +/- 0.023 | -- | 0.459 +/- 0.006 |
+
+Moving stage 1 to GPU 1 reduced async non-streaming wall time by 8.2%. It
+reduced async streaming wall time by 11.4% and TTFA by 6.0%, and reduced sync
+wall time by 1.6%. Single- and dual-GPU outputs were bitwise identical only
+within matching execution and response modes.
+
+| Configuration | GPU 0 | GPU 1 |
+|---|---:|---:|
+| Single-GPU async | 31.52 GiB | -- |
+| Dual-GPU async | 29.23 GiB | 2.91 GiB |
+| Single-GPU sync | 31.47 GiB | -- |
+| Dual-GPU sync | 29.23 GiB | 2.91 GiB |

tests/config/test_environment_variables.py

@@ -164,7 +164,7 @@ def test_inventory_matches_reviewed_snapshot_counts():
     """Make an inventory expansion an explicit review decision."""
     category_counts = Counter(item.category for item in ENVIRONMENT_VARIABLE_INVENTORY.values())
     assert category_counts == {
-        EnvironmentVariableCategory.PUBLIC_OMNI: 23,
+        EnvironmentVariableCategory.PUBLIC_OMNI: 24,
         EnvironmentVariableCategory.INHERITED_VLLM: 20,
         EnvironmentVariableCategory.PLATFORM_EXTERNAL: 27,
         EnvironmentVariableCategory.MODEL_SPECIFIC: 56,

tests/core/sched/test_omni_ar_scheduler_stale_drain.py

@@ -100,6 +100,7 @@ def _make_drain_sched(session: Request) -> MagicMock:
     sched.pending_stop_after_extraction = set()
     sched.connector = None
     sched.kv_cache_manager.take_events.return_value = None
+    sched.kv_cache_manager.estimate_cached_tokens.return_value = 0
     sched.finished_req_ids_dict = {}
     sched.make_stats.return_value = None
     return sched

tests/core/sched/test_omni_ar_scheduler_streaming.py

@@ -93,6 +93,7 @@ def _run_resumable_segment_stop(
     sched.pending_stop_after_extraction = set()
     sched.connector = None
     sched.kv_cache_manager.take_events.return_value = None
+    sched.kv_cache_manager.estimate_cached_tokens.return_value = 0
     sched.finished_req_ids_dict = {}
     sched.make_stats.return_value = None
 
@@ -198,6 +199,7 @@ def test_resumable_segment_boundary_keeps_pre_transition_send_watermark() -> Non
         inter_stage_output,
         session,
         True,
+        new_token_ids=[42],
         confirmed_num_computed_tokens=26,
     )
 
@@ -226,6 +228,7 @@ def test_running_decode_step_without_inter_stage_payload_does_not_raise() -> Non
     sched.pending_stop_after_extraction = set()
     sched.connector = None
     sched.kv_cache_manager.take_events.return_value = None
+    sched.kv_cache_manager.estimate_cached_tokens.return_value = 0
     sched.finished_req_ids_dict = {}
     sched.make_stats.return_value = None
 
@@ -427,3 +430,33 @@ def test_ready_async_chunk_prompt_replacement_releases_stale_kv_once() -> None:
     assert sched.chunk_transfer_adapter.replaced_streaming_prompt_ids == set()
     assert sched.chunk_transfer_adapter.requests_with_ready_chunks == {session.request_id}
     assert sched.chunk_transfer_adapter.requests_num_chunks_sent == {}
+
+
+def test_chunk_segment_cleanup_keeps_requeued_resumable_receiver() -> None:
+    """A WAITING_FOR_CHUNK stop must not delete its newly parked session."""
+    session = _make_request()
+    session.resumable = True
+    session.status = RequestStatus.WAITING_FOR_STREAMING_REQ
+
+    def queue(*requests):
+        result = MagicMock()
+        result.requests = set(requests)
+        result.add_request.side_effect = result.requests.add
+        result.remove_requests.side_effect = result.requests.difference_update
+        return result
+
+    sched = OmniARScheduler.__new__(OmniARScheduler)
+    sched.running = []
+    sched.waiting = queue()
+    sched.skipped_waiting = queue(session)
+    sched.num_waiting_for_streaming_input = 1
+    sched.chunk_transfer_adapter = SimpleNamespace(segment_finished_requests={session.request_id})
+
+    sched._resume_downstream_chunk_receiver(session)
+    sched._remove_stopped_requests_from_queues(set(), {session})
+
+    assert session.status == RequestStatus.WAITING
+    assert session in sched.waiting.requests
+    assert session not in sched.skipped_waiting.requests
+    assert sched.num_waiting_for_streaming_input == 0
+    assert session.request_id not in sched.chunk_transfer_adapter.segment_finished_requests

tests/core/sched/test_omni_generation_scheduler_update_session.py

@@ -138,6 +138,7 @@ def test_resumable_generation_stop_marks_segment_boundary() -> None:
     sched.recompute_kv_load_failures = False
     sched.connector = None
     sched.kv_cache_manager.take_events.return_value = None
+    sched.kv_cache_manager.estimate_cached_tokens.return_value = 0
     sched.finished_req_ids_dict = {}
     sched.make_stats.return_value = None
 

tests/core/sched/test_omni_sched_deferred_free_fence.py

@@ -0,0 +1,168 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""v0.28 deferred KV-block free fence in the Omni schedulers (#6606 review).
+
+Upstream v0.28 (defer_block_free) advances ``sched_step_seq`` for every
+non-empty scheduled step and, at the top of ``update_from_output()``,
+advances ``processed_step_seq`` and drains the deferred frees. The Omni
+schedulers each carried only half of that contract:
+
+- ``OmniARScheduler.schedule()`` delegates to ``super().schedule()`` (fence
+  advances) but its reimplemented ``update_from_output()`` never advanced
+  ``processed_step_seq`` nor drained -> deferred blocks accumulate without
+  bound under async/PP KV-consumer configurations.
+- ``OmniGenerationScheduler`` reimplements both: the fast-path
+  ``schedule()`` never advanced ``sched_step_seq`` (so finished requests
+  freed blocks an in-flight GPU step could still be writing) and its
+  ``update_from_output()`` never drained.
+
+These tests pin the restored contract at the call boundary; the drain and
+free logic itself is upstream-tested.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+import pytest
+
+# Imports must run in this order: vllm_omni applies patches to vllm.v1.request
+# before Request / RequestStatus are bound in this module.
+# isort: off
+import vllm_omni  # noqa: F401 - import for side effects (patch vLLM)
+from vllm.sampling_params import SamplingParams
+from vllm.v1.core.sched.interface import PauseState
+from vllm.v1.core.sched.output import SchedulerOutput
+from vllm.v1.core.sched.request_queue import SchedulingPolicy
+from vllm.v1.outputs import ModelRunnerOutput
+from vllm.v1.request import Request, RequestStatus
+from vllm_omni.core.sched.omni_ar_scheduler import OmniARScheduler
+from vllm_omni.core.sched.omni_generation_scheduler import OmniGenerationScheduler
+
+# isort: on
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+def _make_update_sched(scheduler_cls) -> MagicMock:
+    """MagicMock scheduler driving the real ``update_from_output`` with an
+    empty batch; only the fence state carries real values."""
+    sched = MagicMock()
+    sched.requests = {}
+    sched.perf_metrics = None
+    sched.chunk_transfer_adapter = None
+    sched.defer_block_free = True
+    sched.processed_step_seq = 0
+    return sched
+
+
+def _empty_frame(*, total_num_scheduled_tokens: int):
+    scheduler_output = MagicMock(spec=SchedulerOutput)
+    scheduler_output.num_scheduled_tokens = {}
+    scheduler_output.total_num_scheduled_tokens = total_num_scheduled_tokens
+    model_runner_output = MagicMock(spec=ModelRunnerOutput)
+    model_runner_output.sampled_token_ids = []
+    model_runner_output.logprobs = None
+    model_runner_output.prompt_logprobs_dict = {}
+    model_runner_output.pooler_output = None
+    model_runner_output.num_nans_in_logits = None
+    model_runner_output.kv_connector_output = None
+    model_runner_output.cudagraph_stats = None
+    return scheduler_output, model_runner_output
+
+
+@pytest.mark.parametrize("scheduler_cls", [OmniARScheduler, OmniGenerationScheduler])
+def test_update_from_output_advances_fence_and_drains(scheduler_cls) -> None:
+    sched = _make_update_sched(scheduler_cls)
+    scheduler_output, model_runner_output = _empty_frame(total_num_scheduled_tokens=3)
+
+    scheduler_cls.update_from_output(sched, scheduler_output, model_runner_output)
+
+    assert sched.processed_step_seq == 1
+    sched._drain_deferred_frees.assert_called_once_with()
+
+
+@pytest.mark.parametrize("scheduler_cls", [OmniARScheduler, OmniGenerationScheduler])
+def test_update_from_output_zero_token_step_leaves_fence(scheduler_cls) -> None:
+    """0-token steps do not advance the fence upstream (their seq was never
+    advanced in schedule()); draining on them would run ahead of the GPU."""
+    sched = _make_update_sched(scheduler_cls)
+    scheduler_output, model_runner_output = _empty_frame(total_num_scheduled_tokens=0)
+
+    scheduler_cls.update_from_output(sched, scheduler_output, model_runner_output)
+
+    assert sched.processed_step_seq == 0
+    sched._drain_deferred_frees.assert_not_called()
+
+
+@pytest.mark.parametrize("scheduler_cls", [OmniARScheduler, OmniGenerationScheduler])
+def test_update_from_output_defer_disabled_leaves_fence(scheduler_cls) -> None:
+    sched = _make_update_sched(scheduler_cls)
+    sched.defer_block_free = False
+    scheduler_output, model_runner_output = _empty_frame(total_num_scheduled_tokens=3)
+
+    scheduler_cls.update_from_output(sched, scheduler_output, model_runner_output)
+
+    assert sched.processed_step_seq == 0
+    sched._drain_deferred_frees.assert_not_called()
+
+
+def _make_fast_path_sched(*, defer_block_free: bool) -> tuple[MagicMock, Request]:
+    """MagicMock scheduler that drives the real generation fast-path
+    ``schedule()`` over one running one-shot request."""
+    request = Request(
+        request_id="req-fence",
+        prompt_token_ids=[1, 2, 3],
+        sampling_params=SamplingParams(max_tokens=4),
+        pooling_params=None,
+        arrival_time=100.0,
+        block_hasher=None,
+    )
+    request.status = RequestStatus.RUNNING
+
+    sched = MagicMock()
+    sched.max_num_scheduled_tokens = 8
+    sched._pause_state = PauseState.UNPAUSED
+    sched.requests = {request.request_id: request}
+    sched.running = [request]
+    sched.waiting = []  # falsy: the waiting loop never runs
+    sched.policy = SchedulingPolicy.FCFS
+    sched.chunk_transfer_adapter = None
+    sched.scheduler_config = MagicMock(enable_chunked_prefill=True)
+    sched.num_lookahead_tokens = 0
+    sched.log_stats = False
+    sched.kv_cache_config = MagicMock(kv_cache_groups=[])
+    sched.use_v2_model_runner = False
+    sched.prev_step_scheduled_req_ids = set()
+    sched.needs_kv_cache_zeroing = False
+    sched.connector = None
+    sched.ec_connector = None
+    sched.defer_block_free = defer_block_free
+    sched.sched_step_seq = 0
+    # last_sched_seq stamping order: _update_after_schedule must observe the
+    # already-advanced fence value, as upstream schedule() guarantees.
+    sched.seq_at_update_after_schedule = None
+
+    def _capture(_scheduler_output):
+        sched.seq_at_update_after_schedule = sched.sched_step_seq
+
+    sched._update_after_schedule.side_effect = _capture
+    return sched, request
+
+
+def test_fast_path_schedule_advances_fence_before_update_after_schedule() -> None:
+    sched, request = _make_fast_path_sched(defer_block_free=True)
+
+    OmniGenerationScheduler.schedule(sched)
+
+    assert sched.sched_step_seq == 1
+    assert sched.seq_at_update_after_schedule == 1
+
+
+def test_fast_path_schedule_defer_disabled_leaves_fence() -> None:
+    sched, request = _make_fast_path_sched(defer_block_free=False)
+
+    OmniGenerationScheduler.schedule(sched)
+
+    assert sched.sched_step_seq == 0

tests/core/sched/test_omni_sched_ec_request_finish.py

@@ -0,0 +1,180 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""v0.28 EC request-finish contract in the Omni schedulers (#6606 review).
+
+Upstream v0.28 ``_free_request()`` fires ``ec_connector.request_finished()``
+BEFORE the encoder cache is freed (so the connector can inspect per-request
+state), honors its delayed-free result, and returns the generated
+``ec_transfer_params``; ``update_from_output()`` then carries them into the
+``EngineCoreOutput``. ``OmniARScheduler`` previously skipped the hook and
+always returned ``None``; both Omni schedulers discarded the second tuple
+item, so a remote encoder-cache handle could never reach the frontend.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+import pytest
+
+# Imports must run in this order: vllm_omni applies patches to vllm.v1.request
+# before Request / RequestStatus are bound in this module.
+# isort: off
+import vllm_omni  # noqa: F401 - import for side effects (patch vLLM)
+from vllm.sampling_params import SamplingParams
+from vllm.v1.core.sched.output import SchedulerOutput
+from vllm.v1.outputs import ModelRunnerOutput
+from vllm.v1.request import Request, RequestStatus
+from vllm_omni.core.sched.omni_ar_scheduler import OmniARScheduler
+from vllm_omni.core.sched.omni_generation_scheduler import OmniGenerationScheduler
+
+# isort: on
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+EC_PARAMS = {"remote_encoder_cache_handle": "ec-1"}
+
+
+def _make_free_request_sched(*, ec_connector) -> tuple[OmniARScheduler, MagicMock]:
+    """Minimal real-`_free_request` harness; `order` records the EC hook
+    firing relative to the encoder-cache free."""
+    order = MagicMock()
+    sched = OmniARScheduler.__new__(OmniARScheduler)
+    sched._omits_kv_transfer_cache = {}
+    sched._connector_finished = lambda request: (False, None)
+    sched.ec_connector = ec_connector
+    if ec_connector is not None:
+        order.attach_mock(ec_connector.request_finished, "request_finished")
+    sched.encoder_cache_manager = MagicMock()
+    order.attach_mock(sched.encoder_cache_manager.free, "encoder_cache_free")
+    sched.finished_req_ids = set()
+    sched._new_prompt_len_snapshot = {}
+    sched.finished_req_ids_dict = None
+    sched._should_transfer_kv_for_request = lambda req_id: False
+    sched._free_blocks = MagicMock()
+    sched._free_input_coordinator_request = MagicMock()
+    sched.chunk_transfer_adapter = None
+    return sched, order
+
+
+class _FakeFinishedRequest:
+    def __init__(self, request_id: str) -> None:
+        self.request_id = request_id
+
+    def is_finished(self) -> bool:
+        return True
+
+
+def test_free_request_fires_ec_hook_before_encoder_cache_free() -> None:
+    ec_connector = MagicMock()
+    ec_connector.request_finished.return_value = (False, EC_PARAMS)
+    sched, order = _make_free_request_sched(ec_connector=ec_connector)
+    request = _FakeFinishedRequest("req-ec")
+
+    kv_params, ec_params = sched._free_request(request)
+
+    assert ec_params == EC_PARAMS
+    ec_connector.request_finished.assert_called_once_with(request)
+    hook_calls = [name for name, *_ in order.mock_calls]
+    assert hook_calls.index("request_finished") < hook_calls.index("encoder_cache_free")
+    sched._free_blocks.assert_called_once_with(request)
+
+
+def test_free_request_honors_ec_delayed_free() -> None:
+    ec_connector = MagicMock()
+    ec_connector.request_finished.return_value = (True, EC_PARAMS)
+    sched, _ = _make_free_request_sched(ec_connector=ec_connector)
+    request = _FakeFinishedRequest("req-ec-delay")
+
+    kv_params, ec_params = sched._free_request(request)
+
+    assert ec_params == EC_PARAMS
+    sched._free_blocks.assert_not_called()
+
+
+def test_free_request_without_ec_connector_returns_none() -> None:
+    sched, _ = _make_free_request_sched(ec_connector=None)
+    request = _FakeFinishedRequest("req-no-ec")
+
+    kv_params, ec_params = sched._free_request(request)
+
+    assert ec_params is None
+    sched._free_blocks.assert_called_once_with(request)
+
+
+def _make_finishing_request() -> Request:
+    request = Request(
+        request_id="req-ec-finish",
+        prompt_token_ids=[1, 2, 3],
+        sampling_params=SamplingParams(max_tokens=4),
+        pooling_params=None,
+        arrival_time=100.0,
+        block_hasher=None,
+    )
+    request.status = RequestStatus.RUNNING
+    return request
+
+
+def _make_finish_sched(request: Request) -> MagicMock:
+    """MagicMock scheduler driving the real ``update_from_output`` through a
+    finishing request; ``_free_request`` reports EC transfer params."""
+    sched = MagicMock()
+    sched.requests = {request.request_id: request}
+    sched.perf_metrics = None
+    sched.defer_block_free = False
+    sched.structured_output_manager.should_advance.return_value = False
+    sched._process_kv_transfer_trigger.return_value = False
+    sched._maybe_decode_pooling_output.return_value = None
+    sched._handle_stopped_request.return_value = True
+    sched._free_request.return_value = (None, EC_PARAMS)
+    sched.kv_cache_manager.estimate_cached_tokens.return_value = 0
+    sched.chunk_transfer_adapter = None
+    sched.finished_req_ids_dict = None
+    sched._new_prompt_len_snapshot = {}
+    return sched
+
+
+def test_ar_update_from_output_carries_ec_params_to_engine_output() -> None:
+    request = _make_finishing_request()
+    sched = _make_finish_sched(request)
+    sched._update_request_with_output.return_value = ([42], True)
+
+    engine_core_outputs = OmniARScheduler.update_from_output(sched, *_finish_frame_parts(request))
+
+    (eco,) = engine_core_outputs[request.client_index].outputs
+    assert eco.ec_transfer_params == EC_PARAMS
+    sched._free_request.assert_called_once_with(request)
+
+
+def _finish_frame_parts(request: Request):
+    scheduler_output = MagicMock(spec=SchedulerOutput)
+    scheduler_output.num_scheduled_tokens = {request.request_id: 1}
+    scheduler_output.total_num_scheduled_tokens = 1
+    scheduler_output.scheduled_spec_decode_tokens = {}
+    scheduler_output.num_invalid_spec_tokens = 0
+
+    model_runner_output = MagicMock(spec=ModelRunnerOutput)
+    model_runner_output.sampled_token_ids = [[42]]
+    model_runner_output.logprobs = None
+    model_runner_output.prompt_logprobs_dict = {}
+    model_runner_output.pooler_output = None
+    model_runner_output.num_nans_in_logits = None
+    model_runner_output.kv_connector_output = None
+    model_runner_output.cudagraph_stats = None
+    model_runner_output.req_id_to_index = {request.request_id: 0}
+    model_runner_output.routed_experts = None
+    return scheduler_output, model_runner_output
+
+
+def test_generation_update_from_output_carries_ec_params_to_engine_output() -> None:
+    request = _make_finishing_request()
+    # One-shot generation finish: the whole prompt is computed.
+    request.num_computed_tokens = len(request.prompt_token_ids)
+    sched = _make_finish_sched(request)
+
+    engine_core_outputs = OmniGenerationScheduler.update_from_output(sched, *_finish_frame_parts(request))
+
+    (eco,) = engine_core_outputs[request.client_index].outputs
+    assert eco.ec_transfer_params == EC_PARAMS
+    sched._free_request.assert_called_once_with(request)

tests/core/sched/test_omni_sched_prefill_stats_finalize.py

@@ -0,0 +1,126 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""v0.28 prefill-stats finalization in the Omni schedulers (#6606 review).
+
+Upstream v0.28 finalizes prefill statistics with
+``PrefillStats.finalize(kv_cache_manager.estimate_cached_tokens(request))``
+before emitting them, which computes ``num_cache_creation_tokens`` (tokens
+computed and written to the prefix cache). Both Omni schedulers previously
+emitted ``take_prefill_stats()`` raw, so creation usage was always 0.
+"""
+
+from __future__ import annotations
+
+from unittest.mock import MagicMock
+
+import pytest
+
+# Imports must run in this order: vllm_omni applies patches to vllm.v1.request
+# before Request / RequestStatus are bound in this module.
+# isort: off
+import vllm_omni  # noqa: F401 - import for side effects (patch vLLM)
+from vllm.sampling_params import SamplingParams
+from vllm.v1.core.sched.output import SchedulerOutput
+from vllm.v1.outputs import ModelRunnerOutput
+from vllm.v1.request import Request, RequestStatus
+from vllm_omni.core.sched.omni_ar_scheduler import OmniARScheduler
+from vllm_omni.core.sched.omni_generation_scheduler import OmniGenerationScheduler
+
+# isort: on
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+def _make_request() -> Request:
+    request = Request(
+        request_id="req-prefill-stats",
+        prompt_token_ids=[1, 2, 3],
+        sampling_params=SamplingParams(max_tokens=4),
+        pooling_params=None,
+        arrival_time=100.0,
+        block_hasher=None,
+    )
+    request.status = RequestStatus.RUNNING
+    # A scheduled prefill: 3 prompt tokens, 1 from the local prefix cache.
+    request.prefill_stats.set(
+        num_prompt_tokens=3,
+        num_local_cached_tokens=1,
+        num_external_cached_tokens=0,
+    )
+    return request
+
+
+def _make_sched(request: Request) -> MagicMock:
+    sched = MagicMock()
+    sched.requests = {request.request_id: request}
+    sched.perf_metrics = None
+    sched.defer_block_free = False
+    sched.structured_output_manager.should_advance.return_value = False
+    sched._process_kv_transfer_trigger.return_value = False
+    sched._maybe_decode_pooling_output.return_value = None
+    sched._handle_stopped_request.return_value = True
+    # The whole 3-token prompt ends up cached: 3 - 1 pre-cached = 2 created.
+    sched.kv_cache_manager.estimate_cached_tokens.return_value = 3
+
+    def _free_and_invalidate(req):
+        # Lifecycle guard: freeing releases the KV blocks, after which the
+        # manager reports 0 cached tokens. If finalize ran after
+        # _free_request (the reviewed bug), creation count would compute
+        # against 0 and the assertions below would see 0, not 2.
+        sched.kv_cache_manager.estimate_cached_tokens.return_value = 0
+        return (None, None)
+
+    sched._free_request.side_effect = _free_and_invalidate
+    sched.chunk_transfer_adapter = None
+    sched.finished_req_ids_dict = None
+    sched._new_prompt_len_snapshot = {}
+    return sched
+
+
+def _frame(request: Request):
+    scheduler_output = MagicMock(spec=SchedulerOutput)
+    scheduler_output.num_scheduled_tokens = {request.request_id: 1}
+    scheduler_output.total_num_scheduled_tokens = 1
+    scheduler_output.scheduled_spec_decode_tokens = {}
+    scheduler_output.num_invalid_spec_tokens = 0
+
+    model_runner_output = MagicMock(spec=ModelRunnerOutput)
+    model_runner_output.sampled_token_ids = [[42]]
+    model_runner_output.logprobs = None
+    model_runner_output.prompt_logprobs_dict = {}
+    model_runner_output.pooler_output = None
+    model_runner_output.num_nans_in_logits = None
+    model_runner_output.kv_connector_output = None
+    model_runner_output.cudagraph_stats = None
+    model_runner_output.req_id_to_index = {request.request_id: 0}
+    model_runner_output.routed_experts = None
+    return scheduler_output, model_runner_output
+
+
+def test_ar_scheduler_finalizes_prefill_stats_before_emitting() -> None:
+    request = _make_request()
+    sched = _make_sched(request)
+    sched._update_request_with_output.return_value = ([42], True)
+
+    engine_core_outputs = OmniARScheduler.update_from_output(sched, *_frame(request))
+
+    (eco,) = engine_core_outputs[request.client_index].outputs
+    assert eco.prefill_stats is not None
+    assert eco.prefill_stats.num_cached_tokens == 1
+    assert eco.prefill_stats.num_cache_creation_tokens == 2
+    sched.kv_cache_manager.estimate_cached_tokens.assert_called_once_with(request)
+
+
+def test_generation_scheduler_finalizes_prefill_stats_before_emitting() -> None:
+    request = _make_request()
+    request.num_computed_tokens = len(request.prompt_token_ids)
+    sched = _make_sched(request)
+
+    engine_core_outputs = OmniGenerationScheduler.update_from_output(sched, *_frame(request))
+
+    (eco,) = engine_core_outputs[request.client_index].outputs
+    assert eco.prefill_stats is not None
+    assert eco.prefill_stats.num_cached_tokens == 1
+    assert eco.prefill_stats.num_cache_creation_tokens == 2
+    sched.kv_cache_manager.estimate_cached_tokens.assert_called_once_with(request)

tests/diffusion/attention/test_flash_attn.py

@@ -308,6 +308,91 @@ def test_flash_attn_func_preferred_over_varlen():
     print("✓ flash_attn_func forward works correctly!")
 
 
+@pytest.mark.skipif(not is_gpu, reason="FlashAttention requires CUDA or XPU")
+@pytest.mark.parametrize("k_len", [40, 256])
+def test_cross_attn_key_padding_vs_sdpa(k_len):
+    """
+    Case 3: Cross-attention with a key-padding mask.
+
+    The mask covers only the key sequence; every query position must attend
+    to the valid keys. FA and SDPA outputs should be very close. k_len=256
+    covers equal Q/K lengths, which must not be mistaken for self-attention,
+    and valid key lengths differ per batch element.
+    """
+    device = torch.device(current_omni_platform.device_type)
+    dtype = torch.bfloat16
+
+    batch_size = 2
+    q_len = 256
+    valid_lens = (17, 29)
+    num_heads = 8
+    head_dim = 64
+
+    fa_impl = FlashAttentionImpl(
+        num_heads=num_heads, head_size=head_dim, softmax_scale=1.0 / (head_dim**0.5), causal=False, role="cross"
+    )
+    sdpa_impl = SDPAImpl(num_heads=num_heads, head_size=head_dim, softmax_scale=1.0 / (head_dim**0.5), causal=False)
+
+    torch.manual_seed(7)
+    query = torch.randn(batch_size, q_len, num_heads, head_dim, device=device, dtype=dtype)
+    key = torch.randn(batch_size, k_len, num_heads, head_dim, device=device, dtype=dtype)
+    value = torch.randn(batch_size, k_len, num_heads, head_dim, device=device, dtype=dtype)
+
+    attn_mask = torch.zeros(batch_size, k_len, dtype=torch.bool, device=device)
+    for i, valid_len in enumerate(valid_lens):
+        attn_mask[i, :valid_len] = True
+    attn_metadata = AttentionMetadata(attn_mask=attn_mask)
+
+    output_fa = fa_impl.forward(query=query, key=key, value=value, attn_metadata=attn_metadata)
+    output_sdpa = sdpa_impl.forward(query=query, key=key, value=value, attn_metadata=attn_metadata)
+
+    max_diff = torch.max(torch.abs(output_fa - output_sdpa)).item()
+    mean_diff = torch.mean(torch.abs(output_fa - output_sdpa)).item()
+
+    print("\n=== Case 3: Cross-Attention Key-Padding FA vs SDPA ===")
+    print(f"Max absolute difference: {max_diff:.6f}")
+    print(f"Mean absolute difference: {mean_diff:.6f}")
+
+    assert max_diff < 0.01, f"Max difference {max_diff} exceeds threshold 0.01"
+    assert mean_diff < 0.001, f"Mean difference {mean_diff} exceeds threshold 0.001"
+
+    print("✓ Case 3 PASSED: FA and SDPA cross-attention outputs are very close!")
+
+
+def test_varlen_masked_routing_by_role(monkeypatch):
+    """The unpad route is picked by role, not Q/K length equality; runs without a GPU."""
+    calls = []
+
+    def fake_varlen_func(q, k, v, **kwargs):
+        calls.append((q.shape[0], kwargs["cu_seqlens_q"], kwargs["cu_seqlens_k"]))
+        return q
+
+    monkeypatch.setattr(fa, "HAS_FLASH_ATTN", True)
+    monkeypatch.setattr(fa, "flash_attn_varlen_func", fake_varlen_func)
+
+    batch_size, seq_len, num_heads, head_dim = 2, 4, 2, 8
+    query = torch.randn(batch_size, seq_len, num_heads, head_dim)
+    metadata = AttentionMetadata(attn_mask=torch.tensor([[True, True, False, False], [True, True, True, False]]))
+
+    cross_impl = FlashAttentionImpl(
+        num_heads=num_heads, head_size=head_dim, softmax_scale=0.5, causal=False, role="cross"
+    )
+    output = cross_impl.forward_cuda(query, query, query, metadata)
+    assert output.shape == query.shape
+    num_q_rows, cu_seqlens_q, cu_seqlens_k = calls[-1]
+    assert num_q_rows == batch_size * seq_len
+    assert cu_seqlens_q.tolist() == [0, 4, 8]
+    assert cu_seqlens_k.tolist() == [0, 2, 5]
+
+    self_impl = FlashAttentionImpl(num_heads=num_heads, head_size=head_dim, softmax_scale=0.5, causal=False)
+    output = self_impl.forward_cuda(query, query, query, metadata)
+    assert output.shape == query.shape
+    num_q_rows, cu_seqlens_q, cu_seqlens_k = calls[-1]
+    assert num_q_rows == 5
+    assert cu_seqlens_q.tolist() == [0, 2, 5]
+    assert cu_seqlens_k.tolist() == [0, 2, 5]
+
+
 def test_piecewise_flash_attn_uses_varlen_fallback(monkeypatch):
     calls = []
 

tests/diffusion/kernels/mot/test_mot_linear.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 # ruff: noqa: N803, E741
 """Layer-level correctness & performance test for MoT parallel linear layers.
 
@@ -237,6 +240,53 @@ def _reference_forward(x, text_indices, vae_indices, text_linear, vae_linear):
     return output
 
 
+def _assert_gen_expert_parallel_metadata(layer):
+    """The secondary expert must satisfy vLLM's parallel-linear contract."""
+    for attr in (
+        "tp_rank",
+        "tp_size",
+        "input_size",
+        "input_size_per_partition",
+        "output_size",
+        "output_size_per_partition",
+    ):
+        assert getattr(layer.gen_exp, attr) == getattr(layer, attr)
+
+
+def test_mot_gen_expert_parallel_metadata():
+    """Keep secondary experts compatible with vLLM online quantizers."""
+    from vllm.model_executor.layers.quantization.online.fp8 import (
+        _is_tp_sharded,
+    )
+
+    with set_current_vllm_config(VllmConfig()):
+        layers = (
+            MoTQKVParallelLinear(
+                hidden_size=16,
+                head_size=4,
+                total_num_heads=4,
+                total_num_kv_heads=2,
+                bias=False,
+                vae_bias=False,
+                params_dtype=torch.bfloat16,
+                disable_tp=True,
+            ),
+            MoTRowParallelLinear(
+                16,
+                8,
+                bias=False,
+                vae_bias=False,
+                input_is_parallel=True,
+                params_dtype=torch.bfloat16,
+                disable_tp=True,
+            ),
+        )
+
+    for layer in layers:
+        _assert_gen_expert_parallel_metadata(layer)
+        assert _is_tp_sharded(layer.gen_exp) is False
+
+
 def _check_and_report(ref: torch.Tensor, mot: torch.Tensor, tag: str):
     """Compare outputs, print metrics, and assert correctness.
 
@@ -335,6 +385,8 @@ def test_mot_qkv_parallel(image_num: int, K: int, N: int, dtype: str, bias: bool
             disable_tp=True,
         ).cuda()
 
+        _assert_gen_expert_parallel_metadata(mot_linear)
+
         assert text_linear.output_size_per_partition == N, (
             f"Expected output_size_per_partition={N}, "
             f"got {text_linear.output_size_per_partition}. "
@@ -423,6 +475,8 @@ def test_mot_o_proj(
             disable_tp=True,
         ).cuda()
 
+        _assert_gen_expert_parallel_metadata(mot_linear)
+
         _sync_weights(text_linear, vae_linear, mot_linear)
 
         text_idx, vae_idx, M = _make_indices(image_num, _VAE_CHUNK_SIZE)
@@ -501,6 +555,13 @@ def test_mot_qkv_und_mode(K: int, N: int, dtype: str, bias: bool):
         ).cuda()
 
         with torch.no_grad():
+            # vLLM linear layers leave weights uninitialized (torch.empty), so
+            # a freshly zeroed GPU page would make both outputs zero and the
+            # cosine-similarity gate degenerate. Seed deterministic values so
+            # und-mode is compared against a meaningful reference.
+            ref_linear.weight.normal_(mean=0.0, std=0.02)
+            if ref_linear.bias is not None:
+                ref_linear.bias.normal_(mean=0.0, std=0.02)
             mot_linear.weight.copy_(ref_linear.weight)
             if bias and ref_linear.bias is not None:
                 mot_linear.bias.copy_(ref_linear.bias)
@@ -549,6 +610,13 @@ def test_mot_row_und_mode(K: int, N: int, dtype: str, bias: bool):
         ).cuda()
 
         with torch.no_grad():
+            # vLLM linear layers leave weights uninitialized (torch.empty), so
+            # a freshly zeroed GPU page would make both outputs zero and the
+            # cosine-similarity gate degenerate. Seed deterministic values so
+            # und-mode is compared against a meaningful reference.
+            ref_linear.weight.normal_(mean=0.0, std=0.02)
+            if ref_linear.bias is not None:
+                ref_linear.bias.normal_(mean=0.0, std=0.02)
             mot_linear.weight.copy_(ref_linear.weight)
             if bias and ref_linear.bias is not None:
                 mot_linear.bias.copy_(ref_linear.bias)

tests/diffusion/model_loader/test_diffusers_loader.py

@@ -249,6 +249,30 @@ def test_hwr_commit_failure_discards_model_and_reloads_without_hwr_or_mmap(tmp_p
     assert loader.take_host_weight_plan() is None
 
 
+def test_required_hwr_miss_fails_before_ordinary_loading_or_publication(
+    tmp_path: Path,
+    monkeypatch,
+):
+    canonical_root = tmp_path / "canonical"
+    canonical_root.mkdir()
+    save_file(
+        {"weight": torch.arange(4, dtype=torch.float32).to(torch.bfloat16).reshape(2, 2)},
+        str(canonical_root / "model.safetensors"),
+    )
+    loader = DiffusersPipelineLoader(
+        LoadConfig(),
+        _hwr_config(canonical_root, tmp_path / "empty-store", mode="required"),
+    )
+    pipeline = _HWRPipeline(canonical_root)
+    monkeypatch.setattr(loader, "_init_from_load_format", lambda *args, **kwargs: pipeline)
+
+    with pytest.raises(RuntimeError, match="Host Weight Runtime resolution failed"):
+        loader.load_model(load_device="cpu", device=torch.device("cpu"))
+
+    assert pipeline.load_count == 0
+    assert loader.take_host_weight_plan() is None
+
+
 def _make_dlo_online_quant_config() -> OmniDiffusionConfig:
     return OmniDiffusionConfig(
         model="",
@@ -261,10 +285,11 @@ def _make_dlo_online_quant_config() -> OmniDiffusionConfig:
 
 
 @pytest.mark.parametrize(
-    ("dist_offload", "use_allgather"),
+    ("dist_offload", "use_allgather", "mode"),
     [
-        (False, False),
-        (True, True),
+        (False, False, "preferred"),
+        (True, True, "preferred"),
+        (True, False, "disabled"),
     ],
 )
 def test_hwr_disabled_for_noneligible_dlo_paths_without_store_interaction(
@@ -272,12 +297,13 @@ def test_hwr_disabled_for_noneligible_dlo_paths_without_store_interaction(
     tmp_path,
     dist_offload,
     use_allgather,
+    mode,
 ):
     """Disabled and AllGather paths must never construct or probe HWR."""
     from vllm_omni.host_weight_runtime import HostWeightRuntime
 
     root = tmp_path / "must-not-be-touched"
-    loader = DiffusersPipelineLoader(LoadConfig(), _hwr_config("dummy-model", root))
+    loader = DiffusersPipelineLoader(LoadConfig(), _hwr_config("dummy-model", root, mode=mode))
     model = _DummyPipelineModel(source_prefix="transformer.")
     modules = SimpleNamespace(dit_names=("transformer",), dits=(model.transformer,))
 

tests/diffusion/model_loader/test_final_layout_host_weights.py

@@ -303,6 +303,83 @@ class _FakeH3Attention(nn.Module):
         super().__init__()
 
 
+class _FakeFluxLinear(_FakeH3Linear):
+    def __init__(
+        self,
+        *args: object,
+        bias: bool = False,
+        params_dtype: torch.dtype | None = None,
+        total_num_heads: int | None = None,
+        total_num_kv_heads: int | None = None,
+        **kwargs: object,
+    ) -> None:
+        super().__init__(
+            *args,
+            bias=bias,
+            params_dtype=params_dtype,
+            total_num_heads=total_num_heads,
+            total_num_kv_heads=total_num_kv_heads,
+            **kwargs,
+        )
+        self.weight.weight_loader = self._load_weight
+
+    @staticmethod
+    def _load_weight(
+        parameter: nn.Parameter,
+        loaded_weight: torch.Tensor,
+        shard_id: str | None = None,
+    ) -> None:
+        del shard_id
+        parameter.data.copy_(loaded_weight.to(parameter).reshape_as(parameter))
+
+
+class _FakeFluxAttention(nn.Module):
+    def __init__(self, **kwargs: object) -> None:
+        del kwargs
+        super().__init__()
+        self.register_buffer("beta", torch.tensor([0.5]))
+        self.register_buffer("eps", torch.tensor([-1e-6]))
+
+
+class _Flux2KleinPipeline(nn.Module):
+    def __init__(self, transformer: nn.Module) -> None:
+        super().__init__()
+        self.transformer = transformer
+        self.text_encoder = nn.Linear(2, 2, dtype=torch.bfloat16)
+        self.vae = nn.Module()
+        self.vae.register_buffer("gain", torch.tensor([9.0], dtype=torch.float32))
+
+
+def _reduced_flux2_klein(
+    monkeypatch: pytest.MonkeyPatch,
+    *,
+    device: torch.device | str = "cpu",
+) -> nn.Module:
+    from vllm_omni.diffusion.models.flux2_klein import flux2_klein_transformer as flux2
+
+    monkeypatch.setattr(flux2, "ColumnParallelLinear", _FakeFluxLinear)
+    monkeypatch.setattr(flux2, "MergedColumnParallelLinear", _FakeFluxLinear)
+    monkeypatch.setattr(flux2, "QKVParallelLinear", _FakeFluxLinear)
+    monkeypatch.setattr(flux2, "RowParallelLinear", _FakeFluxLinear)
+    monkeypatch.setattr(flux2, "Attention", _FakeFluxAttention)
+
+    transformer = flux2.Flux2Transformer2DModel(
+        patch_size=1,
+        in_channels=8,
+        out_channels=8,
+        num_layers=1,
+        num_single_layers=1,
+        attention_head_dim=8,
+        num_attention_heads=1,
+        joint_attention_dim=8,
+        timestep_guidance_channels=8,
+        mlp_ratio=1.0,
+        axes_dims_rope=(2, 2, 2, 2),
+        guidance_embeds=False,
+    )
+    return transformer.to(device=device, dtype=torch.bfloat16)
+
+
 class _H3Pipeline(nn.Module):
     def __init__(self, transformer: nn.Module) -> None:
         super().__init__()
@@ -1200,3 +1277,152 @@ def test_minimax_h3_declares_the_final_layout_restore_contract() -> None:
     assert contract.implementation_id == "minimax-h3-dit"
     assert callable(MiniMaxH3DiTModel.validate_restored_host_weights)
     assert "data_parallel" not in FinalLayoutRequest.__dataclass_fields__
+
+
+def test_reduced_flux2_klein_declares_complete_final_layout_contract(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    from vllm_omni.diffusion.models.flux2_klein.flux2_klein_transformer import (
+        Flux2Transformer2DModel,
+    )
+
+    transformer = _reduced_flux2_klein(monkeypatch)
+    pipeline = _Flux2KleinPipeline(transformer)
+    source = _prepared_source(tmp_path, directory="flux2-klein-source")
+    context = _identity(pipeline, source)
+    targets = collect_final_layout_targets(
+        pipeline,
+        (("transformer", transformer),),
+        policy=FINAL_LAYOUT_BF16_POLICY,
+        require_materialized=True,
+    )
+    target_names = {target.name for target in targets}
+
+    contract = Flux2Transformer2DModel.host_weight_restore_contract
+    assert isinstance(contract, FinalLayoutModelContract)
+    assert contract.schema == FINAL_LAYOUT_TENSOR_MODEL_CONTRACT_SCHEMA
+    assert contract.implementation_id == "flux2-klein-dit"
+    assert contract.version == "1"
+    assert context.model_contracts == (contract,)
+    assert transformer.stacked_params_mapping == [
+        (".to_qkv.", ".to_q.", "q"),
+        (".to_qkv.", ".to_k.", "k"),
+        (".to_qkv.", ".to_v.", "v"),
+        (".add_kv_proj", ".add_q_proj", "q"),
+        (".add_kv_proj", ".add_k_proj", "k"),
+        (".add_kv_proj", ".add_v_proj", "v"),
+    ]
+    loaded = transformer.load_weights(
+        [
+            ("transformer_blocks.0.attn.to_q.weight", torch.tensor([1.0], dtype=torch.bfloat16)),
+            ("transformer_blocks.0.attn.add_q_proj.weight", torch.tensor([2.0], dtype=torch.bfloat16)),
+            (
+                "single_transformer_blocks.0.attn.to_qkv_mlp_proj.weight",
+                torch.tensor([3.0], dtype=torch.bfloat16),
+            ),
+        ]
+    )
+    assert loaded == {
+        "transformer_blocks.0.attn.to_qkv.weight",
+        "transformer_blocks.0.attn.add_kv_proj.weight",
+        "single_transformer_blocks.0.attn.to_qkv_mlp_proj.weight",
+    }
+    assert any(name.startswith("transformer.transformer_blocks.0.") for name in target_names)
+    assert any(name.startswith("transformer.single_transformer_blocks.0.") for name in target_names)
+    assert "transformer.transformer_blocks.0.attn.to_qkv.weight" in target_names
+    assert "transformer.transformer_blocks.0.attn.add_kv_proj.weight" in target_names
+    assert "transformer.single_transformer_blocks.0.attn.to_qkv_mlp_proj.weight" in target_names
+    assert "transformer.transformer_blocks.0.attn.attn.beta" in target_names
+    assert "transformer.transformer_blocks.0.attn.attn.eps" in target_names
+    assert all("text_encoder" not in name and "vae" not in name for name in target_names)
+    transformer.validate_restored_host_weights()
+
+
+def test_flux2_klein_restore_validator_rejects_non_ready_state(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    transformer = _reduced_flux2_klein(monkeypatch)
+
+    mapping = transformer.stacked_params_mapping
+    transformer.stacked_params_mapping = None
+    with pytest.raises(ValueError, match="packed QKV mapping"):
+        transformer.validate_restored_host_weights()
+    transformer.stacked_params_mapping = mapping
+
+    weight = transformer.x_embedder.weight
+    weight.data = weight.data.float()
+    with pytest.raises(ValueError, match="must stay bf16"):
+        transformer.validate_restored_host_weights()
+    weight.data = weight.data.to(torch.bfloat16)
+
+    weight.data = weight.data.T
+    with pytest.raises(ValueError, match="contiguous strided layout"):
+        transformer.validate_restored_host_weights()
+    weight.data = weight.data.contiguous()
+
+    attention = transformer.transformer_blocks[0].attn.attn
+    attention._non_persistent_buffers_set.add("beta")
+    with pytest.raises(ValueError, match="must be persistent"):
+        transformer.validate_restored_host_weights()
+    attention._non_persistent_buffers_set.remove("beta")
+
+    attention.beta.data = attention.beta.data.float()
+    with pytest.raises(ValueError, match="scalar bf16"):
+        transformer.validate_restored_host_weights()
+
+
+def test_flux2_klein_publication_restore_fallback_and_teardown(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    source = _prepared_source(tmp_path, directory="flux2-klein-source")
+    cold_transformer = _reduced_flux2_klein(monkeypatch)
+    cold_model = _Flux2KleinPipeline(cold_transformer)
+    with torch.no_grad():
+        for index, parameter in enumerate(cold_transformer.parameters(), start=1):
+            parameter.fill_(index)
+        for name, buffer in cold_transformer.named_buffers():
+            if name.endswith((".beta", ".eps")):
+                buffer.fill_(0.5 if name.endswith(".beta") else -1e-6)
+
+    context = _identity(cold_model, source)
+    runtime = _runtime(tmp_path / "flux2-klein-store")
+    miss = runtime.resolve(context.identity)
+    assert miss.report.outcome is ResolutionOutcome.CANONICAL_FALLBACK
+
+    producer = FinalLayoutBF16Producer(
+        context,
+        cold_model,
+        (("transformer", cold_transformer),),
+        max_shard_bytes=64,
+    )
+    publication = runtime.publish_after_load(context.identity, producer=producer)
+    assert publication.outcome is PostLoadPublicationOutcome.PUBLISHED
+
+    warm_transformer = _reduced_flux2_klein(monkeypatch, device="meta")
+    warm_model = _Flux2KleinPipeline(warm_transformer)
+    warm_context = _identity(warm_model, source)
+    assert warm_context.identity == context.identity
+    hit = runtime.resolve(warm_context.identity)
+    assert hit.report.outcome is ResolutionOutcome.LOCAL_HIT
+    assert hit.lease is not None
+
+    plan = FinalLayoutTensorRestorer(warm_context).plan_restore(warm_model, hit.lease)
+    assert all(tensor.is_meta for name, tensor in warm_model.named_parameters() if name.startswith("transformer."))
+    plan.commit()
+    warm_transformer.validate_restored_host_weights()
+
+    cold_tensors = dict(cold_model.named_parameters()) | dict(cold_model.named_buffers())
+    warm_tensors = dict(warm_model.named_parameters()) | dict(warm_model.named_buffers())
+    for name in warm_context.tensor_names:
+        assert torch.equal(warm_tensors[name], cold_tensors[name])
+        assert warm_tensors[name].untyped_storage().data_ptr() == hit.lease.tensors[name].untyped_storage().data_ptr()
+    assert warm_transformer.stacked_params_mapping == cold_transformer.stacked_params_mapping
+
+    del warm_tensors, warm_model, warm_transformer, plan
+    gc.collect()
+    hit.lease.close()
+    assert hit.lease.closed
+    assert isinstance(runtime.store, FilesystemHostWeightStore)
+    assert runtime.store.cleanup(context.identity) is None

tests/diffusion/models/lingbot_world/test_lingbot_world_camera.py

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 
 from __future__ import annotations
 
@@ -245,7 +245,7 @@ else:
         [sys.executable, "-c", script, str(_MODULE_PATH), str(tmp_path)],
         capture_output=True,
         text=True,
-        timeout=3,
+        timeout=30,
         check=False,
     )
 

tests/diffusion/models/ltx2/test_ltx2_vae.py

@@ -11,6 +11,31 @@ import torch
 pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
 
 
+def test_ltx_base_vocoder_keeps_native_dtype(monkeypatch):
+    import vllm_omni.diffusion.models.ltx2.ltx2_runtime as ltx_runtime
+
+    class FakeBaseVocoder(torch.nn.Module):
+        def __init__(self):
+            super().__init__()
+            self.input_dtype = None
+
+        def forward(self, value):
+            self.input_dtype = value.dtype
+            return value
+
+    monkeypatch.setattr(
+        ltx_runtime.torch,
+        "autocast",
+        lambda **_kwargs: (_ for _ in ()).throw(AssertionError("base vocoder must not use autocast")),
+    )
+    vocoder = FakeBaseVocoder()
+
+    output = ltx_runtime._run_ltx_vocoder(vocoder, torch.ones(1, dtype=torch.bfloat16))
+
+    assert vocoder.input_dtype == torch.bfloat16
+    assert output.dtype == torch.bfloat16
+
+
 class TestLTXOutputRank:
     @pytest.mark.parametrize(
         ("distributed_vae_state", "expected_decode_calls"),

tests/diffusion/models/ltx2/test_ltx2_vocoder_cuda.py

@@ -0,0 +1,48 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+"""CUDA regression tests for LTX-2 BWE vocoder precision."""
+
+import pytest
+import torch
+
+from tests.helpers.mark import hardware_test
+
+pytestmark = [pytest.mark.core_model, pytest.mark.diffusion]
+
+
+class _BWEConvVocoder(torch.nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.conv = torch.nn.Conv1d(
+            1,
+            1,
+            kernel_size=1,
+            bias=False,
+            device="cuda",
+            dtype=torch.bfloat16,
+        )
+        self.bwe_generator = torch.nn.Identity()
+        self.input_dtype = None
+        self.conv_output_dtype = None
+
+    def forward(self, value):
+        self.input_dtype = value.dtype
+        output = self.conv(value)
+        self.conv_output_dtype = output.dtype
+        return output
+
+
+@hardware_test(res={"cuda": "L4"}, num_cards=1)
+def test_ltx_bwe_vocoder_runs_real_cuda_autocast_in_fp32():
+    from vllm_omni.diffusion.models.ltx2.ltx2_runtime import _run_ltx_vocoder
+
+    vocoder = _BWEConvVocoder()
+    generated_mel = torch.ones((1, 1, 4), device="cuda", dtype=torch.bfloat16)
+
+    output = _run_ltx_vocoder(vocoder, generated_mel)
+
+    assert next(vocoder.parameters()).dtype == torch.bfloat16
+    assert vocoder.input_dtype == torch.float32
+    assert vocoder.conv_output_dtype == torch.float32
+    assert output.dtype == torch.bfloat16

tests/diffusion/offloader/test_distributed_layerwise_backend.py

@@ -35,7 +35,10 @@ from vllm_omni.diffusion.offloader.distributed_layerwise_backend import (
     DistributedLayerwiseOffloadHook,
     PinnedResidentLayerGroup,
 )
-from vllm_omni.diffusion.offloader.host_registration import HostRegistrationCleanupError
+from vllm_omni.diffusion.offloader.host_registration import (
+    HostRegistrationCleanupError,
+    HostRegistrationError,
+)
 from vllm_omni.diffusion.offloader.module_residency import PinnedModuleStager
 from vllm_omni.diffusion.offloader.offload_plan import (
     OffloadPlan,
@@ -807,7 +810,10 @@ def test_registered_mmap_resident_group_bypasses_host_staging(patched_offload_ru
     assert torch.equal(block.weight, expected)
 
 
-def test_hwr_registration_uses_transport_budget(monkeypatch: pytest.MonkeyPatch):
+def test_hwr_registration_uses_transport_budget(
+    monkeypatch: pytest.MonkeyPatch,
+    patched_offload_runtime,
+):
     dist_backend_module._ACTIVE_HWR_REGISTRATIONS.clear()
     plan, _, lease = _fake_hwr_plan()
     backend = DistributedLayerwiseOffloadBackend(
@@ -1125,6 +1131,51 @@ class TestMmapWeightLoading:
         backend.disable()
         assert lease.closed
 
+    def test_hwr_registration_failure_falls_back_to_bounded_staging(
+        self,
+        monkeypatch,
+        patched_offload_runtime,
+    ):
+        plan, carrier, lease = _fake_hwr_plan("hwr-registration-fallback")
+        backend = DistributedLayerwiseOffloadBackend(
+            OffloadConfig(
+                strategy=OffloadStrategy.DISTRIBUTED_LAYER_WISE,
+                pin_cpu_memory=True,
+                dp_size=1,
+                dlo_use_allgather=False,
+            ),
+            torch.device("cpu"),
+            host_weight_plan=plan,
+        )
+        pipeline = _HWRPipeline()
+        original_staging_allocator = backend._allocate_shared_cpu_staging_buffers
+
+        def fail_registration(*args, **kwargs):
+            del args, kwargs
+            raise HostRegistrationError("registration unavailable in CPU test")
+
+        def allocate_unpinned_staging(hooks, resident_group=None):
+            # Registration selection still observes pin_cpu_memory=True. The
+            # CPU-only test avoids asking the host for CUDA-pinned allocations.
+            for hook in hooks:
+                hook.pin_memory = False
+            return original_staging_allocator(hooks, resident_group)
+
+        monkeypatch.setattr(dist_backend_module, "register_host_mappings", fail_registration)
+        monkeypatch.setattr(backend, "_allocate_shared_cpu_staging_buffers", allocate_unpinned_staging)
+
+        backend.enable(pipeline)
+
+        assert carrier.taken
+        assert backend._using_rank_local_mmap
+        assert not backend._using_registered_mmap
+        assert backend._host_registration is None
+        assert all(len(hook.cpu_staging_buffers) == 2 for group in backend._all_hook_groups for hook in group)
+        assert not lease.closed
+
+        backend.disable()
+        assert lease.closed
+
     def test_hwr_backend_failure_drains_partial_setup_before_lease_close(
         self,
         monkeypatch,
@@ -2132,8 +2183,9 @@ class TestDynamicSlotTracking:
                 prev_idx = (i - 1) % num_blocks
 
                 # Dynamic slot tracking: read from prev's prefetched slot
-                if prefetched_slots[prev_idx] is not None:
-                    current_slots[i] = prefetched_slots[prev_idx]
+                prefetched_slot = prefetched_slots[prev_idx]
+                if prefetched_slot is not None:
+                    current_slots[i] = prefetched_slot
 
                 read_slot = current_slots[i]
 

tests/diffusion/quantization/test_svdquant_config.py

@@ -0,0 +1,146 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from unittest.mock import Mock
+
+import pytest
+import torch
+from vllm.model_executor.layers.linear import (
+    LinearBase,
+    UnquantizedLinearMethod,
+)
+
+from vllm_omni.quantization import svdquant_config
+from vllm_omni.quantization.factory import (
+    SUPPORTED_QUANTIZATION_METHODS,
+    build_quant_config,
+)
+from vllm_omni.quantization.svdquant_config import (
+    DiffusionSVDQuantConfig,
+    DiffusionSVDQuantLinearMethod,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
+
+
+def test_config_from_checkpoint_dict() -> None:
+    config = DiffusionSVDQuantConfig.from_config(
+        {
+            "rank": 32,
+            "precision": "nvfp4",
+            "act_unsigned": False,
+            "modules_to_not_convert": ["blocks.0.adaln_proj.linear"],
+        }
+    )
+
+    assert config.rank == 32
+    assert config.precision == "nvfp4"
+    assert config.modules_to_not_convert == ["blocks.0.adaln_proj.linear"]
+    assert config.get_supported_act_dtypes() == [torch.bfloat16]
+
+
+@pytest.mark.parametrize(
+    ("kwargs", "message"),
+    [
+        ({"rank": 0}, "rank must be positive"),
+        ({"precision": "int4"}, "NVFP4 checkpoints only"),
+        ({"act_unsigned": True}, "unsigned activations"),
+    ],
+)
+def test_config_rejects_unsupported_phase1_options(
+    kwargs: dict[str, object],
+    message: str,
+) -> None:
+    with pytest.raises(ValueError, match=message):
+        DiffusionSVDQuantConfig(**kwargs)
+
+
+def test_factory_builds_svdquant_with_extra_checkpoint_fields() -> None:
+    config = build_quant_config(
+        {
+            "quant_method": "svdquant",
+            "rank": 16,
+            "precision": "nvfp4",
+            "w4a16_modules": ["adaln_proj.linear"],
+            "w4a16_group_size": 64,
+        }
+    )
+
+    assert isinstance(config, DiffusionSVDQuantConfig)
+    assert config.rank == 16
+    assert "svdquant" in SUPPORTED_QUANTIZATION_METHODS
+
+
+def test_skipped_and_non_linear_modules_remain_unquantized() -> None:
+    config = DiffusionSVDQuantConfig(modules_to_not_convert=["blocks.0.adaln_proj.linear"])
+    linear = Mock(spec=LinearBase)
+
+    assert isinstance(
+        config.get_quant_method(linear, "blocks.0.adaln_proj.linear"),
+        UnquantizedLinearMethod,
+    )
+    assert config.get_quant_method(torch.nn.ReLU(), "blocks.0.act") is None
+
+
+def test_linear_method_creates_canonical_partitioned_weights(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setattr(svdquant_config, "_assert_supported", lambda: None)
+    method = DiffusionSVDQuantLinearMethod(DiffusionSVDQuantConfig(rank=32))
+    layer = torch.nn.Module()
+
+    method.create_weights(
+        layer,
+        input_size_per_partition=128,
+        output_partition_sizes=[64, 64],
+        input_size=128,
+        output_size=128,
+        params_dtype=torch.bfloat16,
+    )
+
+    assert layer.qweight.shape == (128, 64)
+    assert layer.qweight.dtype == torch.int8
+    assert layer.wscales.shape == (8, 128)
+    assert layer.wscales.dtype == torch.float8_e4m3fn
+    assert layer.proj_down.shape == (128, 32)
+    assert layer.proj_up.shape == (128, 32)
+    assert layer.smooth_factor.shape == (128,)
+    assert layer.wcscales.shape == (128,)
+    assert layer.wtscale.shape == (1,)
+
+    assert layer.qweight.input_dim == 1
+    assert layer.qweight.output_dim == 0
+    assert layer.wscales.input_dim == 0
+    assert layer.wscales.output_dim == 1
+    assert layer.proj_down.input_dim == 0
+    assert layer.proj_up.output_dim == 0
+    assert layer.wcscales.output_dim == 0
+
+
+def test_linear_method_rejects_misaligned_input_partition(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setattr(svdquant_config, "_assert_supported", lambda: None)
+    method = DiffusionSVDQuantLinearMethod(DiffusionSVDQuantConfig())
+
+    with pytest.raises(ValueError, match="divisible by the block size 16"):
+        method.create_weights(
+            torch.nn.Module(),
+            input_size_per_partition=24,
+            output_partition_sizes=[32],
+            input_size=24,
+            output_size=32,
+            params_dtype=torch.bfloat16,
+        )
+
+
+def test_active_linear_uses_svdquant_method(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setattr(svdquant_config, "_assert_supported", lambda: None)
+    config = DiffusionSVDQuantConfig()
+    linear = Mock(spec=LinearBase)
+
+    method = config.get_quant_method(linear, "blocks.0.attn.out_proj")
+
+    assert isinstance(method, DiffusionSVDQuantLinearMethod)

tests/diffusion/quantization/test_svdquant_linear.py

@@ -0,0 +1,168 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from types import SimpleNamespace
+
+import pytest
+import torch
+
+from vllm_omni.quantization import svdquant_config as svdquant
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
+
+
+class _FakeNvfp4Kernel:
+    def __init__(self, base_weight: torch.Tensor | None = None) -> None:
+        self.base_weight = base_weight
+        self.processed = False
+        self.last_input: torch.Tensor | None = None
+
+    def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
+        del layer
+        self.processed = True
+
+    def apply_weights(
+        self,
+        layer: torch.nn.Module,
+        x: torch.Tensor,
+        bias: torch.Tensor | None = None,
+    ) -> torch.Tensor:
+        del layer
+        assert bias is None
+        assert self.base_weight is not None
+        self.last_input = x.clone()
+        return torch.mm(x, self.base_weight)
+
+
+def _register_parameter(
+    layer: torch.nn.Module,
+    name: str,
+    value: torch.Tensor,
+) -> None:
+    layer.register_parameter(
+        name,
+        torch.nn.Parameter(value, requires_grad=False),
+    )
+
+
+def test_supports_only_validated_datacenter_blackwell() -> None:
+    assert not svdquant._supports_capability(SimpleNamespace(major=10, minor=0))
+    assert svdquant._supports_capability(SimpleNamespace(major=10, minor=3))
+    assert not svdquant._supports_capability(SimpleNamespace(major=11, minor=0))
+    assert not svdquant._supports_capability(SimpleNamespace(major=12, minor=0))
+    assert not svdquant._supports_capability(None)
+
+
+def test_rejects_nvfp4_backend_with_incompatible_layer_contract(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    svdquant._nvfp4_kernel.cache_clear()
+    monkeypatch.setattr(
+        svdquant,
+        "init_nvfp4_linear_kernel",
+        lambda: _FakeNvfp4Kernel(),
+    )
+
+    with pytest.raises(RuntimeError, match="incompatible with the SVDQuant checkpoint"):
+        svdquant._nvfp4_kernel()
+
+    svdquant._nvfp4_kernel.cache_clear()
+
+
+def test_prepare_weights_uses_existing_nvfp4_layout_contract(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    kernel = _FakeNvfp4Kernel()
+    monkeypatch.setattr(svdquant, "_assert_supported", lambda: None)
+    monkeypatch.setattr(svdquant, "_nvfp4_kernel", lambda: kernel)
+    method = svdquant.DiffusionSVDQuantLinearMethod(svdquant.DiffusionSVDQuantConfig())
+    layer = torch.nn.Module()
+    qweight = torch.arange(16, dtype=torch.int8).reshape(4, 4)
+    wscales = torch.arange(8, dtype=torch.float32).to(torch.float8_e4m3fn).reshape(2, 4)
+    _register_parameter(layer, "qweight", qweight)
+    _register_parameter(layer, "wscales", wscales)
+    _register_parameter(layer, "wtscale", torch.tensor([2.0], dtype=torch.bfloat16))
+    _register_parameter(
+        layer,
+        "wcscales",
+        torch.tensor([1.0, 2.0, 3.0, 4.0], dtype=torch.bfloat16),
+    )
+
+    method.process_weights_after_loading(layer)
+
+    assert kernel.processed
+    assert not hasattr(layer, "qweight")
+    assert not hasattr(layer, "wscales")
+    assert layer.weight.dtype == torch.uint8
+    assert torch.equal(layer.weight.view(torch.int8), qweight)
+    assert torch.equal(layer.weight_scale, wscales.transpose(0, 1))
+    torch.testing.assert_close(layer.alpha, torch.tensor([2.0]))
+    assert layer.input_global_scale_inv.dtype == torch.float32
+    assert layer.output_channel_scale.shape == (4,)
+
+
+def test_prepare_weights_drops_identity_output_scale(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    kernel = _FakeNvfp4Kernel()
+    monkeypatch.setattr(svdquant, "_assert_supported", lambda: None)
+    monkeypatch.setattr(svdquant, "_nvfp4_kernel", lambda: kernel)
+    method = svdquant.DiffusionSVDQuantLinearMethod(svdquant.DiffusionSVDQuantConfig())
+    layer = torch.nn.Module()
+    _register_parameter(layer, "qweight", torch.zeros(4, 4, dtype=torch.int8))
+    _register_parameter(
+        layer,
+        "wscales",
+        torch.ones(2, 4, dtype=torch.float8_e4m3fn),
+    )
+    _register_parameter(layer, "wtscale", torch.ones(1, dtype=torch.bfloat16))
+    _register_parameter(layer, "wcscales", torch.ones(4, dtype=torch.bfloat16))
+
+    method.process_weights_after_loading(layer)
+
+    assert layer.output_channel_scale is None
+
+
+def test_compatibility_path_applies_qkv_scale_and_rank_correction(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    base_weight = torch.tensor(
+        [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
+        dtype=torch.bfloat16,
+    )
+    kernel = _FakeNvfp4Kernel(base_weight)
+    monkeypatch.setattr(svdquant, "_assert_supported", lambda: None)
+    monkeypatch.setattr(svdquant, "_nvfp4_kernel", lambda: kernel)
+    method = svdquant.DiffusionSVDQuantLinearMethod(svdquant.DiffusionSVDQuantConfig())
+
+    layer = torch.nn.Module()
+    layer.smooth_factor = torch.tensor([2.0, 4.0], dtype=torch.bfloat16)
+    layer.proj_down = torch.tensor([[1.0], [2.0]], dtype=torch.bfloat16)
+    layer.proj_up = torch.tensor([[1.0], [2.0], [3.0]], dtype=torch.bfloat16)
+    layer.output_channel_scale = torch.tensor([1.0, 2.0, 4.0], dtype=torch.bfloat16)
+    layer.output_size_per_partition = 3
+    x = torch.tensor([[2.0, 4.0], [4.0, 8.0]], dtype=torch.bfloat16)
+    bias = torch.tensor([1.0, 2.0, 3.0], dtype=torch.bfloat16)
+
+    actual = method.apply(layer, x, bias)
+
+    smoothed = x / layer.smooth_factor
+    expected = torch.mm(smoothed, base_weight)
+    expected = expected * layer.output_channel_scale
+    expected = expected + torch.mm(torch.mm(x, layer.proj_down), layer.proj_up.T)
+    expected = expected + bias
+    torch.testing.assert_close(actual, expected)
+    torch.testing.assert_close(kernel.last_input, smoothed)
+
+
+def test_compatibility_path_rejects_fp16_activations(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setattr(svdquant, "_assert_supported", lambda: None)
+    method = svdquant.DiffusionSVDQuantLinearMethod(svdquant.DiffusionSVDQuantConfig())
+    layer = torch.nn.Module()
+    with pytest.raises(ValueError, match="requires BF16 activations"):
+        method.apply(
+            layer,
+            torch.zeros(1, 16, dtype=torch.float16),
+        )

tests/diffusion/quantization/test_svdquant_tp_loading.py

@@ -0,0 +1,130 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+import pytest
+import torch
+from vllm.model_executor.layers.linear import (
+    ColumnParallelLinear,
+    QKVParallelLinear,
+    RowParallelLinear,
+)
+
+from vllm_omni.quantization import svdquant_config as svdquant
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
+
+
+def _linear_without_init(linear_type: type[torch.nn.Module], tp_rank: int):
+    layer = object.__new__(linear_type)
+    torch.nn.Module.__init__(layer)
+    layer.tp_rank = tp_rank
+    return layer
+
+
+def _values(shape: tuple[int, ...], dtype: torch.dtype) -> torch.Tensor:
+    values = torch.arange(torch.Size(shape).numel(), dtype=torch.int64).remainder(127)
+    return values.reshape(shape).to(dtype)
+
+
+def _method(monkeypatch: pytest.MonkeyPatch):
+    monkeypatch.setattr(svdquant, "_assert_supported", lambda: None)
+    return svdquant.DiffusionSVDQuantLinearMethod(svdquant.DiffusionSVDQuantConfig(rank=8))
+
+
+def test_column_parallel_loader_shards_svdquant_outputs(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    layer = _linear_without_init(ColumnParallelLinear, tp_rank=1)
+    method = _method(monkeypatch)
+    method.create_weights(
+        layer,
+        input_size_per_partition=128,
+        output_partition_sizes=[32],
+        input_size=128,
+        output_size=64,
+        params_dtype=torch.bfloat16,
+        weight_loader=layer.weight_loader,
+    )
+
+    loaded_tensors = {
+        "qweight": _values((64, 64), torch.int8),
+        "wscales": _values((8, 64), torch.float8_e4m3fn),
+        "proj_up": _values((64, 8), torch.bfloat16),
+        "wcscales": _values((64,), torch.bfloat16),
+    }
+    for name, loaded in loaded_tensors.items():
+        parameter = getattr(layer, name)
+        parameter.weight_loader(parameter, loaded)
+        expected = loaded.narrow(parameter.output_dim, 32, 32)
+        torch.testing.assert_close(parameter, expected)
+
+
+def test_row_parallel_loader_shards_svdquant_inputs(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    layer = _linear_without_init(RowParallelLinear, tp_rank=1)
+    method = _method(monkeypatch)
+    method.create_weights(
+        layer,
+        input_size_per_partition=64,
+        output_partition_sizes=[64],
+        input_size=128,
+        output_size=64,
+        params_dtype=torch.bfloat16,
+        weight_loader=layer.weight_loader,
+    )
+
+    loaded_tensors = {
+        "qweight": _values((64, 64), torch.int8),
+        "wscales": _values((8, 64), torch.float8_e4m3fn),
+        "proj_down": _values((128, 8), torch.bfloat16),
+        "smooth_factor": _values((128,), torch.bfloat16),
+    }
+    for name, loaded in loaded_tensors.items():
+        parameter = getattr(layer, name)
+        parameter.weight_loader(parameter, loaded)
+        shard_size = parameter.shape[parameter.input_dim]
+        expected = loaded.narrow(parameter.input_dim, shard_size, shard_size)
+        torch.testing.assert_close(parameter, expected)
+
+
+def test_qkv_parallel_loader_shards_fused_svdquant_outputs(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    layer = _linear_without_init(QKVParallelLinear, tp_rank=1)
+    layer.total_num_heads = 4
+    layer.total_num_kv_heads = 2
+    layer.num_heads = 2
+    layer.num_kv_heads = 1
+    layer.num_kv_head_replicas = 1
+    layer.head_size = 8
+    layer.v_head_size = 8
+    method = _method(monkeypatch)
+    method.create_weights(
+        layer,
+        input_size_per_partition=128,
+        output_partition_sizes=[16, 8, 8],
+        input_size=128,
+        output_size=64,
+        params_dtype=torch.bfloat16,
+        weight_loader=layer.weight_loader,
+    )
+
+    loaded_tensors = {
+        "qweight": _values((64, 64), torch.int8),
+        "wscales": _values((8, 64), torch.float8_e4m3fn),
+        "proj_up": _values((64, 8), torch.bfloat16),
+        "wcscales": _values((64,), torch.bfloat16),
+    }
+    expected_global_outputs = [slice(16, 32), slice(40, 48), slice(56, 64)]
+    for name, loaded in loaded_tensors.items():
+        parameter = getattr(layer, name)
+        parameter.weight_loader(parameter, loaded)
+        expected = torch.cat(
+            [
+                loaded.narrow(parameter.output_dim, part.start, part.stop - part.start)
+                for part in expected_global_outputs
+            ],
+            dim=parameter.output_dim,
+        )
+        torch.testing.assert_close(parameter, expected)

tests/diffusion/test_diffusion_engine_metrics.py

@@ -172,12 +172,29 @@ class TestMetricKeys:
         assert ".cancel(" not in abort_branch
 
     def test_orchestrator_consumes_metrics_only_output_without_routing(self) -> None:
+        """A metrics-only sentinel contributes its queue depth and is not routed.
+
+        Absorption lives in ``_absorb_diffusion_metrics`` rather than inline in a
+        loop, so both orchestration loops share one implementation. The ordering
+        is asserted where it now lives: the snapshot inside the helper, and the
+        absorb-before-route guard inside every loop that polls diffusion output.
+        """
         source = _read_source(_ORCHESTRATOR_PATH)
-        loop_source = _get_function_source(source, "Orchestrator", "_orchestration_loop")
-        snapshot_pos = loop_source.index("_update_stage_replica_waiting(")
-        sentinel_pos = loop_source.index("diffusion_output.request_id == DIFFUSION_METRICS_ONLY_REQUEST_ID")
-        route_pos = loop_source.index("pool.record_output_timestamps([diffusion_output])")
-        assert snapshot_pos < sentinel_pos < route_pos
+
+        # Snapshot before the verdict, so a sentinel still reports its waiting
+        # depth on the way out instead of being dropped unaccounted.
+        absorb_source = _get_function_source(source, "Orchestrator", "_absorb_diffusion_metrics")
+        snapshot_pos = absorb_source.index("_update_stage_replica_waiting(")
+        sentinel_pos = absorb_source.index("diffusion_output.request_id == DIFFUSION_METRICS_ONLY_REQUEST_ID")
+        assert snapshot_pos < sentinel_pos
+
+        # Absorb before routing, or a sentinel reaches the downstream consumer
+        # as if it were a real request output.
+        for loop_name in ("_orchestration_loop", "_orchestration_loop_event_driven"):
+            loop_source = _get_function_source(source, "Orchestrator", loop_name)
+            absorb_pos = loop_source.index("self._absorb_diffusion_metrics(")
+            route_pos = loop_source.index("record_output_timestamps(")
+            assert absorb_pos < route_pos, loop_name
 
 
 class TestVaeDecodeEmit:

tests/distributed/omni_connectors/test_chunk_transfer_adapter.py

@@ -1209,6 +1209,67 @@ def test_cleanup_preserves_pending_save(build_adapter):
     assert len(adapter._pending_save_reqs) == 1
 
 
+def test_abort_invalidates_queued_sender_task_before_external_id_reuse(build_adapter):
+    adapter, connector = build_adapter(stage_id=1)
+    request = _req("req-old", RequestStatus.WAITING, external_req_id="ext-reused")
+    adapter.custom_process_next_stage_input_func = lambda **kwargs: OmniPayloadStruct(
+        codes=CodesStruct(audio=torch.tensor([1], dtype=torch.long))
+    )
+
+    adapter.save_async(multimodal_output=None, request=request)
+    stale_task = adapter._pending_save_reqs.popleft()
+    adapter.cleanup(request.request_id, request.external_req_id)
+    adapter._send_single_request(stale_task)
+
+    connector.put.assert_not_called()
+    assert request.external_req_id not in adapter.put_req_chunk
+
+    replacement = _req("req-new", RequestStatus.WAITING, external_req_id=request.external_req_id)
+    adapter.save_async(multimodal_output=None, request=replacement)
+    current_task = adapter._pending_save_reqs.popleft()
+    adapter._send_single_request(current_task)
+
+    connector.put.assert_called_once()
+    assert connector.put.call_args.kwargs["put_key"] == "ext-reused_1_0"
+
+
+def test_finish_requests_does_not_wait_for_inflight_send(build_adapter):
+    adapter, connector = build_adapter(stage_id=1)
+    first = _req("req-first", RequestStatus.WAITING, external_req_id="ext-first")
+    adapter.custom_process_next_stage_input_func = lambda **kwargs: OmniPayloadStruct()
+    put_started = threading.Event()
+    release_put = threading.Event()
+
+    def blocking_put(**kwargs):
+        put_started.set()
+        release_put.wait(timeout=2)
+        return True, 1, {}
+
+    connector.put.side_effect = blocking_put
+    adapter.save_async(multimodal_output=None, request=first)
+    task = adapter._pending_save_reqs.popleft()
+    sender = threading.Thread(target=adapter._send_single_request, args=(task,))
+    sender.start()
+    assert put_started.wait(timeout=1)
+    cleaner = threading.Thread(
+        target=adapter.finish_requests,
+        args=([first.request_id], RequestStatus.FINISHED_ABORTED, {first.request_id: first}),
+    )
+    cleaner.start()
+    try:
+        cleaner.join(timeout=0.5)
+        assert not cleaner.is_alive()
+        assert task["sender_token"].cancelled
+        assert first.external_req_id in adapter._sender_tokens
+    finally:
+        release_put.set()
+        sender.join(timeout=1)
+        cleaner.join(timeout=1)
+    assert not sender.is_alive()
+    assert first.external_req_id not in adapter._sender_tokens
+    assert first.external_req_id not in adapter.put_req_chunk
+
+
 def test_cleanup_only_affects_target_request(build_adapter):
     """Cleanup for one request must not affect another request's state."""
     adapter, _ = build_adapter(stage_id=1)
@@ -1507,6 +1568,7 @@ def test_generation_scheduler_calls_cleanup_on_finished(monkeypatch, mocker: Moc
     scheduler.structured_output_manager.should_advance.return_value = False
     scheduler.finished_req_ids_dict = {}
     scheduler.kv_cache_manager.take_events.return_value = None
+    scheduler.kv_cache_manager.estimate_cached_tokens.return_value = 0
     scheduler.kv_event_publisher = mocker.MagicMock()
 
     request = _HashableRequest(
@@ -1587,6 +1649,7 @@ def test_ar_scheduler_defers_cleanup_and_queues_save_on_finished(mocker: MockerF
     scheduler.finished_req_ids_dict = {}
     scheduler.kv_cache_manager = mocker.MagicMock()
     scheduler.kv_cache_manager.take_events.return_value = None
+    scheduler.kv_cache_manager.estimate_cached_tokens.return_value = 0
     scheduler.kv_event_publisher = mocker.MagicMock()
     scheduler.waiting_for_transfer_free = set()
     scheduler.transfer_triggered_requests = set()
@@ -1742,6 +1805,7 @@ def _build_deferred_finish_scheduler(mocker, *, running, pending_finish_reqs):
     scheduler.structured_output_manager.should_advance.return_value = False
     scheduler.finished_req_ids_dict = {}
     scheduler.kv_cache_manager.take_events.return_value = None
+    scheduler.kv_cache_manager.estimate_cached_tokens.return_value = 0
     scheduler.kv_event_publisher = mocker.MagicMock()
     scheduler._pending_finish_reqs = list(pending_finish_reqs)
 
@@ -2216,3 +2280,43 @@ def test_expiry_is_per_request(build_adapter):
 
     assert adapter.collect_timed_out_request_ids(timeout_s=600.0) == {"stalled"}
     assert "healthy" in adapter._waiting_since
+
+
+def test_abort_clears_native_codec_state_before_external_id_reuse(build_adapter):
+    from vllm_omni.model_executor.stage_input_processors.nemotron_voicechat import (
+        talker2code2wav_async_chunk,
+    )
+
+    adapter, _ = build_adapter(stage_id=1, connector_extra={"codec_chunk_frames": 1})
+    external_req_id = "ext-native-abort"
+    first = _req("req-native-abort", RequestStatus.WAITING, external_req_id=external_req_id)
+    first.resumable = True
+    first.model_intermediate_buffer = None
+    first.additional_information = {"meta": {"codec_streaming": True}, "nvc_logical_prompt_len": 1}
+    first_frame = torch.full((1, 31), 101, dtype=torch.long)
+
+    talker2code2wav_async_chunk(
+        adapter,
+        {"codes": {"audio": first_frame}, "meta": {"codec_streaming": True}},
+        first,
+        is_finished=False,
+    )
+
+    adapter.requests_num_chunks_sent[external_req_id] = 1
+    adapter.finish_requests([first.request_id], RequestStatus.FINISHED_ABORTED, {first.request_id: first})
+
+    assert external_req_id not in adapter.request_payload
+    assert external_req_id not in adapter.requests_num_chunks_sent
+
+    replacement = _req("req-native-abort", RequestStatus.WAITING, external_req_id=external_req_id)
+    replacement.resumable = True
+    replacement.model_intermediate_buffer = None
+    replacement.additional_information = first.additional_information
+    replacement_payload = talker2code2wav_async_chunk(
+        adapter,
+        {"codes": {"audio": torch.full((1, 31), 202, dtype=torch.long)}, "meta": {"codec_streaming": True}},
+        replacement,
+        is_finished=False,
+    )
+
+    assert torch.equal(replacement_payload.codes.audio, torch.full((1, 31), 202, dtype=torch.long))

tests/e2e/accuracy/ltx/test_ltx25_official_similarity.py

@@ -22,6 +22,7 @@ import shutil
 import subprocess
 import sys
 import time
+from dataclasses import asdict, dataclass
 from pathlib import Path
 
 import numpy as np
@@ -83,35 +84,27 @@ NEGATIVE_PROMPT = (
 # Release parity exercises the accelerated BF16 cuDNN path in both runtimes.
 # Degenerate one-token attention falls back to PyTorch SDPA MATH.
 ATTENTION_BACKEND = "torch_sdpa_cudnn_bf16"
-VIDEO_SSIM_MEAN_THRESHOLD = 0.95
-VIDEO_SSIM_MIN_THRESHOLD = 0.90
-VIDEO_PSNR_MEAN_THRESHOLD = 30.0
-AUDIO_RELATIVE_L2_THRESHOLD = 0.2
-AUDIO_COSINE_THRESHOLD = 0.95
-
-# Decoded-output gates for the pinned official/Omni BF16 cuDNN comparison.
-# The metrics artifact preserves the exact observed values.
-LTX25_VIDEO_SSIM_MEAN_THRESHOLD = 0.99
-LTX25_I2V_VIDEO_SSIM_MEAN_THRESHOLD = 0.99
-LTX25_VIDEO_SSIM_MIN_THRESHOLD = 0.99
-LTX25_VIDEO_PSNR_MEAN_THRESHOLD = 40.0
-LTX25_VIDEO_LPIPS_MEAN_THRESHOLD = 0.01
-LTX25_AUDIO_RELATIVE_L2_THRESHOLD = 0.32
-LTX25_AUDIO_COSINE_THRESHOLD = 0.95
-
-# Full/SFT uses independently converted dev weights, but the release contract
-# still requires decoded-video parity at the same 0.99 SSIM floor as the
-# distilled paths when both runtimes use BF16 cuDNN attention.
-LTX25_FULL_VIDEO_SSIM_MEAN_THRESHOLD = 0.99
-LTX25_FULL_VIDEO_SSIM_MIN_THRESHOLD = 0.99
-LTX25_FULL_VIDEO_PSNR_MEAN_THRESHOLD = 40.0
-LTX25_FULL_AUDIO_RELATIVE_L2_THRESHOLD = 0.3
-LTX25_FULL_VIDEO_LPIPS_MEAN_THRESHOLD = 0.01
-LTX25_FULL_I2V_VIDEO_LPIPS_MEAN_THRESHOLD = 0.02
-LTX25_FULL_AUDIO_COSINE_THRESHOLD = AUDIO_COSINE_THRESHOLD
 LTX25_STAGE_2_SIGMAS = [0.909375, 0.725, 0.421875, 0.0]
 
 
+@dataclass(frozen=True)
+class LTXAccuracyThresholds:
+    video_ssim_mean: float
+    video_ssim_min: float
+    video_psnr_mean_db: float
+    audio_relative_l2: float
+    audio_cosine_similarity: float
+
+
+STRICT_THRESHOLDS = LTXAccuracyThresholds(
+    video_ssim_mean=0.99,
+    video_ssim_min=0.99,
+    video_psnr_mean_db=40.0,
+    audio_relative_l2=0.10,
+    audio_cosine_similarity=0.99,
+)
+
+
 def test_ltx_reference_runner_unwraps_flattened_pipeline_output() -> None:
     from vllm_omni.outputs import OmniRequestOutput
 
@@ -419,8 +412,6 @@ def test_ltx25_i2v_requests_pin_official_crf() -> None:
 
 
 def _video_metrics(reference: np.ndarray, prediction: np.ndarray) -> dict[str, float]:
-    from benchmarks.diffusion.quantization_quality import compute_lpips_video
-
     assert reference.shape == prediction.shape
     assert reference.ndim == 4 and reference.shape[-1] == 3
     ssim_scores: list[float] = []
@@ -435,7 +426,6 @@ def _video_metrics(reference: np.ndarray, prediction: np.ndarray) -> dict[str, f
         "ssim_mean": float(np.mean(ssim_scores)),
         "ssim_min": float(np.min(ssim_scores)),
         "psnr_mean_db": float(np.mean(psnr_scores)),
-        "lpips_alex_mean": compute_lpips_video(reference, prediction),
         "max_abs": float(difference.max()),
         "mean_abs": float(difference.mean()),
     }
@@ -463,6 +453,17 @@ def _audio_metrics(reference: np.ndarray, prediction: np.ndarray) -> dict[str, f
     }
 
 
+def _assert_strict_similarity(
+    video_metrics: dict[str, float],
+    audio_metrics: dict[str, float | bool],
+) -> None:
+    assert video_metrics["ssim_mean"] >= STRICT_THRESHOLDS.video_ssim_mean
+    assert video_metrics["ssim_min"] >= STRICT_THRESHOLDS.video_ssim_min
+    assert video_metrics["psnr_mean_db"] >= STRICT_THRESHOLDS.video_psnr_mean_db
+    assert audio_metrics["relative_l2"] <= STRICT_THRESHOLDS.audio_relative_l2
+    assert audio_metrics["cosine_similarity"] >= STRICT_THRESHOLDS.audio_cosine_similarity
+
+
 @pytest.mark.slow
 @pytest.mark.benchmark
 @pytest.mark.diffusion
@@ -569,6 +570,7 @@ def test_ltx25_distilled_two_stage_matches_official(accuracy_artifact_root: Path
         "omni_model": omni_model_source,
         "omni_model_revision": omni_model_revision,
         "resolved_omni_model_path": str(omni_model),
+        "thresholds": asdict(STRICT_THRESHOLDS),
         "peak_memory_mb": {
             "official_allocated": official_metadata["peak_memory_allocated_mb"],
             "official_reserved": official_metadata["peak_memory_reserved_mb"],
@@ -582,13 +584,7 @@ def test_ltx25_distilled_two_stage_matches_official(accuracy_artifact_root: Path
     (output_root / "metrics.json").write_text(json.dumps(result, indent=2) + "\n")
     print(json.dumps(result, indent=2))
 
-    ssim_mean_threshold = LTX25_VIDEO_SSIM_MEAN_THRESHOLD if task == "t2v" else LTX25_I2V_VIDEO_SSIM_MEAN_THRESHOLD
-    assert video_metrics["ssim_mean"] >= ssim_mean_threshold
-    assert video_metrics["ssim_min"] >= LTX25_VIDEO_SSIM_MIN_THRESHOLD
-    assert video_metrics["psnr_mean_db"] >= LTX25_VIDEO_PSNR_MEAN_THRESHOLD
-    assert video_metrics["lpips_alex_mean"] <= LTX25_VIDEO_LPIPS_MEAN_THRESHOLD
-    assert audio_metrics["cosine_similarity"] >= LTX25_AUDIO_COSINE_THRESHOLD
-    assert audio_metrics["relative_l2"] <= LTX25_AUDIO_RELATIVE_L2_THRESHOLD
+    _assert_strict_similarity(video_metrics, audio_metrics)
 
 
 @pytest.mark.slow
@@ -729,6 +725,7 @@ def test_ltx25_full_matches_official(accuracy_artifact_root: Path, task: str, pi
         "omni_model": omni_model_source,
         "omni_model_revision": omni_model_revision,
         "resolved_omni_model_path": str(omni_model),
+        "thresholds": asdict(STRICT_THRESHOLDS),
         "peak_memory_mb": {
             "official_allocated": official_metadata["peak_memory_allocated_mb"],
             "official_reserved": official_metadata["peak_memory_reserved_mb"],
@@ -742,12 +739,4 @@ def test_ltx25_full_matches_official(accuracy_artifact_root: Path, task: str, pi
     (output_root / "metrics.json").write_text(json.dumps(result, indent=2) + "\n")
     print(json.dumps(result, indent=2))
 
-    assert video_metrics["ssim_mean"] >= LTX25_FULL_VIDEO_SSIM_MEAN_THRESHOLD
-    assert video_metrics["ssim_min"] >= LTX25_FULL_VIDEO_SSIM_MIN_THRESHOLD
-    assert video_metrics["psnr_mean_db"] >= LTX25_FULL_VIDEO_PSNR_MEAN_THRESHOLD
-    lpips_threshold = (
-        LTX25_FULL_VIDEO_LPIPS_MEAN_THRESHOLD if task == "t2v" else LTX25_FULL_I2V_VIDEO_LPIPS_MEAN_THRESHOLD
-    )
-    assert audio_metrics["relative_l2"] <= LTX25_FULL_AUDIO_RELATIVE_L2_THRESHOLD
-    assert video_metrics["lpips_alex_mean"] <= lpips_threshold
-    assert audio_metrics["cosine_similarity"] >= LTX25_FULL_AUDIO_COSINE_THRESHOLD
+    _assert_strict_similarity(video_metrics, audio_metrics)

tests/e2e/accuracy/ltx/test_ltx_official_similarity.py

@@ -3,9 +3,8 @@
 
 """E2E accuracy guard against a pinned Lightricks LTX pipeline revision.
 
-The original reduced one-stage guards remain unchanged. Four complementary
-default-shape cases cover both model versions, both two-stage weight families,
-and T2V/I2V without expanding every combination into a duplicate golden.
+Four complementary default-shape cases cover both model versions, both
+two-stage weight families, and T2V/I2V without duplicating one-stage coverage.
 """
 
 from __future__ import annotations
@@ -69,25 +68,23 @@ class LTXAccuracyThresholds:
     video_ssim_mean: float
     video_ssim_min: float
     video_psnr_mean_db: float
-    # Zero disables gating for waveform-sensitive audio that is not expected
-    # to align with the official output.
     audio_relative_l2: float
     audio_cosine_similarity: float
 
 
 STRICT_THRESHOLDS = LTXAccuracyThresholds(
-    video_ssim_mean=0.95,
-    video_ssim_min=0.90,
-    video_psnr_mean_db=30.0,
-    audio_relative_l2=0.20,
-    audio_cosine_similarity=0.95,
+    video_ssim_mean=0.99,
+    video_ssim_min=0.99,
+    video_psnr_mean_db=40.0,
+    audio_relative_l2=0.10,
+    audio_cosine_similarity=0.99,
 )
 
 
 @dataclass(frozen=True)
 class LTXAccuracyCase:
     name: str
-    pipeline_kind: Literal["one_stage", "distilled", "two_stage"]
+    pipeline_kind: Literal["distilled", "two_stage"]
     model_id: str
     model_revision: str
     model_env: str
@@ -99,8 +96,6 @@ class LTXAccuracyCase:
     num_inference_steps: int
     seed: int
     stg_block: int | None
-    thresholds: LTXAccuracyThresholds
-    prompt: str = PROMPT
     gemma_model_id: str | None = None
     gemma_model_revision: str | None = None
     gemma_model_env: str | None = None
@@ -168,59 +163,6 @@ I2V_IMAGE = LTXArtifact(
 )
 
 
-LEGACY_CASES = (
-    LTXAccuracyCase(
-        name="ltx2",
-        pipeline_kind="one_stage",
-        model_id="Lightricks/LTX-2",
-        model_revision=LTX2_REVISION,
-        model_env="VLLM_TEST_LTX2_MODEL",
-        model_class_name="LTX2Pipeline",
-        checkpoint=LTX2_CHECKPOINT,
-        width=512,
-        height=384,
-        num_frames=25,
-        num_inference_steps=20,
-        seed=42,
-        stg_block=29,
-        thresholds=STRICT_THRESHOLDS,
-    ),
-    LTXAccuracyCase(
-        name="ltx2_3",
-        pipeline_kind="one_stage",
-        model_id="diffusers/LTX-2.3-Diffusers",
-        model_revision="8eee8edcf067e838b843f926ec4d4cc9b2be1aaf",
-        model_env="VLLM_TEST_LTX23_MODEL",
-        model_class_name="LTX2Pipeline",
-        checkpoint=LTX23_CHECKPOINT,
-        width=512,
-        height=384,
-        num_frames=25,
-        num_inference_steps=20,
-        seed=42,
-        stg_block=28,
-        thresholds=STRICT_THRESHOLDS,
-    ),
-    LTXAccuracyCase(
-        name="ltx2_3_i2v",
-        pipeline_kind="one_stage",
-        model_id="diffusers/LTX-2.3-Diffusers",
-        model_revision="8eee8edcf067e838b843f926ec4d4cc9b2be1aaf",
-        model_env="VLLM_TEST_LTX23_MODEL",
-        model_class_name="LTX2Pipeline",
-        checkpoint=LTX23_CHECKPOINT,
-        width=512,
-        height=384,
-        num_frames=25,
-        num_inference_steps=20,
-        seed=42,
-        stg_block=28,
-        thresholds=STRICT_THRESHOLDS,
-        image=I2V_IMAGE,
-    ),
-)
-
-
 TWO_STAGE_CASES = (
     LTXAccuracyCase(
         name="ltx2_distilled_t2v",
@@ -240,7 +182,6 @@ TWO_STAGE_CASES = (
         gemma_model_id="Lightricks/LTX-2",
         gemma_model_revision=LTX2_REVISION,
         gemma_model_env="VLLM_TEST_LTX2_MODEL",
-        thresholds=STRICT_THRESHOLDS,
     ),
     LTXAccuracyCase(
         name="ltx23_distilled_i2v",
@@ -260,7 +201,6 @@ TWO_STAGE_CASES = (
         gemma_model_id="diffusers/LTX-2.3-Diffusers",
         gemma_model_revision="8eee8edcf067e838b843f926ec4d4cc9b2be1aaf",
         gemma_model_env="VLLM_TEST_LTX23_MODEL",
-        thresholds=STRICT_THRESHOLDS,
         image=I2V_IMAGE,
     ),
     LTXAccuracyCase(
@@ -280,7 +220,6 @@ TWO_STAGE_CASES = (
         seed=10,
         stg_block=29,
         enable_layerwise_offload=True,
-        thresholds=STRICT_THRESHOLDS,
     ),
     LTXAccuracyCase(
         name="ltx23_two_stage_layer_fused_i2v",
@@ -299,12 +238,11 @@ TWO_STAGE_CASES = (
         seed=10,
         stg_block=28,
         enable_layerwise_offload=True,
-        thresholds=STRICT_THRESHOLDS,
         image=I2V_IMAGE,
     ),
 )
 
-CASES = (*LEGACY_CASES, *TWO_STAGE_CASES)
+WEEKLY_CASES = TWO_STAGE_CASES
 
 
 def _run(command: list[str], *, env: dict[str, str], timeout: int = 1800) -> None:
@@ -460,10 +398,49 @@ def _resolve_image(case: LTXAccuracyCase) -> Path | None:
     return _resolve_artifact(case.image)
 
 
+def _omni_model_with_pinned_artifacts(
+    model: Path,
+    output_root: Path,
+    artifacts: tuple[Path | None, ...],
+) -> Path:
+    """Expose the already-resolved sidecars to Omni without copying the model."""
+    resolved_artifacts = tuple(artifact for artifact in artifacts if artifact is not None)
+    if not resolved_artifacts:
+        return model
+
+    overlay = output_root / "omni-model"
+    overlay.mkdir()
+    for source in model.iterdir():
+        (overlay / source.name).symlink_to(source.resolve(), target_is_directory=source.is_dir())
+    for artifact in resolved_artifacts:
+        target = overlay / artifact.name
+        if target.exists():
+            assert target.samefile(artifact), f"Model sidecar does not match pinned artifact: {target} != {artifact}"
+        else:
+            target.symlink_to(artifact.resolve())
+    return overlay
+
+
+def test_omni_model_overlay_pins_artifacts(tmp_path: Path) -> None:
+    model = tmp_path / "model"
+    model.mkdir()
+    component = model / "model_index.json"
+    component.write_text("{}\n")
+    artifact = tmp_path / "sidecar.safetensors"
+    artifact.write_bytes(b"pinned")
+    output_root = tmp_path / "output"
+    output_root.mkdir()
+
+    overlay = _omni_model_with_pinned_artifacts(model, output_root, (artifact,))
+
+    assert (overlay / component.name).samefile(component)
+    assert (overlay / artifact.name).samefile(artifact)
+
+
 def _request(case: LTXAccuracyCase, image: Path | None) -> dict[str, object]:
     request: dict[str, object] = {
         "pipeline_kind": case.pipeline_kind,
-        "prompt": case.prompt,
+        "prompt": PROMPT,
         "negative_prompt": "" if case.pipeline_kind == "distilled" else NEGATIVE_PROMPT,
         "width": case.width,
         "height": case.height,
@@ -473,14 +450,17 @@ def _request(case: LTXAccuracyCase, image: Path | None) -> dict[str, object]:
         "seed": case.seed,
     }
     if case.pipeline_kind != "distilled":
-        assert case.stg_block is not None
         request.update(
             video_cfg_scale=3.0,
             audio_cfg_scale=7.0,
-            video_stg_scale=1.0,
-            audio_stg_scale=1.0,
             video_modality_scale=3.0,
             audio_modality_scale=3.0,
+        )
+    if case.pipeline_kind == "two_stage":
+        assert case.stg_block is not None
+        request.update(
+            video_stg_scale=1.0,
+            audio_stg_scale=1.0,
             video_rescale_scale=0.7,
             audio_rescale_scale=0.7,
             video_stg_blocks=[case.stg_block],
@@ -543,7 +523,7 @@ def _audio_metrics(reference: np.ndarray, prediction: np.ndarray) -> dict[str, f
     }
 
 
-@pytest.mark.parametrize("case", CASES, ids=lambda case: case.name)
+@pytest.mark.parametrize("case", WEEKLY_CASES, ids=lambda case: case.name)
 @pytest.mark.slow
 @pytest.mark.benchmark
 @pytest.mark.diffusion
@@ -561,9 +541,11 @@ def test_ltx_matches_official(case: LTXAccuracyCase, accuracy_artifact_root: Pat
     checkpoint = _resolve_artifact(case.checkpoint, model)
     spatial_upsampler = _resolve_artifact(case.spatial_upsampler, model) if case.spatial_upsampler is not None else None
     distilled_lora = _resolve_artifact(case.distilled_lora, model) if case.distilled_lora is not None else None
+    omni_model = _omni_model_with_pinned_artifacts(model, output_root, (spatial_upsampler, distilled_lora))
     image = _resolve_image(case)
+    request = _request(case, image)
     request_path = output_root / "request.json"
-    request_path.write_text(json.dumps(_request(case, image), indent=2) + "\n")
+    request_path.write_text(json.dumps(request, indent=2) + "\n")
 
     runner = Path(__file__).with_name("run_ltx_reference.py")
     runner_args = [
@@ -619,7 +601,7 @@ def test_ltx_matches_official(case: LTXAccuracyCase, accuracy_artifact_root: Pat
             "--output-dir",
             str(omni_output),
             "--model",
-            str(model),
+            str(omni_model),
             "--model-class-name",
             case.model_class_name,
         ]
@@ -651,18 +633,16 @@ def test_ltx_matches_official(case: LTXAccuracyCase, accuracy_artifact_root: Pat
         "spatial_upsampler_revision": (case.spatial_upsampler.revision if case.spatial_upsampler is not None else None),
         "distilled_lora_revision": case.distilled_lora.revision if case.distilled_lora is not None else None,
         "enable_layerwise_offload": enable_layerwise_offload,
-        "thresholds": asdict(case.thresholds),
-        "request": _request(case, image),
+        "thresholds": asdict(STRICT_THRESHOLDS),
+        "request": request,
         "video": video_metrics,
         "audio": audio_metrics,
     }
     (output_root / "metrics.json").write_text(json.dumps(result, indent=2) + "\n")
     print(json.dumps(result, indent=2))
 
-    assert video_metrics["ssim_mean"] >= case.thresholds.video_ssim_mean
-    assert video_metrics["ssim_min"] >= case.thresholds.video_ssim_min
-    assert video_metrics["psnr_mean_db"] >= case.thresholds.video_psnr_mean_db
-    if case.thresholds.audio_relative_l2 > 0:
-        assert audio_metrics["relative_l2"] <= case.thresholds.audio_relative_l2
-    if case.thresholds.audio_cosine_similarity > 0:
-        assert audio_metrics["cosine_similarity"] >= case.thresholds.audio_cosine_similarity
+    assert video_metrics["ssim_mean"] >= STRICT_THRESHOLDS.video_ssim_mean
+    assert video_metrics["ssim_min"] >= STRICT_THRESHOLDS.video_ssim_min
+    assert video_metrics["psnr_mean_db"] >= STRICT_THRESHOLDS.video_psnr_mean_db
+    assert audio_metrics["relative_l2"] <= STRICT_THRESHOLDS.audio_relative_l2
+    assert audio_metrics["cosine_similarity"] >= STRICT_THRESHOLDS.audio_cosine_similarity

tests/e2e/features/fullduplex/nemotron_voicechat/test_data_plane.py

@@ -0,0 +1,94 @@
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+
+from vllm_omni.experimental.fullduplex.engine.duplex_runtime import duplex_resource_request_id
+from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
+from vllm_omni.experimental.fullduplex.nemotron_voicechat.data_plane import (
+    NemotronVoiceChatDataPlaneContext,
+    NemotronVoiceChatDataPlaneSession,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+_RUNTIME = {
+    "nvc_text_bos_id": 0,
+    "nvc_text_eos_id": 1,
+    "nvc_text_pad_id": 12,
+    "nvc_function_sotc_id": 20,
+    "nvc_function_eotc_id": 21,
+    "nvc_tokenizer_ref": "test-tokenizer",
+}
+
+
+def _projector(encode_audio=lambda *_: None) -> NemotronVoiceChatDataPlaneSession:
+    projector = NemotronVoiceChatDataPlaneSession(encode_audio)
+    projector.configure_runtime(_RUNTIME, tokenizer=SimpleNamespace(decode=lambda *_args, **_kwargs: "text"))
+    return projector
+
+
+def _stage0_output(request_id: str = "req-0", *, text_token: int = 12, function_token: int | None = None) -> object:
+    metadata: dict[str, object] = {"nvc_text_token_ids": [text_token]}
+    if function_token is not None:
+        metadata["nvc_function_token"] = [function_token]
+    completion = SimpleNamespace(multimodal_output=metadata)
+    return SimpleNamespace(
+        stage_id=0,
+        request_output=SimpleNamespace(request_id=request_id, outputs=[completion]),
+    )
+
+
+def test_function_channel_projects_completed_call_without_ending_speech() -> None:
+    projector = _projector()
+    projector._decode = lambda token_ids: (
+        '[{"name":"weather","arguments":{"city":"Shanghai"}}]' if token_ids == [99] else ""
+    )
+    context = NemotronVoiceChatDataPlaneContext(epoch=0)
+
+    events = []
+    for function_token in (20, 99, 21):
+        events.extend(projector.project_output(_stage0_output(function_token=function_token), context=context))
+
+    listen = [event for event in events if event.get("is_listen") is True]
+    function = [event for event in events if event.get("function_call") is True]
+    assert len(listen) == 3
+    assert len(function) == 1
+    assert function[0]["name"] == "weather"
+    assert function[0]["arguments"] == '{"city":"Shanghai"}'
+
+    projector._decode = lambda _token_ids: "not-json"
+    bad_events = []
+    for function_token in (20, 98, 21):
+        bad_events.extend(projector.project_output(_stage0_output(function_token=function_token)))
+    assert any(event.get("error_code") == "nemotron_function_call_parse_error" for event in bad_events)
+
+
+def test_new_epoch_request_does_not_inherit_partial_function_or_frame_state() -> None:
+    projector = _projector(lambda *_: "audio")
+    old_request = duplex_resource_request_id(DuplexFence("sid", epoch=1), "stage0")
+    new_request = duplex_resource_request_id(DuplexFence("sid", epoch=2), "stage0")
+
+    projector.begin_request(old_request)
+    list(projector.project_output(_stage0_output(old_request, text_token=42, function_token=20)))
+    projector.begin_request(new_request)
+
+    # EOTC in the new epoch must not close the old epoch's partial function.
+    new_events = list(projector.project_output(_stage0_output(new_request, text_token=1, function_token=21)))
+    assert not [event for event in new_events if event.get("function_call") is True]
+    assert not [event for event in new_events if event.get("end_of_turn") is True]
+
+    audio = SimpleNamespace(
+        stage_id=2,
+        request_output=SimpleNamespace(
+            request_id=new_request,
+            outputs=[
+                SimpleNamespace(multimodal_output={"model_outputs": [np.ones(1764, dtype=np.float32)], "sr": [22050]})
+            ],
+        ),
+    )
+    audio_events = list(projector.project_output(audio))
+    assert sum(event.get("end_of_turn") is True for event in audio_events) == 1

tests/e2e/features/fullduplex/nemotron_voicechat/test_input.py

@@ -0,0 +1,73 @@
+from __future__ import annotations
+
+import base64
+
+import numpy as np
+import pytest
+
+from vllm_omni.experimental.fullduplex.nemotron_voicechat.input import (
+    NemotronVoiceChatPcmAppendBuffer,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+def _payload(samples: np.ndarray, *, sample_rate_hz: int = 16000, fmt: str = "pcm_f32le") -> dict[str, object]:
+    return {
+        "type": "audio",
+        "audio": base64.b64encode(np.asarray(samples, dtype=np.float32).tobytes()).decode("ascii"),
+        "format": fmt,
+        "sample_rate_hz": sample_rate_hz,
+    }
+
+
+def _append(buffer, samples, operation_id):
+    return buffer.prepare_append(
+        _payload(np.asarray(samples, dtype=np.float32)),
+        operation_id=operation_id,
+        chunk_period_ms=80,
+        allow_emit=True,
+    )
+
+
+def test_irregular_browser_packets_emit_exact_ordered_80ms_frame() -> None:
+    buffer = NemotronVoiceChatPcmAppendBuffer()
+    assert _append(buffer, np.arange(1000), "packet-1") is None
+
+    reservation = _append(buffer, np.arange(1000, 1280), "packet-2")
+
+    assert reservation is not None
+    assert reservation.operation_id == "packet-2"
+    decoded = np.frombuffer(base64.b64decode(reservation.payload["audio"]), dtype=np.float32)
+    np.testing.assert_array_equal(decoded, np.arange(1280, dtype=np.float32))
+    assert buffer.pending_byte_count == 0
+
+
+def test_append_rejects_invalid_frame_contract() -> None:
+    cases = (
+        (1281, 16000, "pcm_f32le", 0.0, "at most 1280"),
+        (1, 8000, "pcm_f32le", 0.0, "sample_rate_hz"),
+        (1, 16000, "audio/pcm", 0.0, "format"),
+        (1, 16000, "pcm_f32le", np.nan, "finite"),
+    )
+    for samples, sample_rate_hz, fmt, value, message in cases:
+        with pytest.raises(ValueError, match=message):
+            NemotronVoiceChatPcmAppendBuffer().prepare_append(
+                _payload(np.full(samples, value, dtype=np.float32), sample_rate_hz=sample_rate_hz, fmt=fmt),
+                operation_id="invalid",
+                chunk_period_ms=80,
+                allow_emit=True,
+            )
+
+
+def test_terminal_commit_rejects_more_than_one_frame_tail() -> None:
+    buffer = NemotronVoiceChatPcmAppendBuffer()
+    for index, samples in enumerate((1000, 281)):
+        buffer.prepare_append(
+            _payload(np.zeros(samples, dtype=np.float32)),
+            operation_id=f"tail-{index}",
+            chunk_period_ms=80,
+            allow_emit=False,
+        )
+    with pytest.raises(ValueError, match="more than one 80 ms frame"):
+        buffer.prepare_commit(operation_id="commit", chunk_period_ms=80)

tests/e2e/features/fullduplex/nemotron_voicechat/test_runtime_contract.py

@@ -0,0 +1,115 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from __future__ import annotations
+
+import base64
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+from vllm.sampling_params import SamplingParams
+
+from vllm_omni.experimental.fullduplex.engine.contracts import DuplexInputMode
+from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
+from vllm_omni.experimental.fullduplex.nemotron_voicechat.runtime import (
+    NemotronVoiceChatDuplexRuntimeExtension,
+)
+from vllm_omni.experimental.fullduplex.nemotron_voicechat.serving_adapter import (
+    NemotronVoiceChatServingRuntimeAdapter,
+    _render_tool_response,
+)
+from vllm_omni.model_executor.models.nemotron_voicechat.nemotron_voicechat_thinker import (
+    NemotronVoiceChatThinkerForConditionalGeneration,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+def _frame() -> dict[str, object]:
+    raw = np.zeros(1280, dtype=np.float32).tobytes()
+    return {
+        "type": "audio",
+        "audio": base64.b64encode(raw).decode("ascii"),
+        "format": "pcm_f32le",
+        "sample_rate_hz": 16000,
+    }
+
+
+def _plan(extension, *, input_seq: int):
+    return extension.plan_append(
+        request_id="req",
+        fence=DuplexFence("sid", incarnation=2, epoch=3),
+        session_config={},
+        runtime_config={
+            "nvc_prompt_token_ids": [0, 42, 1],
+            "nvc_text_pad_id": 12,
+            "nvc_max_model_len": 8192,
+        },
+        seq=input_seq,
+        turn_seq=input_seq,
+        mode=DuplexInputMode.APPEND_AUDIO_CHUNK,
+        payload=_frame(),
+        final=False,
+        sampling_params=SamplingParams(),
+    )
+
+
+def test_first_append_prefills_prompt_then_each_append_consumes_one_frame() -> None:
+    extension = NemotronVoiceChatDuplexRuntimeExtension()
+
+    params = extension.configure_sampling_params(
+        runtime_config={}, defaults=(SamplingParams(temperature=0.7, max_tokens=4),)
+    )[0]
+    assert params.temperature == 0.0 and params.max_tokens == 1 and params.top_p == 1.0 and params.top_k == 0
+
+    first = _plan(extension, input_seq=1)
+    later = _plan(extension, input_seq=2)
+
+    assert first.prompt["prompt_token_ids"] == [0, 42, 1, 12]
+    assert later.prompt["prompt_token_ids"] == [12]
+    assert first.prompt["model_intermediate_buffer"]["duplex"]["source_input_seq"] == 1
+    assert later.prompt["model_intermediate_buffer"]["duplex"]["source_input_seq"] == 2
+
+
+def test_append_rejects_stage0_context_overflow() -> None:
+    with pytest.raises(ValueError, match="max_model_len"):
+        _plan(NemotronVoiceChatDuplexRuntimeExtension(), input_seq=8190)
+
+
+def test_function_output_becomes_versioned_nvidia_channel_tokens() -> None:
+    encoded: list[str] = []
+    adapter = NemotronVoiceChatServingRuntimeAdapter(lambda *_: None)
+    adapter._tokenizer = SimpleNamespace(
+        encode=lambda text, **_kwargs: encoded.append(text) or [31, 32, 33],
+    )
+
+    first = adapter.runtime_config_for_function_output(
+        {"type": "function_call_output", "call_id": "call-1", "output": '{"result":20}'},
+        {},
+    )
+    second = adapter.runtime_config_for_function_output(
+        {"type": "function_call_output", "call_id": "call-2", "output": "plain text"},
+        first,
+    )
+
+    assert _render_tool_response('{"result":20}') == '<TOOL_RESPONSE>[{"result":20}]</TOOL_RESPONSE>'
+    assert encoded == [
+        '<TOOL_RESPONSE>[{"result":20}]</TOOL_RESPONSE>',
+        '<TOOL_RESPONSE>["plain text"]</TOOL_RESPONSE>',
+    ]
+    assert first["nvc_function_response_generation"] == 1
+    assert second["nvc_function_response_generation"] == 2
+    assert second["nvc_function_response_token_ids"] == [31, 32, 33]
+    assert second["nvc_function_response_call_id"] == "call-2"
+    assert second["nvc_function_response_batches"] == [
+        {"generation": 1, "call_id": "call-1", "token_ids": [31, 32, 33]},
+        {"generation": 2, "call_id": "call-2", "token_ids": [31, 32, 33]},
+    ]
+
+    session = {"function_response_generation": 0, "forced_function_tokens": []}
+    NemotronVoiceChatThinkerForConditionalGeneration._sync_forced_function_response(session, second)
+
+    assert session["function_response_generation"] == 2
+    assert session["forced_function_token"] == 31
+    assert session["forced_function_tokens"] == [31, 32, 33, 31, 32, 33]

tests/e2e/features/fullduplex/personaplex/test_unified_runtime.py

@@ -18,6 +18,10 @@ from vllm_omni.experimental.fullduplex.engine.duplex_runtime import (
     DuplexInputMode,
 )
 from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
+from vllm_omni.experimental.fullduplex.openai.runtime_adapter import (
+    ServingRuntimeConfigError,
+)
+from vllm_omni.experimental.fullduplex.personaplex.config import DEFAULT_PERSONA
 from vllm_omni.experimental.fullduplex.personaplex.data_plane import (
     PersonaPlexDataPlaneContext,
     PersonaPlexDataPlaneSession,
@@ -242,6 +246,53 @@ def test_personaplex_capabilities_are_honest() -> None:
     assert multi.supports_barge_in is False
 
 
+def test_personaplex_runtime_config_update_rejects_changed_persona() -> None:
+    adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
+    short_persona = "You are a helpful assistant."
+    longer_persona = (
+        "You are now a swashbuckling pirate captain who speaks entirely in "
+        "nautical slang, references buried treasure constantly, and never "
+        "breaks character no matter what the user asks."
+    )
+    current = {"personaplex_persona": short_persona, "personaplex_voice_prompt": "NATF2.pt"}
+
+    with pytest.raises(ServingRuntimeConfigError, match="persona"):
+        adapter.runtime_config_for_update(
+            SimpleNamespace(instructions=longer_persona, extra_body={}),
+            current,
+        )
+
+    unchanged = adapter.runtime_config_for_update(
+        SimpleNamespace(instructions=short_persona, extra_body={}),
+        current,
+    )
+    assert unchanged["personaplex_persona"] == short_persona
+
+
+def test_personaplex_runtime_config_update_rejects_changed_voice() -> None:
+    adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
+    current = {"personaplex_persona": DEFAULT_PERSONA, "personaplex_voice_prompt": "NATF2.pt"}
+
+    with pytest.raises(ServingRuntimeConfigError, match="voice"):
+        adapter.runtime_config_for_update(
+            SimpleNamespace(instructions=None, voice="OtherVoice.pt", extra_body={}),
+            current,
+        )
+
+    unchanged = adapter.runtime_config_for_update(
+        SimpleNamespace(instructions=None, voice="NATF2.pt", extra_body={}),
+        current,
+    )
+    assert unchanged["personaplex_voice_prompt"] == "NATF2.pt"
+
+
+def test_personaplex_rejects_minicpmo_only_opt_out_flag() -> None:
+    adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
+
+    with pytest.raises(ServingRuntimeConfigError, match="minicpmo45_native_duplex"):
+        adapter.validate_client_extra_body({"minicpmo45_native_duplex": False})
+
+
 def test_pcm_buffer_emits_one_80ms_frame_transactionally() -> None:
     buffer = PersonaPlexPcmAppendBuffer()
     reservation = buffer.prepare_append(

tests/e2e/offline_inference/test_flux2_klein.py

@@ -76,6 +76,7 @@ def _send_inpaint_with_generator(
 
 @pytest.mark.slow
 @pytest.mark.diffusion
+@pytest.mark.advanced_model
 @hardware_test(res={"cuda": "L4"}, num_cards=1)
 def test_flux2_klein_inpaint_deterministic(offline_client: OfflineOmniClient):
     input_image, mask_image = _create_test_inputs()
@@ -97,6 +98,7 @@ def test_flux2_klein_inpaint_deterministic(offline_client: OfflineOmniClient):
 
 @pytest.mark.slow
 @pytest.mark.diffusion
+@pytest.mark.advanced_model
 @hardware_test(res={"cuda": "L4"}, num_cards=1)
 def test_flux2_klein_inpaint_different_seeds_different_output(offline_client: OfflineOmniClient):
     input_image, mask_image = _create_test_inputs()

tests/e2e/online_serving/nemotron_voicechat_realtime_duplex.py

@@ -0,0 +1,364 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+import argparse
+import asyncio
+import base64
+import json
+import math
+import uuid
+import wave
+from collections.abc import Sequence
+from contextlib import asynccontextmanager, suppress
+from pathlib import Path
+from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
+
+import numpy as np
+from scipy.signal import resample_poly
+
+from vllm_omni.experimental.fullduplex.client import RealtimeDuplexClient, wait_for, write_pcm16_wav
+
+INPUT_SAMPLE_RATE_HZ = 16_000
+OUTPUT_SAMPLE_RATE_HZ = 22_050
+FRAME_SAMPLES = 1_280
+FRAME_PERIOD_S = FRAME_SAMPLES / INPUT_SAMPLE_RATE_HZ
+DEFAULT_FUNCTION_TOOLS = [
+    {
+        "type": "function",
+        "name": "generate_random_number",
+        "description": "Generate a random integer between min and max (inclusive).",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "min": {"type": "integer", "description": "Minimum value (inclusive)"},
+                "max": {"type": "integer", "description": "Maximum value (inclusive)"},
+            },
+            "required": ["min", "max"],
+        },
+    }
+]
+DEFAULT_INSTRUCTIONS = "You are NVIDIA Voice Chat. Answer briefly. Start by greeting the user."
+DEFAULT_FUNCTION_INSTRUCTIONS = (
+    "You are NVIDIA Voice Chat. If the user's request matches an available tool, "
+    "you MUST call that tool instead of answering from your own knowledge. "
+    "Use only argument values spoken by the user and never invent missing values."
+)
+
+
+def _url(base_url: str, model: str, session_id: str) -> str:
+    parts = urlsplit(base_url)
+    query = dict(parse_qsl(parts.query, keep_blank_values=True))
+    query.update(duplex="1", model=model, autostart="0", session_id=session_id)
+    return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment))
+
+
+def _read_wav(path: Path, *, input_channel: int = 0) -> np.ndarray:
+    with wave.open(str(path), "rb") as wav_file:
+        channels = wav_file.getnchannels()
+        width = wav_file.getsampwidth()
+        source_rate = wav_file.getframerate()
+        raw = wav_file.readframes(wav_file.getnframes())
+    dtypes = {1: np.uint8, 2: np.dtype("<i2"), 4: np.dtype("<i4")}
+    if width not in dtypes:
+        raise ValueError(f"unsupported input sample width: {width}")
+    pcm = np.frombuffer(raw, dtype=dtypes[width]).astype(np.float32)
+    pcm = (pcm - 128.0) / 128.0 if width == 1 else pcm / float(1 << (width * 8 - 1))
+    if not 0 <= input_channel < channels:
+        raise ValueError(f"input channel {input_channel} is outside WAV channel count {channels}")
+    if channels > 1:
+        pcm = pcm.reshape(-1, channels)[:, input_channel]
+    if source_rate != INPUT_SAMPLE_RATE_HZ:
+        divisor = math.gcd(source_rate, INPUT_SAMPLE_RATE_HZ)
+        pcm = resample_poly(pcm, up=INPUT_SAMPLE_RATE_HZ // divisor, down=source_rate // divisor)
+    return np.ascontiguousarray(pcm, dtype="<f4")
+
+
+async def _stream(client: RealtimeDuplexClient, pcm: np.ndarray, *, max_frames: int | None, realtime: bool) -> int:
+    count = math.ceil(pcm.size / FRAME_SAMPLES)
+    if max_frames is not None:
+        count = min(count, max_frames)
+    for seq in range(count):
+        frame = pcm[seq * FRAME_SAMPLES : (seq + 1) * FRAME_SAMPLES]
+        frame = np.pad(frame, (0, FRAME_SAMPLES - frame.size)).astype("<f4")
+        await client.send(
+            {
+                "type": "input_audio_buffer.append",
+                "audio": base64.b64encode(frame).decode("ascii"),
+                "format": "pcm_f32le",
+                "sample_rate_hz": INPUT_SAMPLE_RATE_HZ,
+                "duration_ms": 80,
+                "audio_end_ms": (seq + 1) * 80,
+            }
+        )
+        if realtime:
+            await asyncio.sleep(FRAME_PERIOD_S)
+    return count
+
+
+def _events(client: RealtimeDuplexClient, event_type: str) -> list[dict[str, object]]:
+    return [event for event in client.events.events if event.get("type") == event_type]
+
+
+async def _return_function_output_when_ready(
+    client: RealtimeDuplexClient,
+    *,
+    output: str,
+    timeout_s: float,
+) -> tuple[str, int]:
+    await wait_for(
+        lambda: bool(client.events.errors()) or client.events.count("response.function_call_arguments.done") > 0,
+        timeout_s=timeout_s,
+        label="function call to execute",
+    )
+    if client.events.errors():
+        raise AssertionError(f"function call failed before tool execution: {client.events.errors()}")
+    function_done = _events(client, "response.function_call_arguments.done")[-1]
+    call_id = function_done.get("call_id")
+    if not isinstance(call_id, str) or not call_id:
+        raise AssertionError(f"completed function call has no call_id: {function_done}")
+    event_count_before_output = len(client.events.events)
+    await client.send(
+        {
+            "type": "conversation.item.create",
+            "item": {
+                "type": "function_call_output",
+                "call_id": call_id,
+                "output": output,
+            },
+        }
+    )
+    return call_id, event_count_before_output
+
+
+def _write_events(path: Path, client: RealtimeDuplexClient) -> None:
+    path.write_text(
+        "".join(json.dumps(event, ensure_ascii=False) + "\n" for event in client.events.events),
+        encoding="utf-8",
+    )
+
+
+@asynccontextmanager
+async def _managed_client(client: RealtimeDuplexClient, *, timeout_s: float):
+    async with client:
+        try:
+            yield
+        finally:
+            if client.events.count("session.created") and not client.events.count("session.closed"):
+                with suppress(Exception):
+                    await client.close_session(timeout_s=min(timeout_s, 30.0))
+
+
+async def run(args: argparse.Namespace) -> dict[str, object]:
+    output_dir = Path(args.output_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+    if args.expect_function_call and args.instructions == DEFAULT_INSTRUCTIONS:
+        instructions = DEFAULT_FUNCTION_INSTRUCTIONS
+    else:
+        instructions = args.instructions
+    tools = DEFAULT_FUNCTION_TOOLS if args.expect_function_call else None
+    session_id = f"nemotron-voicechat-{uuid.uuid4().hex}"
+    client = RealtimeDuplexClient(_url(args.url, args.model, session_id))
+    async with _managed_client(client, timeout_s=args.timeout_s):
+        session_payload: dict[str, object] = {
+            "session_id": session_id,
+            "model": args.model,
+            "modalities": ["audio", "text"],
+            "input_audio_format": "pcm_f32le",
+            "output_audio_format": "pcm16",
+            "instructions": instructions,
+            "idle_timeout_s": args.timeout_s,
+            "turn_detection": None,
+            "extra_body": {"auto_response": True},
+        }
+        if tools is not None:
+            session_payload["tools"] = tools
+        await client.send({"type": "session.update", "session": session_payload})
+        await wait_for(
+            lambda: client.events.count("session.created") > 0 or bool(client.events.errors()),
+            timeout_s=args.timeout_s,
+            label="session.created",
+        )
+        if client.events.errors():
+            raise AssertionError(f"session setup failed: {client.events.errors()}")
+        created = _events(client, "session.created")[-1]
+        session = created.get("session")
+        capabilities = session.get("capabilities") if isinstance(session, dict) else None
+        expected = dict(
+            implementation_level="model_native_duplex",
+            chunk_period_ms=80,
+            supports_core_resumable_request=True,
+            supports_core_kv_lease=False,
+            supports_multi_session=False,
+        )
+        if not isinstance(capabilities, dict) or any(capabilities.get(key) != value for key, value in expected.items()):
+            raise AssertionError(f"unexpected capabilities: {capabilities}")
+
+        function_output_task = (
+            asyncio.create_task(
+                _return_function_output_when_ready(
+                    client,
+                    output=args.function_output,
+                    timeout_s=args.timeout_s,
+                )
+            )
+            if args.function_output is not None
+            else None
+        )
+        pcm = _read_wav(Path(args.input_wav), input_channel=args.input_channel)
+        frame_count = await _stream(client, pcm, max_frames=args.max_frames, realtime=not args.no_realtime)
+        completed_responses_at_commit = client.events.count("response.done")
+        await client.send({"type": "input_audio_buffer.commit", "final": True})
+        await wait_for(
+            lambda: (
+                bool(client.events.errors())
+                or (
+                    client.events.count("response.function_call_arguments.done") > 0
+                    if args.expect_function_call
+                    else (
+                        client.events.count("response.audio.delta") >= args.minimum_audio_chunks
+                        and (client.events.count("response.done") > completed_responses_at_commit)
+                    )
+                )
+            ),
+            timeout_s=args.timeout_s,
+            label="model output",
+        )
+        await asyncio.sleep(args.drain_s)
+        if client.events.errors():
+            raise AssertionError(f"Realtime session emitted errors: {client.events.errors()}")
+        done_events = _events(client, "response.done")
+        if not args.expect_function_call:
+            response = done_events[-1].get("response") if done_events else None
+            status = response.get("status") if isinstance(response, dict) else None
+            if status != "completed":
+                raise AssertionError(f"response did not complete successfully: {done_events[-1:]}")
+
+        function_events = [
+            event for event in client.events.events if str(event.get("type", "")).startswith("response.function_call")
+        ]
+        function_items = [
+            event
+            for event in _events(client, "response.output_item.done")
+            if isinstance(event.get("item"), dict) and event["item"].get("type") == "function_call"
+        ]
+        if args.expect_function_call and not any(
+            event.get("type") == "response.function_call_arguments.done" for event in function_events
+        ):
+            raise AssertionError(f"no completed function call: {function_events}")
+        if args.expect_function_call:
+            matching_items = [
+                event for event in function_items if event["item"].get("name") == args.expected_function_name
+            ]
+            if not matching_items:
+                raise AssertionError(f"expected {args.expected_function_name!r}, got {function_items}")
+            function_item = matching_items[-1]["item"]
+            try:
+                function_arguments = json.loads(str(function_item.get("arguments", "")))
+            except json.JSONDecodeError as exc:
+                raise AssertionError(f"function arguments are not JSON: {function_item}") from exc
+            if args.expected_function_arguments is not None:
+                expected_arguments = json.loads(args.expected_function_arguments)
+                if function_arguments != expected_arguments:
+                    raise AssertionError(
+                        f"function arguments differ: expected={expected_arguments!r}, actual={function_arguments!r}"
+                    )
+
+            if args.function_output is not None:
+                call_id = function_item.get("call_id")
+                if not isinstance(call_id, str) or not call_id:
+                    raise AssertionError(f"function item has no call_id: {function_item}")
+                assert function_output_task is not None
+                returned_call_id, event_count_before_output = await function_output_task
+                assert returned_call_id == call_id
+
+                def tool_result_completed() -> bool:
+                    later = client.events.events[event_count_before_output:]
+                    transcript = "".join(
+                        str(event.get("delta", ""))
+                        for event in later
+                        if event.get("type") == "response.audio_transcript.delta"
+                    ).lower()
+                    return (
+                        any(event.get("type") == "response.audio.delta" for event in later)
+                        and any(event.get("type") == "response.done" for event in later)
+                        and (args.expected_post_tool_text is None or args.expected_post_tool_text.lower() in transcript)
+                    )
+
+                await wait_for(
+                    lambda: bool(client.events.errors()) or tool_result_completed(),
+                    timeout_s=args.timeout_s,
+                    label="completed response after function output",
+                )
+                if client.events.errors():
+                    raise AssertionError(f"function output failed: {client.events.errors()}")
+                await asyncio.sleep(args.drain_s)
+
+        audio = client.events.audio_bytes()
+        audio_events = _events(client, "response.audio.delta")
+        rates = {event.get("sample_rate_hz") for event in audio_events}
+        if not args.expect_function_call and audio and rates != {OUTPUT_SAMPLE_RATE_HZ}:
+            raise AssertionError(f"unexpected output sample rates: {rates}")
+        if not args.expect_function_call and args.minimum_audio_chunks and not audio:
+            raise AssertionError("model produced no audio")
+        expected_bytes = 2 * OUTPUT_SAMPLE_RATE_HZ * expected["chunk_period_ms"] // 1000
+        packet_sizes = [len(base64.b64decode(str(event.get("delta", "")), validate=True)) for event in audio_events]
+        if not args.expect_function_call and any(size != expected_bytes for size in packet_sizes):
+            raise AssertionError(f"audio deltas are not fixed 80 ms PCM16 packets: {packet_sizes}")
+        audio_pcm = np.frombuffer(audio, dtype="<i2").astype(np.float32) / 32768.0
+        audio_rms = float(np.sqrt(np.mean(np.square(audio_pcm)))) if audio_pcm.size else 0.0
+        if not args.expect_function_call and audio and audio_rms < args.minimum_audio_rms:
+            raise AssertionError(
+                f"model output RMS {audio_rms:.6f} is below {args.minimum_audio_rms:.6f}; "
+                "received packets contain only silence"
+            )
+    _write_events(output_dir / "events.jsonl", client)
+    if audio:
+        write_pcm16_wav(output_dir / "output.wav", audio, sample_rate_hz=OUTPUT_SAMPLE_RATE_HZ)
+    result = {
+        "ok": True,
+        "session_id": session_id,
+        "input_frames": frame_count,
+        "capabilities": capabilities,
+        "event_counts": {
+            event_type: client.events.count(event_type)
+            for event_type in sorted({str(event.get("type")) for event in client.events.events})
+        },
+        "audio_bytes": len(audio),
+        "audio_rms": audio_rms,
+        "function_events": function_events,
+        "function_items": function_items,
+        "output_dir": str(output_dir),
+    }
+
+    (output_dir / "result.json").write_text(json.dumps(result, ensure_ascii=False, indent=2), encoding="utf-8")
+    return result
+
+
+def parse_args(argv: Sequence[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--url", default="ws://127.0.0.1:8125/v1/realtime")
+    parser.add_argument("--model", required=True)
+    parser.add_argument("--input-wav", required=True)
+    parser.add_argument("--input-channel", type=int, default=0)
+    parser.add_argument("--output-dir", default="/tmp/nemotron-voicechat-duplex")
+    parser.add_argument("--instructions", default=DEFAULT_INSTRUCTIONS)
+    parser.add_argument("--max-frames", type=int)
+    parser.add_argument("--minimum-audio-chunks", type=int, default=1)
+    parser.add_argument("--minimum-audio-rms", type=float, default=1e-4)
+    parser.add_argument("--expect-function-call", action="store_true")
+    parser.add_argument("--expected-function-name", default="generate_random_number")
+    parser.add_argument("--expected-function-arguments")
+    parser.add_argument("--function-output")
+    parser.add_argument("--expected-post-tool-text")
+    parser.add_argument("--no-realtime", action="store_true")
+    parser.add_argument("--drain-s", type=float, default=2.0)
+    parser.add_argument("--timeout-s", type=float, default=600.0)
+    return parser.parse_args(argv)
+
+
+def main() -> None:
+    print(json.dumps(asyncio.run(run(parse_args())), ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+    main()

tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py

@@ -52,9 +52,8 @@ def get_prompt(prompt_type="zh"):
 
 def _set_rocm_request_timeout(request_config: dict) -> None:
     if current_omni_platform.is_rocm():
-        # A cold AITER MHA kernel build took 301 seconds on MI300X.
-        # Allow 600 seconds for the first request; later requests reuse the cache.
-        request_config["timeout"] = 600.0
+        # Cold AITER MHA builds on loaded MI300 agents can exceed 20 minutes.
+        request_config["timeout"] = 1800.0
 
 
 tts_server_params = [

tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py

@@ -0,0 +1,128 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""MiniMax-H3 DLO + DP2 online-serving smoke test."""
+
+from __future__ import annotations
+
+import concurrent.futures
+import io
+import json
+import os
+
+import av
+import pytest
+import requests
+
+from tests.helpers.assertions import assert_video_valid
+from tests.helpers.mark import hardware_marks
+from tests.helpers.runtime import OmniServer, OmniServerParams, OpenAIClientHandler
+
+os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
+
+MODEL = os.environ.get("VLLM_TEST_MINIMAX_H3_MODEL", "MiniMaxAI/MiniMax-H3")
+WIDTH = 1344
+HEIGHT = 768
+FPS = 24
+NUM_INFERENCE_STEPS = 4
+REQUEST_TIMEOUT_SECONDS = 1800
+H100_TWO_CARD_MARKS = hardware_marks(res={"cuda": "H100"}, num_cards=2)
+
+SERVER_ARGS = [
+    "--trust-remote-code",
+    "--task-type",
+    "fl2va",
+    "--num-gpus",
+    "2",
+    "--tensor-parallel-size",
+    "1",
+    "--data-parallel-size",
+    "2",
+    "--request-batch-max-wait-ms",
+    "500",
+    "--usp",
+    "1",
+    "--ring",
+    "1",
+    "--text-encoder-tp-size",
+    "1",
+    "--vae-patch-parallel-size",
+    "1",
+    "--vae-parallel-mode",
+    "tile",
+    "--vae-use-tiling",
+    "--enable-distributed-layerwise-offload",
+]
+
+
+def _assert_audio_stream_present(video: bytes) -> None:
+    """Assert that the generated MP4 contains decodable audio samples."""
+    with av.open(io.BytesIO(video)) as container:
+        audio_streams = [stream for stream in container.streams if stream.type == "audio"]
+        assert audio_streams, "MiniMax-H3 MP4 has no audio stream"
+        audio_frame = next(container.decode(audio=0), None)
+        assert audio_frame is not None and audio_frame.samples > 0, "MiniMax-H3 MP4 audio stream is empty"
+
+
+def _run_t2va_request(client: OpenAIClientHandler, seed: int) -> bytes:
+    """Submit one synchronous T2VA request and return its MP4 body."""
+    request_data = {
+        "model": MODEL,
+        "prompt": "In a snowy blue-purple forest, a traveler walks past a sleeping giant; footsteps crunch in the snow while the creature softly breathes.",
+        "width": str(WIDTH),
+        "height": str(HEIGHT),
+        "fps": str(FPS),
+        "num_inference_steps": str(NUM_INFERENCE_STEPS),
+        "flow_shift": "12",
+        "seed": str(seed),
+        "extra_params": json.dumps(
+            {
+                "task": "t2va",
+                "duration": 4.0,
+                "aspect_ratio": "16:9",
+                "audio_flow_shift": 3.0,
+            },
+            separators=(",", ":"),
+        ),
+    }
+    response = requests.post(
+        f"{client.base_url.rstrip('/')}/v1/videos/sync",
+        data=request_data,
+        headers={"Accept": "video/mp4"},
+        timeout=REQUEST_TIMEOUT_SECONDS,
+    )
+    response.raise_for_status()
+    assert response.headers.get("content-type", "").startswith("video/mp4")
+    assert response.content, "MiniMax-H3 returned an empty video body"
+    return response.content
+
+
+@pytest.mark.core_model
+@pytest.mark.advanced_model
+@pytest.mark.diffusion
+@pytest.mark.slow
+@pytest.mark.parametrize(
+    "omni_server",
+    [
+        pytest.param(
+            OmniServerParams(
+                model=MODEL,
+                server_args=SERVER_ARGS,
+                stage_init_timeout=1800,
+                init_timeout=1800,
+            ),
+            id="minimax_h3_dlo_dp2_t2va",
+            marks=H100_TWO_CARD_MARKS,
+        )
+    ],
+    indirect=True,
+)
+def test_minimax_h3_dlo_dp2_t2va(omni_server: OmniServer, openai_client: OpenAIClientHandler) -> None:
+    """Validate one complete DLO all-gather DP2 wave with two T2VA jobs."""
+    with concurrent.futures.ThreadPoolExecutor(max_workers=2) as executor:
+        futures = [executor.submit(_run_t2va_request, openai_client, seed) for seed in (1101, 1102)]
+        videos = [future.result() for future in futures]
+
+    for video in videos:
+        assert_video_valid(video, width=WIDTH, height=HEIGHT, fps=FPS)
+        _assert_audio_stream_present(video)

tests/e2e/online_serving/test_nemotron_voicechat_duplex.py

@@ -0,0 +1,56 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+from __future__ import annotations
+
+import asyncio
+import os
+from pathlib import Path
+
+import pytest
+from huggingface_hub import snapshot_download
+
+from tests.e2e.online_serving.nemotron_voicechat_realtime_duplex import parse_args, run
+from tests.helpers.mark import hardware_test
+from tests.helpers.runtime import OmniServerParams, get_model_prefix
+from tests.helpers.stage_config import get_deploy_config_path
+
+MODEL = os.environ.get("VLLM_TEST_NEMOTRON_VOICECHAT_MODEL", "nvidia/NVIDIA-NemotronLabs-VoiceChat-11B")
+DEPLOY_CONFIG = get_deploy_config_path("nemotron_labs_voicechat_duplex.yaml")
+TOKENIZER = os.environ.get("VLLM_TEST_NEMOTRON_VOICECHAT_LLM_PATH")
+pytestmark = [pytest.mark.full_model, pytest.mark.omni]
+
+
+@hardware_test(res={"cuda": "H100"}, num_cards=1)
+@pytest.mark.parametrize(
+    "omni_server",
+    [
+        pytest.param(
+            OmniServerParams(
+                model=MODEL,
+                stage_config_path=DEPLOY_CONFIG,
+                env_dict={"NEMOTRON_VOICECHAT_LLM_PATH": TOKENIZER} if TOKENIZER else None,
+            ),
+            id="native-duplex",
+        )
+    ],
+    indirect=True,
+)
+def test_native_duplex_turn_taking_streams_model_audio(omni_server, tmp_path: Path) -> None:
+    model_prefix = get_model_prefix()
+    root = Path(model_prefix) / MODEL if model_prefix else Path(MODEL)
+    if not root.is_dir():
+        root = Path(snapshot_download(MODEL, local_files_only=True))
+    args = parse_args(
+        (
+            f"--url ws://{omni_server.host}:{omni_server.port}/v1/realtime --model {omni_server.model} "
+            f"--input-wav {root / 'turn_taking.wav'} --input-channel 0 --max-frames 190 "
+            f"--minimum-audio-chunks 48 --minimum-audio-rms 0.001 --no-realtime --timeout-s 300 "
+            f"--output-dir {tmp_path / 'native_duplex'}"
+        ).split()
+    )
+    result = asyncio.run(run(args))
+    assert result["ok"] is True
+    assert result["input_frames"] == 190
+    assert result["event_counts"]["response.speak"] > 0
+    assert result["audio_bytes"] >= result["input_frames"] * 1764 * 2 // 4

tests/engine/test_async_omni_engine_stage_init.py

@@ -452,6 +452,10 @@ def test_initialize_local_llm_replica_scopes_runtime_env(monkeypatch):
         "env": {runtime_env_var: "stage-value"},
     }
     monkeypatch.setenv(runtime_env_var, "parent-value")
+    # Pin the logical->physical device mapping so the assertion below is
+    # deterministic regardless of the ambient CUDA_VISIBLE_DEVICES (which on a
+    # rebase host can be e.g. "3,4", mapping logical device 0 -> physical 3).
+    runtime._init_visible_devices_baseline = "0"
 
     captured: dict[str, str | None] = {}
 

tests/engine/test_orchestrator.py

@@ -17,6 +17,7 @@ import janus
 import pytest
 from vllm.outputs import CompletionOutput, RequestOutput
 from vllm.sampling_params import SamplingParams
+from vllm.v1.engine import EngineCoreOutput, EngineCoreOutputs, FinishReason
 from vllm.v1.engine.exceptions import EngineDeadError
 from vllm.v1.metrics.stats import IterationStats
 
@@ -106,6 +107,13 @@ class OrchestratorFixture:
     result_future: concurrent.futures.Future[None]
 
 
+@dataclass
+class FakePromptRequest:
+    request_id: str
+    prompt_token_ids: list[int]
+    resumable: bool = True
+
+
 class FakeStageClient:
     def __init__(
         self,
@@ -291,6 +299,20 @@ def _engine_core_outputs(tag: str, timestamp: float) -> SimpleNamespace:
     return SimpleNamespace(outputs=[tag], timestamp=timestamp, scheduler_stats=None, finished_requests=None)
 
 
+def _terminal_engine_core_outputs(request_id: str) -> EngineCoreOutputs:
+    return EngineCoreOutputs(
+        outputs=[
+            EngineCoreOutput(
+                request_id=request_id,
+                new_token_ids=[],
+                finish_reason=FinishReason.STOP,
+            )
+        ],
+        timestamp=1.0,
+        finished_requests={request_id},
+    )
+
+
 def _build_request_output(
     request_id: str,
     *,
@@ -764,6 +786,148 @@ async def test_run_async_chunk(orchestrator_factory) -> None:
         await _shutdown_orchestrator(orchestrator_fixture)
 
 
+@pytest.mark.asyncio
+async def test_async_chunk_raw_terminal_without_processed_output_finishes_request(orchestrator_factory) -> None:
+    stage0 = FakeStageClient(stage_type="llm", final_output=False)
+    stage1 = FakeStageClient(stage_type="llm", final_output=True, final_output_type="audio")
+    orchestrator_fixture = orchestrator_factory(
+        [stage0, stage1],
+        output_processors=[FakeOutputProcessor(), FakeOutputProcessor()],
+        async_chunk=True,
+    )
+    request = FakePromptRequest(
+        request_id="req-stream-terminal",
+        prompt_token_ids=[1, 2, 3, 4],
+    )
+
+    try:
+        await _enqueue_add_request(
+            orchestrator_fixture,
+            request_id=request.request_id,
+            prompt=request,
+            original_prompt={"prompt": "stream audio"},
+            sampling_params_list=[_sampling_params(), _sampling_params()],
+            final_stage_id=1,
+        )
+
+        await _wait_for(lambda: len(stage1.add_request_calls) == 1)
+        stage1.push_engine_core_outputs(_terminal_engine_core_outputs(request.request_id))
+
+        output_msg = await _get_output_message(orchestrator_fixture)
+
+        assert output_msg.request_id == request.request_id
+        assert output_msg.stage_id == 1
+        assert output_msg.finished is True
+        assert output_msg.engine_outputs.finished is True
+        assert output_msg.engine_outputs.outputs[0].multimodal_output["audio"].numel() == 0
+        await _wait_for(lambda: request.request_id not in orchestrator_fixture.orchestrator.request_states)
+    finally:
+        await _shutdown_orchestrator(orchestrator_fixture)
+
+
+@pytest.mark.asyncio
+async def test_async_chunk_data_then_raw_terminal_finishes_once(orchestrator_factory) -> None:
+    request_id = "req-stream-data-terminal"
+    stage0 = FakeStageClient(stage_type="llm", final_output=False)
+    stage1 = FakeStageClient(stage_type="llm", final_output=True, final_output_type="audio")
+    final_data = _build_request_output(
+        request_id,
+        token_ids=[20, 21],
+        finished=False,
+        text="last audio chunk",
+    )
+    orchestrator_fixture = orchestrator_factory(
+        [stage0, stage1],
+        output_processors=[
+            FakeOutputProcessor(),
+            FakeOutputProcessor(request_outputs=[final_data]),
+        ],
+        async_chunk=True,
+    )
+    request = FakePromptRequest(
+        request_id=request_id,
+        prompt_token_ids=[1, 2, 3, 4],
+    )
+
+    try:
+        await _enqueue_add_request(
+            orchestrator_fixture,
+            request_id=request_id,
+            prompt=request,
+            original_prompt={"prompt": "stream audio"},
+            sampling_params_list=[_sampling_params(), _sampling_params()],
+            final_stage_id=1,
+        )
+
+        await _wait_for(lambda: len(stage1.add_request_calls) == 1)
+        stage1.push_engine_core_outputs(_terminal_engine_core_outputs(request_id))
+
+        data_msg = await _get_output_message(orchestrator_fixture)
+        terminal_msg = await _get_output_message(orchestrator_fixture)
+
+        assert data_msg.engine_outputs is final_data
+        assert data_msg.finished is False
+        assert terminal_msg.request_id == request_id
+        assert terminal_msg.stage_id == 1
+        assert terminal_msg.finished is True
+        assert terminal_msg.engine_outputs.finished is True
+        await _wait_for(lambda: request_id not in orchestrator_fixture.orchestrator.request_states)
+        await asyncio.sleep(0.05)
+        with pytest.raises(queue.Empty):
+            orchestrator_fixture.output_sync_q.get_nowait()
+    finally:
+        await _shutdown_orchestrator(orchestrator_fixture)
+
+
+@pytest.mark.asyncio
+async def test_async_chunk_processed_terminal_and_raw_terminal_finishes_once(orchestrator_factory) -> None:
+    request_id = "req-stream-processed-terminal"
+    stage0 = FakeStageClient(stage_type="llm", final_output=False)
+    stage1 = FakeStageClient(stage_type="llm", final_output=True, final_output_type="audio")
+    processed_terminal = _build_request_output(
+        request_id,
+        token_ids=[20, 21],
+        finished=True,
+        text="last audio chunk",
+    )
+    orchestrator_fixture = orchestrator_factory(
+        [stage0, stage1],
+        output_processors=[
+            FakeOutputProcessor(),
+            FakeOutputProcessor(request_outputs=[processed_terminal]),
+        ],
+        async_chunk=True,
+    )
+    request = FakePromptRequest(
+        request_id=request_id,
+        prompt_token_ids=[1, 2, 3, 4],
+    )
+
+    try:
+        await _enqueue_add_request(
+            orchestrator_fixture,
+            request_id=request_id,
+            prompt=request,
+            original_prompt={"prompt": "stream audio"},
+            sampling_params_list=[_sampling_params(), _sampling_params()],
+            final_stage_id=1,
+        )
+
+        await _wait_for(lambda: len(stage1.add_request_calls) == 1)
+        stage1.push_engine_core_outputs(_terminal_engine_core_outputs(request_id))
+
+        terminal_msg = await _get_output_message(orchestrator_fixture)
+
+        assert terminal_msg.engine_outputs is processed_terminal
+        assert terminal_msg.finished is True
+        await _wait_for(lambda: request_id not in orchestrator_fixture.orchestrator.request_states)
+        await asyncio.sleep(0.05)
+        with pytest.raises(queue.Empty):
+            orchestrator_fixture.output_sync_q.get_nowait()
+    finally:
+        await _shutdown_orchestrator(orchestrator_fixture)
+
+
 @pytest.mark.asyncio
 async def test_run_shutdown(orchestrator_factory) -> None:
     stages = [

tests/engine/test_orchestrator_event_driven.py

@@ -0,0 +1,247 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Event-driven orchestration loop (``VLLM_OMNI_EVENT_DRIVEN_ORCH=1``) tests.
+
+Parity suite: re-runs the legacy orchestration scenarios from
+``test_orchestrator.py`` / ``test_orchestrator_error_handling.py`` with the
+event-driven loop selected, so both loops are held to the same behavior. Plus
+event-driven-specific coverage: reader reconcile on client swap, the blocking
+final-output drain, and flag parsing.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from types import SimpleNamespace
+
+import janus
+import pytest
+
+from vllm_omni.engine.async_omni_engine import AsyncOmniEngine
+from vllm_omni.engine.messages import ShutdownRequestMessage
+from vllm_omni.engine.orchestrator import _event_driven_orch_enabled
+
+from . import test_orchestrator as legacy
+from . import test_orchestrator_error_handling as legacy_errors
+from .test_orchestrator import (
+    FakeOutputProcessor,
+    FakeStageClient,
+    OrchestratorFixture,
+    _build_harness,
+    _build_request_output,
+    _engine_core_outputs,
+    _enqueue_add_request,
+    _get_output_message,
+    _sampling_params,
+    _shutdown_orchestrator,
+    _wait_for,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+@pytest.fixture
+def orchestrator_factory(monkeypatch):
+    """Flag-setting clone of the legacy harness fixture.
+
+    Sets ``VLLM_OMNI_EVENT_DRIVEN_ORCH=1`` before any Orchestrator is
+    constructed and asserts the flag actually took effect, so the parity tests
+    cannot silently exercise the legacy poll loop.
+    """
+    monkeypatch.setenv("VLLM_OMNI_EVENT_DRIVEN_ORCH", "1")
+    fixtures: list[OrchestratorFixture] = []
+
+    def _factory(*args, **kwargs) -> OrchestratorFixture:
+        fixture = _build_harness(*args, **kwargs)
+        assert fixture.orchestrator._event_driven_orch is True
+        fixtures.append(fixture)
+        return fixture
+
+    yield _factory
+
+    for fixture in fixtures:
+        if fixture.thread.is_alive():
+            fixture.request_sync_q.put_nowait(ShutdownRequestMessage())
+            fixture.thread.join(timeout=5)
+        for q in fixture.queues:
+            q.close()
+
+
+# ---------------------------------------------------------------------------
+# Parity: the legacy scenario matrix, re-run through the event-driven loop
+# ---------------------------------------------------------------------------
+
+_PARITY_TESTS = [
+    legacy.test_run_two_stage_llm,
+    legacy.test_run_single_stage_diffusion,
+    legacy.test_run_single_stage_diffusion_streaming_forwards_intermediate_chunks,
+    legacy.test_run_llm_to_diffusion,
+    legacy.test_run_async_chunk,
+    legacy.test_run_shutdown,
+    legacy.test_run_abort,
+    legacy.test_multi_replica_round_robin_distribution,
+    legacy.test_multi_replica_abort_broadcasts_to_all_replicas,
+    legacy.test_multi_replica_shutdown_all_replicas,
+    legacy.test_multi_replica_cfg_companion_inherits_parent_affinity,
+    # Stats plumbing. The scheduler-stats cases matter most here: a batch with
+    # no request outputs still carries SchedulerStats on throttled ticks, and a
+    # reader that drops every output-less batch silently stops reporting
+    # KV/queue gauges under the event-driven loop.
+    legacy.test_orchestrator_records_iteration_stats_without_scheduler_stats,
+    legacy.test_orchestrator_records_scheduler_stats_without_outputs,
+    legacy.test_orchestrator_does_not_build_iteration_stats_for_finished_only_batch,
+    legacy.test_orchestrator_does_not_build_iteration_stats_without_stat_logger,
+    # Per-replica fault isolation (#4285): a dead replica must be evicted and
+    # the server kept up, on both the LLM reader path and the diffusion poller.
+    legacy_errors.test_engine_dead_error_evicts_replica_and_keeps_running,
+    legacy_errors.test_engine_dead_error_fails_only_dead_replica_requests,
+    legacy_errors.test_forward_to_dead_downstream_stage_fails_request_not_server,
+    legacy_errors.test_add_request_to_dead_stage_fails_request_not_server,
+    legacy_errors.test_diffusion_replica_death_on_poll_keeps_server,
+    legacy_errors.test_diffusion_error_output_routed_as_finished,
+    legacy_errors.test_diffusion_client_error_output_propagates_status_code,
+]
+
+
+@pytest.mark.asyncio
+@pytest.mark.parametrize("legacy_test", _PARITY_TESTS, ids=lambda f: f.__name__)
+async def test_event_driven_parity(legacy_test, orchestrator_factory) -> None:
+    await legacy_test(orchestrator_factory)
+
+
+# ---------------------------------------------------------------------------
+# Event-driven-specific behavior
+# ---------------------------------------------------------------------------
+
+
+def test_flag_parsing(monkeypatch) -> None:
+    monkeypatch.delenv("VLLM_OMNI_EVENT_DRIVEN_ORCH", raising=False)
+    assert _event_driven_orch_enabled() is False
+    for value in ("1", "true", "True", "YES", "on"):
+        monkeypatch.setenv("VLLM_OMNI_EVENT_DRIVEN_ORCH", value)
+        assert _event_driven_orch_enabled() is True
+    for value in ("0", "false", "off", ""):
+        monkeypatch.setenv("VLLM_OMNI_EVENT_DRIVEN_ORCH", value)
+        assert _event_driven_orch_enabled() is False
+
+
+def test_default_is_legacy_loop(monkeypatch) -> None:
+    """Without the env flag, the harness runs the legacy poll loop."""
+    monkeypatch.delenv("VLLM_OMNI_EVENT_DRIVEN_ORCH", raising=False)
+    fixture = _build_harness([FakeStageClient(stage_type="llm", final_output=True)])
+    try:
+        assert fixture.orchestrator._event_driven_orch is False
+    finally:
+        fixture.request_sync_q.put_nowait(ShutdownRequestMessage())
+        fixture.thread.join(timeout=5)
+        for q in fixture.queues:
+            q.close()
+
+
+@pytest.mark.asyncio
+async def test_reader_reconcile_picks_up_swapped_client(orchestrator_factory) -> None:
+    """Outputs from a replica whose client object was replaced still flow.
+
+    The event-driven loop binds one reader task per client object; the
+    periodic reconcile must respawn the reader when ``pool.clients[replica]``
+    is swapped (replica replacement), otherwise the new client's outputs
+    would never be drained.
+    """
+    stage0 = FakeStageClient(stage_type="llm", final_output=True)
+    processor = FakeOutputProcessor(request_outputs=[_build_request_output("req-swap", token_ids=[3], finished=True)])
+    orchestrator_fixture = orchestrator_factory([stage0], output_processors=[processor])
+    request = SimpleNamespace(request_id="req-swap", prompt_token_ids=[1, 2])
+
+    try:
+        await _enqueue_add_request(
+            orchestrator_fixture,
+            request_id="req-swap",
+            prompt=request,
+            original_prompt={"prompt": "swap"},
+            sampling_params_list=[_sampling_params()],
+            final_stage_id=0,
+        )
+        await _wait_for(lambda: len(stage0.add_request_calls) == 1)
+
+        # Swap in a fresh client for the same replica slot; keep the pool's
+        # other wiring intact. The reconcile tick (0.5 s) must respawn the
+        # reader bound to the new client object.
+        pool = orchestrator_fixture.orchestrator.stage_pools[0]
+        replacement = FakeStageClient(stage_type="llm", final_output=True)
+        replacement.stage_id = stage0.stage_id
+        replacement.replica_id = stage0.replica_id
+        pool.clients[0] = replacement
+
+        replacement.push_engine_core_outputs(_engine_core_outputs("swapped-raw", 1.0))
+
+        output_msg = await _get_output_message(orchestrator_fixture, timeout=5.0)
+        assert output_msg.request_id == "req-swap"
+        assert output_msg.finished is True
+    finally:
+        await _shutdown_orchestrator(orchestrator_fixture)
+
+
+# ---------------------------------------------------------------------------
+# Blocking final-output drain (AsyncOmniEngine.get_output_blocking_async)
+# ---------------------------------------------------------------------------
+
+
+def _drain_engine(alive: bool = True) -> AsyncOmniEngine:
+    engine = object.__new__(AsyncOmniEngine)
+    engine.output_queue = janus.Queue()
+    engine.orchestrator_thread = SimpleNamespace(is_alive=lambda: alive)
+    return engine
+
+
+def _drain_cleanup(engine: AsyncOmniEngine) -> None:
+    if engine._output_drain_executor is not None:
+        engine._output_drain_executor.shutdown(wait=False)
+        engine._output_drain_executor = None
+    engine.output_queue.close()
+
+
+@pytest.mark.asyncio
+async def test_blocking_drain_returns_queued_message() -> None:
+    engine = _drain_engine()
+    try:
+        engine.output_queue.sync_q.put_nowait("msg-1")
+        assert await engine.get_output_blocking_async(timeout=1.0) == "msg-1"
+    finally:
+        _drain_cleanup(engine)
+
+
+@pytest.mark.asyncio
+async def test_blocking_drain_wakes_on_late_message() -> None:
+    """A message put after the wait starts wakes the drain, no polling."""
+    engine = _drain_engine()
+    try:
+
+        async def _delayed_put() -> None:
+            await asyncio.sleep(0.05)
+            engine.output_queue.sync_q.put_nowait("late-msg")
+
+        put_task = asyncio.create_task(_delayed_put())
+        msg = await engine.get_output_blocking_async(timeout=5.0)
+        await put_task
+        assert msg == "late-msg"
+    finally:
+        _drain_cleanup(engine)
+
+
+@pytest.mark.asyncio
+async def test_blocking_drain_timeout_returns_none_when_alive() -> None:
+    engine = _drain_engine(alive=True)
+    try:
+        assert await engine.get_output_blocking_async(timeout=0.05) is None
+    finally:
+        _drain_cleanup(engine)
+
+
+@pytest.mark.asyncio
+async def test_blocking_drain_raises_when_orchestrator_dead() -> None:
+    engine = _drain_engine(alive=False)
+    try:
+        with pytest.raises(RuntimeError, match="Orchestrator died"):
+            await engine.get_output_blocking_async(timeout=0.05)
+    finally:
+        _drain_cleanup(engine)

tests/engine/test_output_processor.py

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 """Regression tests for OmniRequestState multimodal DELTA drain and consolidation guard."""
 
 from types import SimpleNamespace
@@ -581,18 +581,18 @@ def test_mm_only_terminal_finish_removes_request_state(monkeypatch):
 
 def test_no_detokenizer_make_request_output_with_routed_experts():
     """make_request_output accepts the routed_experts arg that the multimodal
-    output channel (_process_mm_only_outputs) passes positionally, and attaches
-    it to the completion output.
+    output channel (_process_mm_only_outputs) passes, and attaches it to the
+    completion output.
 
-    Regression: the call site passes 6 positional args
-    (..., kv_transfer_params, routed_experts) but the override previously took
-    only 5, raising TypeError on every generation-stage output.
+    routed_experts is omni-specific keyword-only: the 6th positional now
+    matches upstream's ec_transfer_params so that super().process_outputs()
+    calls do not misroute it.
     """
     s = _make_no_detok_state(RequestOutputKind.CUMULATIVE)
     s.add_multimodal_tensor(torch.randn(10), mm_type=AUDIO)
     routed_experts = np.zeros((2, 3), dtype=np.int32)
     # Mirror the exact call shape of _process_mm_only_outputs.
-    result = s.make_request_output([], None, FinishReason.STOP, None, None, routed_experts)
+    result = s.make_request_output([], None, FinishReason.STOP, None, None, routed_experts=routed_experts)
     assert result is not None
     assert not isinstance(result, PoolingRequestOutput)
     assert result.outputs[0].routed_experts is routed_experts
@@ -663,3 +663,40 @@ def test_mm_only_outputs_update_iteration_stats():
     assert finished.finish_reason == FinishReason.STOP
     assert finished.num_prompt_tokens == state.prompt_len
     assert finished.num_generation_tokens == 2
+
+
+def test_ec_transfer_params_survive_both_construction_paths():
+    """v0.28 contract: encoder-cache transfer metadata must reach
+    RequestOutput.ec_transfer_params through BOTH omni construction paths —
+    the upstream-delegating one (logprobs processor present) and the
+    no-detokenizer generation-stage one (direct RequestOutput build).
+    Regression: the override accepted the argument but dropped it."""
+    ec = {"remote_handle": "enc-cache-1"}
+
+    state = _make_state(RequestOutputKind.CUMULATIVE)
+    out = state.make_request_output([1], None, None, None, {"kv": 1}, ec)
+    assert out is not None and out.ec_transfer_params == ec
+
+    kwargs = dict(_DEFAULT_STATE_KWARGS)
+    kwargs.update(logprobs_processor=None, detokenizer=None)
+    gen_state = OmniRequestState(**kwargs, output_kind=RequestOutputKind.CUMULATIVE)
+    completion = gen_state._new_completion_output([1], None, None)
+    direct = gen_state._new_request_output("r", [completion], False, {"kv": 1}, ec)
+    assert direct.ec_transfer_params == ec
+
+
+def test_num_cache_creation_tokens_reaches_direct_request_output():
+    """v0.28 contract: prefix-cache creation usage must reach
+    RequestOutput.num_cache_creation_tokens through the no-detokenizer
+    direct build, which previously passed only num_cached_tokens."""
+    kwargs = dict(_DEFAULT_STATE_KWARGS)
+    kwargs.update(logprobs_processor=None, detokenizer=None)
+    gen_state = OmniRequestState(**kwargs, output_kind=RequestOutputKind.CUMULATIVE)
+    gen_state.num_cached_tokens = 3
+    gen_state.num_cache_creation_tokens = 2
+
+    completion = gen_state._new_completion_output([1], None, None)
+    direct = gen_state._new_request_output("r", [completion], False, None, None)
+
+    assert direct.num_cached_tokens == 3
+    assert direct.num_cache_creation_tokens == 2

tests/engine/test_single_stage_mode.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 """Unit tests for AsyncOmniEngine single-stage mode and OmniMasterServer."""
 
 from __future__ import annotations
@@ -1374,7 +1377,7 @@ class TestConnectRemoteEngineCoresCoordinator:
         omni_master_server.get_zmq_addresses.assert_called_once_with(7, replica_id=2)
         omni_master_server.get_allocation.assert_called_once_with(7, replica_id=2)
         mock_wait.assert_called_once()
-        _, _, core_engines, parallel_config, *_ = mock_wait.call_args.args
+        _, core_engines, parallel_config, *_ = mock_wait.call_args.args
         assert core_engines[0].local is False
         assert parallel_config.data_parallel_size_local == 0
 

tests/engine/test_stage_device_layout.py

@@ -0,0 +1,229 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""Regression tests for issue #5003: a per-stage world size that its assigned
+``devices`` cannot satisfy must fail early in ``build_vllm_config`` with a clear
+message, rather than surfacing as an opaque worker-side ``local rank ... out of
+bounds`` assertion.
+
+Root cause: a top-level ``--tensor-parallel-size`` is broadcast to every stage,
+but each stage's ``devices`` is not adjusted, so a stage can end up with e.g.
+tensor_parallel_size=4 while still holding a single-GPU deploy default. Without
+``--strategy-config`` the strategy-path device check never runs.
+"""
+
+import json
+import re
+import types
+from unittest import mock
+
+import pytest
+
+from vllm_omni.engine import stage_init_utils
+from vllm_omni.engine.stage_init_utils import (
+    _check_stage_device_layout,
+    build_vllm_config,
+    compute_replica_layout,
+    get_stage_devices_per_replica,
+)
+
+pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
+
+
+def _stage(stage_id, devices, num_replicas=1, engine_args=None):
+    return types.SimpleNamespace(
+        stage_id=stage_id,
+        stage_type="llm",
+        engine_args=engine_args or {},
+        runtime=types.SimpleNamespace(devices=devices, num_replicas=num_replicas),
+    )
+
+
+def test_tp_broadcast_without_devices_fails_early():
+    """stage0 gets tensor_parallel_size=4 (broadcast) but only 1 device -> clear error."""
+    stage = _stage(0, devices="0")
+    engine_args = {
+        "tensor_parallel_size": 4,
+        "data_parallel_size": 1,
+        "pipeline_parallel_size": 1,
+    }
+    with pytest.raises(ValueError) as excinfo:
+        _check_stage_device_layout(stage, engine_args)
+    msg = str(excinfo.value)
+    # Message names the stage, the mismatch, and the actionable workaround.
+    assert "Stage 0" in msg
+    assert "1 device" in msg and "4" in msg
+    assert "--stage-overrides" in msg
+
+
+def test_workaround_example_sets_tp_on_every_stage():
+    """The JSON example in the error message must not reproduce the bug: every
+    stage entry it suggests has to set ``tensor_parallel_size`` (a single-GPU
+    stage that only overrides ``devices`` would still inherit the broadcast
+    tp and crash again — the original #5003 failure mode)."""
+    stage = _stage(0, devices="0")
+    with pytest.raises(ValueError) as excinfo:
+        _check_stage_device_layout(stage, {"tensor_parallel_size": 4})
+    msg = str(excinfo.value)
+
+    match = re.search(r"\{.*\}", msg, re.DOTALL)
+    assert match, f"no JSON example found in error message: {msg}"
+    example = json.loads(match.group(0))
+
+    assert example, "example must contain at least one stage"
+    for stage_id, overrides in example.items():
+        assert "tensor_parallel_size" in overrides, (
+            f"stage {stage_id} in the suggested workaround omits "
+            f"tensor_parallel_size, which reproduces #5003: {overrides}"
+        )
+
+
+def test_consistent_tp_and_devices_pass():
+    """tensor_parallel_size=4 with 4 assigned devices is valid."""
+    stage = _stage(0, devices="0,1,2,3")
+    _check_stage_device_layout(
+        stage,
+        {"tensor_parallel_size": 4, "data_parallel_size": 1, "pipeline_parallel_size": 1},
+    )
+
+
+def test_single_gpu_stage_passes():
+    """A TP=1 stage on a single device (talker/code2wav default) is valid."""
+    stage = _stage(1, devices="1")
+    _check_stage_device_layout(stage, {"tensor_parallel_size": 1})
+
+
+def test_missing_devices_is_skipped():
+    """No explicit devices -> vLLM assigns them; nothing to validate here."""
+    stage = _stage(0, devices=None)
+    _check_stage_device_layout(stage, {"tensor_parallel_size": 4})
+
+
+def test_replica_pool_layout_passes():
+    """Pool mode: num_replicas=2 x tp=2 => 4 devices is a valid pool shape."""
+    stage = _stage(0, devices="0,1,2,3", num_replicas=2)
+    _check_stage_device_layout(stage, {"tensor_parallel_size": 2})
+
+
+def test_multinode_dp_uses_local_width_for_device_validation():
+    """One local DP engine needs one device even when global DP spans four nodes."""
+    stage = _stage(0, devices="0")
+    _check_stage_device_layout(
+        stage,
+        {
+            "tensor_parallel_size": 1,
+            "data_parallel_size": 4,
+            "data_parallel_size_local": 1,
+            "pipeline_parallel_size": 1,
+        },
+    )
+
+
+def test_zero_local_dp_skips_local_device_validation():
+    """A head process with no local engines does not own a local device layout."""
+    stage = _stage(0, devices="0")
+    _check_stage_device_layout(
+        stage,
+        {
+            "tensor_parallel_size": 4,
+            "data_parallel_size": 4,
+            "data_parallel_size_local": 0,
+            "pipeline_parallel_size": 1,
+        },
+    )
+
+
+@pytest.mark.parametrize(
+    ("devices", "expected"),
+    [
+        ("0,1", ["0,1", "2,3"]),
+        ("0,1,2,3", ["0,1", "2,3"]),
+    ],
+)
+def test_replica_guard_and_splitter_share_local_world_size(devices, expected):
+    """PP and local DP contribute to the same per-replica size in both paths."""
+    engine_args = {
+        "tensor_parallel_size": 1,
+        "data_parallel_size": 4,
+        "data_parallel_size_local": 1,
+        "pipeline_parallel_size": 2,
+    }
+    stage = _stage(0, devices=devices, num_replicas=2, engine_args=engine_args)
+
+    _check_stage_device_layout(stage, engine_args)
+    assert get_stage_devices_per_replica(stage) == 2
+    _, replica_devices_map = compute_replica_layout([stage])
+    assert replica_devices_map == {0: expected}
+
+
+def test_non_tp_layout_error_uses_generic_guidance():
+    """A PP mismatch must not be reported as a top-level TP broadcast."""
+    stage = _stage(0, devices="0")
+    with pytest.raises(ValueError) as excinfo:
+        _check_stage_device_layout(
+            stage,
+            {
+                "tensor_parallel_size": 1,
+                "data_parallel_size": 1,
+                "pipeline_parallel_size": 2,
+            },
+        )
+
+    msg = str(excinfo.value)
+    assert "local world size" in msg
+    assert "top-level --tensor-parallel-size" not in msg
+
+
+def test_build_vllm_config_fails_before_engine_config_on_mismatch():
+    """Pin the production wiring: ``build_vllm_config`` must run the device-layout
+    guard *before* ``create_engine_config``. If the guard call is dropped or moved
+    after engine-config creation, #5003 regresses while the unit tests above stay
+    green — so assert here that a mismatched stage raises and that neither
+    ``create_engine_config`` nor ``Executor.get_class`` is reached."""
+    stage = _stage(0, devices="0")
+    with (
+        mock.patch.object(stage_init_utils.OmniEngineArgs, "create_engine_config") as create_engine_config,
+        mock.patch.object(stage_init_utils.Executor, "get_class") as get_class,
+    ):
+        with pytest.raises(ValueError) as excinfo:
+            build_vllm_config(
+                stage,
+                model="dummy-model",
+                engine_args_dict={
+                    "tensor_parallel_size": 4,
+                    "data_parallel_size": 1,
+                    "pipeline_parallel_size": 1,
+                },
+            )
+    msg = str(excinfo.value)
+    assert "Stage 0" in msg
+    assert "--stage-overrides" in msg
+    # Guard fired early: worker/config construction was never reached.
+    create_engine_config.assert_not_called()
+    get_class.assert_not_called()
+
+
+def test_build_vllm_config_proceeds_on_consistent_layout():
+    """The guard must not false-positive: a consistent single-GPU stage flows past
+    it and reaches ``create_engine_config`` / ``Executor.get_class`` as usual."""
+    stage = _stage(1, devices="1")
+    fake_config = types.SimpleNamespace(
+        quant_config=None,
+        model_config=types.SimpleNamespace(hf_config=types.SimpleNamespace()),
+    )
+    sentinel_executor = object()
+    with (
+        mock.patch.object(
+            stage_init_utils.OmniEngineArgs, "create_engine_config", return_value=fake_config
+        ) as create_engine_config,
+        mock.patch.object(stage_init_utils.Executor, "get_class", return_value=sentinel_executor),
+        mock.patch.object(stage_init_utils.OmniINCConfig, "maybe_upgrade", side_effect=lambda quant: quant),
+    ):
+        vllm_config, executor_class = build_vllm_config(
+            stage,
+            model="dummy-model",
+            engine_args_dict={"tensor_parallel_size": 1},
+        )
+    create_engine_config.assert_called_once()
+    assert vllm_config is fake_config
+    assert executor_class is sentinel_executor

tests/entrypoints/openai_api/test_duplex_handler.py

@@ -49,6 +49,9 @@ from vllm_omni.experimental.fullduplex.openai.serving import (
 )
 from vllm_omni.experimental.fullduplex.openai.websocket import DuplexWebSocketActor
 from vllm_omni.experimental.fullduplex.output import attach_duplex_output_decision
+from vllm_omni.experimental.fullduplex.personaplex.serving_adapter import (
+    PersonaPlexServingRuntimeAdapter,
+)
 from vllm_omni.outputs import OmniRequestOutput
 
 pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@@ -548,6 +551,54 @@ async def test_native_realtime_protocol_conversation_item_create_commits_user_te
     }
 
 
+@pytest.mark.asyncio
+async def test_function_output_is_recorded_only_after_runtime_delivery():
+    ws = TimedWebSocket()
+    protocol = NativeRealtimeSessionProtocol(ws)  # type: ignore[arg-type]
+    protocol.bind_sender(ws.send_json)
+    protocol.encode_outbound_event(
+        {
+            "type": "conversation.item.created",
+            "item": {
+                "id": "item-call",
+                "type": "function_call",
+                "call_id": "call-1",
+                "name": "weather",
+                "arguments": "{}",
+                "status": "completed",
+            },
+        }
+    )
+    event = {
+        "type": "conversation.item.create",
+        "item": {
+            "id": "item-output",
+            "type": "function_call_output",
+            "call_id": "call-1",
+            "output": '{"temperature":20}',
+        },
+    }
+
+    first = await protocol._to_duplex_event(event)
+    retry = await protocol._to_duplex_event(event)
+
+    assert first is not None
+    assert retry is not None
+    assert "item-output" not in protocol._conversation_items
+
+    delivered_item = first["payload"]["item"]
+    protocol.encode_outbound_event(
+        {
+            "type": "conversation.item.created",
+            "item": delivered_item,
+        }
+    )
+
+    assert protocol._conversation_items["item-output"] == delivered_item
+    assert await protocol._to_duplex_event(event) is None
+    assert ws.sent[-1]["error"]["code"] == "invalid_function_call_output"
+
+
 @pytest.mark.asyncio
 async def test_native_realtime_protocol_audio_commit_requires_non_empty_buffer():
     ws = TimedWebSocket()
@@ -991,6 +1042,178 @@ async def test_minicpmo_native_session_update_requires_ref_audio_before_enabling
     assert ("sid-runtime-update-ref-required", "session.update") not in engine.signals
 
 
+def test_personaplex_candidate_update_does_not_leak_minicpmo_ref_audio_check():
+    handler = OmniDuplexSessionHandler(
+        chat_service=SimpleNamespace(engine_client=SimpleNamespace()),
+        serving_runtime_adapter=PersonaPlexServingRuntimeAdapter(lambda *_: None),
+    )
+    session = DuplexSession(
+        session_id="sid-personaplex-voice-change",
+        config=DuplexSessionConfig(modalities=["audio"], voice="NATF2.pt"),
+        capabilities=PersonaPlexServingRuntimeAdapter.capabilities(max_sessions=1),
+    )
+    candidate_config = DuplexSessionConfig(modalities=["audio"], voice="OtherVoice.pt")
+
+    error = handler._runtime_session_candidate_update_error(session, candidate_config)
+
+    assert error is None
+
+
+@pytest.mark.asyncio
+async def test_minicpmo_native_session_update_rejects_instructions_after_native_context_locked():
+    engine = FakeEngineClient()
+    handler = OmniDuplexSessionHandler(
+        chat_service=FakeChatService(engine),
+        config_timeout_s=0.1,
+        idle_timeout_s=1,
+    )
+    ws = TimedWebSocket()
+    ws.put(_native_session_create("sid-instructions-locked", modalities=["text"]))
+    ws.put(
+        {
+            "type": "input_audio_buffer.append",
+            "audio": _pcm_f32_b64(16000, value=0.05),
+            "format": "pcm_f32le",
+            "sample_rate_hz": 16000,
+            "duration_ms": 1000,
+            "is_speech": True,
+        }
+    )
+    ws.put(
+        {
+            "type": "turn.signal",
+            "event": "session.update",
+            "payload": {"instructions": "You are now a pirate."},
+        }
+    )
+    ws.put({"type": "session.close"})
+
+    await handler.handle_session(ws)
+
+    error = next(message for message in ws.sent if message.get("type") == "error")
+    assert error["code"] == "instructions_update_unsupported"
+    assert "session.updated" not in ws.sent_types()
+    assert ("sid-instructions-locked", "session.update") not in engine.signals
+
+
+@pytest.mark.asyncio
+async def test_minicpmo_native_session_update_allows_unchanged_instructions_after_lock():
+    engine = FakeEngineClient()
+    handler = OmniDuplexSessionHandler(
+        chat_service=FakeChatService(engine),
+        config_timeout_s=0.1,
+        idle_timeout_s=1,
+    )
+    ws = TimedWebSocket()
+    ws.put(_native_session_create("sid-instructions-unchanged", modalities=["text"]))
+    ws.put(
+        {
+            "type": "input_audio_buffer.append",
+            "audio": _pcm_f32_b64(16000, value=0.05),
+            "format": "pcm_f32le",
+            "sample_rate_hz": 16000,
+            "duration_ms": 1000,
+            "is_speech": True,
+        }
+    )
+    ws.put(
+        {
+            "type": "turn.signal",
+            "event": "session.update",
+            "payload": {"instructions": "You are a concise assistant.", "temperature": 0.3},
+        }
+    )
+    ws.put({"type": "session.close"})
+
+    await handler.handle_session(ws)
+
+    assert "session.updated" in ws.sent_types()
+    assert not any(
+        message.get("type") == "error" and message.get("code") == "instructions_update_unsupported"
+        for message in ws.sent
+    )
+
+
+@pytest.mark.asyncio
+async def test_minicpmo_native_session_update_rejects_instructions_before_first_real_append():
+    engine = FakeEngineClient()
+    handler = OmniDuplexSessionHandler(
+        chat_service=FakeChatService(engine),
+        config_timeout_s=0.1,
+        idle_timeout_s=1,
+    )
+    ws = TimedWebSocket()
+    ws.put(_native_session_create("sid-buffered-before-lock", modalities=["text"]))
+    ws.put(
+        {
+            "type": "input_audio_buffer.append",
+            "audio": _pcm_f32_b64(3200, value=0.05),
+            "format": "pcm_f32le",
+            "sample_rate_hz": 16000,
+            "duration_ms": 200,
+            "is_speech": True,
+        }
+    )
+    ws.put(
+        {
+            "type": "turn.signal",
+            "event": "session.update",
+            "payload": {
+                "instructions": (
+                    "You are now a swashbuckling pirate captain who speaks entirely in "
+                    "nautical slang, references buried treasure constantly, and never "
+                    "breaks character no matter what the user asks."
+                )
+            },
+        }
+    )
+    ws.put({"type": "input_audio_buffer.commit", "final": True, "response_create": True})
+    ws.put({"type": "session.close"})
+
+    await handler.handle_session(ws)
+
+    error = next(message for message in ws.sent if message.get("type") == "error")
+    assert error["code"] == "instructions_update_unsupported"
+    assert "session.updated" not in ws.sent_types()
+    assert engine.appended, "the commit must still flush the buffered chunk despite the rejected update"
+
+
+@pytest.mark.asyncio
+async def test_minicpmo_native_session_update_rejects_native_duplex_mode_flip():
+    engine = FakeEngineClient()
+    handler = OmniDuplexSessionHandler(
+        chat_service=FakeChatService(engine),
+        config_timeout_s=0.1,
+        idle_timeout_s=1,
+    )
+    ws = TimedWebSocket()
+    ws.put(_native_session_create("sid-native-duplex-flag-flip", modalities=["text"]))
+    ws.put(
+        {
+            "type": "turn.signal",
+            "event": "session.update",
+            "payload": {"extra_body": {"minicpmo45_native_duplex": False}},
+        }
+    )
+    ws.put(
+        {
+            "type": "turn.signal",
+            "event": "session.update",
+            "payload": {"instructions": "You are now a pirate."},
+        }
+    )
+    ws.put({"type": "session.close"})
+
+    await handler.handle_session(ws)
+
+    errors = [message for message in ws.sent if message.get("type") == "error"]
+    assert [error["code"] for error in errors] == [
+        "native_duplex_mode_update_unsupported",
+        "instructions_update_unsupported",
+    ]
+    assert "session.updated" not in ws.sent_types()
+
+
 def test_native_realtime_protocol_audio_delta_preserves_sample_rate_hz():
     ws = TimedWebSocket()
     protocol = NativeRealtimeSessionProtocol(ws)  # type: ignore[arg-type]
@@ -5306,7 +5529,7 @@ async def test_failed_native_append_prevents_queued_append_from_running():
 
 
 @pytest.mark.asyncio
-async def test_failed_final_append_keeps_committed_audio_for_response_retry():
+async def test_failed_final_append_discards_retained_audio_without_retry():
     failed_result = {
         "operation": "append",
         "session_id": "sid-final-append-retry",
@@ -5315,15 +5538,7 @@ async def test_failed_final_append_keeps_committed_audio_for_response_retry():
         "error_count": 1,
         "stage_results": [],
     }
-    successful_result = {
-        "operation": "append",
-        "session_id": "sid-final-append-retry",
-        "ok": True,
-        "unsupported_count": 0,
-        "error_count": 0,
-        "stage_results": [],
-    }
-    engine = FakeEngineClient(append_results=[failed_result, successful_result])
+    engine = FakeEngineClient(append_result=failed_result)
     handler = OmniDuplexSessionHandler(
         chat_service=FakeChatService(engine),
         config_timeout_s=0.1,
@@ -5348,15 +5563,18 @@ async def test_failed_final_append_keeps_committed_audio_for_response_retry():
 
     handler_task = asyncio.create_task(handler.handle_session(ws))
     for _ in range(100):
-        if len(engine.appended) >= 2:
+        state = handler._serving_runtime_adapter.session_states.get("sid-final-append-retry")
+        if len(engine.appended) >= 1 and state is not None and state.committed_audio_payload is None:
             break
         await asyncio.sleep(0.01)
+    state = handler._serving_runtime_adapter.session_states["sid-final-append-retry"]
+    assert state.committed_audio_payload is None
+    assert state.committed_audio_reserved_bytes == 0
+    assert handler._registry.get("sid-final-append-retry").pending_input_bytes == 0
     ws.put({"type": "session.close"})
     await asyncio.wait_for(handler_task, timeout=2)
 
-    assert len(engine.appended) == 2
-    assert engine.appended[0][2]["audio"] == engine.appended[1][2]["audio"]
-    assert engine.append_operation_ids[0] == engine.append_operation_ids[1]
+    assert len(engine.appended) == 1
 
 
 @pytest.mark.asyncio

tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 """
 E2E online tests for Qwen3-Omni /v1/realtime WebSocket (streaming PCM in, audio out).
 
@@ -38,6 +38,10 @@ MODEL = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
 REALTIME_SYNTH_PHRASE_TEXT = (
     "Translate into Chinese: Beijing is the Capital of China. It is the center of culture and politics"
 )
+ISSUE_6474_SYNTH_PHRASE_TEXT = (
+    "Can you tell me the current temperature and weather conditions in New York City? "
+    "What about Los Angeles? Please compare them in detail using at least eight complete sentences."
+)
 
 # Simulate realtime upload pacing (``openai_realtime_client.py --send-delay-ms``).
 SEND_DELAY_MS = 200
@@ -100,6 +104,7 @@ async def _run_realtime_audio_roundtrip(
     *,
     chunk_ms: int = 100,
     send_delay_ms: int = 0,
+    completion_timeout_s: float = 600,
 ) -> dict:
     uri = f"ws://{host}:{port}/v1/realtime"
     incremental: list[bytes] = []
@@ -131,7 +136,7 @@ async def _run_realtime_audio_roundtrip(
         await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True}))
 
         while True:
-            message = await asyncio.wait_for(ws.recv(), timeout=600)
+            message = await asyncio.wait_for(ws.recv(), timeout=completion_timeout_s)
             if isinstance(message, bytes):
                 continue
 
@@ -178,12 +183,16 @@ async def _run_realtime_audio_roundtrip(
     }
 
 
-def _synthetic_pcm16_input() -> bytes:
+def _synthetic_pcm16_input(
+    *,
+    phrase_text: str = REALTIME_SYNTH_PHRASE_TEXT,
+    duration_s: int = 10,
+) -> bytes:
     syn = generate_synthetic_audio(
-        10,
+        duration_s,
         1,
         sample_rate=16000,
-        phrase_text=REALTIME_SYNTH_PHRASE_TEXT,
+        phrase_text=phrase_text,
     )
     wav_bytes = base64.b64decode(syn["base64"])
     return _pcm16_mono_16k_from_wav_bytes(wav_bytes)
@@ -260,6 +269,33 @@ class TestQwen3OmniRealtimeWebSocket:
         _assert_realtime_smoke(result)
         _assert_realtime_accuracy(result)
 
+    @pytest.mark.advanced_model
+    @pytest.mark.omni
+    @hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2)
+    @pytest.mark.parametrize("omni_server", realtime_async_chunk_server_params, indirect=True)
+    def test_long_audio_response_completes_async_chunk(self, omni_server) -> None:
+        """Regression for #6474: long async-chunk audio must emit response.audio.done."""
+        issue_6474_pcm16 = _synthetic_pcm16_input(
+            phrase_text=ISSUE_6474_SYNTH_PHRASE_TEXT,
+        )
+        issue_6474_result = asyncio.run(
+            _run_realtime_audio_roundtrip(
+                omni_server.host,
+                omni_server.port,
+                omni_server.model,
+                issue_6474_pcm16,
+                chunk_ms=100,
+                send_delay_ms=SEND_DELAY_MS,
+                completion_timeout_s=180,
+            )
+        )
+
+        _assert_realtime_smoke(issue_6474_result)
+        output_duration_s = len(issue_6474_result["output_pcm"]) / (2 * issue_6474_result["output_sample_rate"])
+        assert output_duration_s > 5, (
+            f"Expected an issue-like audio response longer than 5 seconds, got {output_duration_s:.2f}s"
+        )
+
     @pytest.mark.advanced_model
     @pytest.mark.omni
     @hardware_test(res={"cuda": "H100", "rocm": "MI325"}, num_cards=2)

tests/entrypoints/test_omni_snapshot_download.py

@@ -1,11 +1,11 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 """``omni_snapshot_download`` must honor vLLM's ``VLLM_USE_MODELSCOPE`` semantics.
 
-vLLM treats the flag as enabled only for the literal string ``"true"``
-(case-insensitive). Reading ``os.environ`` directly made every non-empty value
-truthy, so an explicit opt-out such as ``VLLM_USE_MODELSCOPE=0`` still took the
-ModelScope path.
+vLLM treats the flag as enabled for the literal strings ``"1"`` or ``"true"``
+(case-insensitive; see vllm.envs). Reading ``os.environ`` directly made every
+non-empty value truthy, so an explicit opt-out such as ``VLLM_USE_MODELSCOPE=0``
+still took the ModelScope path.
 """
 
 import sys
@@ -127,7 +127,7 @@ def download_backend(monkeypatch: pytest.MonkeyPatch):
     return run
 
 
-@pytest.mark.parametrize("value", ["0", "1", "False", "false", "no", "off"])
+@pytest.mark.parametrize("value", ["0", "False", "false", "no", "off"])
 def test_non_true_values_do_not_enable_modelscope(monkeypatch, download_backend, value):
     monkeypatch.setenv("VLLM_USE_MODELSCOPE", value)
 
@@ -135,7 +135,7 @@ def test_non_true_values_do_not_enable_modelscope(monkeypatch, download_backend,
     assert download_backend() == "huggingface"
 
 
-@pytest.mark.parametrize("value", ["true", "True", "TRUE"])
+@pytest.mark.parametrize("value", ["1", "true", "True", "TRUE"])
 def test_true_values_enable_modelscope(monkeypatch, download_backend, value):
     monkeypatch.setenv("VLLM_USE_MODELSCOPE", value)
 

tests/model_executor/models/test_encoder_quant_config.py

@@ -1,12 +1,12 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 """Regression test for #2686: pre-quantized methods must not apply
 quant config to vision / audio encoders.
 
-For modelopt FP8/FP4/MXFP8 checkpoints the Thinker LM is the only
-quantized component.  Vision and audio encoder weights are BF16 with no
-FP8 scale tensors — passing quant_config to them causes FP8 kernels to
-run on BF16 weights, producing garbage embeddings.
+For serialized ModelOpt and SVDQuant checkpoints the target model component is
+the only quantized component. Vision and audio encoder weights are BF16 with no
+serialized scale or correction tensors, so the same config must not be applied
+to them.
 
 **Import note (pytest + ``tests/model_executor`` only):** :mod:`vllm_omni.quantization`
 :file:`__init__` pulls ``factory``/``inc`` and reproduces vLLM CustomOp double
@@ -58,8 +58,7 @@ pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
 @pytest.mark.parametrize("method", sorted(PRE_QUANTIZED_METHODS))
 def test_pre_quantized_returns_none(method: str) -> None:
     """visual_quant_config and audio_quant_config must be None for
-    pre-quantized methods (modelopt, modelopt_fp4, modelopt_mxfp8,
-    modelopt_mixed)."""
+    pre-quantized methods (ModelOpt variants and SVDQuant)."""
     mock_config = MagicMock()
     mock_config.get_name.return_value = method
 
@@ -101,5 +100,5 @@ def test_component_config_passed_through() -> None:
 
 def test_pre_quantized_methods_contains_expected() -> None:
     """Guard against accidental removal of a known pre-quantized method."""
-    expected = {"modelopt", "modelopt_fp4", "modelopt_mxfp8", "modelopt_mixed"}
+    expected = {"modelopt", "modelopt_fp4", "modelopt_mxfp8", "modelopt_mixed", "svdquant"}
     assert PRE_QUANTIZED_METHODS == expected

tests/model_executor/models/test_nemotron_voicechat_shapes.py

@@ -225,3 +225,39 @@ def test_torchaudio_mel_filterbank_matches_librosa() -> None:
     reference = librosa.filters.mel(sr=16000, n_fft=512, n_mels=128, fmin=0.0, fmax=8000.0, norm="slaney")
     assert ta.shape == reference.shape
     assert abs(ta - reference).max() < 1e-6
+
+
+def test_code2wav_reuses_and_clears_per_request_streaming_cache() -> None:
+    import torch
+    from torch import nn
+
+    from vllm_omni.model_executor.models.nemotron_voicechat.nemotron_voicechat_code2wav import (
+        NemotronVoiceChatCode2Wav,
+    )
+
+    class Codec(nn.Module):
+        def __init__(self) -> None:
+            super().__init__()
+            self.anchor = nn.Parameter(torch.zeros(()))
+            self.caches = []
+
+        def decode(self, codes, lengths, *, cache):
+            self.caches.append(cache)
+            wav = torch.zeros((1, int(codes.shape[1]) * 1764), device=codes.device)
+            return wav, lengths * 1764
+
+    model = NemotronVoiceChatCode2Wav.__new__(NemotronVoiceChatCode2Wav)
+    nn.Module.__init__(model)
+    model._sample_rate, model._num_quantizers, model._codebook_size, model._wav_per_frame = 22050, 31, 1024, 1764
+    model._duplex_codec_caches = {}
+    model.audio_codec = Codec()
+
+    def info(codes):
+        return [{"meta": {"codec_streaming": True, "request_id": "req"}, "codes": {"audio": codes}}]
+
+    model.forward(runtime_additional_information=info(torch.zeros((1, 31), dtype=torch.long)))
+    model.forward(runtime_additional_information=info(torch.zeros((2, 31), dtype=torch.long)))
+    assert model.audio_codec.caches[0] is model.audio_codec.caches[1]
+    assert "req" in model._duplex_codec_caches
+    model.on_requests_finished({"req"})
+    assert "req" not in model._duplex_codec_caches

tests/model_executor/stage_input_processors/test_nemotron_voicechat.py

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 
 from types import SimpleNamespace
 
@@ -285,9 +285,46 @@ def test_talker2code2wav_async_chunk_requires_prompt_len() -> None:
         talker2code2wav_async_chunk(tm, {"codes": {"audio": torch.zeros(5, 31, dtype=torch.long)}}, req)
 
 
-# =========================
-# Talker drain-per-wake (async scheduling review P1/P2)
-# =========================
+def test_talker2code2wav_duplex_keeps_cumulative_history_across_wakes() -> None:
+    from vllm_omni.model_executor.stage_input_processors.nemotron_voicechat import (
+        talker2code2wav_async_chunk,
+    )
+
+    tm = _fake_transfer_manager(codec_chunk_frames=2)
+    prompt_len = 3
+    req = _streaming_request(
+        prompt_len=prompt_len,
+        generated=[],
+        info={"meta": {"codec_streaming": torch.tensor([True])}, "nvc_logical_prompt_len": prompt_len},
+    )
+    prompt_rows = torch.zeros((prompt_len - 1, 31), dtype=torch.long)
+    first = torch.full((1, 31), 101, dtype=torch.long)
+    second = torch.full((1, 31), 202, dtype=torch.long)
+
+    for rows, finished in ((prompt_rows, False), (torch.cat([prompt_rows, first]), False)):
+        payload = talker2code2wav_async_chunk(
+            tm,
+            {"codes": {"audio": rows}, "meta": {"codec_streaming": torch.tensor([True])}},
+            req,
+            is_finished=finished,
+        )
+        assert payload is not None and not bool(payload.meta.finished)
+
+    payload = talker2code2wav_async_chunk(
+        tm,
+        {
+            "codes": {"audio": torch.cat([prompt_rows, first, second])},
+            "meta": {"codec_streaming": torch.tensor([True])},
+        },
+        req,
+        is_finished=True,
+    )
+
+    assert payload is not None
+    assert not bool(payload.meta.finished)
+    assert not bool(payload.meta.is_segment_finished)
+    assert torch.equal(payload.codes.audio, torch.cat([first, second]))
+    assert tm.request_payload["req-0"]["nvc_frames_seen"] == 2
 
 
 class _FakeTTSModelOutput:
@@ -342,7 +379,14 @@ def _bare_talker():
     return talker
 
 
-def _async_info(timeline: list[int], prompt_len: int, finished: bool, request_id: str = "req-0") -> dict:
+def _async_info(
+    timeline: list[int],
+    prompt_len: int,
+    finished: bool,
+    request_id: str = "req-0",
+    *,
+    codec_streaming: bool = False,
+) -> dict:
     return {
         "request_id": request_id,
         "additional_information": {
@@ -351,6 +395,7 @@ def _async_info(timeline: list[int], prompt_len: int, finished: bool, request_id
                 "num_processed_tokens": prompt_len,
                 "finished": torch.tensor(finished),
                 "request_id": request_id,
+                "codec_streaming": codec_streaming,
             },
         },
     }
@@ -373,7 +418,7 @@ def test_talker_drains_all_received_positions_per_wake() -> None:
     _, embeds, update = talker.preprocess(ids, None, _omni_is_prefill=False, **info)
     assert session["step"] == prompt_len  # t advanced 1 -> P in one wake
     assert update["codes"]["audio"].shape == (prompt_len - 1, 31)
-    assert float(embeds[0, 0]) == 0.0  # not finished
+    assert float(embeds[0, 0]) == 0.0  # offline stream waits for upstream terminal
 
     # Coalesced chunk: TWO new frame tokens arrive in ONE wake (delayed save
     # thread merged two thinker steps). Both must be consumed by this wake.
@@ -383,14 +428,13 @@ def test_talker_drains_all_received_positions_per_wake() -> None:
     assert update["codes"]["audio"].shape == (prompt_len + 1, 31)
     assert float(embeds[0, 0]) == 0.0
 
-    # Zero-progress wake (no new positions, not finished): plain CONTINUE, no
-    # fabricated frames, no codes update.
+    # Zero-progress wake: never fabricate frames or end the offline stream
+    # before the thinker sends its terminal marker.
     _, embeds, update = talker.preprocess(ids, None, _omni_is_prefill=False, **info2)
     assert session["step"] == prompt_len + 2
     assert update == {}
     assert float(embeds[0, 0]) == 0.0
 
-    # Final chunk: one more token + finished marker -> drain + stop flag.
     info3 = _async_info([pad] * prompt_len + [7, 8, 9], prompt_len, finished=True)
     _, embeds, update = talker.preprocess(ids, None, _omni_is_prefill=False, **info3)
     assert session["step"] == prompt_len + 3
@@ -398,6 +442,18 @@ def test_talker_drains_all_received_positions_per_wake() -> None:
     assert float(embeds[0, 0]) == 1.0  # stop
 
 
+def test_talker_native_codec_stream_ends_each_received_segment() -> None:
+    talker = _bare_talker()
+    ids = torch.zeros(1, dtype=torch.long)
+    info = _async_info([12, 12, 7], 2, finished=False, codec_streaming=True)
+    talker.preprocess(ids, None, _omni_is_prefill=True, **info)
+
+    _, embeds, update = talker.preprocess(ids, None, _omni_is_prefill=False, **info)
+
+    assert update["codes"]["audio"].shape == (2, 31)
+    assert float(embeds[0, 0]) == 1.0
+
+
 def test_talker_sync_mode_steps_once_per_wake() -> None:
     """The verified sync parity path keeps its one-TTS-step-per-engine-step cadence."""
     talker = _bare_talker()

tests/worker/test_gpu_ar_model_runner.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 import asyncio
 import copy
 import re
@@ -894,6 +897,7 @@ def test_sample_tokens_tail_only_prefix_cache_uses_staged_cpu_hidden_states(monk
         lambda *args, **kwargs: (
             0,
             None,
+            None,
             [],
             None,
             ["r1", "r2"],
@@ -1832,6 +1836,7 @@ class TestPreferModelSamplerNoneFallback:
             "hunyuan_image3",
             "minicpmo_4_5",
             "minimax_music3",
+            "nemotron_voicechat",
         }
         assert declarers == expected, (
             "The set of models declaring `prefer_model_sampler` changed:\n"

vllm_omni/config/endpoint_policy.py

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 """Endpoint restriction policy for omni pipelines."""
 
 from dataclasses import dataclass
@@ -8,7 +8,7 @@ from typing import NamedTuple
 
 from fastapi import FastAPI, Request
 from fastapi.responses import JSONResponse
-from vllm.entrypoints.serve.utils.error_response import create_error_response
+from vllm.entrypoints.serve import create_error_response
 
 
 class RouteTarget(NamedTuple):

vllm_omni/config/environment_variable_inventory.py

@@ -64,6 +64,7 @@ _PUBLIC_OMNI = (
     "SPEAKER_MAX_UPLOADED",
     "SPEAKER_SAMPLES_DIR",
     "VLLM_OMNI_ASYNC_OUTPUT_TIMEOUT",
+    "VLLM_OMNI_EVENT_DRIVEN_ORCH",
     "VLLM_OMNI_INPUT_WAIT_TIMEOUT_S",
     "VLLM_OMNI_ORCH_MONITOR_PATH",
     "VLLM_OMNI_SERVER_STORAGE__FILE_CONCURRENCY",

vllm_omni/core/sched/omni_ar_scheduler.py

@@ -344,8 +344,25 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
         inter_stage_outputs = getattr(model_runner_output, "inter_stage_outputs", None)
         num_nans_in_logits = model_runner_output.num_nans_in_logits
         kv_connector_output = model_runner_output.kv_connector_output
+        ec_connector_output = getattr(model_runner_output, "ec_connector_output", None)
         cudagraph_stats: CUDAGraphStat | None = model_runner_output.cudagraph_stats
 
+        # Every GPU write enqueued by this and earlier steps has completed, so
+        # it is safe to return deferred-free blocks to the pool. This is the
+        # update-side half of the upstream v0.28 deferred-free fence; the schedule
+        # half advances sched_step_seq inside super().schedule().
+        # getattr: __new__-constructed test schedulers carry no
+        # defer_block_free attribute.
+        if (
+            getattr(self, "defer_block_free", False)
+            # getattr again: SchedulerOutput is mocked field-by-field in the
+            # scheduler unit tests, and dataclass fields without defaults are
+            # invisible to MagicMock(spec=...).
+            and getattr(scheduler_output, "total_num_scheduled_tokens", 0) > 0
+        ):
+            self.processed_step_seq += 1
+            self._drain_deferred_frees()
+
         perf_stats: PerfStats | None = None
         if self.perf_metrics and self.perf_metrics.is_enabled():
             perf_stats = self.perf_metrics.get_step_perf_stats_per_gpu(scheduler_output)
@@ -491,6 +508,7 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
             mm_output = mm_outputs[req_index] if mm_outputs else None
             inter_stage_output = inter_stage_outputs[req_index] if inter_stage_outputs else None
             kv_transfer_params = None
+            ec_transfer_params = None
             finish_reason = None
             routed_experts = None
 
@@ -542,6 +560,17 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
                     request.resumable = False
                     stopped = True
 
+            # Finalize prefill stats BEFORE stop handling (upstream v0.28
+            # order): _free_request below releases the KV blocks, after which
+            # estimate_cached_tokens(request) reports 0. kv_transfer_params is
+            # omitted from the emission predicate here because it only becomes
+            # non-None when stopped is already True.
+            prefill_stats = None
+            if new_token_ids or mm_output is not None or pooler_output is not None or stopped:
+                prefill_stats = request.take_prefill_stats()
+                if prefill_stats is not None:
+                    prefill_stats.finalize(self.kv_cache_manager.estimate_cached_tokens(request))
+
             confirmed_num_computed_tokens = None
             if stopped:
                 if self.chunk_transfer_adapter is not None:
@@ -561,8 +590,10 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
                     if self.chunk_transfer_adapter:
                         if self.vllm_config.model_config.stage_id != 0:
                             # Downstream async-chunk stages receive real payloads from the
-                            # connector. This update only resumes polling for the next segment.
-                            self.chunk_transfer_adapter.segment_finished_requests.discard(request.request_id)
+                            # connector, not an external StreamingUpdate. Move the
+                            # resumable request back to the ordinary waiting queue so
+                            # connector polling starts again immediately.
+                            self._resume_downstream_chunk_receiver(request)
                     outstanding_async_tokens = request.num_output_placeholders
                     # Always record the discard signal (0 when nothing is in
                     # flight). Upstream a0c092ee72 removed the
@@ -595,7 +626,7 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
                     request.spec_token_ids = []
                     request._output_token_ids.clear()
                 if finished:
-                    kv_transfer_params, _ = self._free_request(request)
+                    kv_transfer_params, ec_transfer_params = self._free_request(request)
                 if status_before_stop == RequestStatus.RUNNING:
                     stopped_running_reqs.add(request)
                 elif status_before_stop == RequestStatus.WAITING_FOR_CHUNK:
@@ -623,8 +654,9 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
                     pooling_output=pooling_output_payload,
                     multimodal_output=mm_output,
                     stop_reason=request.stop_reason,
-                    prefill_stats=request.take_prefill_stats(),
+                    prefill_stats=prefill_stats,
                     kv_transfer_params=kv_transfer_params,
+                    ec_transfer_params=ec_transfer_params,
                     routed_experts=routed_experts,
                     num_nans_in_logits=request.num_nans_in_logits,
                     is_segment_finished=is_segment_finished,
@@ -641,6 +673,7 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
                     inter_stage_output,
                     request,
                     is_segment_finished,
+                    new_token_ids=new_token_ids,
                     confirmed_num_computed_tokens=confirmed_num_computed_tokens,
                 )
 
@@ -663,6 +696,11 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
         if kv_connector_output:
             self._update_from_kv_xfer_finished(kv_connector_output)
 
+        # EC Connector: update state from worker-side EC connector output.
+        # Use getattr for safety with test __new__/SimpleNamespace code paths.
+        if getattr(self, "ec_connector", None) is not None and ec_connector_output:
+            self.ec_connector.update_connector_output(ec_connector_output)
+
         kv_connector_stats = self._aggregate_kv_connector_stats(kv_connector_output)
         self._publish_kv_cache_events()
 
@@ -792,6 +830,17 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
         # 1. Standard cleanup parts from base _free_request
         connector_delay_free_blocks, kv_xfer_params = self._connector_finished(request)
 
+        # EC Connector: mirror the KV hook (upstream v0.28 _free_request).
+        # The contract requires firing before the encoder cache is freed so
+        # the connector can inspect per-request state (e.g. which mm_hashes
+        # it recorded during save_caches()) and emit ec_transfer_params for
+        # the response body. getattr: __new__-constructed test schedulers
+        # carry no ec_connector attribute.
+        ec_xfer_params: dict[str, Any] | None = None
+        if getattr(self, "ec_connector", None) is not None:
+            ec_delay_free, ec_xfer_params = self.ec_connector.request_finished(request)
+            connector_delay_free_blocks |= ec_delay_free
+
         self.encoder_cache_manager.free(request)
         request_id = request.request_id
         self.finished_req_ids.add(request_id)
@@ -817,10 +866,10 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
                         self.waiting_for_transfer_free.add(request_id)
                         self._kv_wait_start_ts[request_id] = time.monotonic()
                         kv_xfer_params = None
-                        return kv_xfer_params, None
+                        return kv_xfer_params, ec_xfer_params
                     elif request_id in self.waiting_for_transfer_free:
                         # Blocks held until KV extraction completes in a future step.
-                        return None, None
+                        return None, ec_xfer_params
                     else:
                         logger.debug(
                             f"[Omni] Request {request_id} finished and transfer no longer ACTIVE (extracted/acked). "
@@ -858,14 +907,14 @@ class OmniARScheduler(OmniSchedulerMixin, VLLMScheduler):
                         if isinstance(add_info, dict):
                             add_info.update(kv_xfer_params)
 
-                    return kv_xfer_params, None
+                    return kv_xfer_params, ec_xfer_params
 
             # 3. Standard Freeing
             delay_free_blocks |= connector_delay_free_blocks
             if not delay_free_blocks:
                 self._free_blocks(request)
 
-            return kv_xfer_params, None
+            return kv_xfer_params, ec_xfer_params
         finally:
             self._free_input_coordinator_request(request_id)
             # Normal completion runs through here, not finish_requests()

vllm_omni/core/sched/omni_generation_scheduler.py

@@ -312,6 +312,16 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
             ec_meta = self.ec_connector.build_connector_meta(scheduler_output)
             scheduler_output.ec_connector_metadata = ec_meta
 
+        # Advance the fence only for non-empty steps (those that actually
+        # write KV and have their output processed later in
+        # update_from_output). Must precede _update_after_schedule, which
+        # stamps request.last_sched_seq from the advanced value; the other
+        # half drains in update_from_output. The fallback path above gets
+        # both halves from super().schedule(). getattr: __new__-constructed
+        # test schedulers carry no defer_block_free attribute.
+        if getattr(self, "defer_block_free", False) and total_num_scheduled_tokens > 0:
+            self.sched_step_seq += 1
+
         # Update internal state (advance num_computed_tokens, free encoder inputs,
         # etc.)
         self._update_after_schedule(scheduler_output)
@@ -348,8 +358,26 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
         mm_outputs = getattr(model_runner_output, "multimodal_outputs", None)
         num_nans_in_logits = model_runner_output.num_nans_in_logits
         kv_connector_output = model_runner_output.kv_connector_output
+        ec_connector_output = getattr(model_runner_output, "ec_connector_output", None)
 
         cudagraph_stats: CUDAGraphStat | None = model_runner_output.cudagraph_stats
+
+        # Every GPU write enqueued by this and earlier steps has completed, so
+        # it is safe to return deferred-free blocks to the pool. This is the
+        # update-side half of the upstream v0.28 deferred-free fence; the schedule
+        # half advances sched_step_seq in the fast path above.
+        # getattr: __new__-constructed test schedulers carry no
+        # defer_block_free attribute.
+        if (
+            getattr(self, "defer_block_free", False)
+            # getattr again: SchedulerOutput is mocked field-by-field in the
+            # scheduler unit tests, and dataclass fields without defaults are
+            # invisible to MagicMock(spec=...).
+            and getattr(scheduler_output, "total_num_scheduled_tokens", 0) > 0
+        ):
+            self.processed_step_seq += 1
+            self._drain_deferred_frees()
+
         perf_stats: PerfStats | None = None
         if self.perf_metrics and self.perf_metrics.is_enabled():
             perf_stats = self.perf_metrics.get_step_perf_stats_per_gpu(scheduler_output)
@@ -441,6 +469,7 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
             new_logprobs = None
             new_token_ids = generated_token_ids
             kv_transfer_params = None
+            ec_transfer_params = None
             pooler_output = pooler_outputs[req_index] if pooler_outputs else None
             mm_output = mm_outputs[req_index] if mm_outputs else None
             status_before_stop = request.status
@@ -477,6 +506,17 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
                 # (does not affect protocol)
                 stopped = True
 
+            # Finalize prefill stats BEFORE stop handling (upstream v0.28
+            # order): _free_request below releases the KV blocks, after which
+            # estimate_cached_tokens(request) reports 0. kv_transfer_params is
+            # omitted from the emission predicate here because it only becomes
+            # non-None when stopped is already True.
+            prefill_stats = None
+            if new_token_ids or mm_output is not None or pooler_output is not None or stopped:
+                prefill_stats = request.take_prefill_stats()
+                if prefill_stats is not None:
+                    prefill_stats.finalize(self.kv_cache_manager.estimate_cached_tokens(request))
+
             if stopped:
                 if model_runner_output.routed_experts is not None:
                     routed_experts = omni_routed_experts_for_request(model_runner_output.routed_experts, request)
@@ -488,7 +528,7 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
                     if self.chunk_transfer_adapter:
                         self.chunk_transfer_adapter.segment_finished_requests.discard(req_id)
                 if finished:
-                    kv_transfer_params, _ = self._free_request(request)
+                    kv_transfer_params, ec_transfer_params = self._free_request(request)
                     if self.chunk_transfer_adapter is not None:
                         self.chunk_transfer_adapter.cleanup(
                             request.request_id,
@@ -530,8 +570,9 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
                     pooling_output=pooling_output_payload,
                     multimodal_output=mm_output,
                     stop_reason=request.stop_reason,
-                    prefill_stats=request.take_prefill_stats(),
+                    prefill_stats=prefill_stats,
                     kv_transfer_params=kv_transfer_params,
+                    ec_transfer_params=ec_transfer_params,
                     routed_experts=routed_experts,
                     num_nans_in_logits=request.num_nans_in_logits,
                     is_segment_finished=is_segment_finished,
@@ -550,8 +591,9 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
             finished = self._handle_stopped_request(request)
             is_segment_finished = not finished
             kv_transfer_params = None
+            ec_transfer_params = None
             if finished:
-                kv_transfer_params, _ = self._free_request(request)
+                kv_transfer_params, ec_transfer_params = self._free_request(request)
                 if self.chunk_transfer_adapter is not None:
                     self.chunk_transfer_adapter.cleanup(
                         request.request_id,
@@ -565,6 +607,7 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
                 finish_reason=finish_reason,
                 stop_reason=request.stop_reason,
                 kv_transfer_params=kv_transfer_params,
+                ec_transfer_params=ec_transfer_params,
                 is_segment_finished=is_segment_finished,
             )
             stopped_running_reqs.add(request)
@@ -590,6 +633,11 @@ class OmniGenerationScheduler(OmniSchedulerMixin, VLLMScheduler):
         if kv_connector_output:
             self._update_from_kv_xfer_finished(kv_connector_output)
 
+        # EC Connector: update state from worker-side EC connector output.
+        # Use getattr for safety with test __new__/SimpleNamespace code paths.
+        if getattr(self, "ec_connector", None) is not None and ec_connector_output:
+            self.ec_connector.update_connector_output(ec_connector_output)
+
         kv_connector_stats = self._aggregate_kv_connector_stats(kv_connector_output)
         self._publish_kv_cache_events()
 

vllm_omni/core/sched/omni_scheduler_mixin.py

@@ -472,6 +472,7 @@ class OmniSchedulerMixin:
         stop_reason: Any = None,
         prefill_stats: Any = None,
         kv_transfer_params: Any = None,
+        ec_transfer_params: Any = None,
         routed_experts: Any = None,
         num_nans_in_logits: int = 0,
         is_segment_finished: bool | None = False,
@@ -490,6 +491,7 @@ class OmniSchedulerMixin:
             events=request.take_events(),
             prefill_stats=prefill_stats,
             kv_transfer_params=kv_transfer_params,
+            ec_transfer_params=ec_transfer_params,
             trace_headers=request.trace_headers,
             routed_experts=routed_experts,
             num_nans_in_logits=num_nans_in_logits,
@@ -564,8 +566,37 @@ class OmniSchedulerMixin:
         if stopped_running_reqs:
             self.running = remove_all(self.running, stopped_running_reqs)
         if stopped_preempted_reqs:
-            self.waiting.remove_requests(stopped_preempted_reqs)
-            self.skipped_waiting.remove_requests(stopped_preempted_reqs)
+            # ``_handle_stopped_request`` re-enqueues a resumable segment as
+            # WAITING_FOR_STREAMING_REQ before this cleanup runs.  A
+            # downstream async-chunk request can have entered the update with
+            # the older WAITING_FOR_CHUNK status, which puts it in
+            # ``stopped_preempted_reqs``; deleting it here would immediately
+            # undo the requeue and the receiver would never poll chunk 1.
+            removable = {
+                request
+                for request in stopped_preempted_reqs
+                if not (
+                    request.resumable
+                    and request.status
+                    in (
+                        RequestStatus.WAITING,
+                        RequestStatus.WAITING_FOR_STREAMING_REQ,
+                    )
+                )
+            }
+            self.waiting.remove_requests(removable)
+            self.skipped_waiting.remove_requests(removable)
+
+    def _resume_downstream_chunk_receiver(self, request: Request) -> None:
+        """Resume connector polling without waiting for an external update."""
+        adapter = self.chunk_transfer_adapter
+        adapter.segment_finished_requests.discard(request.request_id)
+        if request.status != RequestStatus.WAITING_FOR_STREAMING_REQ:
+            return
+        self.num_waiting_for_streaming_input -= 1
+        request.status = RequestStatus.WAITING
+        self.skipped_waiting.remove_requests((request,))
+        self._enqueue_waiting_request(request)
 
     def _aggregate_kv_connector_stats(
         self,

vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml

@@ -0,0 +1,47 @@
+# Nemotron VoiceChat native Full-Duplex deployment.
+#
+# One scheduler segment consumes one 1280-sample (80 ms) microphone frame.
+# Stage 0 remains one resumable request, so its NemotronH KV/Mamba state is
+# preserved between frame appends. Stage 1 and the codec overlap per frame.
+base_config: nemotron_labs_voicechat_streaming.yaml
+pipeline: nemotron_voicechat
+session_mode: duplex
+active_stream_window: 1
+
+duplex_session:
+  idle_ttl_s: 300
+  disconnect_grace_s: 30
+  reaper_interval_s: 5
+  resume_replay_ttl_s: 60
+  resume_replay_max_bytes_per_session: 8388608
+  max_pending_input_bytes_per_session: 16777216
+  max_pending_turns_per_session: 4
+  max_sessions: 1
+  completed_append_cache_size: 256
+
+async_chunk: true
+
+connectors:
+  connector_of_shared_memory:
+    name: SharedMemoryConnector
+    extra:
+      shm_threshold_bytes: 65536
+      codec_chunk_frames: 1
+
+stages:
+  - stage_id: 0
+    max_num_seqs: 1
+    async_scheduling: false
+    default_sampling_params:
+      temperature: 0.0
+      max_tokens: 1
+      ignore_eos: true
+      seed: 0
+
+  - stage_id: 1
+    max_num_seqs: 1
+    async_scheduling: false
+
+  - stage_id: 2
+    max_num_seqs: 1
+    async_scheduling: false

vllm_omni/diffusion/attention/backends/flash_attn.py

@@ -93,12 +93,14 @@ class FlashAttentionImpl(AttentionImpl):
         prefix: str = "",
         qkv_layout: str | None = None,
         backend_kwargs: dict | None = None,
+        role: str = "self",
         **extra_impl_args,
     ) -> None:
         self.num_heads = num_heads
         self.causal = causal
         self.softmax_scale = softmax_scale
         self.qkv_layout = qkv_layout
+        self.is_cross_attn = role == "cross"
         cfg = get_current_diffusion_config_or_none()
         self.fa_deterministic = bool(getattr(cfg, "fa_deterministic", False)) if cfg is not None else False
         if backend_kwargs:
@@ -152,6 +154,7 @@ class FlashAttentionImpl(AttentionImpl):
         attention_mask: torch.Tensor,
     ) -> torch.Tensor:
         from vllm_omni.diffusion.attention.backends.utils.fa import (
+            _index_first_axis,
             _pad_input,
             _unpad_input,
             _upad_input,
@@ -159,10 +162,21 @@ class FlashAttentionImpl(AttentionImpl):
         )
 
         assert attention_mask.ndim == 2, "attention_mask must be 2D, (batch_size, seq_len)"
-        query_length = query.size(1)
-        q, k, v, indices_q, (cu_seq_lens_q, cu_seq_lens_k), (max_length_q, max_length_k) = _upad_input(
-            query, key, value, attention_mask, query_length, _unpad_input
-        )
+        batch_size, query_length = query.shape[:2]
+        if not self.is_cross_attn and query_length == key.size(1):
+            q, k, v, indices_q, (cu_seq_lens_q, cu_seq_lens_k), (max_length_q, max_length_k) = _upad_input(
+                query, key, value, attention_mask, query_length, _unpad_input
+            )
+        else:
+            # Cross-attention: the mask covers keys only, so keep every query row.
+            k, indices_k, cu_seq_lens_k, max_length_k, _ = _unpad_input(key, attention_mask)
+            v = _index_first_axis(value, indices_k)
+            q = query.flatten(0, 1)
+            cu_seq_lens_q = torch.arange(
+                0, (batch_size + 1) * query_length, query_length, dtype=torch.int32, device=query.device
+            )
+            max_length_q = query_length
+            indices_q = None
 
         out_unpad = flash_attn_varlen_func(
             q,
@@ -178,7 +192,9 @@ class FlashAttentionImpl(AttentionImpl):
             },
         )
         out_unpad = self._unwrap_flash_output(out_unpad)
-        return _pad_input(out_unpad, indices_q, query.size(0), query_length)
+        if indices_q is None:
+            return out_unpad.reshape(batch_size, query_length, *out_unpad.shape[1:])
+        return _pad_input(out_unpad, indices_q, batch_size, query_length)
 
     def _forward_varlen_packed(
         self,

vllm_omni/diffusion/layers/mot/mot_qkv_parallel_linear.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 from __future__ import annotations
 
 import torch
@@ -91,6 +94,21 @@ class MoTQKVParallelLinear(QKVParallelLinear):
         # boilerplate Module subclass just for the VAE pathway.
         self.gen_exp = torch.nn.Module()
 
+        # ``gen_exp`` participates in the same tensor-parallel layout as this
+        # layer.  vLLM's online quantizers inspect these attributes while
+        # processing weights (for example, FP8 uses them to share scales across
+        # TP ranks), so the lightweight container must expose the parallel
+        # linear metadata rather than only holding parameters.
+        for attr in (
+            "tp_rank",
+            "tp_size",
+            "input_size",
+            "input_size_per_partition",
+            "output_size",
+            "output_size_per_partition",
+        ):
+            setattr(self.gen_exp, attr, getattr(self, attr))
+
         # Use the same weight_loader as text
         vae_weight_loader = (
             self.weight_loader_v2

vllm_omni/diffusion/layers/mot/mot_row_parallel_linear.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 from __future__ import annotations
 
 import torch
@@ -84,6 +87,21 @@ class MoTRowParallelLinear(RowParallelLinear):
         # boilerplate Module subclass just for the VAE pathway.
         self.gen_exp = torch.nn.Module()
 
+        # ``gen_exp`` participates in the same tensor-parallel layout as this
+        # layer.  vLLM's online quantizers inspect these attributes while
+        # processing weights (for example, FP8 uses them to share scales across
+        # TP ranks), so the lightweight container must expose the parallel
+        # linear metadata rather than only holding parameters.
+        for attr in (
+            "tp_rank",
+            "tp_size",
+            "input_size",
+            "input_size_per_partition",
+            "output_size",
+            "output_size_per_partition",
+        ):
+            setattr(self.gen_exp, attr, getattr(self, attr))
+
         # Select weight_loader consistent with text
         vae_weight_loader = (
             self.weight_loader_v2

vllm_omni/diffusion/models/flux2_klein/flux2_klein_transformer.py

@@ -49,12 +49,23 @@ from vllm_omni.diffusion.distributed.sp_plan import (
 )
 from vllm_omni.diffusion.forward_context import get_forward_context
 from vllm_omni.diffusion.layers.rope import RotaryEmbedding, apply_rope_to_qk
+from vllm_omni.diffusion.models.host_weight_contract import FinalLayoutModelContract
 
 logger = init_logger(__name__)
 if TYPE_CHECKING:
     from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
 
 
+_FLUX2_STACKED_PARAMS_MAPPING = (
+    (".to_qkv.", ".to_q.", "q"),
+    (".to_qkv.", ".to_k.", "k"),
+    (".to_qkv.", ".to_v.", "v"),
+    (".add_kv_proj", ".add_q_proj", "q"),
+    (".add_kv_proj", ".add_k_proj", "k"),
+    (".add_kv_proj", ".add_v_proj", "v"),
+)
+
+
 def _join_prefix(prefix: str, name: str) -> str:
     return f"{prefix}.{name}" if prefix else name
 
@@ -770,6 +781,14 @@ class Flux2Transformer2DModel(nn.Module):
     Supports Sequence Parallelism (Ulysses and Ring) when configured via OmniDiffusionConfig.
     """
 
+    # Constructor state plus complete final-layout parameters and persistent
+    # buffers is sufficient to reconstruct a ready base-model transformer.
+    # Changes to the packed mapping or restore invariants require a version bump.
+    host_weight_restore_contract = FinalLayoutModelContract(
+        implementation_id="flux2-klein-dit",
+        version="1",
+    )
+
     _cache_dit_adapter_config = CacheDiTAdapterConfig(
         block_forward_patterns={
             "transformer_blocks": ForwardPattern.Pattern_1,
@@ -817,6 +836,9 @@ class Flux2Transformer2DModel(nn.Module):
         quant_config: "QuantizationConfig | None" = None,
     ):
         super().__init__()
+        # Warm HWR restoration skips load_weights(), so loader consumers such
+        # as LoRA discovery must not depend on that method to create this state.
+        self.stacked_params_mapping = list(_FLUX2_STACKED_PARAMS_MAPPING)
         self.out_channels = out_channels or in_channels
         self.inner_dim = num_attention_heads * attention_head_dim
         self.config = SimpleNamespace(
@@ -1004,18 +1026,78 @@ class Flux2Transformer2DModel(nn.Module):
             return (output,)
         return Transformer2DModelOutput(sample=output)
 
-    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
-        stacked_params_mapping = [
-            (".to_qkv.", ".to_q.", "q"),
-            (".to_qkv.", ".to_k.", "k"),
-            (".to_qkv.", ".to_v.", "v"),
-            (".add_kv_proj", ".add_q_proj", "q"),
-            (".add_kv_proj", ".add_k_proj", "k"),
-            (".add_kv_proj", ".add_v_proj", "v"),
-        ]
-        # Expose packed shard mappings for LoRA handling of fused projections.
-        self.stacked_params_mapping = stacked_params_mapping
+    def validate_restored_host_weights(self) -> None:
+        """Validate the tensor-complete BF16 base-model restore contract."""
+        if not isinstance(self.stacked_params_mapping, (list, tuple)) or (
+            tuple(self.stacked_params_mapping) != _FLUX2_STACKED_PARAMS_MAPPING
+        ):
+            raise ValueError("FLUX.2-klein packed QKV mapping is not constructor-stable")
 
+        stack_specs = (
+            ("transformer_blocks", self.config.num_layers, Flux2TransformerBlock),
+            ("single_transformer_blocks", self.config.num_single_layers, Flux2SingleTransformerBlock),
+        )
+        for name, expected_count, block_type in stack_specs:
+            blocks = getattr(self, name, None)
+            if not isinstance(blocks, nn.ModuleList) or len(blocks) != expected_count or expected_count <= 0:
+                raise ValueError(
+                    f"FLUX.2-klein {name} is incomplete: expected {expected_count} blocks, "
+                    f"got {len(blocks) if isinstance(blocks, nn.ModuleList) else 'missing'}"
+                )
+            if any(not isinstance(block, block_type) for block in blocks):
+                raise ValueError(f"FLUX.2-klein {name} contains an unexpected block implementation")
+
+        required_modules = (
+            "pos_embed",
+            "rope_prepare",
+            "time_guidance_embed",
+            "double_stream_modulation_img",
+            "double_stream_modulation_txt",
+            "single_stream_modulation",
+            "x_embedder",
+            "context_embedder",
+            "norm_out",
+            "proj_out",
+        )
+        for name in required_modules:
+            if not isinstance(getattr(self, name, None), nn.Module):
+                raise ValueError(f"FLUX.2-klein required module {name!r} is missing")
+
+        parameter_count = 0
+        stack_parameter_coverage = {name: False for name, _, _ in stack_specs}
+        for name, parameter in self.named_parameters():
+            parameter_count += 1
+            for stack_name in stack_parameter_coverage:
+                if name.startswith(f"{stack_name}."):
+                    stack_parameter_coverage[stack_name] = True
+            if parameter.is_meta or parameter.device.type != "cpu":
+                raise ValueError(f"FLUX.2-klein parameter {name!r} is not materialized on CPU")
+            if parameter.dtype is not torch.bfloat16:
+                raise ValueError(f"FLUX.2-klein parameter {name!r} must stay bf16, got {parameter.dtype}")
+            if parameter.layout is not torch.strided or not parameter.is_contiguous():
+                raise ValueError(f"FLUX.2-klein parameter {name!r} must use contiguous strided layout")
+            if parameter.numel() == 0:
+                raise ValueError(f"FLUX.2-klein parameter {name!r} is empty")
+        if parameter_count == 0 or not all(stack_parameter_coverage.values()):
+            raise ValueError("FLUX.2-klein restore did not materialize both transformer block stacks")
+
+        for name, buffer in self.named_buffers():
+            parent_path, _, leaf_name = name.rpartition(".")
+            owner = self.get_submodule(parent_path)
+            persistent = leaf_name not in owner._non_persistent_buffers_set
+            loader_buffer = name.endswith((".beta", ".eps"))
+            if loader_buffer and not persistent:
+                raise ValueError(f"FLUX.2-klein loader buffer {name!r} must be persistent")
+            if not persistent:
+                continue
+            if buffer.is_meta or buffer.device.type != "cpu":
+                raise ValueError(f"FLUX.2-klein persistent buffer {name!r} is not materialized on CPU")
+            if buffer.layout is not torch.strided or not buffer.is_contiguous():
+                raise ValueError(f"FLUX.2-klein persistent buffer {name!r} must use contiguous strided layout")
+            if loader_buffer and (buffer.dtype is not torch.bfloat16 or buffer.numel() != 1):
+                raise ValueError(f"FLUX.2-klein loader buffer {name!r} must be a scalar bf16 tensor")
+
+    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
         params_dict = dict(self.named_parameters())
 
         for name, buffer in self.named_buffers():
@@ -1026,7 +1108,7 @@ class Flux2Transformer2DModel(nn.Module):
         for name, loaded_weight in weights:
             original_name = name
             mapped = False
-            for param_name, weight_name, shard_id in stacked_params_mapping:
+            for param_name, weight_name, shard_id in self.stacked_params_mapping:
                 if weight_name not in original_name:
                     continue
                 name = original_name.replace(weight_name, param_name)

vllm_omni/diffusion/models/ltx2/ltx2_components.py

@@ -637,14 +637,18 @@ def initialize_pipeline_components(pipeline: Any, od_config: Any) -> None:
         upsampler_config = os.path.join(model, "latent_upsampler", "config.json")
         if os.path.isfile(upsampler_config) or not local_files_only:
             try:
-                pipeline.latent_upsampler = _load_component(
-                    LTX2LatentUpsamplerModel,
-                    model,
-                    "latent_upsampler",
-                    local_files_only=local_files_only,
-                    dtype=dtype,
-                    revision=revision,
-                )
+                # Diffusers builds the rational-resampler kernel with Long
+                # tensors; constructing it under a CUDA default-device context
+                # attempts an unsupported Long addmm before weights are loaded.
+                with torch.device("cpu"):
+                    pipeline.latent_upsampler = _load_component(
+                        LTX2LatentUpsamplerModel,
+                        model,
+                        "latent_upsampler",
+                        local_files_only=local_files_only,
+                        dtype=dtype,
+                        revision=revision,
+                    )
             except (OSError, ValueError):
                 if profile.latent_upsampler_filename is None or profile.artifact_repo_id is None:
                     raise

vllm_omni/diffusion/models/ltx2/ltx2_runtime.py

@@ -60,6 +60,29 @@ from .ltx2_request import (
 )
 
 
+def _run_ltx_vocoder(vocoder: nn.Module, generated_mel: torch.Tensor) -> torch.Tensor:
+    """Run the BWE vocoder in FP32, matching the official LTX pipeline."""
+    if not hasattr(vocoder, "bwe_generator"):
+        return vocoder(generated_mel)
+
+    input_dtype = generated_mel.dtype
+    device_type = generated_mel.device.type
+    module_dtype = next(vocoder.parameters()).dtype
+    if device_type == "mps":
+        if module_dtype != torch.float32:
+            vocoder.float()
+        try:
+            return vocoder(generated_mel.float()).to(input_dtype)
+        finally:
+            if module_dtype != torch.float32:
+                vocoder.to(module_dtype)
+
+    # BF16 errors compound through the BWE model's long convolution stack.
+    # FP32 autocast upcasts each op without materializing a second FP32 model.
+    with torch.autocast(device_type=device_type, dtype=torch.float32):
+        return vocoder(generated_mel.float()).to(input_dtype)
+
+
 def _expand_per_prompt_decode_value(
     value: float | list[float],
     *,
@@ -558,7 +581,7 @@ class LTXRuntime(
         if video.numel() > 0:
             video = self.video_processor.postprocess_video(video, output_type=output_type)
         generated_mel = self.audio_vae.decode(audio_latents.to(self.audio_vae.dtype), return_dict=False)[0]
-        audio = self.vocoder(generated_mel)
+        audio = _run_ltx_vocoder(self.vocoder, generated_mel)
         return self._make_output((video, audio))
 
     def _prepare_video_latents_stage(

vllm_omni/diffusion/stage_diffusion_proc.py

@@ -25,7 +25,12 @@ from vllm.logger import init_logger
 from vllm.utils.network_utils import get_open_zmq_ipc_path, zmq_socket_ctx
 from vllm.utils.system_utils import get_mp_context
 from vllm.v1.engine.core import EngineCoreProc
-from vllm.v1.engine.utils import CoreEngine, EngineZmqAddresses, wait_for_engine_startup
+from vllm.v1.engine.utils import (
+    CoreEngine,
+    CoreEngineLaunch,
+    EngineZmqAddresses,
+    wait_for_engine_startup,
+)
 from vllm.v1.utils import shutdown
 
 from vllm_omni.diffusion.data import DiffusionRequestAbortedError
@@ -764,7 +769,6 @@ class StageDiffusionProcManager:
             with zmq_socket_ctx(handshake_address, zmq.ROUTER, bind=True) as handshake_socket:
                 wait_for_engine_startup(
                     handshake_socket,
-                    self.addresses,
                     [CoreEngine(index=0, local=True)],
                     SimpleNamespace(
                         data_parallel_size_local=1,
@@ -773,8 +777,17 @@ class StageDiffusionProcManager:
                     ),
                     False,
                     None,
-                    self,
-                    None,
+                    CoreEngineLaunch(
+                        engine_manager=self,
+                        coordinator=None,
+                        addresses=self.addresses,
+                        tensor_queue=None,
+                        # StageDiffusionProcManager is not a CoreEngineProcManager,
+                        # so upstream's isinstance-gated sentinel registration would
+                        # be skipped; watch the subprocess directly so a proc death
+                        # during handshake is still detected.
+                        watched_frontend_processes=[self.proc],
+                    ),
                 )
         except Exception:
             shutdown([self.proc])

vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py

@@ -3,9 +3,11 @@
 
 import copy
 import importlib
+import inspect
+import threading
 import time
 from collections import defaultdict, deque
-from collections.abc import Callable, Mapping
+from collections.abc import Callable, Iterable, Mapping
 from typing import Any
 
 import torch
@@ -24,6 +26,14 @@ from .base import OmniTransferAdapterBase
 logger = get_connector_logger(__name__)
 
 
+class _SenderGeneration:
+    """Fence one external request generation without blocking cleanup."""
+
+    def __init__(self) -> None:
+        self.in_flight = False
+        self.cancelled = False
+
+
 class OmniChunkTransferAdapter(OmniTransferAdapterBase):
     """Chunk-level transfer adapter for Omni connector pipelines.
 
@@ -44,6 +54,12 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
 
     def __init__(self, vllm_config: Any):
         model_config = vllm_config.model_config
+        # The base constructor starts the save thread, so sender-generation
+        # state must exist before it can observe an enqueued task. The global
+        # lock only protects this map and short state transitions. Slow payload
+        # construction and connector writes never hold it.
+        self._sender_state_lock = threading.Lock()
+        self._sender_tokens: dict[str, _SenderGeneration] = {}
         self.scheduler_max_num_seqs = vllm_config.scheduler_config.max_num_seqs
         active_stream_window = int(getattr(model_config, "active_stream_window", 0) or 0)
         model_max_num_seqs = int(getattr(model_config, "max_num_seqs", self.scheduler_max_num_seqs) or 0)
@@ -70,6 +86,7 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
             module_path, func_name = custom_process_next_stage_input_func.rsplit(".", 1)
             module = importlib.import_module(module_path)
             self.custom_process_next_stage_input_func = getattr(module, func_name)
+        self._processor_accepts_step_tokens: dict[Callable[..., Any], bool] = {}
         # mapping for request id and chunk id
         self.put_req_chunk: dict[str, int] = defaultdict(int)
         self.get_req_chunk: dict[str, int] = defaultdict(int)
@@ -198,11 +215,28 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
         with self._recv_cond:
             self._recv_cond.notify()
 
+    def _accepts_new_token_ids(self, processor: Callable[..., Any]) -> bool:
+        cached = self._processor_accepts_step_tokens.get(processor)
+        if cached is not None:
+            return cached
+        try:
+            parameters = inspect.signature(processor).parameters.values()
+        except (TypeError, ValueError):
+            accepts = False
+        else:
+            accepts = any(
+                parameter.name == "new_token_ids" or parameter.kind is inspect.Parameter.VAR_KEYWORD
+                for parameter in parameters
+            )
+        self._processor_accepts_step_tokens[processor] = accepts
+        return accepts
+
     def save_async(
         self,
         multimodal_output: dict[str, Any] | None = None,
         request: Request | None = None,
         is_segment_finished: bool = False,
+        new_token_ids: Iterable[int] | None = None,
         confirmed_num_computed_tokens: int | None = None,
     ):
         """Build and enqueue one chunk for asynchronous sending.
@@ -221,38 +255,56 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
             multimodal_output: Per-request multimodal output dictionary
             request: Request object
             is_segment_finished: whether the segment of request is finished
+            new_token_ids: immutable snapshot of tokens generated by this
+                scheduler step, before resumable cleanup mutates the request
             confirmed_num_computed_tokens: committed token count captured
                 before a streaming transition can mutate ``request``
         """
         is_finished = request.is_finished() and not request.resumable
-
+        external_req_id = request.external_req_id
         if confirmed_num_computed_tokens is None:
             confirmed_num_computed_tokens = self._confirmed_num_computed_tokens(request)
-
-        # If the request is preempted, skip the already saved chunks.
-        if confirmed_num_computed_tokens < self.requests_num_chunks_sent.get(request.external_req_id, 0):
-            logger.warning(
-                f"Enqueue save_async for request {request.external_req_id}, "
-                f"request.num_computed_tokens={request.num_computed_tokens}, "
-                f"request.num_output_placeholders={getattr(request, 'num_output_placeholders', 0)}, "
-                f"previous_chunks_sent={self.requests_num_chunks_sent.get(request.external_req_id, 0)}"
-            )
-            return
-
-        self.requests_num_chunks_sent[request.external_req_id] = confirmed_num_computed_tokens
+        processor_request = (
+            self._snapshot_processor_request(request) if self.custom_process_next_stage_input_func else request
+        )
         task = {
             "multimodal_output": multimodal_output,
-            "request": (
-                self._snapshot_processor_request(request) if self.custom_process_next_stage_input_func else request
-            ),
+            "request": processor_request,
             "is_finished": is_finished,
             "is_segment_finished": is_segment_finished,
+            "new_token_ids": tuple(int(token_id) for token_id in (new_token_ids or ())),
         }
-        self._pending_save_reqs.append(task)
-        if is_segment_finished:
-            # The queued FIFO item now owns the old segment. Start the next
-            # segment's deduplication watermark before the worker sends it.
-            self.requests_num_chunks_sent.pop(request.external_req_id, None)
+
+        reject_reason = None
+        with self._sender_state_lock:
+            sender_token = self._sender_tokens.get(external_req_id)
+            if sender_token is not None and sender_token.cancelled:
+                reject_reason = "previous sender generation is still draining"
+            # If the request is preempted, skip the already saved chunks.
+            elif confirmed_num_computed_tokens < self.requests_num_chunks_sent.get(external_req_id, 0):
+                logger.warning(
+                    f"Enqueue save_async for request {external_req_id}, "
+                    f"request.num_computed_tokens={request.num_computed_tokens}, "
+                    f"request.num_output_placeholders={getattr(request, 'num_output_placeholders', 0)}, "
+                    f"previous_chunks_sent={self.requests_num_chunks_sent.get(external_req_id, 0)}"
+                )
+                return
+
+            else:
+                self.requests_num_chunks_sent[external_req_id] = confirmed_num_computed_tokens
+                if sender_token is None:
+                    sender_token = _SenderGeneration()
+                    self._sender_tokens[external_req_id] = sender_token
+                task["sender_token"] = sender_token
+                self._pending_save_reqs.append(task)
+                if is_segment_finished:
+                    # The queued FIFO item now owns the old segment. Start the next
+                    # segment's deduplication watermark before the worker sends it.
+                    self.requests_num_chunks_sent.pop(external_req_id, None)
+        if reject_reason is not None:
+            logger.error("Cannot enqueue %s: %s", external_req_id, reject_reason)
+            self.record_send_failure(request.request_id, reject_reason)
+            return
         with self._save_cond:
             self._save_cond.notify()
 
@@ -378,6 +430,44 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
         return False
 
     def _send_single_request(self, task: dict):
+        request = task["request"]
+        external_req_id = request.external_req_id
+        sender_token = task.get("sender_token")
+        if sender_token is None:
+            self._send_single_request_for_generation(task)
+            return
+        with self._sender_state_lock:
+            is_current = self._sender_tokens.get(external_req_id) is sender_token
+            if is_current and not sender_token.cancelled:
+                sender_token.in_flight = True
+            else:
+                is_current = False
+        if not is_current:
+            logger.debug("Discarding stale queued chunk for aborted request %s", external_req_id)
+            return
+        try:
+            self._send_single_request_for_generation(task, sender_token)
+        finally:
+            with self._sender_state_lock:
+                if self._sender_tokens.get(external_req_id) is sender_token:
+                    sender_token.in_flight = False
+                    if sender_token.cancelled:
+                        self._sender_tokens.pop(external_req_id, None)
+                        self._clear_sender_state_locked(external_req_id)
+
+    def _sender_generation_is_active(
+        self,
+        external_req_id: str,
+        sender_token: _SenderGeneration,
+    ) -> bool:
+        with self._sender_state_lock:
+            return self._sender_tokens.get(external_req_id) is sender_token and not sender_token.cancelled
+
+    def _send_single_request_for_generation(
+        self,
+        task: dict,
+        sender_token: _SenderGeneration | None = None,
+    ):
         raw_mm = task["multimodal_output"]
         multimodal_output = unflatten_payload(raw_mm) if isinstance(raw_mm, Mapping) else raw_mm
         request = task["request"]
@@ -392,10 +482,11 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
         payload_data: OmniPayloadStruct | None = None
         if self.custom_process_next_stage_input_func:
             try:
-                payload_data = self.custom_process_next_stage_input_func(
-                    transfer_manager=self,
-                    multimodal_output=multimodal_output,
-                    request=request,
+                processor = self.custom_process_next_stage_input_func
+                processor_kwargs = {
+                    "transfer_manager": self,
+                    "multimodal_output": multimodal_output,
+                    "request": request,
                     # Existing processors use is_finished as a flush signal.
                     # Terminal stops no longer count as segment boundaries
                     # (is_segment_finished is False when the request finishes,
@@ -403,8 +494,11 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
                     # accumulated tail on the terminal chunk — otherwise the
                     # downstream stage receives the finished marker without
                     # the final payload (#5413).
-                    is_finished=is_segment_finished or is_finished,
-                )
+                    "is_finished": is_segment_finished or is_finished,
+                }
+                if self._accepts_new_token_ids(processor):
+                    processor_kwargs["new_token_ids"] = task.get("new_token_ids", ())
+                payload_data = processor(**processor_kwargs)
 
             except Exception as e:
                 logger.error(f"Failed to use custom_process_input_func for payload extraction: {e}")
@@ -417,10 +511,23 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
             payload_data = OmniPayloadStruct()
         if payload_data.meta is None:
             payload_data.meta = MetaStruct()
-        payload_data.meta.finished = torch.tensor(is_finished, dtype=torch.bool)
+        # A model-native codec stream may span many resumable scheduler
+        # requests.  Its processor is the only layer that knows whether a
+        # request finish is a transport wake or the lifetime of the codec
+        # state itself.  Preserve an explicit streaming lifetime decision;
+        # all ordinary producers keep the scheduler-derived default.
+        processor_controls_codec_lifetime = payload_data.meta.finished is not None and self._is_truthy_scalar(
+            payload_data.meta.codec_streaming
+        )
+        if not processor_controls_codec_lifetime:
+            payload_data.meta.finished = torch.tensor(is_finished, dtype=torch.bool)
         if payload_data.meta.is_segment_finished is None:
             payload_data.meta.is_segment_finished = torch.tensor(is_segment_finished, dtype=torch.bool)
 
+        if sender_token is not None and not self._sender_generation_is_active(external_req_id, sender_token):
+            logger.debug("Skipping cancelled chunk for request %s before connector put", external_req_id)
+            return
+
         success, size, metadata = self.connector.put(
             from_stage=str(stage_id),
             to_stage=str(next_stage_id),
@@ -428,6 +535,13 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
             data=payload_data,
         )
 
+        if sender_token is not None and not self._sender_generation_is_active(external_req_id, sender_token):
+            # cleanup_sender() may cancel this generation while put() is
+            # blocked. Do not let that stale completion recreate state for a
+            # request whose cleanup is already in progress.
+            logger.debug("Ignoring completed put for cancelled request %s", external_req_id)
+            return
+
         if success:
             self.put_req_chunk[external_req_id] += 1
             self.ramp_chunk_count[external_req_id] += 1
@@ -535,11 +649,29 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
     def cleanup_sender(self, external_req_id: str) -> None:
         """Reclaim sender-side per-request state (keyed by external id).
 
-        Must only be called after the terminal chunk has actually been
-        sent (i.e. from ``_send_single_request``), not before.
+        Called after a terminal chunk is sent or when the scheduler aborts the
+        request before a terminal chunk can be produced. In-flight sends are
+        cancelled here and reclaim their own state in ``finally``; cleanup
+        never waits for connector I/O on the scheduler thread.
 
         Idempotent: calling with an already-cleaned or unknown id is safe.
         """
+        with self._sender_state_lock:
+            sender_token = self._sender_tokens.get(external_req_id)
+            if sender_token is None:
+                self._clear_sender_state_locked(external_req_id)
+                return
+            sender_token.cancelled = True
+            if sender_token.in_flight:
+                # The sender's finally block is the first point where no
+                # in-flight code can mutate this request's state. Let it own
+                # cleanup without making the scheduler wait for connector I/O.
+                return
+            self._sender_tokens.pop(external_req_id, None)
+            self._clear_sender_state_locked(external_req_id)
+
+    def _clear_sender_state_locked(self, external_req_id: str) -> None:
+        """Clear sender state while ``_sender_state_lock`` is held."""
         self.put_req_chunk.pop(external_req_id, None)
         self.request_payload.pop(external_req_id, None)
         self.code_prompt_token_ids.pop(external_req_id, None)
@@ -959,6 +1091,7 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
         }
 
         # First pass: collect requests to remove from queues
+        request_ids = set(request_ids)
         for req_id in request_ids:
             request = requests.get(req_id) if requests else None
             if request is None:
@@ -974,7 +1107,18 @@ class OmniChunkTransferAdapter(OmniTransferAdapterBase):
             if req_id in self.requests_origin_status and req_id in connector_owned_ids:
                 request.status = self.requests_origin_status.pop(req_id)
 
-        request_ids = set(request_ids)
+        # An abort can terminate a long-lived native codec stream before it
+        # emits a terminal payload. Reclaim both sides of the adapter so a
+        # later request reusing the same external id starts with empty state.
+        for req_id in request_ids:
+            request = requests.get(req_id) if requests else None
+            external_req_id = (
+                getattr(request, "external_req_id", None)
+                if request is not None
+                else self.request_ids_mapping.get(req_id)
+            ) or req_id
+            self.cleanup_receiver(req_id)
+            self.cleanup_sender(external_req_id)
 
         self.waiting_for_chunk_waiting_requests = deque(
             request for request in self.waiting_for_chunk_waiting_requests if request.request_id not in request_ids

vllm_omni/engine/async_omni_engine.py

@@ -109,6 +109,8 @@ class AsyncOmniEngine:
     _transfer_emitter: Any = None
     _prom_metrics: Any = None
     _enable_orch_monitor: bool = False
+    # Lazily created by get_output_blocking_async().
+    _output_drain_executor: concurrent.futures.ThreadPoolExecutor | None = None
 
     def __init__(
         self,
@@ -1739,6 +1741,42 @@ class AsyncOmniEngine:
                 raise RuntimeError("Orchestrator died unexpectedly. See logs above.")
             return None
 
+    async def get_output_blocking_async(self, timeout: float = 1.0) -> EngineQueueMessage | None:
+        """Blocking-wait read from the Orchestrator output queue.
+
+        Waits up to ``timeout`` seconds in a dedicated drain thread for the
+        next message (condition-variable wakeup instead of a poll cadence);
+        returns ``None`` on timeout so the caller keeps its liveness check,
+        mirroring ``try_get_output_async``'s contract. Used by the serving
+        final-output drain when ``VLLM_OMNI_EVENT_DRIVEN_ORCH`` is on.
+        """
+        executor = self._output_drain_executor
+        if executor is None:
+            executor = concurrent.futures.ThreadPoolExecutor(
+                max_workers=1,
+                thread_name_prefix="omni-output-drain",
+            )
+            self._output_drain_executor = executor
+
+        sync_q = self.output_queue.sync_q
+
+        def _drain_get() -> EngineQueueMessage | None:
+            # Exceptions are swallowed to a None sentinel: the queue may be
+            # closed mid-shutdown, and an exception left on an executor future
+            # after task cancellation would warn as never-retrieved.
+            try:
+                return sync_q.get(timeout=timeout)
+            except queue.Empty:
+                return None
+            except Exception:
+                return None
+
+        loop = asyncio.get_running_loop()
+        msg = await loop.run_in_executor(executor, _drain_get)
+        if msg is None and not self.is_alive():
+            raise RuntimeError("Orchestrator died unexpectedly. See logs above.")
+        return msg
+
     def get_stage_metadata(self, stage_id: int) -> StageRuntimeInfo:
         """Get cached metadata for a stage."""
         return self.stage_metadata[stage_id]
@@ -1928,6 +1966,12 @@ class AsyncOmniEngine:
             except Exception:
                 pass
 
+        if self._output_drain_executor is not None:
+            # Any in-flight blocking get bails out within its ≤1 s timeout
+            # (or immediately via the queue close above), so don't wait.
+            self._output_drain_executor.shutdown(wait=False)
+            self._output_drain_executor = None
+
         if hasattr(self, "_runtime") and self._runtime is not None and orchestrator_stopped:
             try:
                 self._runtime.shutdown()

vllm_omni/engine/orchestrator.py

@@ -15,6 +15,7 @@ handled by :class:`MembershipController`, which is injected optionally.
 from __future__ import annotations
 
 import asyncio
+import os
 import time as _time
 from collections.abc import Awaitable, Callable
 from dataclasses import dataclass, field
@@ -65,6 +66,23 @@ from vllm_omni.outputs import OmniRequestOutput
 
 logger = init_logger(__name__)
 
+# VLLM_OMNI_EVENT_DRIVEN_ORCH=1 switches the orchestration loop (and the
+# serving-side final-output drain in entrypoints/async_omni.py) from the legacy
+# 1 ms poll cadence to event-driven wakeups: one reader task per live LLM stage
+# replica awaits `client.get_output_async()` directly — the same pattern vLLM's
+# own AsyncLLM output handler uses — and feeds a single serial dispatch queue.
+# Default off; the legacy poll loop remains the fallback.
+_EVENT_DRIVEN_ORCH_ENV = "VLLM_OMNI_EVENT_DRIVEN_ORCH"
+
+# How often the event-driven loop reconciles its reader-task set against
+# `available_replica_ids()` (elastic membership, replica eviction) while idle.
+_ORCH_READER_RECONCILE_INTERVAL_S = 0.5
+
+
+def _event_driven_orch_enabled() -> bool:
+    return os.environ.get(_EVENT_DRIVEN_ORCH_ENV, "0").strip().lower() in ("1", "true", "yes", "on")
+
+
 if TYPE_CHECKING:
     from vllm_omni.experimental.fullduplex.engine.contracts import (
         DuplexControlPlanePort,
@@ -449,6 +467,7 @@ class Orchestrator:
 
         self._shutdown_event = asyncio.Event()
         self._stages_shutdown = False
+        self._event_driven_orch = _event_driven_orch_enabled()
 
         # Distributed membership (optional, injected by DistStageRuntime)
         self._membership = membership_controller
@@ -964,11 +983,105 @@ class Orchestrator:
     async def _orchestration_output_handler(self) -> None:
         """Poll all stages, handle transfers, send final outputs to main."""
         try:
-            await self._orchestration_loop()
+            if self._event_driven_orch:
+                await self._orchestration_loop_event_driven()
+            else:
+                await self._orchestration_loop()
         except asyncio.CancelledError:
             logger.debug("[Orchestrator] _orchestration_output_handler cancelled")
             return
 
+    async def _process_llm_stage_outputs(
+        self,
+        stage_id: int,
+        replica_id: int,
+        raw_outputs: Any,
+        raw_terminal_request_ids: set[str],
+    ) -> list[Any]:
+        """Process one raw LLM poll result; returns processed request outputs.
+
+        Shared by the legacy poll loop and the event-driven dispatcher so both
+        run the exact same per-output handling. Callers own the
+        ``EngineDeadError`` catch, since eviction needs the replica id, and
+        supply ``raw_terminal_request_ids`` — the per-poll accumulator this
+        fills for the caller to drain through
+        ``_finish_raw_terminal_requests`` once routing is done.
+        """
+        pool = self.stage_pools[stage_id]
+        await self._handle_kv_ready_raw_outputs(stage_id, raw_outputs)
+        for eco in raw_outputs.outputs:
+            # Emit kv_wait_s before _handle_kv_ready_raw_outputs'
+            # async_chunk early-return so it lands in all modes.
+            kv_params = getattr(eco, "kv_transfer_params", None)
+            if (
+                self._prom_metrics is not None
+                and isinstance(kv_params, dict)
+                and (kv_wait_s := kv_params.get("kv_wait_s")) is not None
+            ):
+                self._prom_metrics.observe_kv_wait(
+                    kv_params.get("connector_type") or "unknown",
+                    float(kv_wait_s),
+                )
+            req_state = self.request_states.get(getattr(eco, "request_id", None))
+            if req_state is None or not req_state.streaming.enabled:
+                continue
+            req_state.streaming.segment_finished = bool(getattr(eco, "is_segment_finished", False))
+            req_state.streaming.segment_token_ids = (
+                self._coerce_int_list(getattr(eco, "new_token_ids", None))
+                if req_state.streaming.segment_finished
+                else []
+            )
+            raw_mm = self._completion_multimodal_output(eco, None)
+            req_state.streaming.segment_output_metadata = (
+                dict(raw_mm) if req_state.streaming.segment_finished and isinstance(raw_mm, dict) else {}
+            )
+            req_state.streaming.new_prompt_len_snapshot = getattr(
+                eco,
+                "new_prompt_len_snapshot",
+                None,
+            )
+            if await self._apply_raw_terminal_stage_finish(stage_id, eco, req_state):
+                raw_terminal_request_ids.add(req_state.request_id)
+        iteration_stats = IterationStats() if (self._stat_logger is not None and raw_outputs.outputs) else None
+        processed = await pool.process_llm_raw_outputs(
+            replica_id,
+            raw_outputs,
+            iteration_stats=iteration_stats,
+        )
+        if self._stat_logger is not None and (raw_outputs.scheduler_stats is not None or iteration_stats is not None):
+            self._stat_logger.record(
+                raw_outputs.scheduler_stats,
+                iteration_stats,
+                engine_idx=self._stage_replica_to_engine_idx[(stage_id, replica_id)],
+            )
+        _sched_stats = raw_outputs.scheduler_stats
+        if (
+            self._prom_metrics is not None
+            and _sched_stats is not None
+            and getattr(_sched_stats, "num_waiting_reqs", None) is not None
+        ):
+            self._update_stage_replica_waiting(
+                stage_id,
+                replica_id,
+                int(_sched_stats.num_waiting_reqs),
+            )
+        return processed
+
+    def _absorb_diffusion_metrics(self, stage_id: int, replica_id: int, diffusion_output: Any) -> bool:
+        """Drain a diffusion output's piggybacked metrics.
+
+        Returns ``True`` when the output carried nothing but metrics and must
+        not be routed. Shared by both orchestration loops: the event-driven
+        poller would otherwise hand a metrics-only sentinel to
+        ``_handle_processed_outputs`` as if it were a real request output.
+        """
+        output_metrics = getattr(diffusion_output, "metrics", None)
+        if isinstance(output_metrics, dict):
+            n_waiting = output_metrics.pop(metric_defs.DIFFUSION_SCHEDULER_WAITING_KEY, None)
+            if n_waiting is not None:
+                self._update_stage_replica_waiting(stage_id, replica_id, int(n_waiting))
+        return diffusion_output.request_id == DIFFUSION_METRICS_ONLY_REQUEST_ID
+
     async def _orchestration_loop(self) -> None:
         """Poll stage pools and route logical outputs."""
         while not self._shutdown_event.is_set():
@@ -978,6 +1091,7 @@ class Orchestrator:
                 for replica_id in pool.available_replica_ids():
                     if self._shutdown_event.is_set():
                         return
+                    raw_terminal_request_ids: set[str] = set()
 
                     # Shared catch so a dead replica on either poll path is
                     # evicted rather than tearing down every stage (#4285).
@@ -987,13 +1101,7 @@ class Orchestrator:
                             if diffusion_output is None:
                                 continue
 
-                            output_metrics = getattr(diffusion_output, "metrics", None)
-                            if isinstance(output_metrics, dict):
-                                n_waiting = output_metrics.pop(metric_defs.DIFFUSION_SCHEDULER_WAITING_KEY, None)
-                                if n_waiting is not None:
-                                    self._update_stage_replica_waiting(stage_id, replica_id, int(n_waiting))
-
-                            if diffusion_output.request_id == DIFFUSION_METRICS_ONLY_REQUEST_ID:
+                            if self._absorb_diffusion_metrics(stage_id, replica_id, diffusion_output):
                                 idle = False
                                 continue
 
@@ -1004,69 +1112,9 @@ class Orchestrator:
                             if raw_outputs is None:
                                 continue
 
-                            await self._handle_kv_ready_raw_outputs(stage_id, raw_outputs)
-                            for eco in raw_outputs.outputs:
-                                # Emit kv_wait_s before _handle_kv_ready_raw_outputs'
-                                # async_chunk early-return so it lands in all modes.
-                                kv_params = getattr(eco, "kv_transfer_params", None)
-                                if (
-                                    self._prom_metrics is not None
-                                    and isinstance(kv_params, dict)
-                                    and (kv_wait_s := kv_params.get("kv_wait_s")) is not None
-                                ):
-                                    self._prom_metrics.observe_kv_wait(
-                                        kv_params.get("connector_type") or "unknown",
-                                        float(kv_wait_s),
-                                    )
-                                req_state = self.request_states.get(getattr(eco, "request_id", None))
-                                if req_state is None or not req_state.streaming.enabled:
-                                    continue
-                                req_state.streaming.segment_finished = bool(getattr(eco, "is_segment_finished", False))
-                                req_state.streaming.segment_token_ids = (
-                                    self._coerce_int_list(getattr(eco, "new_token_ids", None))
-                                    if req_state.streaming.segment_finished
-                                    else []
-                                )
-                                raw_mm = self._completion_multimodal_output(eco, None)
-                                req_state.streaming.segment_output_metadata = (
-                                    dict(raw_mm)
-                                    if req_state.streaming.segment_finished and isinstance(raw_mm, dict)
-                                    else {}
-                                )
-                                req_state.streaming.new_prompt_len_snapshot = getattr(
-                                    eco,
-                                    "new_prompt_len_snapshot",
-                                    None,
-                                )
-                                if req_state.streaming.enabled:
-                                    await self._apply_raw_terminal_stage_finish(stage_id, eco, req_state)
-                            iteration_stats = (
-                                IterationStats() if (self._stat_logger is not None and raw_outputs.outputs) else None
+                            processed = await self._process_llm_stage_outputs(
+                                stage_id, replica_id, raw_outputs, raw_terminal_request_ids
                             )
-                            processed = await pool.process_llm_raw_outputs(
-                                replica_id,
-                                raw_outputs,
-                                iteration_stats=iteration_stats,
-                            )
-                            if self._stat_logger is not None and (
-                                raw_outputs.scheduler_stats is not None or iteration_stats is not None
-                            ):
-                                self._stat_logger.record(
-                                    raw_outputs.scheduler_stats,
-                                    iteration_stats,
-                                    engine_idx=self._stage_replica_to_engine_idx[(stage_id, replica_id)],
-                                )
-                            _sched_stats = raw_outputs.scheduler_stats
-                            if (
-                                self._prom_metrics is not None
-                                and _sched_stats is not None
-                                and getattr(_sched_stats, "num_waiting_reqs", None) is not None
-                            ):
-                                self._update_stage_replica_waiting(
-                                    stage_id,
-                                    replica_id,
-                                    int(_sched_stats.num_waiting_reqs),
-                                )
                     except asyncio.CancelledError:
                         raise
                     except EngineDeadError as e:
@@ -1083,6 +1131,7 @@ class Orchestrator:
                         raise
 
                     await self._handle_processed_outputs(stage_id, replica_id, processed)
+                    await self._finish_raw_terminal_requests(stage_id, replica_id, raw_terminal_request_ids)
                     idle = False
 
             self._orch_monitor.note_loop(idle=idle)
@@ -1091,6 +1140,258 @@ class Orchestrator:
             else:
                 await asyncio.sleep(0)
 
+    async def _orchestration_loop_event_driven(self) -> None:
+        """Event-driven variant of ``_orchestration_loop``.
+
+        Selected by ``VLLM_OMNI_EVENT_DRIVEN_ORCH=1``. One reader task per
+        available LLM stage replica awaits ``client.get_output_async()``
+        directly — the same pattern vLLM's own ``AsyncLLM`` output handler uses
+        — and feeds a single dispatch queue. This coroutine consumes that queue
+        serially, so routing/handling semantics are identical to the legacy
+        loop; only the 1 ms poll cadence (and its per-tick ``asyncio.wait_for``
+        task churn) is removed. Diffusion stages keep their nowait-poll contract
+        via a per-pool poller task feeding the same queue.
+        """
+        ready_q: asyncio.Queue[tuple[str, int, int, Any]] = asyncio.Queue()
+        readers: dict[tuple[int, int], tuple[asyncio.Task, Any]] = {}
+        pollers: dict[int, asyncio.Task] = {}
+        reaped: list[asyncio.Task] = []
+
+        async def _llm_replica_reader(stage_id: int, replica_id: int, client: Any) -> None:
+            try:
+                while not self._shutdown_event.is_set():
+                    raw_outputs = await client.get_output_async()
+                    # Same keep/drop rule as StagePool._poll_stage_raw: a batch
+                    # with no request outputs still carries SchedulerStats on
+                    # throttled ticks, and dropping it loses the KV/queue gauges
+                    # for that interval. Only a fully empty batch is dropped. A
+                    # poll-style client returns one instead of blocking, so keep
+                    # the legacy 1 ms cadence for those; a blocking client only
+                    # lands here rarely, where 1 ms is irrelevant.
+                    if (
+                        not raw_outputs.outputs
+                        and raw_outputs.scheduler_stats is None
+                        and not raw_outputs.finished_requests
+                    ):
+                        await asyncio.sleep(0.001)
+                        continue
+                    await ready_q.put(("llm", stage_id, replica_id, raw_outputs))
+            except asyncio.CancelledError:
+                raise
+            except BaseException as e:  # noqa: BLE001 - routed to the dispatcher
+                await ready_q.put(("error", stage_id, replica_id, e))
+
+        async def _diffusion_poller(stage_id: int, pool: StagePool) -> None:
+            try:
+                while not self._shutdown_event.is_set():
+                    got = False
+                    for replica_id in pool.available_replica_ids():
+                        try:
+                            output = pool.poll_diffusion_output(replica_id)
+                        except EngineDeadError as e:
+                            await ready_q.put(("error", stage_id, replica_id, e))
+                            continue
+                        if output is None:
+                            continue
+                        await ready_q.put(("diffusion", stage_id, replica_id, output))
+                        got = True
+                    await asyncio.sleep(0 if got else 0.001)
+            except asyncio.CancelledError:
+                raise
+            except BaseException as e:  # noqa: BLE001 - routed to the dispatcher
+                await ready_q.put(("error", stage_id, -1, e))
+
+        def _reconcile_readers() -> None:
+            """Track available replicas: spawn/reap LLM readers and diffusion pollers.
+
+            Walks ``available_replica_ids()`` for parity with the legacy loop,
+            so a replica evicted by ``_handle_dead_replica`` (or marked
+            unavailable by membership) stops being read here too.
+
+            Diffusion pollers are reconciled here rather than spawned once at
+            startup: a pool with no live replica yet reports ``stage_type is
+            None``, so a stage whose first replica registers at runtime would
+            otherwise never get a poller and its outputs would never drain.
+            """
+            live: set[tuple[int, int]] = set()
+            live_pools: set[int] = set()
+            for stage_id in range(self.num_stages):
+                pool = self.stage_pools[stage_id]
+                stage_type = pool.stage_type
+                if stage_type is None:
+                    # No live replica yet; nothing to attach to. A later
+                    # reconcile picks the stage up once one registers.
+                    continue
+                if stage_type == "diffusion":
+                    live_pools.add(stage_id)
+                    existing_poller = pollers.get(stage_id)
+                    if existing_poller is None or existing_poller.done():
+                        pollers[stage_id] = asyncio.create_task(
+                            _diffusion_poller(stage_id, pool),
+                            name=f"orch-diffusion-poller-s{stage_id}",
+                        )
+                    continue
+                for replica_id in pool.available_replica_ids():
+                    client = pool.clients[replica_id]
+                    if client is None:
+                        continue
+                    key = (stage_id, replica_id)
+                    live.add(key)
+                    existing = readers.get(key)
+                    if existing is None:
+                        readers[key] = (
+                            asyncio.create_task(
+                                _llm_replica_reader(stage_id, replica_id, client),
+                                name=f"orch-reader-s{stage_id}r{replica_id}",
+                            ),
+                            client,
+                        )
+                        continue
+                    if existing[0].done():
+                        # The reader exited, which means it already queued its
+                        # failure. Leave the slot alone until the dispatcher
+                        # handles that error and evicts the replica; respawning
+                        # now just re-raises the same error against the same
+                        # dead client and queues a duplicate eviction.
+                        continue
+                    if existing[1] is client:
+                        continue
+                    # Client swapped underneath us: retire the old reader.
+                    existing[0].cancel()
+                    reaped.append(existing[0])
+                    readers[key] = (
+                        asyncio.create_task(
+                            _llm_replica_reader(stage_id, replica_id, client),
+                            name=f"orch-reader-s{stage_id}r{replica_id}",
+                        ),
+                        client,
+                    )
+            for key in list(readers):
+                if key not in live:
+                    task, _ = readers.pop(key)
+                    task.cancel()
+                    reaped.append(task)
+            for stage_id in list(pollers):
+                if stage_id not in live_pools:
+                    task = pollers.pop(stage_id)
+                    task.cancel()
+                    reaped.append(task)
+
+        _reconcile_readers()
+        logger.info(
+            "[Orchestrator] Event-driven orchestration loop enabled (%s): %d LLM replica readers, %d diffusion pollers",
+            _EVENT_DRIVEN_ORCH_ENV,
+            len(readers),
+            len(pollers),
+        )
+
+        shutdown_task = asyncio.create_task(self._shutdown_event.wait(), name="orch-shutdown-wait")
+        pending_get: asyncio.Task | None = None
+        next_reconcile = _time.monotonic() + _ORCH_READER_RECONCILE_INTERVAL_S
+        try:
+            while not self._shutdown_event.is_set():
+                if pending_get is None:
+                    pending_get = asyncio.create_task(ready_q.get(), name="orch-dispatch-get")
+                done, _ = await asyncio.wait(
+                    {pending_get, shutdown_task},
+                    timeout=max(0.0, next_reconcile - _time.monotonic()),
+                    return_when=asyncio.FIRST_COMPLETED,
+                )
+                if shutdown_task in done:
+                    return
+                # Reconcile on a wall-clock schedule, not only when the queue
+                # goes idle: under continuous traffic `asyncio.wait` returns on
+                # every output and a timeout-only reconcile would never fire,
+                # so a replica that registers at runtime would never get a
+                # reader and its outputs would never drain.
+                if _time.monotonic() >= next_reconcile:
+                    next_reconcile = _time.monotonic() + _ORCH_READER_RECONCILE_INTERVAL_S
+                    _reconcile_readers()
+                if not done:
+                    self._orch_monitor.note_loop(idle=True)
+                    continue
+                kind, stage_id, replica_id, payload = pending_get.result()
+                pending_get = None
+
+                if kind == "error":
+                    # replica_id < 0 means the failure was raised by a poller
+                    # task itself rather than attributed to one replica, so
+                    # there is nothing to evict -- and evict_replica() rejects
+                    # an out-of-range id, which would turn a survivable death
+                    # into a ValueError out of the dispatcher.
+                    if isinstance(payload, EngineDeadError) and replica_id >= 0:
+                        # Same per-replica isolation as the legacy loop (#4285):
+                        # evict and keep serving. Reconcile immediately so the
+                        # dead replica's reader is reaped now rather than at the
+                        # next scheduled tick. Skip a replica already evicted by
+                        # an earlier error from the same slot -- evict_replica()
+                        # would shut a None client down a second time.
+                        if replica_id in self.stage_pools[stage_id].live_replica_ids():
+                            await self._handle_dead_replica(stage_id, replica_id, payload)
+                        _reconcile_readers()
+                        continue
+                    if self._shutdown_event.is_set():
+                        return
+                    logger.error(
+                        "[Orchestrator] Stage-%s replica-%s reader failed: %r",
+                        stage_id,
+                        replica_id,
+                        payload,
+                    )
+                    raise payload
+
+                raw_terminal_request_ids: set[str] = set()
+                try:
+                    if kind == "diffusion":
+                        pool = self.stage_pools[stage_id]
+                        if self._absorb_diffusion_metrics(stage_id, replica_id, payload):
+                            self._orch_monitor.note_loop(idle=False)
+                            continue
+                        pool.record_output_timestamps([payload])
+                        processed = [payload]
+                    else:
+                        processed = await self._process_llm_stage_outputs(
+                            stage_id, replica_id, payload, raw_terminal_request_ids
+                        )
+                except asyncio.CancelledError:
+                    raise
+                except EngineDeadError as e:
+                    if replica_id in self.stage_pools[stage_id].live_replica_ids():
+                        await self._handle_dead_replica(stage_id, replica_id, e)
+                    _reconcile_readers()
+                    continue
+                except Exception:
+                    if self._shutdown_event.is_set():
+                        return
+                    logger.exception(
+                        "[Orchestrator] Stage-%s replica-%s processing failed",
+                        stage_id,
+                        replica_id,
+                    )
+                    raise
+
+                await self._handle_processed_outputs(stage_id, replica_id, processed)
+                await self._finish_raw_terminal_requests(stage_id, replica_id, raw_terminal_request_ids)
+                # Mirrors the legacy loop, which sets idle=False only after a
+                # poll produced outputs that were routed -- an evicted replica
+                # `continue`s above without marking the tick active.
+                self._orch_monitor.note_loop(idle=False)
+        finally:
+            shutdown_task.cancel()
+            if pending_get is not None:
+                pending_get.cancel()
+            reader_tasks = [task for task, _ in readers.values()]
+            poller_tasks = list(pollers.values())
+            for task in (*reader_tasks, *poller_tasks):
+                task.cancel()
+            # `reaped` holds readers/pollers retired mid-run (client swap,
+            # eviction, stage teardown). They were cancelled but never awaited,
+            # so gather them here too rather than leaving dangling tasks.
+            cleanup = [shutdown_task, *reader_tasks, *poller_tasks, *reaped]
+            if pending_get is not None:
+                cleanup.append(pending_get)
+            await asyncio.gather(*cleanup, return_exceptions=True)
+
     async def _handle_processed_outputs(self, stage_id: int, replica_id: int, outputs: list[Any]) -> None:
         """Route processed stage outputs produced by one stage poll."""
         pool = self.stage_pools[stage_id]
@@ -1434,8 +1735,8 @@ class Orchestrator:
         stage_id: int,
         eco: Any,
         req_state: OrchestratorRequestState,
-    ) -> None:
-        """Record session-level finish markers dropped by the streaming output processor.
+    ) -> bool:
+        """Record a session-level finish marker and report whether it was terminal.
 
         Streaming segment stops set ``is_segment_finished=True`` and are handled
         via processed outputs. Session termination (e.g. ``finish_requests`` after
@@ -1448,14 +1749,60 @@ class Orchestrator:
         outputs after stage-0 session end.
         """
         if getattr(eco, "finish_reason", None) is None:
-            return
+            return False
         if getattr(eco, "is_segment_finished", False):
-            return
+            return False
 
         final_output_stage_ids = req_state.final_output_stage_ids or {req_state.final_stage_id}
         if stage_id not in final_output_stage_ids:
-            return
+            return False
         req_state.finished_final_output_stage_ids.add(stage_id)
+        return True
+
+    async def _finish_raw_terminal_requests(
+        self,
+        stage_id: int,
+        replica_id: int,
+        request_ids: set[str],
+    ) -> None:
+        """Finish streaming requests whose raw terminal had no processed output."""
+        pool = self.stage_pools[stage_id]
+        if not pool.final_output:
+            return
+
+        for request_id in request_ids:
+            req_state = self.request_states.get(request_id)
+            if req_state is None or self._is_duplex_session_request(req_state):
+                continue
+
+            final_output_stage_ids = req_state.final_output_stage_ids or {req_state.final_stage_id}
+            if not final_output_stage_ids.issubset(req_state.finished_final_output_stage_ids):
+                continue
+
+            final_output_type = getattr(pool.stage_client, "final_output_type", None)
+            logger.info(
+                "[Orchestrator] req=%s stage-%s raw terminal produced no processed terminal; returning empty %s output",
+                request_id,
+                stage_id,
+                final_output_type or "text",
+            )
+            terminal_output = _build_terminal_empty_output(
+                request_id,
+                final_output_type=final_output_type,
+                audio_sample_rate=pool._infer_audio_sample_rate(),
+            )
+            await self.output_async_queue.put(
+                OutputMessage(
+                    request_id=request_id,
+                    stage_id=stage_id,
+                    replica_id=replica_id,
+                    engine_outputs=terminal_output,
+                    metrics=None,
+                    finished=True,
+                    stage_submit_ts=req_state.stage_submit_ts.get(stage_id),
+                )
+            )
+            await self._cleanup_request_ids([request_id, *self._cfg_tracker.cleanup_parent(request_id)])
 
     def _maybe_clone_diffusion_params_for_cfg(self, request_id: str, params: Any) -> Any:
         """Attach CFG companion ids to diffusion sampling params when needed."""

vllm_omni/engine/stage_engine_startup.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 """Helpers for launching and handshaking omni engine cores."""
 
 from __future__ import annotations
@@ -22,6 +25,7 @@ from vllm.utils.network_utils import get_open_ports_list, zmq_socket_ctx
 from vllm.v1.engine.coordinator import DPCoordinator
 from vllm.v1.engine.utils import (
     CoreEngine,
+    CoreEngineLaunch,
     CoreEngineProcManager,
     EngineZmqAddresses,
     get_engine_zmq_addresses,
@@ -833,13 +837,16 @@ def connect_remote_engine_cores(
             )
             wait_for_engine_startup(
                 handshake_socket,
-                addresses,
                 engines_to_handshake,
                 vllm_config.parallel_config,
                 False,  # coordinated_dp
                 vllm_config.cache_config,
-                None,  # proc_manager (remote — no local procs)
-                None,  # coord_process
+                CoreEngineLaunch(
+                    engine_manager=None,  # remote — no local procs
+                    coordinator=None,
+                    addresses=addresses,
+                    tensor_queue=None,
+                ),
             )
     finally:
         omni_master_server.release_route_port_reservations(
@@ -875,13 +882,16 @@ def connect_remote_diffusion_proc(
             yield StageReplicaResources(addresses=addresses)
             wait_for_engine_startup(
                 handshake_socket,
-                addresses,
                 [CoreEngine(index=0, local=False)],
                 _single_diffusion_parallel_config(local_client=False),
                 False,
                 None,
-                None,
-                None,
+                CoreEngineLaunch(
+                    engine_manager=None,  # remote — no local procs
+                    coordinator=None,
+                    addresses=addresses,
+                    tensor_queue=None,
+                ),
             )
     finally:
         omni_master_server.release_route_port_reservations(
@@ -1077,13 +1087,16 @@ def _launch_omni_core_engines(
             yield local_engine_manager, coordinator, addresses
             wait_for_engine_startup(
                 handshake_socket,
-                addresses,
                 engines_to_handshake,
                 parallel_config,
                 parallel_config.data_parallel_size > 1 and vllm_config.model_config.is_moe,
                 vllm_config.cache_config,
-                local_engine_manager,
-                coordinator.proc if coordinator else None,
+                CoreEngineLaunch(
+                    engine_manager=local_engine_manager,
+                    coordinator=coordinator,
+                    addresses=addresses,
+                    tensor_queue=None,
+                ),
             )
     finally:
         omni_master_server.release_route_port_reservations(
@@ -1166,13 +1179,16 @@ def launch_stage_replica(
         )
         wait_for_engine_startup(
             handshake_socket,
-            addresses,
             engines_to_handshake,
             vllm_config.parallel_config,
             False,  # coordinated_dp
             vllm_config.cache_config,
-            engine_manager,
-            None,  # coordinator_proc
+            CoreEngineLaunch(
+                engine_manager=engine_manager,
+                coordinator=None,
+                addresses=addresses,
+                tensor_queue=None,
+            ),
         )
 
 

vllm_omni/engine/stage_init_utils.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 """
 Stage initialization helpers for vLLM-Omni multi-stage runtime.
 
@@ -565,7 +568,7 @@ def _maybe_set_qwen3_omni_moe_env(engine_args_dict: dict[str, Any]) -> None:
 def split_devices_for_replicas(
     devices_str: str | None,
     num_replicas: int,
-    tp_size: int,
+    devices_per_replica: int,
     stage_id: int,
 ) -> list[str | None]:
     """Split a devices string into per-replica subsets.
@@ -578,19 +581,19 @@ def split_devices_for_replicas(
     When ``num_replicas`` is 1, returns ``[devices_str]`` unchanged.
     Otherwise, two YAML shapes are accepted:
 
-    1. **Legacy / pool mode** — ``len(devices) == num_replicas * tp_size``:
+    1. **Legacy / pool mode** — ``len(devices) == num_replicas * devices_per_replica``:
        the string enumerates the full per-stage pool. Each replica gets
-       ``tp_size`` consecutive entries. The values are logical indices
+       ``devices_per_replica`` consecutive entries. The values are logical indices
        into the launcher's ``CUDA_VISIBLE_DEVICES``.
 
        ``split_devices_for_replicas("1,2,3,4", 2, 2, 1) → ["1,2", "3,4"]``
 
-    2. **Template mode** — ``len(devices) == tp_size``: the YAML declares
+    2. **Template mode** — ``len(devices) == devices_per_replica``: the YAML declares
        a single per-replica template (the same shape one replica would
-       use), and is **dp-independent**. Each replica r gets the offsets
-       ``[r*tp_size + a for a in template]`` of the launcher's
+       use), and is **replica-count-independent**. Each replica r gets the offsets
+       ``[r*devices_per_replica + a for a in template]`` of the launcher's
        ``CUDA_VISIBLE_DEVICES``. The template's entries must lie in
-       ``[0, tp_size)``.
+       ``[0, devices_per_replica)``.
 
        ``split_devices_for_replicas("0,1", 2, 2, 1) → ["0,1", "2,3"]``
        ``split_devices_for_replicas("0,1", 4, 2, 1) → ["0,1", "2,3", "4,5", "6,7"]``
@@ -613,27 +616,29 @@ def split_devices_for_replicas(
 
     device_list = [d.strip() for d in devices_str.split(",") if d.strip()]
 
-    if len(device_list) == num_replicas * tp_size:
-        return [",".join(device_list[r * tp_size : (r + 1) * tp_size]) for r in range(num_replicas)]
+    if len(device_list) == num_replicas * devices_per_replica:
+        return [
+            ",".join(device_list[r * devices_per_replica : (r + 1) * devices_per_replica]) for r in range(num_replicas)
+        ]
 
-    if len(device_list) == tp_size:
+    if len(device_list) == devices_per_replica:
         try:
             offsets = [int(a) for a in device_list]
         except ValueError as e:
             raise ValueError(f"Stage {stage_id}: template-mode devices must be ints, got {devices_str!r}") from e
-        bad = [a for a in offsets if not (0 <= a < tp_size)]
+        bad = [a for a in offsets if not (0 <= a < devices_per_replica)]
         if bad:
             raise ValueError(
                 f"Stage {stage_id}: template-mode device offset(s) {bad} "
-                f"out of range [0, {tp_size}); devices={devices_str!r}"
+                f"out of range [0, {devices_per_replica}); devices={devices_str!r}"
             )
-        return [",".join(str(r * tp_size + a) for a in offsets) for r in range(num_replicas)]
+        return [",".join(str(r * devices_per_replica + a) for a in offsets) for r in range(num_replicas)]
 
     raise ValueError(
         f"Stage {stage_id}: devices={devices_str!r} has {len(device_list)} id(s); "
-        f"need either {tp_size} (template, dp-independent) or "
-        f"{num_replicas * tp_size} (pool / legacy). "
-        f"num_replicas={num_replicas}, tensor_parallel_size={tp_size}."
+        f"need either {devices_per_replica} (per-replica template) or "
+        f"{num_replicas * devices_per_replica} (pool / legacy). "
+        f"num_replicas={num_replicas}, devices_per_replica={devices_per_replica}."
     )
 
 
@@ -645,9 +650,32 @@ def get_stage_tp_size(stage_cfg: Any) -> int:
     return int(getattr(engine_args, "tensor_parallel_size", 1) or 1)
 
 
-def get_stage_devices_per_replica(stage_cfg: Any) -> int:
+def _get_local_llm_parallel_sizes(
+    stage_cfg: Any,
+    engine_args: Any | None = None,
+) -> tuple[int, int, int]:
+    """Return ``(tp, local_dp, pp)`` for one local LLM replica.
+
+    ``data_parallel_size`` is cluster-wide, whereas ``runtime.devices`` is
+    local to this process.  Prefer an explicitly resolved
+    ``data_parallel_size_local`` (including zero for a head process that owns
+    no local engines), and only fall back to the global DP width when it is
+    unset.
+    """
+    if engine_args is None:
+        engine_args = getattr(stage_cfg, "engine_args", {})
+    tp_size = int(_get_attr_or_item(engine_args, "tensor_parallel_size", 1) or 1)
+    pp_size = int(_get_attr_or_item(engine_args, "pipeline_parallel_size", 1) or 1)
+    local_dp_size = _get_attr_or_item(engine_args, "data_parallel_size_local", None)
+    if local_dp_size is None:
+        local_dp_size = _get_attr_or_item(engine_args, "data_parallel_size", 1)
+    return tp_size, int(local_dp_size if local_dp_size is not None else 1), pp_size
+
+
+def get_stage_devices_per_replica(stage_cfg: Any, engine_args: Any | None = None) -> int:
     """Return the number of devices consumed by one replica of *stage_cfg*."""
-    engine_args = getattr(stage_cfg, "engine_args", {})
+    if engine_args is None:
+        engine_args = getattr(stage_cfg, "engine_args", {})
     if getattr(stage_cfg, "stage_type", "llm") == "diffusion":
         parallel_config = _get_attr_or_item(engine_args, "parallel_config")
         if parallel_config is None:
@@ -664,14 +692,15 @@ def get_stage_devices_per_replica(stage_cfg: Any) -> int:
         except Exception:
             return 1
 
-    return get_stage_tp_size(stage_cfg)
+    tp_size, local_dp_size, pp_size = _get_local_llm_parallel_sizes(stage_cfg, engine_args)
+    return tp_size * max(1, local_dp_size) * pp_size
 
 
 def compute_replica_layout(
     stage_configs: Sequence[Any],
     *,
     allow_zero: bool = False,
-) -> tuple[list[int], dict[int, list[str]]]:
+) -> tuple[list[int], dict[int, list[str | None]]]:
     """Compute per-stage replica counts and device assignments.
 
     Args:
@@ -820,7 +849,7 @@ def _project_upstream_config_fields(
     field_map: Mapping[str, str],
 ) -> dict[str, Any]:
     """Project every explicit upstream input, including newly added fields."""
-    explicit_fields = getattr(config, "_omni_explicit_fields", frozenset())
+    explicit_fields: frozenset[str] = getattr(config, "_omni_explicit_fields", frozenset())
     unprojected_fields = explicit_fields - frozenset(field_map)
     if unprojected_fields:
         names = ", ".join(sorted(unprojected_fields))
@@ -838,9 +867,9 @@ def _project_omni_stage_engine_args(
     """Read backend inputs from one structured stage config."""
     engine_args: dict[str, Any] = {}
     is_diffusion = isinstance(stage_config, VllmOmniDiffusionStageConfig)
-
     if is_diffusion:
-        engine_args.update(_project_omni_config_fields(stage_config.diffusion_config))
+        diffusion_stage = cast(VllmOmniDiffusionStageConfig, stage_config)
+        engine_args.update(_project_omni_config_fields(diffusion_stage.diffusion_config))
 
     for config, excluded_fields in (
         (
@@ -1112,6 +1141,74 @@ def build_engine_args_dict_from_omni_stage_config(
     )
 
 
+def _count_stage_devices(devices: Any) -> int | None:
+    if devices is None:
+        return None
+    if isinstance(devices, (list, tuple)):
+        return len(devices)
+    values = [device for device in str(devices).split(",") if device.strip()]
+    return len(values) or None
+
+
+def _check_stage_device_layout(stage_config: Any, engine_args_dict: dict[str, Any]) -> None:
+    """Fail early when a stage's world size cannot fit its assigned ``devices``.
+
+    Re-runs :func:`check_device_layout` (normally only reached on the
+    ``--strategy-config`` path) against the fully resolved per-stage layout, so
+    an inconsistent ``tensor_parallel_size`` vs ``devices`` (issue #5003) is
+    reported here with a clear message instead of surfacing later as an opaque
+    worker-side ``local rank ... out of bounds`` assertion.
+    """
+    from vllm_omni.config.composable_parallel import StrategyApplyError, check_device_layout
+
+    runtime = getattr(stage_config, "runtime", None)
+    devices = _get_attr_or_item(runtime, "devices", None) if runtime is not None else None
+    if devices is None:
+        # No explicit placement -> vLLM assigns devices itself; nothing to check.
+        return
+
+    num_replicas = _get_attr_or_item(runtime, "num_replicas", 1) if runtime is not None else 1
+    stage_id = getattr(stage_config, "stage_id", "?")
+    tp_size, local_dp_size, pp_size = _get_local_llm_parallel_sizes(stage_config, engine_args_dict)
+    if local_dp_size == 0:
+        # This process hosts no local DP engines, so its local device list does
+        # not describe the cluster-wide DP layout and must not be validated.
+        return
+
+    try:
+        check_device_layout(
+            devices,
+            tensor_parallel_size=tp_size,
+            data_parallel_size=local_dp_size,
+            pipeline_parallel_size=pp_size,
+            num_replicas=int(num_replicas or 1),
+            role=f"stage-{stage_id}",
+        )
+    except StrategyApplyError as e:
+        message = (
+            f"Stage {stage_id}: device layout is inconsistent — {e} "
+            "Set devices and the per-stage TP, local DP, PP, and replica counts "
+            "so the declared device count matches the local world size."
+        )
+        device_count = _count_stage_devices(devices)
+        world_without_tp = local_dp_size * pp_size
+        valid_without_tp = {world_without_tp, int(num_replicas or 1) * world_without_tp}
+        if tp_size > 1 and device_count in valid_without_tp:
+            message += (
+                " This layout is consistent with issue #5003: a top-level "
+                "--tensor-parallel-size is applied to every stage, but each stage's "
+                "`devices` is not adjusted automatically. Pass --stage-overrides "
+                "to set tensor_parallel_size and devices together on every stage, "
+                "so single-GPU stages get tensor_parallel_size=1, e.g. "
+                '\'{"0": {"tensor_parallel_size": 4, "devices": "0,1,2,3"}, '
+                '"1": {"tensor_parallel_size": 1, "devices": "0"}, '
+                '"2": {"tensor_parallel_size": 1, "devices": "1"}}\'. '
+                "Or omit the top-level --tensor-parallel-size and set it only in "
+                "stage-0's override."
+            )
+        raise ValueError(message) from e
+
+
 def build_vllm_config(
     stage_config: Any,
     model: str,
@@ -1147,6 +1244,16 @@ def build_vllm_config(
 
     omni_engine_args = OmniEngineArgs(**filtered_engine_args_dict)
 
+    # Guard against a per-stage world size that its assigned ``devices`` cannot
+    # satisfy (issue #5003). A top-level ``--tensor-parallel-size`` is broadcast
+    # to every stage, but ``devices`` is not, so a stage can end up with e.g.
+    # tensor_parallel_size=4 while still holding a single-GPU deploy default.
+    # Without --strategy-config the strategy-path device check never runs, so
+    # the mismatch used to surface only as an opaque worker-side assertion
+    # ("DP adjusted local rank N is out of bounds for M devices."). Re-run the
+    # same check here, before workers spawn, to fail early with a clear message.
+    _check_stage_device_layout(stage_config, filtered_engine_args_dict)
+
     # Multi-stage pipelines (qwen3_tts code2wav, etc.) set max_model_len
     # larger than HF max_position_embeddings by design. vLLM's validator
     # rejects that without the env flag.
@@ -1473,6 +1580,7 @@ def build_diffusion_config(
     num_devices_per_stage = od_config.parallel_config.world_size
     device_control_env = current_omni_platform.device_control_env_var
     visible_devices_str = os.environ.get(device_control_env) if device_control_env else None
+    physical_devices: list[str | int]
     if visible_devices_str:
         physical_devices = [device.strip() for device in visible_devices_str.split(",") if device.strip()]
     else:

vllm_omni/entrypoints/async_omni.py

@@ -53,6 +53,11 @@ if TYPE_CHECKING:
 
 logger = init_logger(__name__)
 _FINAL_OUTPUT_IDLE_SLEEP_S = 0.001
+# Blocking-wait interval for the event-driven final-output drain
+# (VLLM_OMNI_EVENT_DRIVEN_ORCH=1): a message wakes the drain immediately via
+# the janus queue's condition variable; this timeout only bounds how often the
+# orchestrator liveness check runs while the pipeline is idle.
+_FINAL_OUTPUT_BLOCKING_WAIT_S = 1.0
 
 
 class AsyncEventResolver:
@@ -880,14 +885,29 @@ class AsyncOmni(EngineClient, OmniBase):
 
         engine = self.engine
 
+        # Event-driven drain (VLLM_OMNI_EVENT_DRIVEN_ORCH=1): block on the
+        # queue's condition variable in a dedicated thread instead of the
+        # get_nowait + 1 ms sleep cadence. Same flag as the orchestrator-side
+        # event-driven loop (vllm_omni/engine/orchestrator.py).
+        from vllm_omni.engine.orchestrator import _event_driven_orch_enabled
+
+        event_driven_drain = _event_driven_orch_enabled() and hasattr(engine, "get_output_blocking_async")
+
         async def _final_output_loop():
             """Background coroutine that dispatches final outputs to request queues."""
             try:
                 while True:
-                    msg = await engine.try_get_output_async()
-                    if msg is None:
-                        await asyncio.sleep(_FINAL_OUTPUT_IDLE_SLEEP_S)
-                        continue
+                    if event_driven_drain:
+                        msg = await engine.get_output_blocking_async(timeout=_FINAL_OUTPUT_BLOCKING_WAIT_S)
+                        if msg is None:
+                            # Timed out with the orchestrator alive; loop for
+                            # the periodic liveness check.
+                            continue
+                    else:
+                        msg = await engine.try_get_output_async()
+                        if msg is None:
+                            await asyncio.sleep(_FINAL_OUTPUT_IDLE_SLEEP_S)
+                            continue
 
                     if isinstance(msg, dict) and msg.get("type") == "ack":
                         ack_data = msg.get("ack")

vllm_omni/entrypoints/openai/api_server.py

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: Apache-2.0
-# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
 import asyncio
 import base64
 import dataclasses
@@ -64,6 +64,7 @@ from vllm.entrypoints.pooling.embed.serving import ServingEmbedding as OpenAISer
 from vllm.entrypoints.pooling.pooling.serving import ServingPooling
 from vllm.entrypoints.pooling.scoring.serving import ServingScores
 from vllm.entrypoints.scale_out.token_in_token_out.serving import ServingTokens
+from vllm.entrypoints.serve import create_error_response
 
 # vLLM moved `base` from openai.basic.api_router to serve.instrumentator.basic.
 # Keep a fallback for older/newer upstream layouts during rebase windows.
@@ -75,7 +76,6 @@ from vllm.entrypoints.serve.utils.api_utils import (
     validate_json_request,
     with_cancellation,
 )
-from vllm.entrypoints.serve.utils.error_response import create_error_response
 from vllm.entrypoints.serve.utils.orca_metrics import metrics_header
 from vllm.entrypoints.serve.utils.request_logger import RequestLogger
 from vllm.entrypoints.serve.utils.server_utils import get_uvicorn_log_config
@@ -997,11 +997,9 @@ async def omni_init_app_state(
     )
 
     # Warm up chat template processing to avoid first-request latency
-    # Upstream f5ffc59b6a moved warmup onto OnlineRenderer. Accelerator
-    # images can temporarily lag that renderer API, where warmup is optional.
-    renderer_warmup = getattr(state.online_renderer, "warmup", None)
-    if renderer_warmup is not None:
-        renderer_warmup()
+    # Upstream f5ffc59b6a removed OpenAIServingChat.warmup() and moved the
+    # warmup onto the renderer (OnlineRenderer.warmup()); mirror upstream.
+    state.online_renderer.warmup()
 
     state.openai_serving_completion = (
         OpenAIServingCompletion(

vllm_omni/entrypoints/openai/serving_chat.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 import asyncio
 import base64
 import json
@@ -19,9 +22,11 @@ from vllm.entrypoints.chat_utils import (
     ChatCompletionMessageParam,
     ChatTemplateContentFormatOption,
     ConversationMessage,
-    get_history_tool_calls_cnt,
     make_tool_call_id,
 )
+from vllm.parser.utils import (
+    count_chat_history_tool_calls as get_history_tool_calls_cnt,
+)
 
 from vllm_omni.diffusion.utils.param_utils import apply_declared_extra_args
 from vllm_omni.entrypoints.async_omni import AsyncOmni
@@ -86,14 +91,7 @@ from vllm.inputs import PromptType
 from vllm.logger import init_logger
 from vllm.multimodal.media.connector import MediaConnector
 from vllm.outputs import RequestOutput
-
-try:
-    from vllm.parser.mistral import MistralToolCall
-except ImportError:
-    # Accelerator images can lag the core vLLM release by one version.
-    from vllm.tool_parsers.mistral_tool_parser import MistralToolCall
-
-
+from vllm.parser.mistral import MistralToolCall
 from vllm.reasoning import ReasoningParser
 from vllm.renderers import BaseRenderer, merge_kwargs
 from vllm.renderers.inputs import TokPrompt

vllm_omni/experimental/fullduplex/minicpmo45/adapter.py

@@ -14,6 +14,7 @@ from vllm_omni.experimental.fullduplex.minicpmo45.policy import MiniCPMO45Duplex
 from vllm_omni.experimental.fullduplex.openai.protocol import DuplexSessionConfig
 from vllm_omni.experimental.fullduplex.openai.runtime_adapter import (
     ServingRuntimeConfigError,
+    reject_changed_runtime_value,
 )
 
 
@@ -69,7 +70,13 @@ class MiniCPMO45NativeDuplexServingAdapter:
         current: object,
     ) -> dict[str, object]:
         runtime_config = deepcopy(dict(current)) if isinstance(current, dict) else {}
-        runtime_config["instructions"] = config.instructions
+        reject_changed_runtime_value(
+            config.instructions,
+            runtime_config.get("instructions"),
+            message="instructions cannot be changed after the session is created",
+            code="instructions_update_unsupported",
+            error_cls=MiniCPMO45ClientRuntimeConfigError,
+        )
         stage_max_tokens = runtime_config.get("duplex_stage_max_tokens")
         stage_max_tokens = deepcopy(stage_max_tokens) if isinstance(stage_max_tokens, dict) else {}
         stage_max_tokens["0"] = (

vllm_omni/experimental/fullduplex/minicpmo45/session.py

@@ -16,6 +16,7 @@ class MiniCPMO45ServingSessionState:
     audio_buffer: MiniCPMO45PcmAppendBuffer = field(default_factory=MiniCPMO45PcmAppendBuffer)
     input_since_commit: bool = False
     speech_since_commit: bool = False
+    native_context_locked: bool = False
     committed_audio_payload: dict[str, object] | None = None
     committed_audio_operation_id: str | None = None
     committed_audio_reserved_bytes: int = 0

vllm_omni/experimental/fullduplex/nemotron_voicechat/__init__.py

@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+"""Native Full-Duplex integration for NVIDIA NemotronLabs VoiceChat."""

vllm_omni/experimental/fullduplex/nemotron_voicechat/data_plane.py

@@ -0,0 +1,403 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+"""Output projection for Nemotron VoiceChat's independent channels."""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Callable, Iterator, Mapping
+from dataclasses import dataclass, field
+from typing import Any
+from uuid import uuid4
+
+import numpy as np
+import torch
+from vllm.logger import init_logger
+
+from vllm_omni.experimental.fullduplex.engine.contracts import (
+    duplex_resource_request_belongs_to_session,
+)
+from vllm_omni.experimental.fullduplex.output import (
+    get_duplex_output_decision,
+)
+
+logger = init_logger(__name__)
+
+EncodeAudio = Callable[[object, int, str, float | None], str | None]
+
+
+@dataclass(frozen=True, slots=True)
+class NemotronVoiceChatDataPlaneContext:
+    epoch: int = 0
+    turn_id: int = 0
+    auto_responds: bool = True
+    response_format: str = "wav"
+    speed: float | None = None
+    modalities: tuple[str, ...] = ("text", "audio")
+
+
+@dataclass(slots=True)
+class _RequestState:
+    text_frames: int = 0
+    audio_frames: int = 0
+    pending_speech_end_frames: list[int] = field(default_factory=list)
+    function_active: bool = False
+    function_tokens: list[int] = field(default_factory=list)
+    function_call_id: str | None = None
+    terminal: bool = False
+
+
+def _coerce_ints(value: object) -> list[int]:
+    if value is None:
+        return []
+    if isinstance(value, torch.Tensor):
+        value = value.detach().cpu().reshape(-1).tolist()
+    elif isinstance(value, np.ndarray):
+        value = value.reshape(-1).tolist()
+    if isinstance(value, int):
+        return [value]
+    if not isinstance(value, list | tuple):
+        return []
+    values: list[int] = []
+    for item in value:
+        if isinstance(item, torch.Tensor):
+            if item.numel() != 1:
+                continue
+            item = item.item()
+        elif isinstance(item, np.generic):
+            item = item.item()
+        try:
+            values.append(int(item))
+        except (TypeError, ValueError):
+            continue
+    return values
+
+
+def _unwrap(output: object) -> tuple[object, object | None, int | None]:
+    stage_id = getattr(output, "stage_id", None)
+    inner = getattr(output, "request_output", None)
+    if inner is not None and inner is not output:
+        output = inner
+    outputs = getattr(output, "outputs", None)
+    completion = outputs[0] if isinstance(outputs, list) and outputs else None
+    if stage_id is None:
+        stage_id = getattr(output, "stage_id", None)
+    return output, completion, int(stage_id) if isinstance(stage_id, int) else None
+
+
+def _multimodal(output: object, completion: object | None) -> dict[str, object]:
+    decision = get_duplex_output_decision(output)
+    metadata = getattr(decision, "metadata", None)
+    if isinstance(metadata, Mapping):
+        return dict(metadata)
+    for candidate in (
+        getattr(output, "multimodal_output", None),
+        getattr(completion, "multimodal_output", None) if completion is not None else None,
+    ):
+        if isinstance(candidate, Mapping):
+            return dict(candidate)
+    return {}
+
+
+def _audio_value(metadata: Mapping[str, object]) -> object | None:
+    value = next((metadata[key] for key in ("audio", "model_outputs", "latent") if key in metadata), None)
+    if isinstance(value, list) and len(value) == 1:
+        return value[0]
+    return value
+
+
+def _sample_rate(metadata: Mapping[str, object]) -> int:
+    value = metadata.get("sr", metadata.get("sample_rate_hz", 22050))
+    if isinstance(value, list) and value:
+        value = value[0]
+    if hasattr(value, "item"):
+        value = value.item()
+    return int(value) if isinstance(value, int | float) else 22050
+
+
+def _audio_samples(audio: object | None) -> int:
+    if audio is None:
+        return 0
+    if isinstance(audio, torch.Tensor):
+        return int(audio.numel())
+    try:
+        return int(np.asarray(audio, dtype=np.float32).size)
+    except (TypeError, ValueError):
+        return 0
+
+
+def _speech_end_event(request_id: str) -> dict[str, object]:
+    return {
+        "stage_role": "tts",
+        "is_listen": False,
+        "data_plane_request_id": request_id,
+        "text": "",
+        "end_of_turn": True,
+    }
+
+
+class NemotronVoiceChatDataPlaneSession:
+    """Join frame-locked text/function outputs with Stage-2 audio."""
+
+    def __init__(self, encode_audio: EncodeAudio) -> None:
+        self._encode_audio = encode_audio
+        self._requests: dict[str, _RequestState] = {}
+        self._tokenizer = None
+        self._tokenizer_ref: str | None = None
+        self._special_ids: dict[str, int] | None = None
+
+    def configure_runtime(self, runtime_config: Mapping[str, object], *, tokenizer: Any | None = None) -> None:
+        """Install the serving-resolved tokenizer contract as the sole truth."""
+        try:
+            special_ids = {
+                "bos": int(runtime_config["nvc_text_bos_id"]),
+                "eos": int(runtime_config["nvc_text_eos_id"]),
+                "pad": int(runtime_config["nvc_text_pad_id"]),
+                "sotc": int(runtime_config["nvc_function_sotc_id"]),
+                "eotc": int(runtime_config["nvc_function_eotc_id"]),
+            }
+        except (KeyError, TypeError, ValueError) as exc:
+            raise ValueError("Nemotron VoiceChat data plane requires resolved special-token ids") from exc
+        tokenizer_ref = runtime_config.get("nvc_tokenizer_ref")
+        if not isinstance(tokenizer_ref, str) or not tokenizer_ref:
+            raise ValueError("Nemotron VoiceChat data plane requires nvc_tokenizer_ref")
+        self._special_ids = special_ids
+        self._tokenizer_ref = tokenizer_ref
+        self._tokenizer = tokenizer
+
+    def begin_request(self, request_id: str) -> None:
+        self._requests.setdefault(request_id, _RequestState()).terminal = False
+
+    def is_terminal(self, request_id: str | None) -> bool:
+        return bool(request_id and self._requests.get(request_id) and self._requests[request_id].terminal)
+
+    def mark_terminal(self, request_id: str) -> None:
+        self._requests.setdefault(request_id, _RequestState()).terminal = True
+
+    def close_stream(self, request_id: str) -> None:
+        self._requests.pop(request_id, None)
+
+    def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None:
+        if active_request_id:
+            self._requests.pop(active_request_id, None)
+        for request_id in tuple(self._requests):
+            if duplex_resource_request_belongs_to_session(request_id, session_id):
+                self._requests.pop(request_id, None)
+
+    def _load_tokenizer(self):
+        if self._tokenizer is None:
+            from transformers import AutoTokenizer
+
+            if self._tokenizer_ref is None:
+                raise RuntimeError("Nemotron VoiceChat data plane was used before runtime configuration")
+            self._tokenizer = AutoTokenizer.from_pretrained(self._tokenizer_ref, trust_remote_code=False)
+        return self._tokenizer
+
+    def _ids(self) -> dict[str, int]:
+        if self._special_ids is None:
+            raise RuntimeError("Nemotron VoiceChat data plane was used before runtime configuration")
+        return self._special_ids
+
+    def _decode(self, token_ids: list[int]) -> str:
+        if not token_ids:
+            return ""
+        return str(self._load_tokenizer().decode(token_ids, skip_special_tokens=True))
+
+    def project(
+        self,
+        result: object,
+        *,
+        context: NemotronVoiceChatDataPlaneContext | None = None,
+    ) -> Iterator[dict[str, object]]:
+        if not isinstance(result, dict):
+            return
+        outputs = result.get("data_plane_outputs")
+        if not isinstance(outputs, list):
+            return
+        for output in outputs:
+            yield from self.project_output(output, context=context)
+
+    def project_output(
+        self,
+        output: object,
+        *,
+        context: NemotronVoiceChatDataPlaneContext | None = None,
+    ) -> Iterator[dict[str, object]]:
+        context = context or NemotronVoiceChatDataPlaneContext()
+        outer = output
+        output, completion, stage_id = _unwrap(output)
+        request_id = getattr(output, "request_id", None) or getattr(outer, "request_id", None)
+        request_id = str(request_id) if request_id is not None else ""
+        state = self._requests.setdefault(request_id, _RequestState())
+        metadata = _multimodal(outer, completion)
+        modalities = {modality.lower() for modality in context.modalities}
+        wants_text = "text" in modalities
+        wants_audio = "audio" in modalities
+        if stage_id == 0 or "nvc_text_token_ids" in metadata:
+            ids = self._ids()
+            text_ids = _coerce_ints(metadata.get("nvc_text_token_ids"))
+            if not text_ids and completion is not None:
+                text_ids = _coerce_ints(
+                    getattr(completion, "token_ids", None) or getattr(completion, "cumulative_token_ids", None)
+                )
+            for token_id in text_ids[-1:]:
+                state.text_frames += 1
+                if token_id == ids["eos"]:
+                    if state.audio_frames >= state.text_frames:
+                        yield _speech_end_event(request_id)
+                    else:
+                        state.pending_speech_end_frames.append(state.text_frames)
+                elif token_id == ids["pad"]:
+                    yield {
+                        "stage_role": "llm",
+                        "is_listen": True,
+                        "model_listen": True,
+                        "listen_source": "model_listen",
+                        "data_plane_request_id": request_id,
+                        "end_of_turn": False,
+                    }
+                elif token_id != ids["bos"] and wants_text:
+                    text = self._decode([token_id])
+                    if text:
+                        yield {
+                            "stage_role": "llm",
+                            "is_listen": False,
+                            "data_plane_request_id": request_id,
+                            "text": text,
+                            "end_of_turn": False,
+                        }
+
+            for function_id in _coerce_ints(metadata.get("nvc_function_token"))[-1:]:
+                yield from self._project_function_token(
+                    function_id,
+                    state=state,
+                    request_id=request_id,
+                    ids=ids,
+                )
+            return
+
+        audio = _audio_value(metadata)
+        sample_rate = _sample_rate(metadata)
+        sample_count = _audio_samples(audio)
+        encoded = (
+            self._encode_audio(audio, sample_rate, context.response_format, context.speed)
+            if wants_audio and audio is not None
+            else None
+        )
+        audio_chunk_complete = sample_count > 0
+        if audio_chunk_complete:
+            # A native-duplex Stage-2 segment can contain one or more 80 ms
+            # codec frames.  Count them against the frame-locked Stage-0 text
+            # channel. The long-lived codec request deliberately keeps
+            # ``finished=False`` across scheduler wakes; a non-empty decoded
+            # chunk, not request lifetime, is the audio-coverage signal.
+            duration_ms = sample_count * 1000 / max(1, sample_rate)
+            state.audio_frames += max(1, round(duration_ms / 80))
+        end_of_turn = bool(state.pending_speech_end_frames) and state.audio_frames >= state.pending_speech_end_frames[0]
+        if encoded:
+            yield {
+                "stage_role": "tts",
+                "is_listen": False,
+                "data_plane_request_id": request_id,
+                "text": "",
+                "audio_data": encoded,
+                "audio_format": context.response_format,
+                "sample_rate_hz": sample_rate,
+                "audio_duration_ms": round(sample_count * 1000 / max(1, sample_rate)),
+                "audio_text_mark": True,
+                "end_of_turn": end_of_turn,
+            }
+        elif end_of_turn:
+            yield _speech_end_event(request_id)
+        if end_of_turn:
+            state.pending_speech_end_frames.pop(0)
+
+    def _project_function_token(
+        self,
+        token_id: int,
+        *,
+        state: _RequestState,
+        request_id: str,
+        ids: Mapping[str, int],
+    ) -> Iterator[dict[str, object]]:
+        if token_id == ids["sotc"]:
+            state.function_active = True
+            state.function_tokens.clear()
+            state.function_call_id = f"call_{uuid4().hex}"
+            return
+        if token_id == ids["eotc"] and state.function_active:
+            state.function_active = False
+            raw = self._decode(state.function_tokens)
+            state.function_tokens.clear()
+            try:
+                calls = self._parse_calls(raw)
+            except ValueError as exc:
+                state.function_call_id = None
+                yield {
+                    "stage_role": "function",
+                    "data_plane_request_id": request_id,
+                    "error_code": "nemotron_function_call_parse_error",
+                    "error": str(exc),
+                }
+                return
+            for index, call in enumerate(calls):
+                yield {
+                    "stage_role": "function",
+                    "data_plane_request_id": request_id,
+                    "function_call": True,
+                    "call_id": (
+                        state.function_call_id if index == 0 and state.function_call_id else f"call_{uuid4().hex}"
+                    ),
+                    "name": str(call["name"]),
+                    "arguments": (
+                        call.get("arguments")
+                        if isinstance(call.get("arguments"), str)
+                        else json.dumps(call.get("arguments", {}), separators=(",", ":"))
+                    ),
+                }
+            state.function_call_id = None
+            return
+        if state.function_active and token_id != ids["pad"]:
+            state.function_tokens.append(token_id)
+
+    @staticmethod
+    def _parse_calls(value: str) -> list[dict[str, object]]:
+        text = value.strip()
+        if "<TOOLCALL>" in text:
+            text = text.split("<TOOLCALL>", 1)[1]
+        if "</TOOLCALL>" in text:
+            text = text.split("</TOOLCALL>", 1)[0]
+        try:
+            parsed = json.loads(text)
+        except json.JSONDecodeError as exc:
+            raise ValueError("Nemotron VoiceChat produced malformed function-call JSON") from exc
+        if isinstance(parsed, dict):
+            parsed = [parsed]
+        if not isinstance(parsed, list) or not parsed:
+            raise ValueError("Nemotron VoiceChat function-call payload must be a non-empty object or list")
+        calls: list[dict[str, object]] = []
+        for item in parsed:
+            if not isinstance(item, dict):
+                raise ValueError("Nemotron VoiceChat function-call entries must be objects")
+            name = item.get("name")
+            if not isinstance(name, str) or not name.strip():
+                raise ValueError("Nemotron VoiceChat function-call entries require a non-empty name")
+            arguments = item.get("arguments", {})
+            if isinstance(arguments, str):
+                try:
+                    decoded_arguments = json.loads(arguments)
+                except json.JSONDecodeError as exc:
+                    raise ValueError("Nemotron VoiceChat function-call arguments contain malformed JSON") from exc
+                if not isinstance(decoded_arguments, dict):
+                    raise ValueError("Nemotron VoiceChat function-call arguments must encode a JSON object")
+            elif not isinstance(arguments, dict):
+                raise ValueError("Nemotron VoiceChat function-call arguments must be a JSON object")
+            calls.append({"name": name.strip(), "arguments": arguments})
+        return calls
+
+
+__all__ = [
+    "NemotronVoiceChatDataPlaneContext",
+    "NemotronVoiceChatDataPlaneSession",
+]

vllm_omni/experimental/fullduplex/nemotron_voicechat/input.py

@@ -0,0 +1,225 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+"""80 ms PCM packetization for Nemotron VoiceChat native duplex."""
+
+from __future__ import annotations
+
+import binascii
+
+import numpy as np
+import pybase64 as base64
+
+NEMOTRON_VOICECHAT_SAMPLE_RATE = 16000
+NEMOTRON_VOICECHAT_FRAME_SAMPLES = 1280
+_SAMPLE_BYTES = 4
+_FRAME_BYTES = NEMOTRON_VOICECHAT_FRAME_SAMPLES * _SAMPLE_BYTES
+
+
+def _require_terminal_tail(buffered_bytes: int) -> None:
+    if buffered_bytes > _FRAME_BYTES:
+        raise ValueError(
+            "Nemotron VoiceChat manual/deferred commit cannot contain more than one 80 ms frame; "
+            "enable extra_body.auto_response for native full-duplex streaming"
+        )
+
+
+def decode_pcm_f32le(payload: object, *, exact_frame: bool = False) -> bytes:
+    if not isinstance(payload, dict):
+        raise ValueError("Nemotron VoiceChat duplex audio payload must be a mapping")
+    if payload.get("format") != "pcm_f32le":
+        raise ValueError("Nemotron VoiceChat duplex audio format must be pcm_f32le")
+    if payload.get("sample_rate_hz") != NEMOTRON_VOICECHAT_SAMPLE_RATE:
+        raise ValueError("Nemotron VoiceChat duplex audio sample_rate_hz must be 16000")
+    encoded = payload.get("audio")
+    if not isinstance(encoded, str):
+        raise ValueError("Nemotron VoiceChat duplex audio must be base64 pcm_f32le")
+    try:
+        raw = base64.b64decode(encoded, validate=True)
+    except (binascii.Error, ValueError) as exc:
+        raise ValueError("Nemotron VoiceChat duplex audio is not valid base64") from exc
+    if len(raw) % _SAMPLE_BYTES:
+        raise ValueError("Nemotron VoiceChat pcm_f32le payload has a partial sample")
+    values = np.frombuffer(raw, dtype="<f4")
+    if values.size and not bool(np.isfinite(values).all()):
+        raise ValueError("Nemotron VoiceChat duplex audio samples must be finite")
+    if exact_frame and len(raw) != _FRAME_BYTES:
+        raise ValueError(
+            f"Nemotron VoiceChat native duplex append must contain exactly {NEMOTRON_VOICECHAT_FRAME_SAMPLES} samples"
+        )
+    return raw
+
+
+def _frame_payload(raw: bytes, *, final: bool) -> dict[str, object]:
+    return {
+        "type": "audio",
+        "audio": base64.b64encode(raw).decode("ascii"),
+        "format": "pcm_f32le",
+        "sample_rate_hz": NEMOTRON_VOICECHAT_SAMPLE_RATE,
+        "final": final,
+    }
+
+
+class NemotronVoiceChatPcmAppendReservation:
+    def __init__(
+        self,
+        owner: NemotronVoiceChatPcmAppendBuffer,
+        *,
+        operation_id: str,
+        payload: dict[str, object] | None,
+        raw: bytes,
+    ) -> None:
+        self._owner = owner
+        self.operation_id = operation_id
+        self.payload = payload
+        self.raw = raw
+        self._active = True
+
+    @property
+    def active(self) -> bool:
+        return self._active
+
+    @property
+    def byte_count(self) -> int:
+        return len(self.raw)
+
+    def commit(self) -> None:
+        if not self._active:
+            return
+        self._active = False
+        if self in self._owner._reservations:
+            self._owner._reservations.remove(self)
+
+    def rollback(self) -> None:
+        if not self._active:
+            return
+        try:
+            index = self._owner._reservations.index(self)
+        except ValueError:
+            self._active = False
+            return
+        restore = bytearray()
+        for reservation in self._owner._reservations[index:]:
+            if reservation._active:
+                restore.extend(reservation.raw)
+                reservation._active = False
+        del self._owner._reservations[index:]
+        self._owner._buffer[:0] = restore
+
+
+class NemotronVoiceChatPcmAppendBuffer:
+    """Adapt Realtime packets to the current full-duplex PCM buffer contract.
+
+    The model consumes exactly one 1280-sample frame per scheduler append.
+    Realtime packets may split a frame arbitrarily, but a single packet is
+    capped at one frame so the serving contract never silently drops a second
+    ready frame behind a single-reservation API.
+    """
+
+    def __init__(self) -> None:
+        self._buffer = bytearray()
+        self._reservations: list[NemotronVoiceChatPcmAppendReservation] = []
+
+    @property
+    def pending_byte_count(self) -> int:
+        return len(self._buffer)
+
+    def clear(self) -> None:
+        for reservation in self._reservations:
+            reservation._active = False
+        self._reservations.clear()
+        self._buffer.clear()
+
+    def clear_force_listen(self) -> None:
+        # Nemotron VoiceChat has no serving-side force-listen packet bit.
+        return
+
+    def has_pending(self) -> bool:
+        return bool(self._buffer)
+
+    def has_reserved(self) -> bool:
+        return any(reservation.active for reservation in self._reservations)
+
+    def _reserve_frame(
+        self,
+        *,
+        operation_id: str,
+        final: bool,
+    ) -> NemotronVoiceChatPcmAppendReservation | None:
+        if len(self._buffer) < _FRAME_BYTES:
+            return None
+        raw = bytes(self._buffer[:_FRAME_BYTES])
+        del self._buffer[:_FRAME_BYTES]
+        reservation = NemotronVoiceChatPcmAppendReservation(
+            self,
+            operation_id=operation_id,
+            payload=_frame_payload(raw, final=final),
+            raw=raw,
+        )
+        self._reservations.append(reservation)
+        return reservation
+
+    def prepare_append(
+        self,
+        payload: dict[str, object],
+        *,
+        operation_id: str,
+        chunk_period_ms: int,
+        allow_emit: bool,
+    ) -> NemotronVoiceChatPcmAppendReservation | None:
+        if chunk_period_ms != 80:
+            raise ValueError("Nemotron VoiceChat native duplex chunk_period_ms must be 80")
+        raw = decode_pcm_f32le(payload)
+        if len(raw) > _FRAME_BYTES:
+            raise ValueError("Nemotron VoiceChat Realtime append must contain at most 1280 samples")
+        self._buffer.extend(raw)
+        if not allow_emit:
+            return None
+        return self._reserve_frame(operation_id=operation_id, final=False)
+
+    def prepare_commit(
+        self,
+        *,
+        operation_id: str,
+        chunk_period_ms: int,
+    ) -> NemotronVoiceChatPcmAppendReservation:
+        if chunk_period_ms != 80:
+            raise ValueError("Nemotron VoiceChat native duplex chunk_period_ms must be 80")
+        _require_terminal_tail(len(self._buffer))
+        if not self._buffer:
+            reservation = NemotronVoiceChatPcmAppendReservation(
+                self,
+                operation_id=operation_id,
+                payload=None,
+                raw=b"",
+            )
+        else:
+            raw = bytes(self._buffer)
+            self._buffer.clear()
+            reservation = NemotronVoiceChatPcmAppendReservation(
+                self,
+                operation_id=operation_id,
+                payload=_frame_payload(raw + bytes(_FRAME_BYTES - len(raw)), final=True),
+                raw=raw,
+            )
+        self._reservations.append(reservation)
+        return reservation
+
+    def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
+        if chunk_period_ms != 80:
+            raise ValueError("Nemotron VoiceChat native duplex chunk_period_ms must be 80")
+        _require_terminal_tail(len(self._buffer))
+        if not self._buffer:
+            return None
+        raw = bytes(self._buffer)
+        self._buffer.clear()
+        return _frame_payload(raw + bytes(_FRAME_BYTES - len(raw)), final=True)
+
+
+__all__ = [
+    "NEMOTRON_VOICECHAT_FRAME_SAMPLES",
+    "NEMOTRON_VOICECHAT_SAMPLE_RATE",
+    "NemotronVoiceChatPcmAppendBuffer",
+    "NemotronVoiceChatPcmAppendReservation",
+    "decode_pcm_f32le",
+]

vllm_omni/experimental/fullduplex/nemotron_voicechat/runtime.py

@@ -0,0 +1,167 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+
+"""Engine policy for Nemotron VoiceChat's frame-locked duplex timeline."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from vllm.sampling_params import RequestOutputKind, SamplingParams
+
+from vllm_omni.experimental.fullduplex.engine.contracts import (
+    DuplexAppendPlan,
+    DuplexInputMode,
+    DuplexOutputAction,
+    DuplexOutputDecision,
+)
+from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
+from vllm_omni.experimental.fullduplex.nemotron_voicechat.input import (
+    decode_pcm_f32le,
+)
+
+
+def _plain_token_ids(value: object, *, name: str) -> list[int]:
+    if not isinstance(value, list | tuple) or not value:
+        raise ValueError(f"Nemotron VoiceChat runtime requires non-empty {name}")
+    try:
+        return [int(token_id) for token_id in value]
+    except (TypeError, ValueError) as exc:
+        raise ValueError(f"Nemotron VoiceChat {name} must contain token ids") from exc
+
+
+def _positive_int(value: object, *, name: str) -> int:
+    try:
+        parsed = int(value)
+    except (TypeError, ValueError) as exc:
+        raise ValueError(f"Nemotron VoiceChat requires a positive {name}") from exc
+    if parsed <= 0:
+        raise ValueError(f"Nemotron VoiceChat requires a positive {name}")
+    return parsed
+
+
+class NemotronVoiceChatDuplexRuntimeExtension:
+    """Append one 80 ms acoustic frame to one resumable thinker request."""
+
+    def configure_sampling_params(
+        self,
+        *,
+        runtime_config: dict[str, Any],
+        defaults: tuple[object, ...],
+    ) -> tuple[object, ...]:
+        del runtime_config
+        if not defaults:
+            return defaults
+        # Every scheduler segment is one transport wake of a long-lived
+        # request.  CUMULATIVE output makes the multimodal output processor
+        # concatenate all earlier Stage-2 waveforms and resend them on every
+        # wake (80, 160, 240, ... ms), causing O(n^2) audio replay.  Duplex
+        # consumers require per-wake deltas at every stage.
+        configured = [params.clone() if isinstance(params, SamplingParams) else params for params in defaults]
+        for params in configured:
+            if isinstance(params, SamplingParams):
+                params.output_kind = RequestOutputKind.DELTA
+        stage0 = defaults[0]
+        if isinstance(stage0, SamplingParams):
+            stage0 = stage0.clone()
+            stage0.temperature = 0.0
+            stage0.top_p = 1.0
+            stage0.top_k = 0
+            stage0.max_tokens = 1
+            stage0.ignore_eos = True
+            stage0.output_kind = RequestOutputKind.DELTA
+            configured[0] = stage0
+        return tuple(configured)
+
+    def plan_append(
+        self,
+        *,
+        request_id: str,
+        fence: DuplexFence,
+        session_config: dict[str, Any],
+        runtime_config: dict[str, Any],
+        seq: int,
+        turn_seq: int,
+        mode: DuplexInputMode,
+        payload: object,
+        final: bool,
+        sampling_params: object,
+    ) -> DuplexAppendPlan:
+        del sampling_params, turn_seq
+        if mode is not DuplexInputMode.APPEND_AUDIO_CHUNK:
+            raise ValueError(f"Nemotron VoiceChat does not support duplex input mode {mode.value!r}")
+        decode_pcm_f32le(payload, exact_frame=True)
+        normalized_payload = dict(payload)
+        prompt_ids = _plain_token_ids(
+            runtime_config.get("nvc_prompt_token_ids"),
+            name="nvc_prompt_token_ids",
+        )
+        max_model_len = _positive_int(
+            runtime_config.get("nvc_max_model_len"),
+            name="nvc_max_model_len",
+        )
+        required_model_len = len(prompt_ids) + seq + 1
+        if required_model_len > max_model_len:
+            raise ValueError(
+                "Nemotron VoiceChat native duplex session exceeds the Stage-0 "
+                f"max_model_len: prompt_tokens={len(prompt_ids)} + input_frames={seq} + "
+                f"sampled_token=1 gives {required_model_len} > {max_model_len}; "
+                "start a new session or raise Stage 0 max_model_len"
+            )
+        try:
+            pad_id = int(runtime_config["nvc_text_pad_id"])
+        except (KeyError, TypeError, ValueError) as exc:
+            raise ValueError("Nemotron VoiceChat runtime requires nvc_text_pad_id") from exc
+        scheduler_prompt = prompt_ids + [pad_id] if seq <= 1 else [pad_id]
+        return DuplexAppendPlan(
+            prompt={
+                "prompt_token_ids": scheduler_prompt,
+                "model_intermediate_buffer": {
+                    "request_id": request_id,
+                    "global_request_id": [fence.session_id],
+                    "duplex": {
+                        "data_plane": True,
+                        "fence": fence,
+                        "session_id": fence.session_id,
+                        "incarnation": fence.incarnation,
+                        "epoch": fence.epoch,
+                        "source_input_seq": seq,
+                        "seq": seq,
+                        "mode": mode.value,
+                        "payload": normalized_payload,
+                        "final": final,
+                        "session_config": dict(session_config),
+                        "runtime_config": dict(runtime_config),
+                        "scheduler_token_budget": len(scheduler_prompt),
+                    },
+                },
+            }
+        )
+
+    def decide_output(
+        self,
+        *,
+        stage_id: int,
+        final_stage_id: int,
+        segment_finished: bool,
+        segment_token_ids: tuple[int, ...],
+        segment_output_metadata: dict[str, Any],
+        output: object,
+    ) -> DuplexOutputDecision | None:
+        del final_stage_id, segment_finished, output
+        if stage_id != 0:
+            return None
+        metadata = dict(segment_output_metadata)
+        metadata["nvc_text_token_ids"] = list(segment_token_ids)
+        # Stage 0 is a client-visible side channel even though Stage 2 is the
+        # configured audio response stage. Mark it explicitly so the shared
+        # collector does not discard it while waiting for the final stage.
+        metadata["duplex_direct_response"] = True
+        return DuplexOutputDecision(
+            action=DuplexOutputAction.DIRECT_RESPONSE,
+            metadata=metadata,
+            final_output_type="text",
+        )
+
+
+__all__ = ["NemotronVoiceChatDuplexRuntimeExtension"]

vllm_omni/experimental/fullduplex/nemotron_voicechat/serving_adapter.py

@@ -0,0 +1,363 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
+"""Realtime serving adapter for Nemotron VoiceChat native duplex."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+from collections.abc import Callable, Mapping
+from copy import deepcopy
+from typing import Any
+
+from vllm_omni.experimental.fullduplex.minicpmo45.session import (
+    MiniCPMO45ServingSessionState,
+)
+from vllm_omni.experimental.fullduplex.nemotron_voicechat.data_plane import (
+    NemotronVoiceChatDataPlaneContext,
+    NemotronVoiceChatDataPlaneSession,
+)
+from vllm_omni.experimental.fullduplex.nemotron_voicechat.input import (
+    NemotronVoiceChatPcmAppendBuffer,
+)
+from vllm_omni.experimental.fullduplex.openai.protocol import DuplexCapabilities
+from vllm_omni.experimental.fullduplex.openai.runtime_adapter import ServingRuntimeConfigError
+
+EncodeAudio = Callable[[object, int, str, float | None], str | None]
+
+_DEFAULT_SYSTEM_PROMPT = (
+    "You are an AI voice assistant developed by NVIDIA. "
+    "Your name is NVIDIA Voice Chat. "
+    "Answer in a spoken, conversational style rather than a written one. "
+    "Do not repeat the same sentence over and over again. "
+    "Start the conversation by greeting the user."
+)
+_PRIVATE_KEYS = frozenset(
+    {
+        "nvc_prompt_token_ids",
+        "nvc_text_bos_id",
+        "nvc_text_eos_id",
+        "nvc_text_pad_id",
+        "nvc_function_sotc_id",
+        "nvc_function_eotc_id",
+        "nvc_function_eotr_id",
+        "nvc_max_model_len",
+        "nvc_tokenizer_ref",
+        "nvc_tools_signature",
+        "nvc_function_response_generation",
+        "nvc_function_response_token_ids",
+        "nvc_function_response_call_id",
+        "nvc_function_response_batches",
+    }
+)
+
+
+def _stt_config(model_config: Any) -> dict[str, Any]:
+    hf_config = getattr(model_config, "hf_config", None)
+    stt_cfg = getattr(hf_config, "stt_cfg", None)
+    if not isinstance(stt_cfg, dict):
+        raise ServingRuntimeConfigError("Nemotron VoiceChat checkpoint STT configuration is unavailable")
+    return stt_cfg
+
+
+def _normalized_tools(config: object) -> tuple[list[dict[str, object]], str]:
+    extra_body = getattr(config, "extra_body", None)
+    raw_tools = extra_body.get("realtime_tools") if isinstance(extra_body, dict) else None
+    if raw_tools is None:
+        return [], "[]"
+    if not isinstance(raw_tools, list):
+        raise ServingRuntimeConfigError("Nemotron VoiceChat tools must be a list")
+    if len(raw_tools) > 5:
+        raise ServingRuntimeConfigError("Nemotron VoiceChat supports at most 5 tools per session")
+
+    normalized: list[dict[str, object]] = []
+    for index, tool in enumerate(raw_tools):
+        if not isinstance(tool, dict):
+            raise ServingRuntimeConfigError(f"Nemotron VoiceChat tool {index} must be an object")
+        function = tool.get("function", tool)
+        if not isinstance(function, dict):
+            raise ServingRuntimeConfigError(f"Nemotron VoiceChat tool {index} has no function definition")
+        definition = {key: value for key, value in function.items() if key != "type"}
+        if not isinstance(definition.get("name"), str) or not str(definition["name"]).strip():
+            raise ServingRuntimeConfigError(f"Nemotron VoiceChat tool {index} requires a name")
+        normalized.append(definition)
+    signature = json.dumps(normalized, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
+    return normalized, signature
+
+
+def _render_tool_prompt(instructions: str, tools: list[dict[str, object]]) -> str:
+    if not tools:
+        return instructions
+    available = json.dumps(tools, ensure_ascii=False, sort_keys=True)
+    return (
+        f"{instructions}\n\n"
+        "You can use the following tools to assist the user if required:"
+        f"\n<AVAILABLE_TOOLS>{available}</AVAILABLE_TOOLS>\n\n"
+        "If you decide to call any tool(s), use the following format:\n"
+        '<TOOLCALL>[{"name": "tool_name1", "arguments": "tool_args1"}, '
+        '{"name": "tool_name2", "arguments": "tool_args2"}]</TOOLCALL>\n\n'
+        "The user will execute tool-calls and return responses from tool(s) in this format:\n"
+        '<TOOL_RESPONSE>[{"tool_response1"}, {"tool_response2"}]</TOOL_RESPONSE>\n\n'
+        "Based on the tool responses, you can call additional tools if needed, correct tool calls if any "
+        "errors are found, or just respond to the user."
+    )
+
+
+def _render_tool_response(output: str) -> str:
+    """Serialize one Realtime function result using NVIDIA's function channel."""
+    try:
+        value = json.loads(output)
+    except json.JSONDecodeError:
+        value = output
+    payload = json.dumps([value], ensure_ascii=True, separators=(",", ":"))
+    return f"<TOOL_RESPONSE>{payload}</TOOL_RESPONSE>"
+
+
+def _require_native_full_duplex(config: object) -> None:
+    extra_body = getattr(config, "extra_body", None)
+    enabled = isinstance(extra_body, dict) and (
+        extra_body.get("auto_response") is True or extra_body.get("full_duplex") is True
+    )
+    if not enabled:
+        raise ServingRuntimeConfigError(
+            "Nemotron VoiceChat currently supports model-native full-duplex streaming only; "
+            "set extra_body.auto_response=true",
+            code="unsupported_nemotron_duplex_mode",
+        )
+
+
+def _tokenize_runtime(model_config: Any, instructions: str) -> tuple[dict[str, object], Any]:
+    from transformers import AutoTokenizer
+
+    stt_cfg = _stt_config(model_config)
+    tokenizer_ref = os.environ.get("NEMOTRON_VOICECHAT_LLM_PATH") or stt_cfg.get(
+        "pretrained_llm", "nvidia/NVIDIA-Nemotron-Nano-9B-v2"
+    )
+    tokenizer = AutoTokenizer.from_pretrained(tokenizer_ref, trust_remote_code=False)
+
+    def token(name: str, default: str) -> int:
+        value = tokenizer.convert_tokens_to_ids(stt_cfg.get(name, default))
+        if value is None:
+            raise ServingRuntimeConfigError(f"Nemotron VoiceChat tokenizer does not define {name}")
+        return int(value)
+
+    bos_id = token("bos_token", "<s>")
+    eos_id = token("eos_token", "</s>")
+    pad_id = token("pad_token", "<SPECIAL_12>")
+    prompt_ids = [bos_id] + list(tokenizer.encode(instructions, add_special_tokens=False)) + [eos_id]
+    runtime = {
+        "instructions": instructions,
+        "nvc_prompt_token_ids": prompt_ids,
+        "nvc_text_bos_id": bos_id,
+        "nvc_text_eos_id": eos_id,
+        "nvc_text_pad_id": pad_id,
+        "nvc_function_sotc_id": int(tokenizer.convert_tokens_to_ids("<SPECIAL_20>")),
+        "nvc_function_eotc_id": int(tokenizer.convert_tokens_to_ids("<SPECIAL_21>")),
+        "nvc_function_eotr_id": int(tokenizer.convert_tokens_to_ids("<SPECIAL_22>")),
+        "nvc_tokenizer_ref": str(tokenizer_ref),
+    }
+    return runtime, tokenizer
+
+
+class NemotronVoiceChatServingRuntimeAdapter:
+    adapter_id = "nemotron_voicechat"
+    silence_continuation_samples = 1280
+    # The session-owned persistent drain is the sole data-plane output
+    # consumer. Collecting on append would race that drain and can consume a
+    # Stage2 audio output before the realtime projector observes it.
+    collect_outputs_on_append = False
+    clean_response_done_prefix = ""
+    interrupted_tts_prefix = ""
+    private_runtime_config_keys = _PRIVATE_KEYS
+
+    def __init__(self, encode_audio: EncodeAudio) -> None:
+        self.session_states: dict[str, MiniCPMO45ServingSessionState] = {}
+        self.data_plane = NemotronVoiceChatDataPlaneSession(encode_audio)
+        self._tokenizer: Any | None = None
+
+    def create_session_state(self) -> MiniCPMO45ServingSessionState:
+        return MiniCPMO45ServingSessionState(
+            audio_buffer=NemotronVoiceChatPcmAppendBuffer(),
+        )
+
+    def session_state(self, session_id: str) -> MiniCPMO45ServingSessionState:
+        return self.session_states.setdefault(session_id, self.create_session_state())
+
+    def remove_session_state(self, session_id: str) -> None:
+        self.session_states.pop(session_id, None)
+
+    @staticmethod
+    def is_enabled(config: object) -> bool:
+        del config
+        return True
+
+    @staticmethod
+    def capabilities(*, max_sessions: int) -> DuplexCapabilities:
+        supports_multi_session = max_sessions > 1
+        return DuplexCapabilities(
+            supports_model_native_turn_policy=True,
+            supports_external_turn_signal=False,
+            supports_client_commit=True,
+            supports_barge_in=False,
+            supports_playback_ack=True,
+            supports_input_append=True,
+            supports_replace_latest_chunk=False,
+            supports_reencode_context=False,
+            supports_rollback_to_checkpoint=False,
+            supports_turn_commit_only=False,
+            supports_kv_lease=False,
+            supports_core_kv_lease=False,
+            supports_model_internal_state=True,
+            supports_stage_resumption=True,
+            supports_scheduler_native_append=False,
+            supports_core_resumable_request=True,
+            supports_stage_connector_handoff=True,
+            supports_independent_io_streams=True,
+            supports_realtime_endpoint=True,
+            supports_multi_session=supports_multi_session,
+            supports_multi_session_same_replica=False,
+            supports_session_lease=True,
+            supports_session_resume=True,
+            session_admission_mode="engine_managed",
+            supports_audio_truncate=False,
+            requires_model_runner_kv=True,
+            requires_native_stage_role=True,
+            implementation_level="model_native_duplex",
+            adapter_patterns=["scheduler_data_plane"],
+            input_modes=["append_audio_chunk"],
+            signal_sources=["model_native", "client_event"],
+            stage_handoff_transport="scheduler_data_plane",
+            chunk_period_ms=80,
+            target_barge_in_latency_ms=None,
+        )
+
+    @staticmethod
+    def validate_client_extra_body(extra_body: object) -> None:
+        if not isinstance(extra_body, dict):
+            return
+        private = sorted(_PRIVATE_KEYS.intersection(extra_body))
+        if private:
+            raise ServingRuntimeConfigError(
+                "Nemotron VoiceChat runtime configuration is server-owned: " + ", ".join(private)
+            )
+
+    async def prepare_runtime_config(
+        self,
+        config: object,
+        *,
+        model_config: Any,
+    ) -> dict[str, object]:
+        self.validate_client_extra_body(getattr(config, "extra_body", None))
+        _require_native_full_duplex(config)
+        instructions = str(getattr(config, "instructions", None) or _DEFAULT_SYSTEM_PROMPT)
+        tools, tools_signature = _normalized_tools(config)
+        rendered_prompt = _render_tool_prompt(instructions, tools)
+        runtime, tokenizer = await asyncio.to_thread(_tokenize_runtime, model_config, rendered_prompt)
+        max_model_len = getattr(model_config, "max_model_len", None)
+        if not isinstance(max_model_len, int) or max_model_len <= 0:
+            raise ServingRuntimeConfigError("Nemotron VoiceChat requires a positive Stage-0 max_model_len")
+        runtime["nvc_max_model_len"] = max_model_len
+        # Keep raw session values for immutable-in-incarnation update checks;
+        # only the rendered prompt token ids enter Stage-0 KV.
+        runtime["instructions"] = instructions
+        runtime["nvc_tools_signature"] = tools_signature
+        self.data_plane.configure_runtime(runtime, tokenizer=tokenizer)
+        self._tokenizer = tokenizer
+        return runtime
+
+    def runtime_config_for_function_output(
+        self,
+        item: Mapping[str, object],
+        current: Mapping[str, object],
+    ) -> dict[str, object]:
+        """Queue a client tool result for frame-locked function-channel injection."""
+        call_id = item.get("call_id")
+        output = item.get("output")
+        if not isinstance(call_id, str) or not call_id:
+            raise ServingRuntimeConfigError(
+                "function_call_output requires call_id",
+                code="invalid_function_call_output",
+            )
+        if not isinstance(output, str):
+            raise ServingRuntimeConfigError(
+                "function_call_output requires a string output",
+                code="invalid_function_call_output",
+            )
+        if self._tokenizer is None:
+            raise ServingRuntimeConfigError(
+                "Nemotron VoiceChat tokenizer is unavailable for function output",
+                code="invalid_function_call_output",
+            )
+        response_token_ids = list(
+            self._tokenizer.encode(
+                _render_tool_response(output),
+                add_special_tokens=False,
+            )
+        )
+        if not response_token_ids:
+            raise ServingRuntimeConfigError(
+                "function_call_output tokenized to an empty response",
+                code="invalid_function_call_output",
+            )
+        runtime = deepcopy(dict(current))
+        generation = int(runtime.get("nvc_function_response_generation", 0)) + 1
+        token_ids = [int(token_id) for token_id in response_token_ids]
+        batches = runtime.get("nvc_function_response_batches")
+        if not isinstance(batches, list):
+            batches = []
+        batches = [*batches, {"generation": generation, "call_id": call_id, "token_ids": token_ids}]
+        runtime["nvc_function_response_generation"] = generation
+        runtime["nvc_function_response_token_ids"] = token_ids
+        runtime["nvc_function_response_call_id"] = call_id
+        runtime["nvc_function_response_batches"] = batches
+        return runtime
+
+    @staticmethod
+    def runtime_config_for_update(
+        config: object,
+        current: Mapping[str, object],
+    ) -> dict[str, object]:
+        NemotronVoiceChatServingRuntimeAdapter.validate_client_extra_body(getattr(config, "extra_body", None))
+        _require_native_full_duplex(config)
+        runtime = deepcopy(dict(current))
+        instructions = str(getattr(config, "instructions", None) or _DEFAULT_SYSTEM_PROMPT)
+        _, tools_signature = _normalized_tools(config)
+        # The system/tool prompt is already resident in Stage-0 KV. Updating it
+        # without a new incarnation would make the advertised config disagree
+        # with model state, so reject such updates explicitly.
+        if runtime and instructions != runtime.get("instructions"):
+            raise ServingRuntimeConfigError(
+                "Nemotron VoiceChat instructions cannot change inside an active duplex incarnation"
+            )
+        if runtime and tools_signature != runtime.get("nvc_tools_signature", "[]"):
+            raise ServingRuntimeConfigError(
+                "Nemotron VoiceChat tools cannot change inside an active duplex incarnation"
+            )
+        return runtime
+
+    @staticmethod
+    def data_plane_context(
+        *,
+        epoch: int,
+        turn_id: int,
+        active_response_turn_id: int | None,
+        active_response_id: str | None,
+        auto_responds: bool,
+        response_format: str,
+        speed: float | None,
+        modalities: tuple[str, ...],
+    ) -> NemotronVoiceChatDataPlaneContext:
+        del active_response_turn_id, active_response_id
+        return NemotronVoiceChatDataPlaneContext(
+            epoch=epoch,
+            turn_id=turn_id,
+            auto_responds=auto_responds,
+            response_format=response_format,
+            speed=speed,
+            modalities=modalities,
+        )
+
+
+__all__ = ["NemotronVoiceChatServingRuntimeAdapter"]

vllm_omni/experimental/fullduplex/openai/realtime_input.py

@@ -1,3 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
+
 from __future__ import annotations
 
 import asyncio
@@ -1052,6 +1055,46 @@ class RealtimeInputTranslator:
         item_id = str(item["id"])
         item_type = item.get("type")
         role = item.get("role")
+        if item_type == "function_call_output":
+            call_id = item.get("call_id")
+            output = item.get("output")
+            matching_call = any(
+                known.get("type") == "function_call" and known.get("call_id") == call_id
+                for known in self._conversation_items.values()
+            )
+            duplicate_output = any(
+                known.get("type") == "function_call_output" and known.get("call_id") == call_id
+                for known in self._conversation_items.values()
+            )
+            if not isinstance(call_id, str) or not call_id or not matching_call:
+                await self._send_realtime_payload(
+                    self._realtime_error_payload(
+                        "invalid_function_call_output",
+                        "function_call_output requires the call_id of a completed function call",
+                        event_id=event.get("event_id"),
+                        param="item.call_id",
+                    )
+                )
+                return None
+            if not isinstance(output, str) or duplicate_output:
+                await self._send_realtime_payload(
+                    self._realtime_error_payload(
+                        "invalid_function_call_output",
+                        (
+                            "function_call_output requires a string output"
+                            if not isinstance(output, str)
+                            else f"function_call_output already exists for call_id {call_id}"
+                        ),
+                        event_id=event.get("event_id"),
+                        param="item.output",
+                    )
+                )
+                return None
+            # Do not retain the result until SessionRunner has delivered its
+            # runtime update. On success, the resulting
+            # ``conversation.item.created`` event records it in the shared
+            # input/output protocol state; on failure the client may retry the
+            # same call_id instead of being rejected by provisional history.
         if item_type != "message" or role in {"assistant", "system"}:
             return {
                 "type": "turn.signal",

vllm_omni/experimental/fullduplex/openai/realtime_output.py

@@ -346,6 +346,67 @@ class RealtimeOutputProjector:
                     "reason": event.get("finish_reason") or "stop",
                 },
             )
+        if event_type == "function_call.done":
+            call_id = event.get("call_id")
+            name = event.get("name")
+            arguments = event.get("arguments", "")
+            if not isinstance(call_id, str) or not call_id or not isinstance(name, str) or not name:
+                return [{"type": "duplex.function_call.done", "event": event}]
+            if not isinstance(arguments, str):
+                arguments = str(arguments)
+            response_id = f"resp_{uuid4().hex}"
+            item = {
+                "id": f"item_{uuid4().hex}",
+                "object": "realtime.item",
+                "type": "function_call",
+                "status": "completed",
+                "name": name,
+                "call_id": call_id,
+                "arguments": arguments,
+            }
+            self._conversation_items[str(item["id"])] = item
+            response = {
+                "id": response_id,
+                "object": "realtime.response",
+                "status": "completed",
+                "status_details": {"type": "completed", "reason": "completed"},
+                "output": [dict(item)],
+                "metadata": {},
+            }
+            return [
+                {"type": "response.created", "response": {**response, "status": "in_progress", "output": []}},
+                *self._conversation_item_added_events(item),
+                {
+                    "type": "response.output_item.added",
+                    "response_id": response_id,
+                    "output_index": 0,
+                    "item": dict(item),
+                },
+                {
+                    "type": "response.function_call_arguments.delta",
+                    "response_id": response_id,
+                    "item_id": item["id"],
+                    "output_index": 0,
+                    "call_id": call_id,
+                    "delta": arguments,
+                },
+                {
+                    "type": "response.function_call_arguments.done",
+                    "response_id": response_id,
+                    "item_id": item["id"],
+                    "output_index": 0,
+                    "call_id": call_id,
+                    "arguments": arguments,
+                },
+                {
+                    "type": "response.output_item.done",
+                    "response_id": response_id,
+                    "output_index": 0,
+                    "item": dict(item),
+                },
+                self._conversation_item_done_event(item),
+                {"type": "response.done", "response": response},
+            ]
         if event_type == "error":
             raw_error = event.get("error")
             if isinstance(raw_error, dict):

vllm_omni/experimental/fullduplex/openai/runtime_adapter.py

@@ -14,6 +14,18 @@ class ServingRuntimeConfigError(ValueError):
         self.code = code
 
 
+def reject_changed_runtime_value(
+    new_value: object,
+    current_value: object,
+    *,
+    message: str,
+    code: str,
+    error_cls: type[ServingRuntimeConfigError] = ServingRuntimeConfigError,
+) -> None:
+    if new_value != current_value:
+        raise error_cls(message, code=code)
+
+
 class PcmAppendReservation(Protocol):
     operation_id: str
     payload: dict[str, object] | None
@@ -64,6 +76,7 @@ class ServingRuntimeSessionState(Protocol):
     audio_buffer: PcmAppendBuffer
     input_since_commit: bool
     speech_since_commit: bool
+    native_context_locked: bool
     committed_audio_payload: dict[str, object] | None
     committed_audio_operation_id: str | None
     committed_audio_reserved_bytes: int
@@ -111,6 +124,7 @@ class ServingRuntimeAdapter(Protocol):
     clean_response_done_prefix: str
     interrupted_tts_prefix: str
     private_runtime_config_keys: frozenset[str]
+    collect_outputs_on_append: bool
 
     def create_session_state(self) -> ServingRuntimeSessionState: ...
 

vllm_omni/experimental/fullduplex/openai/runtime_bridge.py

@@ -135,7 +135,9 @@ class NativeRuntimeBridgeMixin:
                     incarnation=session.incarnation,
                 )
             if self._callable_accepts_keyword(append_input, "collect_outputs"):
-                append_kwargs["collect_outputs"] = False
+                append_kwargs["collect_outputs"] = bool(
+                    getattr(self._serving_runtime_adapter, "collect_outputs_on_append", False)
+                )
             result = await append_input(session.session_id, **append_kwargs)
         except Exception as exc:
             logger.exception("Failed to append duplex runtime input: %s", exc)
@@ -315,16 +317,21 @@ class NativeRuntimeBridgeMixin:
         native.data_plane_task = task
         return True
 
-    # One model unit (1 s at 16 kHz) of pcm_f32le silence, matching the
-    # official full-duplex behavior where the microphone keeps streaming
-    # silence while the assistant speaks; replies span multiple units.
+    # One MiniCPM model unit (1 s at 16 kHz) is the compatibility default.
+    # Other native-duplex adapters may use a different frame-locked unit.
     _NATIVE_SILENCE_UNIT_PAYLOAD_AUDIO = base64.b64encode(bytes(16000 * 4)).decode("ascii")
     _NATIVE_RESPONSE_MAX_CONTINUATION_UNITS = 8
 
     def _native_silence_unit_payload(self) -> dict[str, object]:
+        samples = int(getattr(self._serving_runtime_adapter, "silence_continuation_samples", 16000))
+        audio = (
+            self._NATIVE_SILENCE_UNIT_PAYLOAD_AUDIO
+            if samples == 16000
+            else base64.b64encode(bytes(samples * 4)).decode("ascii")
+        )
         return {
             "type": "audio",
-            "audio": self._NATIVE_SILENCE_UNIT_PAYLOAD_AUDIO,
+            "audio": audio,
             "format": "pcm_f32le",
             "sample_rate_hz": 16000,
         }
@@ -826,6 +833,18 @@ class NativeRuntimeBridgeMixin:
                     }
                 )
             return close_reason, True
+        if native_result.get("function_call") is True:
+            await send_json(
+                {
+                    "type": "function_call.done",
+                    "session_id": session.session_id,
+                    "epoch": session.epoch,
+                    "call_id": native_result.get("call_id"),
+                    "name": native_result.get("name"),
+                    "arguments": native_result.get("arguments", ""),
+                }
+            )
+            return close_reason, True
         if native_result.get("requires_stage_handoff") is True or native_result.get("requires_tts_stage") is True:
             # Stage0 announces a talker handoff before Stage1 has produced
             # client-visible audio. A single client input stream may contain