anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196)

Browse Source 
Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
This commit is contained in:
chickeytonClaude Fable 5Gao Han
 authored and GitHub committed 2026-09-05 08:20:21 +00:00
1 parent d4096e01fe
commit 5d7fdf9350
179 files changed
+6462 -3832
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.buildkite/cuda/test-merge.yml
+15
View File
@@ -183,6 +183,21 @@ steps:
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py
- vllm_omni/deploy/minicpmo_4_5.yaml
- tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
- vllm_omni/engine/duplex/
- vllm_omni/entrypoints/duplex/
# Public duplex client library + its unit suites: client-side
# changes must re-run the duplex e2e group. The live client probe
# (test_duplex_client_live.py, single GPU, core tier) runs in the
# ready pipeline's duplex group.
- vllm_omni/clients/
- tests/clients/
- vllm_omni/entrypoints/duplex_request_client.py
- vllm_omni/outputs/duplex.py
- vllm_omni/model_executor/duplex_sampling.py
- tests/engine/duplex/
- tests/entrypoints/duplex/
- tests/model_executor/models/minicpmo_4_5/duplex/
# Legacy Realtime probe client + video stacking used by the e2e drivers.
- vllm_omni/experimental/fullduplex/
- vllm_omni/model_executor/models/minicpmo_4_5/
- vllm_omni/model_executor/models/step_audio2/
.buildkite/cuda/test-ready.yml
+17 -2
View File
@@ -214,6 +214,21 @@ steps:
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py
- vllm_omni/deploy/minicpmo_4_5.yaml
- tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
- vllm_omni/engine/duplex/
- vllm_omni/entrypoints/duplex/
# Public duplex client library + its unit suites: client-side
# changes must re-run the duplex e2e group, which includes the
# live client probe (test_duplex_client_live.py, single GPU).
- vllm_omni/clients/
- tests/clients/
- tests/e2e/online_serving/test_duplex_client_live.py
- vllm_omni/entrypoints/duplex_request_client.py
- vllm_omni/outputs/duplex.py
- vllm_omni/model_executor/duplex_sampling.py
- tests/engine/duplex/
- tests/entrypoints/duplex/
- tests/model_executor/models/minicpmo_4_5/duplex/
# Legacy Realtime probe client + video stacking used by the e2e drivers.
- vllm_omni/experimental/fullduplex/
- vllm_omni/model_executor/models/minicpmo_4_5/
- vllm_omni/model_executor/models/step_audio2/
@@ -231,8 +246,8 @@ steps:
- vllm_omni/config/stage_config.py
commands:
- |
timeout 20m bash -c "
pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py -m 'core_model and cuda' --run-level 'core_model'
timeout 35m bash -c "
pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'
"
mirror_hardwares: h100_1
README.md
+1 -1
View File
@@ -53,7 +53,7 @@ vLLM-Omni is flexible and easy to use with:
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output (experimental)
- [Full-duplex realtime serving](docs/serving/realtime_duplex_api.md) with streaming audio input and output
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
docs/.nav.yml
+1
View File
@@ -17,6 +17,7 @@ nav:
- Streaming and Realtime:
- Realtime Audio: serving/realtime_api.md
- Full Duplex: serving/full_duplex_api.md
- Realtime Duplex Reference: serving/realtime_duplex_api.md
- Streaming Video Input: serving/video_stream_api.md
- Streaming Video Output: serving/streaming_video_output_api.md
- OpenPI Robot Policy: serving/openpi_api.md
docs/README.md
+1 -2
View File
@@ -23,7 +23,6 @@ Easy, fast, and cheap omni-modality model serving for everyone
<a class="github-button" href="https://github.com/vllm-project/vllm-omni/fork" data-show-count="true" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork">Fork</a>
</p>
## About
[vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
@@ -52,7 +51,7 @@ vLLM-Omni is flexible and easy to use with:
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output (experimental)
- [Full-duplex realtime serving](serving/realtime_duplex_api.md) with streaming audio input and output
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
docs/api/README.md
+30
View File
@@ -15,6 +15,36 @@ Main entry points for vLLM-Omni inference and serving.
- [vllm_omni.entrypoints.omni_base.OmniBase][]
- [vllm_omni.entrypoints.pd_utils.PDDisaggregationMixin][]
## Clients
Client libraries that talk to a running vLLM-Omni server (the full-duplex Realtime client and its per-model session presets).
- [vllm_omni.clients.duplex.AudioDelta][]
- [vllm_omni.clients.duplex.AudioFormat][]
- [vllm_omni.clients.duplex.ConnectionResumed][]
- [vllm_omni.clients.duplex.DuplexClient][]
- [vllm_omni.clients.duplex.DuplexClientError][]
- [vllm_omni.clients.duplex.DuplexConnectionError][]
- [vllm_omni.clients.duplex.DuplexEvent][]
- [vllm_omni.clients.duplex.DuplexProtocolError][]
- [vllm_omni.clients.duplex.DuplexSessionClosedError][]
- [vllm_omni.clients.duplex.ErrorEvent][]
- [vllm_omni.clients.duplex.EventCollector][]
- [vllm_omni.clients.duplex.ListenDecision][]
- [vllm_omni.clients.duplex.ReconnectPolicy][]
- [vllm_omni.clients.duplex.ResponseCreated][]
- [vllm_omni.clients.duplex.ResponseDone][]
- [vllm_omni.clients.duplex.ResponseHandle][]
- [vllm_omni.clients.duplex.SessionClosed][]
- [vllm_omni.clients.duplex.SessionConfig][]
- [vllm_omni.clients.duplex.SessionCreated][]
- [vllm_omni.clients.duplex.SessionExpired][]
- [vllm_omni.clients.duplex.SessionResumed][]
- [vllm_omni.clients.duplex.SpeakDecision][]
- [vllm_omni.clients.duplex.TextDelta][]
- [vllm_omni.clients.duplex.TranscriptDelta][]
- [vllm_omni.clients.duplex.WebSocketTransport][]
## Inputs
Input data structures for multi-modal inputs.
docs/cli/bench/serve.md
+25 -7
View File
@@ -1,11 +1,15 @@
# vLLM-Omni Benchmark CLI Guide
The vllm bench command launches the vLLM-Omni benchmark to evaluate the performance of multimodal models.
## Notes
vLLM-Omni registers the `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, `daily-omni`, and `openai-realtime-duplex` serving benchmark backends. It also adds the `omniinteract` dataset.
## Basic Parameter Description
You can use `vllm bench serve --omni --help=all` to get descriptions of all parameters. The commonly used parameters are described below:
- `--omni`
Enable Omni (multimodal) mode, supporting multimodal inputs and outputs such as images, videos, and audio.
@@ -77,8 +81,8 @@ Specify to save benchmark results to a json file
- `--random-prefix-len`
Number of fixed prefix tokens before the random context in a request.
The total input length is the sum of random-prefix-len and a random
context length sampled from [input_len * (1 - range_ratio),
input_len * (1 + range_ratio)].Only the random and random-mm modes
context length sampled from [input_len *(1 - range_ratio),
input_len* (1 + range_ratio)].Only the random and random-mm modes
support this parameter.
- `--random-input-len`
@@ -91,7 +95,7 @@ Specify to save benchmark results to a json file
Range ratio for sampling input/output length,
used only for random sampling. Must be in the range [0, 1) to define
a symmetric sampling range
[length * (1 - range_ratio), length * (1 + range_ratio)].
[length *(1 - range_ratio), length* (1 + range_ratio)].
Only the random and random-mm modes support this parameter.
- `--random-mm-base-items-per-request`
@@ -137,6 +141,7 @@ Specify to save benchmark results to a json file
## Usage Examples
### Online Benchmark
<details class="admonition abstract" markdown="1">
<summary>Show more</summary>
@@ -162,7 +167,9 @@ vllm bench serve \
--dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
--percentile-metrics ttft,tpot,itl,e2el
```
If successful, you will see the following output:
```text
============ Serving Benchmark Result ============
Successful requests: 2
@@ -264,6 +271,7 @@ Median AUDIO_RTF: 0.47
P99 AUDIO_RTF: 0.48
==================================================
```
Notes:
We use audio generation time / audio duration to calculate RTF.
@@ -301,8 +309,12 @@ options apply. Use `--omniinteract-require-response` only for functional E2E cas
Each completed case writes `output.wav`, `wav_transcript.json`, `events.json`, `result.json`, and a final `.done` marker under
`--omniinteract-output-dir`. The root also contains `batch_summary.json` and `official_eval_manifest.jsonl`; failed cases write
`.failed.json`. Runs sharing one output root are serialized. Completion validates transport, response lifecycle, and artifacts,
not answer accuracy. Transcript timestamps are serialized playback-queue times. Clipped or cancelled outputs are ineligible and
omitted from the official manifest; `audio_clipped_bytes` records output beyond the rounded video horizon.
not answer accuracy. Transcript timestamps are serialized playback-queue times. Playback ACKs report cumulative progress
incrementally along that serialized clock, like a live listener; the first ack for a response goes out as soon as its audio
arrives, checkpointing the response's history position so a later committed user input updates it in place. A residual
`playback_ack_too_late` rejection is recorded as an artifact warning rather than failing the case. Clipped or cancelled
outputs are ineligible and omitted from the official manifest; `audio_clipped_bytes` records output beyond the rounded video
horizon.
TTFT, TTFP, and RTF start at client receipt of `response.created`. TPOT/ITL use engine stage-0 timing; ITL is emitted only when
every token interval is present.
@@ -339,8 +351,8 @@ Generate synthetic image、video、audio inputs alongside random text prompts to
Notes:
- Works only with online benchmark via:
- the OpenAI chat backend (`--backend openai-chat-omni`) and endpoint `/v1/chat/completions`.
- the OpenAI edit image backend (`--backend openai-image-edits-omni`) and endpoint `/v1/images/edits`.
- the OpenAI chat backend (`--backend openai-chat-omni`) and endpoint `/v1/chat/completions`.
- the OpenAI edit image backend (`--backend openai-image-edits-omni`) and endpoint `/v1/images/edits`.
Start the server (example):
@@ -351,6 +363,7 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni
It is recommended to use the flag `--ignore-eos` to simulate real responses. You can set the size of the output via the arg `random-output-len`.
Then run the benchmarking script:
```bash
vllm bench serve \
--omni \
@@ -420,6 +433,7 @@ How sampling works:
- If a modality (e.g., image) reaches its limit from `--random-mm-limit-mm-per-prompt`, all buckets of that modality are excluded and the remaining bucket probabilities are renormalized before continuing.
This should be seen as an edge case, and if this behavior can be avoided by setting `--random-mm-limit-mm-per-prompt` to a large number. Note that this might result in errors due to engine config `--limit-mm-per-prompt`.
- The resulting request contains synthetic image data in `multi_modal_data` (OpenAI Chat format). When `random-mm` is used with the OpenAI Chat backend, prompts remain text and MM content is attached via `multi_modal_data`.
</details>
### Multi-Stage Benchmark
@@ -434,6 +448,7 @@ vllm serve --omni --port 29999 --model ~/Qwen3-Omni-30B-A3B-Instruct
```
Then run the benchmarking script (remember to add the flag `--print-stage`):
```bash
vllm bench serve --omni \
--dataset-name random \
@@ -453,6 +468,7 @@ vllm bench serve --omni \
```
Besides the "Serving Benchmark Result", there will be "Stage Benchmark Result" below:
```text
============= Stage Benchmark Result =============
=============== Stage 0 (thinker) ================
@@ -509,7 +525,9 @@ Mean AUDIO_RTF: 0.24
Median AUDIO_RTF: 0.26
P99 AUDIO_RTF: 0.27
```
Explanation:
- stage_gen_time: Time from submitting a request to a specific stage to that stage finishing generation.
- Serving TTFC (Time to First Chunk): Time from the HTTP request being accepted by the serving frontend to
docs/design/architecture_overview.md
+1 -1
View File
@@ -418,7 +418,7 @@ The public interfaces map onto the same engine and stage boundaries:
flowchart LR
offline["Offline Python<br/>Omni.generate()"] --> engine["AsyncOmniEngine"]
online["OpenAI-compatible serving<br/>vllm serve ... --omni"] --> engine
duplex["Experimental duplex WebSocket<br/>/v1/duplex or realtime duplex"] --> engine
duplex["Duplex WebSocket<br/>/v1/realtime?duplex=1 or /v1/duplex"] --> engine
engine --> stages["Configured AR and diffusion stages"]
stages --> result["Streaming or final multimodal output"]
```
vllm_omni/experimental/fullduplex/personaplex/DESIGN.md → docs/design/fullduplex-personaplex.md
+29 -15
View File
@@ -23,16 +23,18 @@ The target is the engine-native path:
-> response.audio.delta + response.audio_transcript.delta
```
The standalone `/api/chat` and `/v1/audio/duplex` server remains a compatibility
surface. It is not evidence that the unified engine path works.
The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
original PR was demo-only and has been removed from the tree; the unified
engine path above is the only serving surface. It was never evidence that the
unified engine path works.
## Why configuration-only enablement is invalid
## Why configuration-only enablement was invalid
The current staged pipeline is explicitly turn based. Its Talker reads
`pplex_user_codes`, `pplex_prefill_text`, and `pplex_silence_codes`, but no
production staged input path writes those fields. The voice prompt, persona
prefill, and streaming Mimi state live only in the standalone
`PersonaPlexEngine`.
The staged pipeline as shipped by the original PR was explicitly turn based.
Its Talker read `pplex_user_codes`, `pplex_prefill_text`, and
`pplex_silence_codes`, but no production staged input path wrote those fields.
The voice prompt, persona prefill, and streaming Mimi state lived only in the
standalone `PersonaPlexEngine` (demo-only, since removed).
Setting only the following fields would therefore advertise an endpoint whose
model never receives the live microphone stream:
@@ -56,7 +58,12 @@ The unified implementation supports:
- continuous user input while assistant audio is generated or played;
- bundled `.pt` voice prompts and a session persona;
- greedy text and depformer sampling, matching the current PersonaPlex port;
- `/v1/realtime?duplex=1`;
- `/v1/realtime?duplex=1` (the wire vocabulary is catalogued in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide);
- the public client preset
`vllm_omni.clients.personaplex.create_duplex_session_config()` (24 kHz
`pcm_f32le` input format, voice prompt, persona) as the canonical
session-config source for `DuplexClient` consumers;
- engine lease close, disconnect cleanup, reconnect after cleanup, and explicit
response cancellation without cross-session state reuse.
@@ -189,10 +196,17 @@ model-neutral native results:
```
It owns per-request audio and text cursors so a cumulative output cannot replay
old audio. PersonaPlex keeps one visible response open while continuous output
arrives. Session close or cancellation terminates that response through the
generic Realtime lifecycle; a codec segment finishing is not a conversational
turn boundary.
old audio. The generic projector turns each native result into the Realtime
`response.audio.delta` / `response.audio_transcript.delta` pair (and
`response.output_text.delta` for text) under one `response_id`; the full
mapping is the name map in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
PersonaPlex keeps one visible response open while continuous output arrives.
Session close or cancellation terminates that response through the generic
Realtime lifecycle; a codec segment finishing is not a conversational turn
boundary. PersonaPlex advertises `supports_barge_in=false` and
`supports_session_resume=false`, so `barge_in`, `turn_detection.server_vad`,
and `session.resume` are rejected on this model.
## Error and lifecycle contracts
@@ -210,8 +224,8 @@ turn boundary.
- New sessions cannot observe the previous voice, persona, PCM tail, Mimi
convolution state, or Talker delayed code frame.
The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path
must follow the same exception-safe rule while they remain in the tree.
(The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path,
which followed the same exception-safe rule, have been removed from the tree.)
## Testing and acceptance
vllm_omni/experimental/fullduplex/DESIGN.md → docs/design/fullduplex.md
+81 -39
View File
@@ -1,5 +1,13 @@
# MiniCPM-o 4.5 Full-Duplex Runtime Architecture
> Relocation note: after the H20 validation recorded below, the full-duplex
> implementation moved out of `vllm_omni/experimental/fullduplex/` into the
> stable tree (`vllm_omni/engine/duplex/`, `vllm_omni/entrypoints/duplex/`,
> `vllm_omni/model_executor/models/minicpmo_4_5/duplex/`, and companion
> modules). Module paths in this document reflect the new layout; the runtime
> evidence was produced on the pre-move tree, which is file-identical apart
> from module paths.
## Purpose
This document records the runtime architecture implemented by PR #3907, the
@@ -31,8 +39,8 @@ session was rejected and capacity returned after one session closed. The two
accepted sessions independently completed their audio, transcript, response,
playback, and close lifecycles. This is isolation and admission evidence for the
tested two-session deployment, not a production fairness, arbitrary-capacity,
or failure-recovery claim. The validated paths include the experimental runtime
extension, typed resumable policy, extracted control plane and clients, request
or failure-recovery claim. The validated paths include the MiniCPM duplex
runtime extension, typed resumable policy, extracted control plane and clients, request
preregistration, typed direct-output decision, engine-managed two-session
capacity in the checked-in deploy profile, separate public/runtime configuration
channels, private Session ledgers, ordered `session.update`, the MiniCPM runner
@@ -67,7 +75,9 @@ The checkpoint does not claim:
- an acoustic-onset-to-cancel latency target across arbitrary client chunk sizes;
- production multi-session admission, fairness, capacity, or failure recovery;
- bounded long-session KV;
- video input or audio/video synchronization.
- audio/video synchronization beyond one camera frame per model unit (camera
frames ride the audio appends as `video_frames`; the wire contract is in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) guide).
## Active Runtime Path
@@ -81,7 +91,7 @@ WebSocket
-> AsyncOmni thin open/append/signal/close proxies
-> DuplexControlClient over the engine-owned correlated RPC transport
-> DuplexControlPlane + DuplexSessionRuntimeManager
-> experimental DuplexRuntimeExtension + engine session/stage bindings
-> MiniCPM DuplexRuntimeExtension + engine session/stage bindings
-> StagePool
-> resumable scheduler request
-> MiniCPM Stage0
@@ -177,21 +187,21 @@ ordering guarantee.
### Extracted control ownership
The generic engine mechanism and the experimental duplex plugin are split by
The generic engine mechanism and the duplex plugin are split by
responsibility:
| Module | Responsibility |
| --- | --- |
| `engine.messages` | ordinary engine queue messages and collective RPC envelopes |
| `engine.rpc_result_router` | generic correlation routing through each result message's `rpc_correlation_key` |
| `experimental.fullduplex.engine.contracts` | immutable duplex DTOs, extension/stage protocols, typed decisions, request-ID codec |
| `experimental.fullduplex.engine.lease` | duplex lease activity, TTL/grace configuration, and expiry records |
| `experimental.fullduplex.engine.messages` | identity fence and duplex control message/result envelopes |
| `experimental.fullduplex.engine.duplex_runtime` | extension loading, validation, and compatibility exports |
| `experimental.fullduplex.engine.duplex_session` | session resources, fences, append reservations, idempotency cache, binding cleanup |
| `experimental.fullduplex.engine.duplex_control_plane` | control algorithms and the narrow `DuplexStagePort` used to submit and clean up stage requests |
| `experimental.fullduplex.engine.duplex_control_client` | typed control-message construction and calls through the engine-owned correlated RPC transport |
| `experimental.fullduplex.request_client` | deterministic request identity, client-state preregistration, acknowledgement validation, data-plane collection, rollback |
| `engine.duplex.contracts` | immutable duplex DTOs, extension/stage protocols, typed decisions, request-ID codec |
| `engine.duplex.lease` | duplex lease activity, TTL/grace configuration, and expiry records |
| `engine.duplex.messages` | identity fence and duplex control message/result envelopes |
| `engine.duplex.runtime` | extension loading, validation, and compatibility exports |
| `engine.duplex.session` | session resources, fences, append reservations, idempotency cache, binding cleanup |
| `engine.duplex.control_plane` | control algorithms and the narrow `DuplexStagePort` used to submit and clean up stage requests |
| `engine.duplex.control_client` | typed control-message construction and calls through the engine-owned correlated RPC transport |
| `entrypoints.duplex_request_client` | deterministic request identity, client-state preregistration, acknowledgement validation, data-plane collection, rollback |
`Orchestrator`, `AsyncOmniEngine`, and `AsyncOmni` retain explicit wiring and
compatibility proxies. They no longer own the corresponding duplex algorithms.
@@ -362,33 +372,34 @@ vllm_omni.engine.rpc_result_router # generic correlation routing
vllm_omni.engine.async_engine_utils # ordinary engine lifecycle helpers
```
The full-duplex implementation remains experimental:
The full-duplex implementation lives in dedicated duplex modules:
```text
vllm_omni.experimental.fullduplex.engine.duplex_runtime
vllm_omni.engine.duplex.runtime
# extension loading and validation
vllm_omni.experimental.fullduplex.engine.contracts
vllm_omni.engine.duplex.contracts
# immutable DTOs and narrow protocols
vllm_omni.experimental.fullduplex.engine.lease
vllm_omni.engine.duplex.lease
# duplex lease primitives
vllm_omni.experimental.fullduplex.engine.messages
vllm_omni.engine.duplex.messages
# duplex identity and queue envelopes
vllm_omni.experimental.fullduplex.engine.duplex_session
vllm_omni.engine.duplex.session
# session transaction implementation
vllm_omni.experimental.fullduplex.engine.duplex_control_plane
vllm_omni.engine.duplex.control_plane
# engine-side control algorithms
vllm_omni.experimental.fullduplex.engine.duplex_control_client
vllm_omni.engine.duplex.control_client
# correlated control client
vllm_omni.experimental.fullduplex.request_client
vllm_omni.entrypoints.duplex_request_client
# entrypoint request/output lifecycle
```
Stable engine and entrypoint modules import only the kernel at module load.
They load the experimental ControlPlane, ControlClient, request client, and
runtime extension inside duplex-only construction or call paths. Importing
`Orchestrator`, `AsyncOmniEngine`, or `AsyncOmni` for an ordinary deployment
therefore does not import `vllm_omni.experimental.fullduplex`. Compatibility
exports remain lazy and are not part of ordinary startup.
Stable engine and entrypoint modules import the duplex kernel eagerly at
module load (top-of-file imports). Duplex behavior remains opt-in: the
ControlPlane, ControlClient, and request client are *constructed* only on
duplex-enabled deployments, and the model runtime extension still loads only
through the dotted-string plugin path. The client-side package
(`vllm_omni/clients/`, home of the public `DuplexClient`) is never imported
by runtime code.
`PipelineConfig.duplex_control_enabled` enables the generic mechanism, while
`PipelineConfig.duplex_runtime_extension` separately selects the model adapter.
@@ -452,7 +463,7 @@ sampling generation unchanged.
`extra_body` or session payloads:
| Field | Default | Owner | Lifecycle or resource effect |
|---|---:|---|---|
| --- | ---: | --- | --- |
| `idle_ttl_s` | `300.0` | Engine lease manager | Expires an attached or detached session after inactivity; `null` disables only idle expiry. |
| `disconnect_grace_s` | `30.0` | Engine lease manager and API attachment registry | Bounds how long a detached session may retain engine resources before cleanup. |
| `reaper_interval_s` | `5.0` | Orchestrator | Sets the cadence for lease expiry and pending cleanup retries. |
@@ -467,6 +478,30 @@ Deploy profiles may lower or raise these values to match scheduler capacity,
but clients cannot override them. The engine remains authoritative for
admission and cleanup ownership even when the API performs an early check.
### Client-side companion package
The public client surface ships under `vllm_omni/clients/` and stays on the
consumer side of the boundary above: `vllm_omni.clients.duplex` provides the
model-agnostic `DuplexClient` (typed events, per-response demux via
`ResponseHandle`, `session.resume` with acknowledgement-based replay dedup)
and the `EventCollector` helper used for timing and request-metric summaries.
Model-specific session defaults live in thin per-model preset modules —
`vllm_omni.clients.minicpmo_4_5.create_duplex_session_config()` and
`vllm_omni.clients.personaplex.create_duplex_session_config()` — so the core
client never imports model packages and the model duplex plugin packages
never import the client (both directions are enforced by tests:
`tests/engine/test_duplex_import_boundary.py`,
`tests/entrypoints/openai_api/test_duplex_capability.py`, and the subprocess
purity guards in `tests/clients/test_model_session_configs.py`).
`examples/online_serving/barge_in_client.py` (with its flow companion
`barge_in_client_flow.md`) is the runnable reference for the client library
against a live duplex deployment. The wire vocabulary the client speaks —
every `/v1/realtime?duplex=1` event with its OpenAI-compatibility tier and a
JSON example, plus the native-to-Realtime name map — is catalogued in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide; that
page derives from the normative contract table above and must be updated
with it.
## Generic-Path Cleanup
Model-specific `MINICPMO45_PROFILE_LOGS` probes were removed from:
@@ -508,14 +543,16 @@ The MiniCPM Stage1 input processor consumes that capability for token-faithful
delta transcripts. The generic orchestrator no longer installs or names a
MiniCPM-specific bridge-state key.
The stable API server inspects the deployment `session_mode` before importing or
constructing the experimental duplex handler. Ordinary deployments do not load
the full-duplex serving package or create its registries and background state.
The stable API server imports the duplex serving package eagerly and inspects
the deployment `session_mode` before constructing the duplex handler. Ordinary
deployments do not create its registries and background state.
For an enabled deployment, the client still selects the Realtime duplex route
with `?duplex=1` (or an equivalent explicit true value). Model-name matching is
not used for routing or native-runtime activation. MiniCPM clients explicitly
set `extra_body.minicpmo45_native_duplex=true`; repository demos do so in their
session payloads.
set `extra_body.native_duplex=true`; the canonical way to do so is
the `vllm_omni.clients.minicpmo_4_5.create_duplex_session_config()` preset,
which the repository's barge-in example
(`examples/online_serving/barge_in_client.py`) uses in its session payload.
`DuplexRequestClient` derives the data-plane request identity from the accepted
fence and preregisters a resumable `ClientRequestState` before the append crosses
@@ -623,7 +660,7 @@ continues to return `input_audio_buffer_empty`.
The commit-to-response contract is:
| Session state at `input_audio_buffer.commit` | Runtime action | Required event outcome |
|---|---|---|
| --- | --- | --- |
| Auto-response disabled, valid idle input | Commit and retain the input item. | Emit `input_audio_buffer.committed`; emit no response until an explicit `response.create`. |
| Auto-response enabled, valid idle speech | Submit the final native append for the current model turn. | Emit `input_audio_buffer.committed` before model output. Create a response only on the first visible text/audio output; a model listen decision may end with `response.listen` and no response. |
| Auto-response enabled, short overlap during an active response | Discard the short overlap as an acknowledgement. | Emit the committed/no-response acknowledgement; do not create or defer a response. |
@@ -770,9 +807,12 @@ Graph capture. This is a model execution constraint, not an E2E-only override.
### Resource capabilities
The checked-in MiniCPM duplex deploy profile sets Stage0 and Stage1
`max_num_seqs` to two and configures the engine runtime manager with
`max_sessions=2`. The Realtime capability response therefore advertises
At the time of this snapshot the checked-in MiniCPM duplex deploy profile set
Stage0 and Stage1 `max_num_seqs` to two and configured the engine runtime
manager with `max_sessions=2`; the duplex settings have since been folded into
`vllm_omni/deploy/minicpmo_4_5.yaml`, which admits four sessions
(`duplex_session.max_sessions: 4`, matching its stage `max_num_seqs`). The
Realtime capability response advertises
multi-session and same-replica multi-session support with
`session_admission_mode="engine_managed"`. Client session fields cannot raise
this server-owned limit. Fresh E2E evidence covers two concurrent sessions in
@@ -1025,4 +1065,6 @@ Passing this checkpoint supports the statement:
It does not support claims for an acoustic-onset barge-in latency target across
arbitrary client chunk sizes, multi-session production concurrency, bounded
long-session KV, scheduler-native append, or video input.
long-session KV, or scheduler-native append. Camera-frame input
(`video_frames`) landed after this snapshot and is covered by its own E2E
test rather than by the evidence above.
docs/design/index.md
+2
View File
@@ -17,6 +17,8 @@ implementation contract; it is not, by itself, a general support claim.
### Runtime and stage execution
- [Full-Duplex Runtime (MiniCPM-o 4.5)](fullduplex.md)
- [Full-Duplex Runtime (PersonaPlex)](fullduplex-personaplex.md)
- [Disaggregated Inference](feature/disaggregated_inference.md)
- [Host Weight Runtime](feature/host_weight_runtime.md)
- [Async Chunk](feature/async_chunk.md)
docs/design/module/engine_orchestration.md
+5 -1
View File
@@ -28,6 +28,9 @@ related_code_paths:
- vllm_omni/engine/stage_pool.py
- vllm_omni/outputs/output_processor.py
- vllm_omni/distributed/omni_coordinator/**
# Duplex kernel design is owned by ../fullduplex.md; listed here because the
# orchestrator hosts the duplex control plane.
- vllm_omni/engine/duplex/**
depends_on:
- input_output_modality_contracts.md
- error_contracts.md
@@ -44,7 +47,8 @@ validation_paths:
- tests/engine/test_orchestrator_stage_input_bridge.py
- tests/engine/test_cfg_companion_lifecycle.py
- tests/engine/test_rpc_result_router.py
- tests/e2e/features/fullduplex/engine/**
- tests/engine/duplex/**
- tests/engine/test_duplex_import_boundary.py
upstream_refs:
- vllm.v1.engine.EngineCoreRequest
- vllm.v1.engine.EngineCoreOutputs
docs/design/module/entrypoints.md
+9
View File
@@ -26,9 +26,18 @@ primary_code_paths:
- vllm_omni/entrypoints/client_request_state.py
- vllm_omni/entrypoints/stage_utils.py
- vllm_omni/entrypoints/utils.py
- vllm_omni/entrypoints/duplex/**
- vllm_omni/entrypoints/duplex_request_client.py
- vllm_omni/clients/**
primary_path_exceptions:
- path: vllm_omni/entrypoints/openai/errors.py
owner: error_contracts.md
- path: vllm_omni/entrypoints/duplex/**
owner: ../fullduplex.md
- path: vllm_omni/entrypoints/duplex_request_client.py
owner: ../fullduplex.md
- path: vllm_omni/clients/**
owner: ../fullduplex.md
related_code_paths:
- vllm_omni/errors.py
- vllm_omni/inputs/**
docs/mkdocs/hooks/generate_api_readme.py
+7
View File
@@ -22,6 +22,13 @@ CATEGORIES = {
"name": "Entry Points",
"description": "Main entry points for vLLM-Omni inference and serving.",
},
"clients": {
"name": "Clients",
"description": (
"Client libraries that talk to a running vLLM-Omni server "
"(the full-duplex Realtime client and its per-model session presets)."
),
},
"inputs": {
"name": "Inputs",
"description": "Input data structures for multi-modal inputs.",
docs/serving/README.md
+6 -7
View File
@@ -43,7 +43,7 @@ Completions for conversational or heterogeneous omni pipelines, rather than as
the default wrapper for every generation task.
| Task | Endpoint | Request | Response | Details |
|------|----------|---------|----------|---------|
| ------ | ---------- | --------- | ---------- | --------- |
| Conversation or multimodal understanding/generation | `POST /v1/chat/completions` | JSON | OpenAI-style JSON or SSE | [Chat Completions](chat_completions_api.md) |
| Text-to-speech | `POST /v1/audio/speech` | JSON | Audio bytes or SSE | [Speech](speech_api.md) |
| Sound, music, or ambient audio generation | `POST /v1/audio/generate` | JSON | Audio bytes | [Audio Generation](audio_generate_api.md) |
@@ -141,7 +141,7 @@ These WebSocket APIs have different event schemas and cannot be used
interchangeably.
| Workload | Endpoint | Interaction model | Details |
|----------|----------|-------------------|---------|
| ---------- | ---------- | ------------------- | --------- |
| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
@@ -158,7 +158,7 @@ ordinary realtime handler when duplex is unavailable.
## Related Endpoints
| Purpose | Endpoints | Reference |
|---------|-----------|-----------|
| --------- | ----------- | ----------- |
| Discovery and readiness | `GET /health`, `GET /v1/models` | This page |
| Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) |
| Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) |
@@ -168,11 +168,10 @@ ordinary realtime handler when duplex is unavailable.
## Standalone Experimental Servers
PersonaPlex also provides a Moshi-compatible WebSocket server, while JoyVL
provides a stateful interaction orchestrator in front of another model server.
These are separate processes with their own routes; they are not additional
JoyVL provides a stateful interaction orchestrator in front of another model
server. It is a separate process with its own routes; they are not additional
paths on every `vllm serve --omni` instance. See [Standalone Experimental
Servers](standalone_servers.md) before deploying either one.
Servers](standalone_servers.md) before deploying it.
## Compatibility Notes
docs/serving/full_duplex_api.md
+12 -6
View File
@@ -1,16 +1,20 @@
# Full-Duplex WebSocket API
vLLM-Omni provides an experimental full-duplex runtime for models that can
vLLM-Omni provides a full-duplex runtime for models that can
continue receiving speech while producing speech. It adds persistent session
state, model-specific turn policy, overlap handling, playback acknowledgement,
and optional session resume.
Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md).
This page is the endpoint overview. For the complete wire contract, the
`vllm_omni.clients.duplex.DuplexClient` Python library, and the per-model
capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
## Choose an Endpoint
| Endpoint | Protocol | Recommended use |
|----------|----------|-----------------|
| --- | --- | --- |
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
@@ -41,8 +45,10 @@ and control plane as part of their registered pipeline.
The current unified-runtime integrations are:
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`;
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex.
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5.yaml`
(`session_mode: duplex`);
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex;
- Nemotron VoiceChat, via its registered duplex plugin package.
JoyVL is a separate HTTP interaction orchestrator and does not use these
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
@@ -53,7 +59,7 @@ Start the duplex deployment:
```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni \
--deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \
--deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
--trust-remote-code \
--port 8091
```
@@ -112,5 +118,5 @@ applications should use the provided Realtime client where possible.
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
and [full-duplex runtime design](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/experimental/fullduplex/DESIGN.md)
and the [full-duplex runtime design](../design/fullduplex.md)
for model-specific validation and architecture details.
docs/serving/realtime_api.md
+2 -2
View File
@@ -5,7 +5,7 @@ receive incremental transcription and synthesized response audio. This is a
turn-based streaming API: it starts one generation over an audio stream and
finishes after the client closes that input stream.
For a model that listens while it is already speaking, use the experimental
For a model that listens while it is already speaking, use the
[Full-Duplex API](full_duplex_api.md) instead.
## Quick Start
@@ -35,7 +35,7 @@ Messages in both directions are JSON text frames. Audio carried inside an
event is base64-encoded raw PCM16.
| Direction | Event | Purpose |
|-----------|-------|---------|
| ----------- | ------- | --------- |
| Server to client | `session.created` | Confirms that the WebSocket connection is ready |
| Client to server | `session.update` | Selects and validates the served model |
| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream |
docs/serving/realtime_duplex_api.md
+1217
View File
File diff suppressed because it is too large. Load diff
docs/serving/standalone_servers.md
+10 -52
View File
@@ -1,61 +1,19 @@
# Standalone Experimental Servers
Most vLLM-Omni endpoints run on the unified server started with
`vllm serve <model> --omni`. The experimental full-duplex package also contains
two separate user-facing server processes. Their routes are not added to the
unified API server.
`vllm serve <model> --omni`. The experimental package also contains one
separate user-facing server process; its routes are not added to the unified
API server.
| Server | Transport | Role |
|--------|-----------|------|
| PersonaPlex compatibility server | WebSocket | Hosts the official PersonaPlex browser protocol and a raw-PCM alternative |
| --- | --- | --- |
| JoyVL interaction server | HTTP | Adds state, memory, proactive decisions, and delegation in front of a separate OpenAI-compatible model backend |
## PersonaPlex Compatibility Server
Use this server when a client requires the official Moshi/PersonaPlex wire
protocol or the bundled browser UI. For the engine-owned unified duplex path,
use [`WS /v1/realtime?duplex=1`](full_duplex_api.md) instead.
Start it with:
```bash
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \
vllm_omni.experimental.fullduplex.personaplex.serving.server \
--port 8091
```
| Route | Purpose |
|-------|---------|
| `GET /` | Official PersonaPlex browser client, when its assets are available |
| `GET /health` | Readiness check |
| `WS /api/chat` | Moshi-compatible tagged binary protocol with Opus audio |
| `WS /v1/audio/duplex` | Simpler float32 raw-PCM protocol |
### `/api/chat` Protocol
The first byte of every binary frame is a message tag:
| Direction | Tag | Payload |
|-----------|-----|---------|
| Server to client | `0x00` | Ready handshake |
| Client to server | `0x01` | Opus microphone audio at 24 kHz |
| Server to client | `0x01` | Opus response audio |
| Server to client | `0x02` | UTF-8 inner-monologue text |
The optional `text_prompt` and `voice_prompt` query parameters select a persona
and voice. The default server admits one conversation; use `--batch-size N` to
configure multiple lockstep slots. Connections beyond capacity close with
WebSocket code 1013.
### `/v1/audio/duplex` Protocol
Send `{"type":"open","persona":"...","voice":"..."}` as a JSON text frame,
then exchange binary float32 PCM frames. The server returns generated float32
audio as binary frames and text as `{"type":"text","text":"..."}`. Send
`{"type":"close"}` and wait for `{"type":"done"}` to flush and finish.
See the [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
for the browser and headless clients.
PersonaPlex no longer ships a standalone compatibility server: it is served
through the unified duplex path (`vllm_omni/deploy/personaplex.yaml`) over
[`WS /v1/realtime?duplex=1`](full_duplex_api.md); see the
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
for its clients.
## JoyVL Interaction Server
@@ -80,7 +38,7 @@ python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \
```
| Method and route | Purpose |
|------------------|---------|
| ------------------ | --------- |
| `GET /health` | Readiness check |
| `GET /v1/models` | Reports the configured interaction model |
| `POST /v1/chat/completions` | Processes one frame or interaction tick |
docs/usage/faq.md
+1 -1
View File
@@ -18,4 +18,4 @@ A: AWQ is available through the inherited vLLM quantization registry, and vLLM-O
> Q: Does vLLM-Omni support multimodal streaming input and output?
A: Yes, for specific models and serving paths. Current capabilities include [streaming video input](../serving/video_stream_api.md), [streaming diffusion output](../user_guide/diffusion/execution_modes.md#streaming-output), and experimental full-duplex real-time audio described in the [project overview](../README.md). Check the model and API documentation because input and output support is model-specific.
A: Yes, for specific models and serving paths. Current capabilities include [streaming video input](../serving/video_stream_api.md), [streaming diffusion output](../user_guide/diffusion/execution_modes.md#streaming-output), and [full-duplex realtime audio](../serving/realtime_duplex_api.md). Check the model and API documentation because input and output support is model-specific.
examples/offline_inference/personaplex/personaplex_offline.py
-77
View File
@@ -1,77 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Offline PersonaPlex inference through the vllm-omni fullduplex backend.
This is the runnable end-to-end proof for the integration: it drives a user WAV
through :class:`PersonaPlexEngine` + :class:`PersonaPlexSession` (our wrappers
around the Moshi/Mimi lockstep loop) and writes the agent's audio + inner-monologue
text. It mirrors ``python -m moshi.offline`` but goes through our ``FrameStepper``
seam, so a green run validates that the session/adapter plumbing is faithful.
Prereqs: HF access to ``nvidia/personaplex-7b-v1`` (``export HF_TOKEN=...``). The
stack is moshi-free (native temporal + depformer + streaming Mimi via
``transformers``); decoding is greedy.
Example:
python examples/offline_inference/personaplex/personaplex_offline.py \\
--input-wav input_assistant.wav --output-wav out.wav \\
--voice-prompt NATF2.pt --persona "You are a wise and friendly teacher."
"""
from __future__ import annotations
import argparse
import json
import numpy as np
from vllm_omni.experimental.fullduplex.personaplex import (
PersonaPlexConfig,
PersonaPlexEngine,
PersonaPlexSession,
)
from vllm_omni.experimental.fullduplex.personaplex.config import DEFAULT_PERSONA
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--input-wav", required=True, help="User-side input WAV")
parser.add_argument("--output-wav", required=True, help="Agent audio output WAV")
parser.add_argument("--output-text", default=None, help="Optional JSON for agent text")
parser.add_argument("--voice-prompt", default="NATF2.pt", help="Voice clone (bundled .pt or .wav)")
parser.add_argument("--persona", default=DEFAULT_PERSONA, help="System role text")
parser.add_argument("--hf-repo", default="nvidia/personaplex-7b-v1")
parser.add_argument("--device", default="cuda")
parser.add_argument("--cpu-offload", action="store_true")
args = parser.parse_args()
import sphn # local import: only the runnable driver needs the audio I/O dep
config = PersonaPlexConfig(
hf_repo=args.hf_repo,
voice_prompt=args.voice_prompt,
persona=args.persona,
device=args.device,
cpu_offload=args.cpu_offload,
)
engine = PersonaPlexEngine(config).load()
session = PersonaPlexSession(engine, config)
session.open()
# Load exactly like the reference (moshi.lm.load_audio): native read then
# explicit resample, so greedy runs stay bit-faithful to `python -m moshi.offline`.
user_pcm, src_sr = sphn.read(args.input_wav)
user_pcm = sphn.resample(user_pcm, src_sample_rate=src_sr, dst_sample_rate=config.sample_rate)
audio, text = session.run(np.asarray(user_pcm[0], dtype=np.float32))
sphn.write_wav(args.output_wav, audio, config.sample_rate)
print(f"wrote {audio.shape[0] / config.sample_rate:.2f}s of agent audio to {args.output_wav}")
print(f"agent text: {text!r}")
if args.output_text:
with open(args.output_text, "w") as f:
json.dump({"text": text}, f, ensure_ascii=False)
if __name__ == "__main__":
main()
examples/online_serving/barge_in_client.py
+272
View File
@@ -0,0 +1,272 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Full-duplex use case: talk over the assistant (barge-in).
Scenario (see ``barge_in_client_flow.md`` for the diagram)
---------------------------------------------------------------
1. The user asks a question (``--question-wav``); the model answers in speech.
2. Mid-answer, the user starts talking again (``--interrupt-wav``) — a
follow-up or correction — while assistant audio is still streaming.
3. The model handles the overlap: depending on the session overlap policy and
speech length, the in-flight response is barged in (cancelled) or the
commit is deferred to the turn end, and a new response answers the
interruption. Either way the user never waits for the first answer.
The demo is model-neutral: it drives any duplex model through the public
:class:`vllm_omni.clients.duplex.DuplexClient` over
``/v1/realtime?duplex=1``. Model-specific session shape comes from the
per-model presets under ``vllm_omni.clients.<model>`` selected with
``--preset`` (default ``minicpmo_4_5``, whose model decides listen/speak on
its own — no VAD).
Run the server first (for MiniCPM-o 4.5: ``bash minicpmo/barge_in_serve.sh``, which
starts the ``vllm_omni/deploy/minicpmo_4_5.yaml`` profile with ``session_mode: duplex``),
then:
python barge_in_client.py \
--url ws://127.0.0.1:8099/v1/realtime \
--model openbmb/MiniCPM-o-4_5 \
--ref-audio /path/to/reference_voice.wav \
--question-wav question_16k.wav \
--interrupt-wav follow_up_16k.wav \
--output-dir ./duplex_out
Input WAVs must be mono 16 kHz PCM16; when a preset's session takes a
different capture format (the PersonaPlex preset streams 24 kHz float32),
the audio is converted before streaming. ``--ref-audio`` is required by the
MiniCPM-o preset (without it the session is rejected with
``ref_audio_required``). Outputs land in ``--output-dir`` as one WAV per
response — ``response_1_cancelled.wav`` for a barged-in answer,
``response_1_completed.wav`` for one that finished — plus ``summary.json``.
"""
from __future__ import annotations
import argparse
import asyncio
import json
import sys
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from vllm_omni.clients.duplex import ( # noqa: E402
AudioFormat,
DuplexClient,
EventCollector,
SessionConfig,
acknowledge_collected_playback,
audio_data_url,
read_pcm16_wav,
wait_for_condition,
write_pcm16_wav,
)
_TRANSCRIPT_DELTA_EVENTS = {
"response.audio_transcript.delta",
"response.output_text.delta",
"response.text.delta",
}
# Input WAVs are mono 16 kHz PCM16; each preset's session may negotiate a
# different capture format (PersonaPlex takes 24 kHz float32) and a different
# model-unit-aligned chunk size (PersonaPlex frames are exactly 80 ms).
_PRESET_DEFAULT_CHUNK_MS = {"minicpmo_4_5": 100, "personaplex": 80, "none": 100}
def _convert_to_session_format(pcm16_16k: bytes, target: AudioFormat) -> bytes:
"""Convert mono 16 kHz PCM16 to the session's input format."""
if target.encoding == "pcm16" and target.sample_rate_hz == 16_000:
return pcm16_16k
import numpy as np
samples = np.frombuffer(pcm16_16k, dtype=np.int16).astype(np.float32) / 32768.0
if target.sample_rate_hz != 16_000 and len(samples) > 1:
target_len = round(len(samples) * target.sample_rate_hz / 16_000)
positions = np.linspace(0.0, len(samples) - 1, num=target_len)
samples = np.interp(positions, np.arange(len(samples)), samples).astype(np.float32)
if target.encoding == "pcm_f32le":
return samples.tobytes()
if target.encoding == "pcm16":
return (np.clip(samples, -1.0, 1.0) * 32767.0).astype(np.int16).tobytes()
raise ValueError(f"Unsupported session input encoding: {target.encoding!r}")
def _session_config(preset: str, *, ref_audio_path: str | None) -> SessionConfig:
ref_audio = audio_data_url(Path(ref_audio_path).expanduser()) if ref_audio_path else None
if preset == "minicpmo_4_5":
from vllm_omni.clients.minicpmo_4_5 import create_duplex_session_config
return create_duplex_session_config(ref_audio=ref_audio)
if preset == "personaplex":
from vllm_omni.clients.personaplex import create_duplex_session_config
return create_duplex_session_config()
return SessionConfig(ref_audio=ref_audio)
def _fold_responses(collector: EventCollector) -> dict[str, dict[str, object]]:
"""Fold the event stream into per-response audio, text, and status."""
out: dict[str, dict[str, object]] = {}
for response_id in collector.response_ids:
out[response_id] = {
"audio": collector.audio_bytes(response_id),
"text": "",
"status": "in_progress",
"sample_rate_hz": collector.output_sample_rate_hz,
}
for event in collector.events:
etype = event.get("type")
response_id = collector.response_id(event)
item = out.get(response_id) if isinstance(response_id, str) else None
if etype in _TRANSCRIPT_DELTA_EVENTS and item is not None:
delta = event.get("delta")
if isinstance(delta, str):
item["text"] += delta
elif etype == "response.done" and item is not None:
response = event.get("response")
status = response.get("status") if isinstance(response, dict) else None
if item["status"] == "in_progress":
item["status"] = "cancelled" if status == "cancelled" else "completed"
elif etype == "audio.cancelled":
targets = [item] if item is not None else list(out.values())
for target in targets:
if target["status"] == "in_progress":
target["status"] = "cancelled"
return out
async def run(args: argparse.Namespace) -> int:
config = _session_config(args.preset, ref_audio_path=args.ref_audio)
question = _convert_to_session_format(read_pcm16_wav(Path(args.question_wav)), config.input_audio)
interrupt = _convert_to_session_format(read_pcm16_wav(Path(args.interrupt_wav)), config.input_audio)
chunk_ms = args.chunk_ms if args.chunk_ms is not None else _PRESET_DEFAULT_CHUNK_MS.get(args.preset, 100)
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
collector = EventCollector()
client = DuplexClient(
args.url,
model=args.model,
config=config,
reconnect=None,
heartbeat_interval_s=None,
handshake_timeout_s=args.timeout_s,
)
# GREET: open the voice session (handshake happens on enter).
async with client:
consume_task = asyncio.create_task(collector.consume(client))
# ASK: stream the question at realtime pace, then commit the turn.
# The model decides on its own to answer (no VAD).
print("[1] streaming question ...", file=sys.stderr)
await client.stream_pcm(question, chunk_ms=chunk_ms)
await client.commit(create_response=False)
await wait_for_condition(
lambda: collector.count("response.created") > 0,
timeout_s=args.timeout_s,
label="first response.created",
)
print("[1] assistant is answering", file=sys.stderr)
# Let the answer play for a moment, as a listening user would.
await asyncio.sleep(args.listen_s)
# INTERRUPT: talk over the assistant while its audio still streams.
# The serving overlap policy decides: enough overlapped speech barges
# in (the first response is cancelled) or a short remark is deferred
# until the model yields the turn.
print("[2] interrupting mid-answer ...", file=sys.stderr)
ids_before_interrupt = set(collector.response_ids)
await client.stream_pcm(interrupt, chunk_ms=chunk_ms)
await client.commit(create_response=False)
# ANSWER AGAIN: wait for a response created *after* the interruption
# to finish. The first answer may still be streaming here; matching on
# any response.done would let its completion satisfy the wait and cut
# off the actual follow-up answer at close.
def follow_up_answer_done() -> bool:
return any(
event.get("type") == "response.done"
and collector.response_id(event) is not None
and collector.response_id(event) not in ids_before_interrupt
for event in collector.events
)
await wait_for_condition(
follow_up_answer_done,
timeout_s=args.timeout_s,
label="response.done for the response answering the interruption",
)
# HANG UP: report playback, close the session cleanly.
await acknowledge_collected_playback(client, collector)
await client.close(timeout_s=args.timeout_s)
try:
await asyncio.wait_for(consume_task, timeout=5.0)
except (asyncio.TimeoutError, asyncio.CancelledError):
consume_task.cancel()
# Report: one WAV per response; statuses show what the overlap did.
responses = _fold_responses(collector)
summary: list[dict[str, object]] = []
for index, (response_id, item) in enumerate(responses.items(), start=1):
audio = item["audio"]
assert isinstance(audio, bytes)
sample_rate_hz = int(item["sample_rate_hz"]) # type: ignore[arg-type]
wav_path = output_dir / f"response_{index}_{item['status']}.wav"
if audio:
write_pcm16_wav(wav_path, audio, sample_rate_hz=sample_rate_hz)
summary.append(
{
"response_id": response_id,
"status": item["status"],
"audio_s": round(len(audio) / (sample_rate_hz * 2), 2),
"text": item["text"],
"wav": wav_path.name if audio else None,
}
)
(output_dir / "summary.json").write_text(json.dumps(summary, indent=2, ensure_ascii=False))
(output_dir / "events.jsonl").write_text(
"".join(json.dumps(event, ensure_ascii=False) + "\n" for event in collector.events),
encoding="utf-8",
)
print(json.dumps(summary, indent=2, ensure_ascii=False))
cancelled = sum(1 for item in summary if item["status"] == "cancelled")
print(
f"\n{len(summary)} responses ({cancelled} interrupted); outputs in {output_dir}",
file=sys.stderr,
)
return 0
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--url", default="ws://127.0.0.1:8099/v1/realtime")
parser.add_argument("--model", default="openbmb/MiniCPM-o-4_5")
parser.add_argument(
"--preset",
choices=["minicpmo_4_5", "personaplex", "none"],
default="minicpmo_4_5",
help="per-model session preset from vllm_omni.clients.<model>",
)
parser.add_argument("--ref-audio", help="reference voice WAV (the MiniCPM-o preset requires it)")
parser.add_argument("--question-wav", required=True, help="mono 16 kHz PCM16 question")
parser.add_argument("--interrupt-wav", required=True, help="mono 16 kHz PCM16 follow-up spoken mid-answer")
parser.add_argument("--output-dir", default="./duplex_out")
parser.add_argument(
"--chunk-ms",
type=int,
default=None,
help="append chunk size; defaults to the preset's model-unit-aligned size (minicpmo_4_5: 100, personaplex: 80)",
)
parser.add_argument("--listen-s", type=float, default=2.0, help="how long to listen before interrupting")
parser.add_argument("--timeout-s", type=float, default=120.0)
return asyncio.run(run(parser.parse_args()))
if __name__ == "__main__":
raise SystemExit(main())
examples/online_serving/barge_in_client_flow.md
+54
View File
@@ -0,0 +1,54 @@
# `barge_in_client.py` — what happens, at a glance
The demo plays one everyday conversation pattern against the full-duplex
server: ask something, get talked-over mid-answer, and see the assistant
cope. High-level flow below — for the exact wire events, read the
docstring in `barge_in_client.py`.
```mermaid
sequenceDiagram
autonumber
participant U as user (client script)
participant A as assistant (server + model)
Note over U,A: GREET
U->>A: start a voice session (with the reference voice)
A-->>U: ready
Note over U,A: ASK
U->>A: ask a question (speak the question WAV)
Note right of A: decides on its own to answer<br/>(no VAD — the model chooses)
A-->>U: starts answering aloud
Note over U,A: INTERRUPT
Note over U: listens for ~2 seconds
par both talk at once
U->>A: interrupts with a follow-up
and
A-->>U: still speaking the first answer
end
alt the interruption is substantial
Note right of A: stops mid-sentence (barge-in),<br/>first answer is cut off
else it was just a short remark
Note right of A: finishes the first answer,<br/>then takes the follow-up
end
Note over U,A: ANSWER AGAIN
A-->>U: answers the follow-up aloud
Note over U,A: HANG UP
U->>A: goodbye (close the session)
Note over U: saves one recording per answer<br/>plus a summary of what happened
```
Outcome: the output directory shows which branch ran — a cut-off first
answer is saved as `response_1_cancelled.wav`, a completed one as
`response_1_completed.wav`, and `summary.json` records status, text, and
duration per answer.
Companion: the runtime architecture lives in
[`docs/design/fullduplex.md`](../../docs/design/fullduplex.md); the client
API is `vllm_omni.clients.duplex.DuplexClient`, documented with the wire
protocol in
[`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md).
examples/online_serving/minicpmo/README.md
+5 -3
View File
@@ -4,9 +4,11 @@ This directory contains MiniCPM-o 4.5 online serving demos for vLLM-Omni.
Inputs can include text, image, audio, or video; outputs are text and optional
24 kHz speech.
For the experimental native duplex runtime architecture, lifecycle invariants,
For the native duplex runtime architecture, lifecycle invariants,
capability boundary, and validation scope, see
[`vllm_omni/experimental/fullduplex/DESIGN.md`](../../../vllm_omni/experimental/fullduplex/DESIGN.md).
[`docs/design/fullduplex.md`](../../../docs/design/fullduplex.md); for the
`DuplexClient` API and the `/v1/realtime?duplex=1` wire protocol, see
[`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md).
## Installation
@@ -231,7 +233,7 @@ Start the duplex generation server:
```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni --trust-remote-code \
--deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \
--deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
--served-model-name openbmb/MiniCPM-o-4_5 \
--host 0.0.0.0 --port 8099
```
examples/online_serving/minicpmo/barge_in_serve.sh
+25
View File
@@ -0,0 +1,25 @@
#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
# Server side for barge_in_client.py: start MiniCPM-o 4.5 in native
# full-duplex mode.
#
# The duplex deploy profile registers the WS /v1/duplex endpoint (with
# /v1/realtime?duplex=1 projecting onto the same handler), enables the
# engine duplex control plane, and bounds the Thinker/Talker/Code2Wav
# stages to four live duplex sessions on one GPU. See
# vllm_omni/deploy/minicpmo_4_5.yaml (session_mode: duplex) for the session
# limits; every other duplex_session knob keeps its runtime default
# (idle TTL 300s, disconnect grace 30s, 16MiB pending input per session).
set -euo pipefail
MODEL="${MODEL:-openbmb/MiniCPM-o-4_5}"
PORT="${PORT:-8099}"
exec vllm-omni serve "$MODEL" \
--omni \
--deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
--trust-remote-code \
--host 0.0.0.0 --port "$PORT"
# duplex endpoints: ws://<host>:$PORT/v1/duplex and /v1/realtime?duplex=1
examples/online_serving/minicpmo/realtime_duplex_demo.py
+71 -60
View File
@@ -23,23 +23,26 @@ REPO_ROOT = Path(__file__).resolve().parents[3]
if str(REPO_ROOT) not in sys.path:
sys.path.insert(0, str(REPO_ROOT))
from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
from vllm_omni.clients.duplex import ( # noqa: E402
PCM16_BYTES_PER_SAMPLE,
PCM16_SAMPLE_RATE,
RealtimeDuplexClient,
RealtimeEventCollector,
build_realtime_url,
DuplexClient,
EventCollector,
acknowledge_collected_playback,
read_pcm16_wav,
wait_for,
wait_for_condition,
write_pcm16_wav,
)
from vllm_omni.experimental.fullduplex.client import chunk_period_ms as _chunk_period_ms # noqa: E402
from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
from vllm_omni.clients.duplex import chunk_period_ms as _chunk_period_ms # noqa: E402
from vllm_omni.clients.duplex import ( # noqa: E402
has_residual_model_unit as _has_residual_model_unit,
)
from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
from vllm_omni.clients.duplex import ( # noqa: E402
reference_audio_data_url as _ref_audio_data_url,
)
from vllm_omni.clients.minicpmo_4_5 import ( # noqa: E402
create_duplex_session_config,
)
from vllm_omni.experimental.fullduplex.video_stacking import ( # noqa: E402
concat_frames_b64,
unit_subframe_offsets,
@@ -96,7 +99,7 @@ class _StreamingOutputWriter:
)
class _StreamingEventCollector(RealtimeEventCollector):
class _StreamingEventCollector(EventCollector):
def __init__(self, writer: _StreamingOutputWriter) -> None:
super().__init__()
self._writer = writer
@@ -155,7 +158,7 @@ def _response_in_progress(events: list[dict[str, object]]) -> bool:
)
async def _commit_input_after_playback_checkpoint(client: RealtimeDuplexClient) -> float:
async def _commit_input_after_playback_checkpoint(client: DuplexClient, collector: EventCollector) -> float:
"""Commit input only after preserving already-played assistant history.
Native duplex can finish several responses while the user is still
@@ -166,7 +169,7 @@ async def _commit_input_after_playback_checkpoint(client: RealtimeDuplexClient)
original history position. A second acknowledgement after the commit can
then safely advance response-local playback that drained in the meantime.
"""
await client.acknowledge_playback()
await acknowledge_collected_playback(client, collector)
commit_sent_at_s = time.monotonic()
await client.commit()
return commit_sent_at_s
@@ -434,87 +437,95 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
if not input_pcm16:
raise ValueError("input WAV has no audio")
url = build_realtime_url(
collector = _StreamingEventCollector(stream_writer)
client = DuplexClient(
args.url,
args.model,
autostart=False if args.ref_audio else None,
session_id=args.session_id,
)
client = RealtimeDuplexClient(url)
client.events = _StreamingEventCollector(stream_writer)
async with client:
await client.configure(
args.model,
model=args.model,
config=create_duplex_session_config(
ref_audio=_ref_audio_data_url(args.ref_audio),
session_id=args.session_id,
temperature=args.temperature,
timeout_s=args.timeout_s,
)
stream_event_cursor = len(client.events.events)
frames_sent = await client.stream_pcm16(
),
session_id=args.session_id,
reconnect=None,
heartbeat_interval_s=None,
handshake_timeout_s=args.timeout_s,
)
async with client:
consume_task = asyncio.create_task(collector.consume(client))
# The handshake consumed session.created before the collector
# subscribed; seed it so counts and capability lookups still work.
collector.add({"type": "session.created", "session": client.session_info})
stream_event_cursor = len(collector.events)
frames_sent = await client.stream_pcm(
input_pcm16,
chunk_ms=args.chunk_ms,
realtime=not args.no_realtime_pacing,
video_frames=video_frames,
stacked_video_frames=stacked_video_frames,
)
commit_event_cursor = len(client.events.events)
stream_decision = _latest_model_decision(client.events.events, stream_event_cursor)
commit_event_cursor = len(collector.events)
stream_decision = _latest_model_decision(collector.events, stream_event_cursor)
input_has_residual_model_unit = _has_residual_model_unit(
input_pcm16,
chunk_period_ms=_chunk_period_ms(client.events.events),
chunk_period_ms=_chunk_period_ms(collector.events),
)
wait_for_post_commit_decision = False
commit_sent_at_s = await _commit_input_after_playback_checkpoint(client)
commit_sent_at_s = await _commit_input_after_playback_checkpoint(client, collector)
wait_error: str | None = None
committed_index: int | None = None
post_commit_decision: str | None = None
try:
await wait_for(
lambda: _input_committed_index(client.events.events, commit_event_cursor) is not None,
await wait_for_condition(
lambda: _input_committed_index(collector.events, commit_event_cursor) is not None,
timeout_s=args.timeout_s,
label="input_audio_buffer.committed",
)
committed_index = _input_committed_index(client.events.events, commit_event_cursor)
stream_decision = _latest_model_decision(client.events.events[: committed_index + 1], stream_event_cursor)
committed_index = _input_committed_index(collector.events, commit_event_cursor)
stream_decision = _latest_model_decision(collector.events[: committed_index + 1], stream_event_cursor)
wait_for_post_commit_decision = input_has_residual_model_unit or _response_in_progress(
client.events.events[: committed_index + 1]
collector.events[: committed_index + 1]
)
if wait_for_post_commit_decision:
await wait_for(
lambda: _post_commit_model_decision(client.events.events, committed_index) is not None,
await wait_for_condition(
lambda: _post_commit_model_decision(collector.events, committed_index) is not None,
timeout_s=args.timeout_s,
label="post-commit model decision or response drain",
)
post_commit_decision = _post_commit_model_decision(client.events.events, committed_index)
post_commit_decision = _post_commit_model_decision(collector.events, committed_index)
except TimeoutError as exc:
wait_error = str(exc)
await client.acknowledge_playback()
await acknowledge_collected_playback(client, collector)
close_error: str | None = None
await client.close(timeout_s=args.timeout_s)
# close() returns when the client's reader observed session.closed;
# the collector is a separate subscriber task, so wait for it to
# drain before checking, or a normal close reads as a failure.
try:
await client.close_session(timeout_s=args.timeout_s)
except TimeoutError as exc:
close_error = str(exc)
await asyncio.wait_for(consume_task, timeout=5.0)
except (asyncio.TimeoutError, asyncio.CancelledError):
consume_task.cancel()
if collector.count("session.closed") == 0:
close_error = "Timed out waiting for session.closed"
audio = client.events.audio_bytes()
first_text_at_s = client.events.first_received_at(
audio = collector.audio_bytes()
first_text_at_s = collector.first_received_at(
"response.audio_transcript.delta",
"response.output_text.delta",
after_s=commit_sent_at_s,
)
first_audio_at_s = client.events.first_received_at(
first_audio_at_s = collector.first_received_at(
"response.audio.delta",
after_s=commit_sent_at_s,
)
response_created_at_s = client.events.first_received_at(
response_created_at_s = collector.first_received_at(
"response.created",
after_s=commit_sent_at_s,
)
response_done_at_s = client.events.first_received_at(
response_done_at_s = collector.first_received_at(
"response.done",
after_s=commit_sent_at_s,
)
audio_duration_s = len(audio) / (client.events.output_sample_rate_hz * 2)
audio_duration_s = len(audio) / (collector.output_sample_rate_hz * 2)
response_generation_s = (
response_done_at_s - response_created_at_s
if response_done_at_s is not None and response_created_at_s is not None
@@ -522,27 +533,27 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
)
transcript_deltas = [
str(event.get("delta", ""))
for event in client.events.events
for event in collector.events
if event.get("type")
in {
"response.audio_transcript.delta",
"response.output_text.delta",
}
]
response_id = client.events.response_ids[0] if client.events.response_ids else None
timing = client.events.timing_summary(
response_id = collector.response_ids[0] if collector.response_ids else None
timing = collector.timing_summary(
after_s=commit_sent_at_s,
input_committed_at_s=commit_sent_at_s,
response_id=response_id,
)
errors = client.events.errors()
errors = collector.errors()
if wait_error:
errors.append({"type": "client.timeout", "message": wait_error})
if close_error:
errors.append({"type": "client.timeout", "message": close_error})
model_decision = post_commit_decision or stream_decision
(output_dir / "events.jsonl").write_text(
"".join(json.dumps(event, ensure_ascii=False) + "\n" for event in client.events.events),
"".join(json.dumps(event, ensure_ascii=False) + "\n" for event in collector.events),
encoding="utf-8",
)
(output_dir / "output.pcm").write_bytes(audio)
@@ -550,13 +561,13 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
write_pcm16_wav(
output_dir / "output.wav",
audio,
sample_rate_hz=client.events.output_sample_rate_hz,
sample_rate_hz=collector.output_sample_rate_hz,
)
result = {
"ok": (
client.events.count("session.created") > 0
and client.events.count("session.closed") > 0
collector.count("session.created") > 0
and collector.count("session.closed") > 0
and not errors
and model_decision is not None
and (bool(audio) or not args.require_audio)
@@ -568,12 +579,12 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
"decision_required": wait_for_post_commit_decision,
"input_had_residual_model_unit": input_has_residual_model_unit,
"response_listen_count": _event_count_after(
client.events.events,
collector.events,
"response.listen",
committed_index,
),
"response_done_count": _event_count_after(
client.events.events,
collector.events,
"response.done",
committed_index,
),
@@ -585,7 +596,7 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
"audio_bytes": len(audio),
"audio_chunk_count": len(stream_writer.audio_chunk_paths),
"audio_chunk_files": [str(path) for path in stream_writer.audio_chunk_paths],
"output_sample_rate_hz": client.events.output_sample_rate_hz,
"output_sample_rate_hz": collector.output_sample_rate_hz,
"latency": {
"ttft_ms": (
round((first_text_at_s - commit_sent_at_s) * 1000, 2) if first_text_at_s is not None else None
@@ -611,7 +622,7 @@ async def run_demo(args: argparse.Namespace) -> dict[str, object]:
"measurement_origin": "input_audio_buffer.commit send",
},
"timing": timing,
"response_ids": client.events.response_ids,
"response_ids": collector.response_ids,
"transcript": "".join(transcript_deltas),
"errors": errors,
"output_dir": str(output_dir),
examples/online_serving/minicpmo/realtime_web/app/static/app.js
+2 -2
View File
@@ -86,7 +86,7 @@
url.protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
url.searchParams.set('duplex', '1');
url.searchParams.set('model', config.model || 'openbmb/MiniCPM-o-4_5');
url.searchParams.set('minicpmo45_native_duplex', '1');
url.searchParams.set('native_duplex', '1');
url.searchParams.set('autostart', '0');
return url.toString();
}
@@ -458,7 +458,7 @@
settled = true;
const extraBody = {
auto_response: true,
minicpmo45_native_duplex: true,
native_duplex: true,
};
const session = {
modalities: ['audio', 'text'],
examples/online_serving/personaplex/README.md
+31 -52
View File
@@ -1,47 +1,40 @@
# PersonaPlex full-duplex online serving
Serve [`nvidia/personaplex-7b-v1`](https://huggingface.co/nvidia/personaplex-7b-v1)
(a Moshi-based full-duplex speech-to-speech model) with the native vLLM-Omni engine.
(a Moshi-based full-duplex speech-to-speech model) with the native vLLM-Omni engine
through the unified duplex serving stack (`/v1/duplex` and `/v1/realtime?duplex=1`).
The server hosts the **official PersonaPlex web client** at `/` (auto-downloaded from
the model repo) and implements its WebSocket protocol at `/api/chat`, so you talk to the
model live in the browser — mic in, agent speech + inner-monologue text out.
> Experimental. Requires a GPU and Hugging Face access to the gated repo
> Requires a GPU and Hugging Face access to the gated repo
> (`HF_TOKEN` with access to `nvidia/personaplex-7b-v1`).
## Start the server
```bash
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 bash run_server.sh
# or directly:
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \
vllm_omni.experimental.fullduplex.personaplex.serving.server --port 8124
```
## Talk to it
- **Browser (recommended):** open `http://localhost:8124/` and allow the microphone.
Use headphones so the agent does not hear itself.
- **Headless:** stream a 24 kHz mono WAV and save the reply:
```bash
python duplex_client.py --url ws://localhost:8124/api/chat --input user.wav --out reply.wav
```
## Unified Realtime API
The default `vllm_omni/deploy/personaplex.yaml` also enables main's engine-owned
full-duplex control plane:
The default `vllm_omni/deploy/personaplex.yaml` enables the engine-owned
full-duplex control plane (`session_mode: duplex`):
```bash
CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
/path/to/personaplex-7b-v1 \
--omni \
--deploy-config vllm_omni/deploy/personaplex.yaml
```
Validate the actual `/v1/realtime?duplex=1` scheduler path with paced 24 kHz
This exposes:
- `WS /v1/duplex` — the native duplex session dialect
(`session.create` / `input_audio_buffer.append` / `response.output_audio.delta` ...);
- `WS /v1/realtime?duplex=1` — the same sessions projected onto the
OpenAI Realtime event vocabulary (client API and wire protocol:
[`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md)).
PersonaPlex is a pure-lockstep model: every session on a PersonaPlex deployment is
native duplex (`is_enabled()` is unconditionally true), audio flows continuously in
both directions, and there are no client commits or external turn signals
(`supports_client_commit=false`, `supports_external_turn_signal=false`).
## Validate the serving path
Validate the `/v1/realtime?duplex=1` scheduler path with paced 24 kHz
PCM, two concurrent sessions, overflow admission, per-session slot recycling,
and non-silent output:
@@ -52,33 +45,19 @@ python tests/e2e/online_serving/personaplex_realtime_duplex.py \
--output-dir /tmp/personaplex-realtime-duplex
```
This is distinct from the standalone `/api/chat` compatibility server above.
The unified endpoint advertises `supports_barge_in=false`: overlapping speech
is native model behavior, but destructive output interruption and model-state
rewind have not been validated for PersonaPlex.
## Protocol (`/api/chat`)
Binary WebSocket messages, first byte is a tag (identical to `moshi.server`):
| Direction | Tag | Payload |
| --- | --- | --- |
| server → client | `\x00` | ready handshake (after the system-prompt prefill) |
| client → server | `\x01` | Opus-encoded mic audio (24 kHz) |
| server → client | `\x01` | Opus-encoded agent audio |
| server → client | `\x02` | UTF-8 inner-monologue text |
Connect with query params `text_prompt` (persona) and `voice_prompt` (voice file).
A raw-PCM endpoint (`/v1/audio/duplex`, JSON `open` + float32 frames) is also available
for clients that do not want an Opus dependency.
## Notes
- **Run the client near the server.** Real-time 80 ms audio is sensitive to network
latency/jitter; over a high-latency remote link playback can stutter regardless of
engine speed. On localhost it is smooth.
- Greedy decoding by default. One conversation per server by default (a second
concurrent connection is rejected with WS close 1013 until the first ends);
pass `--batch-size N` to host N concurrent conversations on one engine
(elastic batching; connections beyond N are rejected until a slot frees).
- **Run the client near the server.** Real-time 80 ms frame audio is sensitive to
network latency/jitter; over a high-latency remote link playback can stutter
regardless of engine speed. On localhost it is smooth.
- The earlier standalone Moshi-web compatibility server (browser client at `/`,
binary WS protocol at `/api/chat`, raw-PCM `/v1/audio/duplex`) was demo-only
and has been removed; use the unified endpoints above.
- Session config (voice / persona / sampling) is passed per session via
`extra_body`; see
`vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py`.
- Full runbook: [`recipes/NVIDIA/PersonaPlex.md`](../../../recipes/NVIDIA/PersonaPlex.md).
examples/online_serving/personaplex/duplex_client.py
-114
View File
@@ -1,114 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Headless client for the PersonaPlex duplex server (official Moshi protocol).
Streams a 24 kHz mono WAV to ``/api/chat`` exactly as the browser client does
(Opus over WebSocket, tags ``\\x00`` ready / ``\\x01`` audio / ``\\x02`` text),
collects the agent's reply audio + inner-monologue text, and writes a WAV.
The browser UI at ``http://<host>:<port>/`` is the primary demo; this script is
for headless testing / scripting.
python duplex_client.py --url ws://localhost:8124/api/chat \
--input user.wav --out reply.wav
Deps: ``websockets``, ``sphn``, ``soundfile``, ``numpy``.
"""
from __future__ import annotations
import argparse
import asyncio
import numpy as np
import soundfile as sf
import sphn
SR = 24000 # PersonaPlex / Mimi codec rate
async def converse(url: str, pcm: np.ndarray) -> tuple[np.ndarray, str]:
import websockets
reader = sphn.OpusStreamReader(SR) # decode agent audio
writer = sphn.OpusStreamWriter(SR) # encode our mic audio
text: list[str] = []
ready = asyncio.Event()
last = {"t": 0.0}
async with websockets.connect(url, max_size=None) as ws:
async def receive() -> None:
loop = asyncio.get_event_loop()
while True:
try:
msg = await ws.recv()
except Exception:
return
if not isinstance(msg, (bytes, bytearray)) or not msg:
continue
tag, payload = msg[0], msg[1:]
if tag == 0:
ready.set()
elif tag == 1:
reader.append_bytes(bytes(payload))
last["t"] = loop.time()
elif tag == 2:
text.append(payload.decode("utf8", errors="replace"))
recv_task = asyncio.create_task(receive())
await asyncio.wait_for(ready.wait(), timeout=120) # waits for system-prompt prefill
frame = 1920 # 80 ms
for i in range(0, len(pcm), frame):
chunk = pcm[i : i + frame]
if len(chunk) < frame:
chunk = np.concatenate([chunk, np.zeros(frame - len(chunk), dtype=np.float32)])
writer.append_pcm(chunk)
data = writer.read_bytes()
if data:
await ws.send(b"\x01" + data)
await asyncio.sleep(frame / SR) # pace at real time
loop = asyncio.get_event_loop()
last["t"] = loop.time()
while loop.time() - last["t"] < 2.5: # drain until the agent stops talking
await asyncio.sleep(0.1)
recv_task.cancel()
chunks: list[np.ndarray] = []
while True:
out = reader.read_pcm()
if out is None or out.shape[-1] == 0:
break
chunks.append(np.asarray(out, dtype=np.float32))
audio = np.concatenate(chunks) if chunks else np.zeros(0, dtype=np.float32)
return audio, "".join(text)
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--url", default="ws://localhost:8124/api/chat")
ap.add_argument("--input", required=True, help="24 kHz mono WAV of the user turn")
ap.add_argument("--out", default="personaplex_reply.wav")
ap.add_argument("--tail", type=float, default=10.0, help="trailing silence (s) for the agent reply")
ap.add_argument("--persona", default="")
ap.add_argument("--voice", default="NATF2.pt")
args = ap.parse_args()
wav, sr = sf.read(args.input, dtype="float32")
if wav.ndim > 1:
wav = wav.mean(axis=1)
if sr != SR:
raise SystemExit(f"input must be {SR} Hz mono; got {sr} Hz. Resample it first.")
wav = np.concatenate([wav, np.zeros(int(args.tail * SR), dtype=np.float32)])
url = f"{args.url}?text_prompt={args.persona}&voice_prompt={args.voice}"
audio, text = asyncio.run(converse(url, wav))
sf.write(args.out, audio if audio.size else np.zeros(SR, dtype=np.float32), SR)
print(f"agent reply: {audio.size / SR:.2f}s -> {args.out}")
print(f"inner-monologue: {text.strip()!r}")
if __name__ == "__main__":
main()
examples/online_serving/personaplex/run_server.sh
-15
View File
@@ -1,15 +0,0 @@
#!/bin/bash
# PersonaPlex full-duplex online serving.
#
# Serves the OFFICIAL PersonaPlex web client at / (auto-downloaded from
# nvidia/personaplex-7b-v1) and the Moshi WebSocket protocol at /api/chat,
# backed by the native vLLM-Omni engine. Requires HF access to the gated repo.
HOST="${HOST:-0.0.0.0}"
PORT="${PORT:-8124}"
echo "Starting PersonaPlex duplex server on ${HOST}:${PORT}..."
echo "Open http://localhost:${PORT}/ for the official web client (run it near the server)."
python -m vllm_omni.experimental.fullduplex.personaplex.serving.server \
--host "$HOST" --port "$PORT"
recipes/NVIDIA/PersonaPlex.md
+46 -66
View File
@@ -1,7 +1,7 @@
# PersonaPlex
> Live full-duplex speech-to-speech (talk to the model in your browser, it listens
> while it speaks) with persona and voice control, on the native vLLM-Omni engine.
> Live full-duplex speech-to-speech (the model listens while it speaks) with
> persona and voice control, on the native vLLM-Omni engine.
## Summary
@@ -11,8 +11,8 @@
- Task: Full-duplex speech-to-speech. 24 kHz mic audio in, agent speech +
inner-monologue text out, in 80 ms lockstep. Persona (role text) and voice
(zero-shot clone) are set per session.
- Mode: Live duplex WebSocket server with the official browser client, single or
multi-session (`--batch-size`), plus an offline WAV-in / WAV-out example
- Mode: Live duplex WebSocket serving through the unified duplex stack
(`/v1/duplex` and `/v1/realtime?duplex=1`)
- Maintainer: [`@linyueqian`](https://github.com/linyueqian)
## When to use this recipe
@@ -24,23 +24,21 @@ streaming Mimi codec are all native vLLM-Omni modules (Mimi runs on
matching the reference implementation frame for frame on the golden replays used
as the acceptance gate.
PersonaPlex is the first Moshi-class (pure-lockstep) model in the experimental
full-duplex framework (`vllm_omni/experimental/fullduplex/`), and the first with
elastic multi-session batching: one engine hosts `--batch-size` concurrent
conversations, callers join or leave on any 80 ms tick without disturbing live
slots (greedy isolation is bit-exact).
PersonaPlex is the first Moshi-class (pure-lockstep) model on the vLLM-Omni
full-duplex serving stack: it plugs into the generic `/v1/duplex` handler
through the standard plugin seams (`duplex_serving_adapter` /
`duplex_runtime_extension` in its `pipeline.py`) with a model-specific
package at `vllm_omni/model_executor/models/personaplex/duplex/`.
## References
- Serving package:
[`vllm_omni/experimental/fullduplex/personaplex/`](../../vllm_omni/experimental/fullduplex/personaplex/)
- Duplex serving package:
[`vllm_omni/model_executor/models/personaplex/duplex/`](../../vllm_omni/model_executor/models/personaplex/duplex/)
- Native model modules (temporal / depformer / streaming Mimi):
[`vllm_omni/model_executor/models/personaplex/`](../../vllm_omni/model_executor/models/personaplex/)
- Online example (server + headless client):
- Online serving example:
[`examples/online_serving/personaplex/`](../../examples/online_serving/personaplex/)
- Offline example:
[`examples/offline_inference/personaplex/personaplex_offline.py`](../../examples/offline_inference/personaplex/personaplex_offline.py)
- Staged offline pipeline (talker -> Mimi code2wav, async-chunk streaming):
- Staged pipeline (talker -> Mimi code2wav, async-chunk streaming):
[`vllm_omni/deploy/personaplex.yaml`](../../vllm_omni/deploy/personaplex.yaml)
- Integration PR: [vllm-project/vllm-omni#4771](https://github.com/vllm-project/vllm-omni/pull/4771)
- Framework context: [RFC #3745](https://github.com/vllm-project/vllm-omni/issues/3745)
@@ -53,18 +51,20 @@ slots (greedy isolation is bit-exact).
## Why it is different from MiniCPM-o / JoyVL duplex
| | MiniCPM-o 4.5 (#3907) | JoyVL | **PersonaPlex (this)** |
|---|---|---|---|
| --- | --- | --- | --- |
| Cadence | 1 s chunk groups | ~1 fps frames | **80 ms lockstep (12.5 Hz)** |
| Turn control | learned `⟨listen⟩`/`⟨speak⟩` | `</silence>`/`</response>` | **none, pure lockstep** |
| Per step | variable-length token group | text decision | **1 user frame in -> 1 agent frame + 1 text token** |
| Barge-in | at chunk boundary | n/a | **native (model always hears the user)** |
| Session state | chunk-group KV | per-tick HTTP | **persistent ring KV + streaming Mimi state** |
This is the lockstep ("parallel-frame joint") shape of the duplex adapter
patterns: the adapter declares `DuplexCapability.continuous = True`, and
`core.DuplexRuntime` runs ONE eternal response that consumes input frames as they
arrive and drains on close. The flag is a small, default-off lifecycle mode, so
turn-based adapters (JoyVL, MiniCPM-o) are unaffected.
This is the lockstep ("parallel-frame joint") shape of full duplex: audio flows
continuously in both directions with no client commits or external turn signals.
On the unified stack this is expressed purely through the capability payload
(`supports_client_commit=false`, `supports_external_turn_signal=false`,
`supports_barge_in=false`); every session on a PersonaPlex deployment is native
duplex (`is_enabled()` is unconditionally true). Turn-based models
(MiniCPM-o 4.5) are unaffected.
## Architecture (Moshi RQ-Transformer, all native)
@@ -95,9 +95,7 @@ native modules, so other CUDA GPUs with enough memory are expected to work.
- OS: Linux
- Python: 3.10+ (CI covers 3.11 / 3.12)
- vLLM-Omni: PR #4771 branch or current `main` once merged
- Extra dependency for the live server: `pip install sphn` (Opus framing;
`aiohttp` already ships with vLLM)
- vLLM-Omni: current `main`
- `export HF_TOKEN=...` with access to the gated
`nvidia/personaplex-7b-v1` repo (accept the license on the model page)
@@ -107,67 +105,49 @@ First run auto-downloads from the model repo: `model.safetensors`,
#### Command
Live duplex server (official browser client served at `/`):
Live duplex serving through the unified stack:
```bash
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 \
python -m vllm_omni.experimental.fullduplex.personaplex.serving.server \
--port 8124 --voice NATF2.pt
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
/path/to/personaplex-7b-v1 \
--omni \
--deploy-config vllm_omni/deploy/personaplex.yaml
```
Multi-session on the same GPU (elastic batching, callers join/leave any tick):
```bash
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 \
python -m vllm_omni.experimental.fullduplex.personaplex.serving.server \
--port 8124 --batch-size 4
```
Then open `http://localhost:8124/` and allow the microphone (use headphones so
the agent does not hear itself). Headless alternative:
```bash
python examples/online_serving/personaplex/duplex_client.py \
--url ws://localhost:8124/api/chat --input user.wav --out reply.wav
```
Offline WAV-in / WAV-out (no server):
```bash
python examples/offline_inference/personaplex/personaplex_offline.py \
--input-wav question.wav --output-wav reply.wav --output-text reply.json \
--voice-prompt NATF2.pt --persona "You are a wise and friendly teacher."
```
This exposes `WS /v1/duplex` (native duplex dialect) and
`WS /v1/realtime?duplex=1` (OpenAI Realtime projection; client API and wire
protocol in [`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md)).
Voice and persona are set per session via `extra_body`.
#### Verification
```bash
# GPU-free contract tests (stubbed FrameStepper)
pytest tests/e2e/features/fullduplex/test_personaplex_adapter.py \
tests/e2e/features/fullduplex/test_personaplex_session.py \
tests/e2e/features/fullduplex/test_personaplex_batched.py -q
# GPU-free contract tests (stage0 runtime + unified serving adapter)
pytest tests/model_executor/models/personaplex/duplex/ -q
# GPU e2e: batched isolation + slot recycle (coherence check needs a real
# question WAV via PPLEX_QUESTION_WAV)
pytest tests/e2e/features/fullduplex/test_personaplex_elastic.py -q
# GPU e2e: paced 24 kHz PCM over /v1/realtime?duplex=1, two concurrent
# sessions, overflow admission, slot recycling, non-silent output
python tests/e2e/online_serving/personaplex_realtime_duplex.py \
--model /path/to/personaplex-7b-v1 \
--input-wav /path/to/speech.wav \
--output-dir /tmp/personaplex-realtime-duplex
```
A green offline run producing a coherent spoken reply (and inner-monologue text
in `reply.json`) is the quickest end-to-end check.
#### Notes
- Realtime budget is 80 ms/frame. Verified eager per-tick latency at
`--batch-size 4` is ~70-74 ms on this hardware class, i.e. all four
- Realtime budget is 80 ms/frame. Verified eager per-tick latency at four
concurrent sessions is ~70-74 ms on this hardware class, i.e. all four
conversations stay realtime; single-session has comfortable headroom.
- Decoding is greedy only (temperature/top-k knobs are intentionally not
exposed): greedy is what the parity gates pin against the reference
implementation, and sampling amplifies sub-bit numeric drift into divergence.
- With `--batch-size N`, connections beyond N are rejected until a slot frees;
a recycled slot is bit-exact with a fresh engine for the same inputs.
- Session concurrency is governed by the deploy config
(`vllm_omni/deploy/personaplex.yaml`); connections beyond the limit are
rejected until a slot frees. A recycled slot is bit-exact with a fresh
engine for the same inputs.
- Voice prompts: bundled `voices.tgz` names (`NATF*`/`NATM*` natural,
`VARF*`/`VARM*` varied) or a path to your own `.pt`/`.wav`.
- Run the browser client near the server. The 80 ms cadence is sensitive to
- Run the client near the server. The 80 ms cadence is sensitive to
network jitter; over a high-latency link playback can stutter regardless of
engine speed. On localhost it is smooth.
- Sliding window is 3000 frames (~4 min). The ring KV recycles beyond that;
tests/benchmarks/patch/test_patch.py
+3 -3
View File
@@ -28,7 +28,7 @@ from vllm_omni.benchmarks.patch.patch import (
async_request_openai_realtime_duplex,
should_request_stage_metrics,
)
from vllm_omni.experimental.fullduplex.client import RealtimeEventCollector
from vllm_omni.clients.duplex import EventCollector
pytestmark = [pytest.mark.core_model, pytest.mark.benchmark, pytest.mark.cpu]
@@ -131,7 +131,7 @@ async def test_seed_tts_realtime_duplex_exports_per_request_metrics(monkeypatch)
def __init__(self, url):
assert url == "ws://localhost:8000/v1/realtime?duplex=1"
self.events = RealtimeEventCollector()
self.events = EventCollector()
self.configure_kwargs = None
self.sent = []
self.response_count = 0
@@ -206,7 +206,7 @@ async def test_seed_tts_realtime_duplex_exports_per_request_metrics(monkeypatch)
return None
monkeypatch.setattr(
"vllm_omni.benchmarks.patch.patch.RealtimeDuplexClient",
"vllm_omni.benchmarks.patch.patch._RealtimeTTSProbe",
FakeRealtimeClient,
)
request_input = RequestFuncInput(
tests/benchmarks/test_omniinteract.py
+96 -22
View File
@@ -25,9 +25,9 @@ from vllm_omni.benchmarks import serve as benchmark_serve
from vllm_omni.benchmarks.data_modules import omniinteract_dataset as data
from vllm_omni.benchmarks.metrics.metrics import calculate_metrics
from vllm_omni.benchmarks.patch import patch as benchmark_patch
from vllm_omni.clients.duplex import EventCollector
from vllm_omni.entrypoints.cli.benchmark.cli_args import preprocess_serve_args
from vllm_omni.entrypoints.cli.benchmark.serve import OmniBenchmarkServingSubcommand
from vllm_omni.experimental.fullduplex.client import RealtimeEventCollector
from vllm_omni.utils.tracking_parser import TrackingArgumentParser
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.benchmark]
@@ -42,8 +42,8 @@ def _case(tmp_path: Path, *, subset: str = "1q1a", name: str = "video.mp4") -> d
return data.OmniInteractCase(subset, name, video, annotation, "multi_turn")
def _collector(*events: tuple[dict[str, object], float]) -> RealtimeEventCollector:
collector = RealtimeEventCollector()
def _collector(*events: tuple[dict[str, object], float]) -> EventCollector:
collector = EventCollector()
for event, received_at in events:
collector.add(event, received_at_s=received_at)
return collector
@@ -136,7 +136,7 @@ def _successful_output(
*,
audio_time: float = 10.0,
status: str = "completed",
) -> tuple[data.OmniInteractCase, RealtimeEventCollector, oi.OmniInteractCaseResult]:
) -> tuple[data.OmniInteractCase, EventCollector, oi.OmniInteractCaseResult]:
case = _case(tmp_path)
collector = _collector(
(_created(), 9.9),
@@ -151,7 +151,7 @@ def _successful_output(
def _write_success(
root: Path,
case: data.OmniInteractCase,
collector: RealtimeEventCollector,
collector: EventCollector,
result: oi.OmniInteractCaseResult,
**kwargs,
):
@@ -285,7 +285,7 @@ def test_response_ledger_rejects_identity_errors(events, match: str):
class _CompletionClient:
def __init__(self, collector: RealtimeEventCollector):
def __init__(self, collector: EventCollector):
self.events, self.acks = collector, []
def raise_if_reader_stopped(self) -> None:
@@ -334,13 +334,71 @@ async def test_deferred_commit_ignores_the_prior_response_terminal():
@pytest.mark.asyncio
async def test_playback_acks_only_after_serial_audio_drain():
collector = _collector((_audio(samples=2400), 10.0), (_done(), 10.01))
async def test_playback_acks_incrementally_along_the_serial_clock():
# 1 s of audio playing from t=10.0: the first ack checkpoints the
# response as soon as audio arrives, later acks report cumulative
# progress at least every _ACK_STEP_MS of new audio, and the completion
# ack lands exactly at the total once the serial clock drains it.
collector = _collector((_audio(samples=24_000), 10.0), (_done(), 10.01))
client, playback = _CompletionClient(collector), oi._Playback()
await playback.acknowledge(client, now=10.05)
assert client.acks == []
assert client.acks == [("r1", 50)] # checkpoint ack, reserves the slot
await playback.acknowledge(client, now=10.1)
assert client.acks == [("r1", 100)]
assert client.acks == [("r1", 50)] # step not reached, no spam
await playback.acknowledge(client, now=10.3)
assert client.acks == [("r1", 50), ("r1", 300)]
await playback.acknowledge(client, now=11.0)
assert client.acks == [("r1", 50), ("r1", 300), ("r1", 1000)]
assert playback.completion_acked == {"r1"}
await playback.acknowledge(client, now=11.5)
assert client.acks[-1] == ("r1", 1000) # completion acked once
@pytest.mark.asyncio
async def test_playback_checkpoint_ack_fires_before_audio_plays():
# Audio queued but none played yet: a 0 ms checkpoint ack still goes out
# so a later input commit cannot displace the response's history slot.
collector = _collector((_audio(samples=24_000), 10.0), (_done(), 10.01))
client, playback = _CompletionClient(collector), oi._Playback()
await playback.acknowledge(client, now=10.0)
assert client.acks == [("r1", 0)]
@pytest.mark.asyncio
async def test_acks_continue_after_committed_input():
# A later committed user input must not stop the acks: the checkpointed
# response's history slot is updated in place, so the drain ack for the
# tail is still delivered afterwards.
collector = _collector(
(_audio(samples=24_000), 10.0),
(_done(), 10.01),
({"type": "input_audio_buffer.committed"}, 10.4),
)
client, playback = _CompletionClient(collector), oi._Playback()
await playback.acknowledge(client, now=10.05)
assert client.acks == [("r1", 50)]
await playback.acknowledge(client, now=11.5)
assert client.acks == [("r1", 50), ("r1", 1000)]
assert playback.completion_acked == {"r1"}
def test_tolerated_playback_ack_rejection_is_a_warning_not_a_failure():
collector = _collector(
(
{
"type": "error",
"error": {"type": "invalid_request_error", "code": "playback_ack_too_late"},
},
1.0,
),
)
warnings: list[str] = []
oi._raise_if_session_terminated(collector, 0, warnings=warnings)
assert warnings == ["tolerated in-flight server rejection: playback_ack_too_late"]
collector.add({"type": "error", "error": {"code": "bad_event", "message": "boom"}}, received_at_s=1.1)
with pytest.raises(RuntimeError, match="bad_event"):
oi._raise_if_session_terminated(collector, 0, warnings=warnings)
@pytest.mark.parametrize(
@@ -421,30 +479,34 @@ def test_publication_failure_removes_partial_success_bundle(tmp_path: Path, monk
def test_atomic_write_preserves_destination_on_failure(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
destination = tmp_path / "output.wav"
destination.write_bytes(b"old")
monkeypatch.setattr(oi, "write_pcm16_wav", lambda *a, **k: (_ for _ in ()).throw(OSError("write")))
monkeypatch.setattr(
"vllm_omni.clients.duplex.write_pcm16_wav",
lambda *a, **k: (_ for _ in ()).throw(OSError("write")),
)
with pytest.raises(OSError):
oi._atomic_write_wav(destination, b"\0\0", 24_000)
assert destination.read_bytes() == b"old"
assert not list(tmp_path.glob(".*.tmp"))
class _RealtimeClient:
class _RealtimeClient(oi._RealtimeSession):
"""Fake transport over the real session wrapper: URL and session-config
construction stay real, only the network-facing calls are stubbed."""
instances: list[_RealtimeClient] = []
def __init__(self, url: str, **kwargs):
self.url, self.events, self.acks, self.configure_kwargs = url, RealtimeEventCollector(), [], {}
def __init__(self, config: oi.OmniInteractBenchmarkConfig, session_id: str, reference_audio: str):
super().__init__(config, session_id, reference_audio)
self.acks: list[tuple[str, int]] = []
self.instances.append(self)
async def __aenter__(self):
self.events.add({"type": "session.created", "session": {"capabilities": {"chunk_period_ms": 1000}}})
return self
async def __aexit__(self, *args):
return None
async def configure(self, model: str, **kwargs) -> None:
self.configure_kwargs = kwargs
self.events.add({"type": "session.created", "session": {"capabilities": {"chunk_period_ms": 1000}}})
async def send(self, event: dict[str, object]) -> None:
return None
@@ -480,7 +542,7 @@ async def test_public_runner_executes_one_prepared_session(tmp_path: Path, monke
extra_body={"custom": "value"},
)
prepared = data.OmniInteractPreparedInput(1.0, b"\0\0\0\0", ("frame",), "data:audio/wav;base64,ref")
monkeypatch.setattr(oi, "RealtimeDuplexClient", _RealtimeClient)
monkeypatch.setattr(oi, "_RealtimeSession", _RealtimeClient)
monkeypatch.setattr(oi, "_COMPLETION_SETTLE_S", 0.0)
result = await oi.run_omniinteract_case(
case,
@@ -491,8 +553,15 @@ async def test_public_runner_executes_one_prepared_session(tmp_path: Path, monke
assert result.success and result.transcript == "answer" and result._artifact_context is not None
assert not config.output_root.exists()
assert "autostart=0" in _RealtimeClient.instances[-1].url
assert _RealtimeClient.instances[-1].configure_kwargs["extra_body"] == {"custom": "value"}
assert _RealtimeClient.instances[-1].acks == [("r1", 1)]
session_config = _RealtimeClient.instances[-1].session_config
assert session_config.extra_body["custom"] == "value"
assert session_config.extra_body["native_duplex"] is True
assert session_config.ref_audio == "data:audio/wav;base64,ref"
acks = _RealtimeClient.instances[-1].acks
# Cumulative acks for the one response: an optional 0 ms checkpoint the
# moment its audio arrived, then the exact total once drained.
assert acks[-1] == ("r1", 1)
assert acks in ([("r1", 1)], [("r1", 0), ("r1", 1)])
def test_standard_sample_loading_prepares_media_before_timing(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
@@ -521,7 +590,12 @@ def test_standard_sample_loading_prepares_media_before_timing(tmp_path: Path, mo
)
preprocess_serve_args(args)
assert args.endpoint == "/v1/realtime"
monkeypatch.setattr(benchmark_patch, "reference_audio_data_url", lambda _: "data:audio/wav;base64,ref")
# The benchmark patch module imports the duplex client library lazily
# (it must stay out of the CLI import graph), so patch the provider.
monkeypatch.setattr(
"vllm_omni.clients.duplex.reference_audio_data_url",
lambda _: "data:audio/wav;base64,ref",
)
monkeypatch.setattr(benchmark_patch, "prepare_media", lambda *a, **k: (1.0, b"pcm", ["frame"]))
samples = benchmark_patch.get_samples(args, None)
assert len(samples) == args.num_prompts == 3
tests/clients/__init__.py
+2
View File
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
tests/clients/test_duplex_client.py
+895
View File
@@ -0,0 +1,895 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Unit tests for the public duplex client (fake transport, no server)."""
from __future__ import annotations
import asyncio
import base64
import json
import wave
from urllib.parse import parse_qs, urlsplit
import pytest
from vllm_omni.clients.duplex import (
AudioFormat,
ConnectionResumed,
DuplexClient,
DuplexProtocolError,
DuplexSessionClosedError,
EventCollector,
ReconnectPolicy,
SessionConfig,
SessionResumed,
build_realtime_url,
chunk_period_ms,
duplex_unit_boundary_ms,
has_residual_model_unit,
read_pcm16_wav,
reference_audio_data_url,
summarize_session_request_metrics,
write_pcm16_wav,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
SESSION_CREATED = {
"type": "session.created",
"session": {"session_id": "sess-1"},
"incarnation": 0,
"resume_token": "tok-1",
"server_event_seq": 1,
}
SESSION_CLOSED = {"type": "session.closed", "session_id": "sess-1", "server_event_seq": 99}
class FakeSocket:
def __init__(self) -> None:
self.sent: list[dict[str, object]] = []
self.incoming: asyncio.Queue = asyncio.Queue()
self.closed = False
async def send(self, raw: str) -> None:
if self.closed:
raise RuntimeError("socket closed")
self.sent.append(json.loads(raw))
async def recv(self) -> str:
item = await self.incoming.get()
if isinstance(item, BaseException):
raise item
return json.dumps(item)
async def close(self) -> None:
self.closed = True
def feed(self, event: dict[str, object] | BaseException) -> None:
self.incoming.put_nowait(event)
def sent_types(self) -> list[object]:
return [event.get("type") for event in self.sent]
def make_client(*sockets: FakeSocket, **kwargs) -> tuple[DuplexClient, list[str]]:
remaining = list(sockets)
calls: list[str] = []
async def connect(url: str):
calls.append(url)
if not remaining:
raise ConnectionError("no more sockets")
return remaining.pop(0)
kwargs.setdefault("heartbeat_interval_s", None)
kwargs.setdefault("reconnect", None)
kwargs.setdefault("handshake_timeout_s", 5.0)
client = DuplexClient("ws://test-host:8099", model="test-model", connect=connect, **kwargs)
return client, calls
# ---------------------------------------------------------------------------
# SessionConfig
def test_session_config_payload_defaults():
payload = SessionConfig().to_session_payload(model="m", session_id="s")
assert payload["model"] == "m"
assert payload["session_id"] == "s"
assert payload["modalities"] == ["audio", "text"]
assert payload["input_audio_format"] == "pcm16"
assert payload["output_audio_format"] == "pcm16"
# The encodings do not pin the rates; the payload must carry both (the
# server reads the input rate top-level and the output rate from audio).
assert payload["sample_rate_hz"] == 16_000
assert payload["audio"] == {
"input": {"sample_rate_hz": 16_000},
"output": {"sample_rate_hz": 24_000},
}
assert payload["turn_detection"] is None
assert payload["extra_body"] == {"auto_response": True}
assert "voice" not in payload
assert "ref_audio" not in payload
def test_session_config_payload_carries_preset_rates():
config = SessionConfig(
input_audio=AudioFormat("pcm_f32le", 24_000),
output_audio=AudioFormat("pcm16", 24_000),
)
payload = config.to_session_payload(model="m")
assert payload["input_audio_format"] == "pcm_f32le"
assert payload["sample_rate_hz"] == 24_000
assert payload["audio"] == {
"input": {"sample_rate_hz": 24_000},
"output": {"sample_rate_hz": 24_000},
}
def test_audio_format_math():
fmt = AudioFormat("pcm16", 16_000)
assert fmt.byte_count(100) == 3200
assert fmt.duration_ms(3200) == 100.0
f32 = AudioFormat("pcm_f32le", 24_000)
assert f32.byte_count(80) == 1920 * 4
with pytest.raises(ValueError):
_ = AudioFormat("mp3", 16_000).bytes_per_sample
# ---------------------------------------------------------------------------
# Handshake and lifecycle
async def test_handshake_adopts_session_and_acks():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, calls = make_client(sock)
async with client:
assert client.session_id == "sess-1"
assert client.resume_token == "tok-1"
assert calls == ["ws://test-host:8099/v1/realtime?duplex=1&model=test-model&autostart=0"]
assert sock.sent[0]["type"] == "session.update"
assert sock.sent[0]["session"]["model"] == "test-model"
await _drain(lambda: {"type": "session.event_ack", "server_event_seq": 1} in _acks(sock))
close_task = asyncio.create_task(client.close())
await _drain(lambda: "session.close" in sock.sent_types())
sock.feed(SESSION_CLOSED)
await close_task
assert "session.close" in sock.sent_types()
async def test_handshake_error_raises_protocol_error():
sock = FakeSocket()
sock.feed({"type": "error", "error": {"code": "unsupported_audio_format", "message": "bad"}})
client, _ = make_client(sock)
with pytest.raises(DuplexProtocolError) as excinfo:
async with client:
pass
assert excinfo.value.code == "unsupported_audio_format"
async def test_session_expired_raises_from_event_stream():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
sock.feed({"type": "session.expired", "reason": "lease_expired", "server_event_seq": 2})
with pytest.raises(DuplexSessionClosedError) as excinfo:
async for _ in client.events():
pass
assert "lease_expired" in excinfo.value.reason
async def test_resync_required_drops_resume_credential():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, calls = make_client(sock, reconnect=ReconnectPolicy(max_attempts=2, backoff_s=(0.0, 0.0)))
async with client:
assert client.resume_token == "tok-1"
stream = client.events()
sock.feed({"type": "session.resync_required", "session_id": "sess-1", "server_event_seq": 2})
async for event in stream:
if event.type == "session.resync_required":
break
assert client.resume_token is None
# With the credential gone, a transport drop must finalize instead of
# attempting session.resume against a server that stopped journaling.
sock.feed(RuntimeError("transport dropped"))
with pytest.raises(DuplexSessionClosedError):
async for _ in stream:
pass
assert len(calls) == 1
# ---------------------------------------------------------------------------
# Input events
async def test_append_audio_tracks_cumulative_end_ms():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
pcm = b"\x01\x02" * 1600 # 100 ms of pcm16 @ 16 kHz
await client.append_audio(pcm)
await client.append_audio(pcm, is_speech=True)
appends = [event for event in sock.sent if event.get("type") == "input_audio_buffer.append"]
assert [a["audio_end_ms"] for a in appends] == [100, 200]
assert appends[0]["format"] == "pcm16"
assert appends[0]["sample_rate_hz"] == 16_000
assert appends[0]["duration_ms"] == 100
assert "is_speech" not in appends[0]
assert appends[1]["is_speech"] is True
assert base64.b64decode(appends[0]["audio"]) == pcm
sock.feed(SESSION_CLOSED)
async def test_append_audio_strips_video_frame_data_url_prefix():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
jpeg_b64 = base64.b64encode(b"\xff\xd8fake").decode("ascii")
# The wire contract carries bare base64; image_data_url output must
# still be accepted (the server validator rejects data-URL prefixes).
await client.append_audio(b"\x00\x00", video_frames=[f"data:image/jpeg;base64,{jpeg_b64}", jpeg_b64])
append = next(event for event in sock.sent if event.get("type") == "input_audio_buffer.append")
assert append["video_frames"] == [jpeg_b64, jpeg_b64]
sock.feed(SESSION_CLOSED)
async def test_stream_pcm_chunking():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
await client.stream_pcm(b"\x00" * 8000, chunk_ms=100, realtime=False)
appends = [event for event in sock.sent if event.get("type") == "input_audio_buffer.append"]
assert [a["duration_ms"] for a in appends] == [100, 100, 50]
sock.feed(SESSION_CLOSED)
async def test_interruption_primitives_send_documented_events():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
await client.cancel_response()
await client.clear_input()
types = sock.sent_types()
assert types.index("response.cancel") < types.index("input_audio_buffer.clear")
sock.feed(SESSION_CLOSED)
# ---------------------------------------------------------------------------
# Response demultiplexing
def _b64(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
async def test_response_handle_flow():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
chunk = b"\x00\x01" * 2400 # 100 ms of pcm16 @ 24 kHz
client, _ = make_client(sock)
async with client:
sock.feed({"type": "response.created", "response": {"id": "resp-1"}, "server_event_seq": 2})
sock.feed({"type": "response.speak", "response_id": "resp-1", "server_event_seq": 3})
sock.feed(
{
"type": "response.audio.delta",
"response_id": "resp-1",
"delta": _b64(chunk),
"sample_rate_hz": 24_000,
"server_event_seq": 4,
}
)
sock.feed(
{
"type": "response.audio_transcript.delta",
"response_id": "resp-1",
"delta": "hi there",
"server_event_seq": 5,
}
)
sock.feed({"type": "response.done", "response_id": "resp-1", "server_event_seq": 6})
async for response in client.responses():
chunks = [piece async for piece in response.audio()]
assert chunks == [chunk]
assert response.decision == "speak"
assert response.transcript == "hi there"
assert response.played_ms == pytest.approx(100.0)
assert response.finished
break
sock.feed(SESSION_CLOSED)
async def test_listen_decision_yields_finished_silent_handle():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
sock.feed({"type": "response.listen", "server_event_seq": 2})
async for response in client.responses():
assert response.decision == "listen"
assert response.finished
assert [piece async for piece in response.audio()] == []
break
sock.feed(SESSION_CLOSED)
async def test_listen_terminates_active_response():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
sock.feed({"type": "response.created", "response": {"id": "resp-1"}, "server_event_seq": 2})
sock.feed({"type": "response.listen", "response_id": "resp-1", "server_event_seq": 3})
async for response in client.responses():
await response.wait(timeout_s=5.0)
assert response.decision == "listen"
break
sock.feed(SESSION_CLOSED)
async def test_id_less_listen_never_closes_an_in_flight_response():
# An id-less listen is a standalone decision beat (e.g. a silence-skip
# while a response streams); it must surface as its own finished handle
# and leave the in-flight response to its real terminal.
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
sock.feed({"type": "response.created", "response": {"id": "resp-1"}, "server_event_seq": 2})
sock.feed({"type": "response.speak", "response_id": "resp-1", "server_event_seq": 3})
sock.feed({"type": "response.listen", "server_event_seq": 4})
sock.feed({"type": "response.done", "response_id": "resp-1", "server_event_seq": 5})
seen: list[tuple[str | None, str | None, bool]] = []
async for response in client.responses():
await response.wait(timeout_s=5.0)
seen.append((response.response_id, response.decision, response.finished))
if len(seen) == 2:
break
assert ("resp-1", "speak", True) in seen
assert (None, "listen", True) in seen
sock.feed(SESSION_CLOSED)
async def test_terminal_listen_keeps_spoken_decision():
# A terminal listen after the response spoke is the model yielding the
# turn; the handle must stay decision="speak" with its audio intact.
sock = FakeSocket()
sock.feed(SESSION_CREATED)
chunk = b"\x00\x01" * 2400
client, _ = make_client(sock)
async with client:
sock.feed({"type": "response.created", "response": {"id": "resp-1"}, "server_event_seq": 2})
sock.feed({"type": "response.speak", "response_id": "resp-1", "server_event_seq": 3})
sock.feed(
{
"type": "response.audio.delta",
"response_id": "resp-1",
"delta": _b64(chunk),
"sample_rate_hz": 24_000,
"server_event_seq": 4,
}
)
sock.feed({"type": "response.listen", "response_id": "resp-1", "server_event_seq": 5})
async for response in client.responses():
chunks = [piece async for piece in response.audio()]
assert chunks == [chunk]
assert response.decision == "speak"
assert response.finished
break
sock.feed(SESSION_CLOSED)
async def test_error_event_surfaces_on_response_path():
# A rejected send produces no response; a consumer waiting in responses()
# must see the rejection instead of waiting forever.
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
sock.feed(
{
"type": "error",
"error": {"code": "input_audio_buffer_empty", "message": "empty commit"},
"server_event_seq": 2,
}
)
with pytest.raises(DuplexProtocolError) as excinfo:
async for _ in client.responses():
pass
assert excinfo.value.code == "input_audio_buffer_empty"
sock.feed(SESSION_CLOSED)
async def test_slow_audio_consumer_drops_oldest_instead_of_stalling():
from vllm_omni.clients.duplex import AudioDelta, ResponseHandle
handle = ResponseHandle(
response_id="r1",
output_format=AudioFormat("pcm16", 24_000),
max_buffered_events=2,
)
for payload in (b"c1", b"c2", b"c3"):
# _feed must never block the reader, even against a full queue.
handle._feed(AudioDelta({"type": "response.audio.delta", "delta": _b64(payload)}))
handle._finish(None)
chunks = [chunk async for chunk in handle.audio()]
assert chunks == [b"c3"] # oldest chunks were dropped, the sentinel landed
# ---------------------------------------------------------------------------
# Resume
async def test_resume_after_transport_drop():
first = FakeSocket()
first.feed(SESSION_CREATED)
second = FakeSocket()
# Mirror the real resume activation payload
# (entrypoints/duplex/serving.py, activation_payload_factory): it carries
# no nested "session" object, only the identity fields.
second.feed(
{
"type": "session.resumed",
"session_id": "sess-1",
"incarnation": 0,
"attachment_generation": 1,
"resume_token": "tok-2",
"server_event_seq": 5,
}
)
client, calls = make_client(
first,
second,
reconnect=ReconnectPolicy(max_attempts=2, backoff_s=(0.0, 0.0)),
)
async with client:
stream = client.events()
async def take_two():
seen = []
async for event in stream:
seen.append(event)
if len(seen) == 2:
return seen
consumer = asyncio.create_task(take_two())
await asyncio.sleep(0) # let the consumer subscribe before the drop
await asyncio.sleep(0)
first.feed(RuntimeError("transport dropped"))
seen = await asyncio.wait_for(consumer, timeout=5.0)
assert isinstance(seen[0], ConnectionResumed)
assert isinstance(seen[1], SessionResumed)
assert client.resume_token == "tok-2"
# The activation payload has no session object; the info captured at
# the original handshake must survive the resume.
assert client.session_info == {"session_id": "sess-1"}
assert len(calls) == 2
resume = second.sent[0]
assert resume["type"] == "session.resume"
assert resume["session_id"] == "sess-1"
assert resume["incarnation"] == 0
assert resume["resume_token"] == "tok-1"
assert resume["last_received_server_event_seq"] == 1
second.feed(SESSION_CLOSED)
async def test_no_reconnect_policy_surfaces_closed():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock) # reconnect=None
async with client:
stream = client.events()
sock.feed(RuntimeError("transport dropped"))
with pytest.raises(DuplexSessionClosedError):
async for _ in stream:
pass
async def test_heartbeat_survives_transient_send_failure():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
armed = {"fail": True}
original_send = sock.send
async def flaky_send(raw: str) -> None:
if json.loads(raw).get("type") == "session.heartbeat" and armed["fail"]:
armed["fail"] = False
raise RuntimeError("transport hiccup")
await original_send(raw)
sock.send = flaky_send # type: ignore[method-assign]
client, _ = make_client(sock, heartbeat_interval_s=0.01)
async with client:
# The first tick fails; the loop must keep ticking (a resumed but
# quiet session relies on heartbeats to reset the server timeout).
await _drain(lambda: "session.heartbeat" in sock.sent_types())
sock.feed(SESSION_CLOSED)
async def test_aexit_sends_session_close_on_error_path():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
with pytest.raises(RuntimeError):
async with client:
raise RuntimeError("application failure")
# The server must not be left holding the session for the disconnect
# grace period; the error path still announces the close.
assert "session.close" in sock.sent_types()
def test_target_url_forces_autostart_off():
client = DuplexClient("ws://test-host:8099/v1/realtime?autostart=1", model="m")
query = parse_qs(urlsplit(client._target_url()).query)
# autostart would race the session.update handshake and silently drop
# ref_audio/extra_body; the client overrides it even when the URL asks.
assert query["autostart"] == ["0"]
async def test_fatal_resume_error_gives_up():
first = FakeSocket()
first.feed(SESSION_CREATED)
second = FakeSocket()
second.feed({"type": "error", "error": {"code": "invalid_resume_token", "message": "nope"}})
client, calls = make_client(
first,
second,
reconnect=ReconnectPolicy(max_attempts=3, backoff_s=(0.0, 0.0)),
)
async with client:
stream = client.events()
first.feed(RuntimeError("transport dropped"))
with pytest.raises(DuplexSessionClosedError):
async for _ in stream:
pass
assert len(calls) == 2 # no retry after a fatal resume error
# ---------------------------------------------------------------------------
# Collector
def test_event_collector_accumulates_audio():
collector = EventCollector()
collector.add({"type": "response.created", "response": {"id": "r1"}}, received_at_s=1.0)
collector.add(
{"type": "response.audio.delta", "response_id": "r1", "delta": _b64(b"ab"), "sample_rate_hz": 16_000},
received_at_s=1.1,
)
collector.add(
{"type": "response.output_audio.delta", "response_id": "r1", "audio": _b64(b"cd")},
received_at_s=1.2,
)
assert collector.count("response.created") == 1
assert collector.audio_bytes() == b"abcd"
assert collector.output_sample_rate_hz == 16_000
summary = collector.timing_summary(after_s=0.0)
assert summary["audio_output"]["chunk_count"] == 2
assert summary["audio_output"]["response_created_to_first_audio_ms"] == pytest.approx(100.0)
# ---------------------------------------------------------------------------
# helpers
def _acks(sock: FakeSocket) -> list[dict[str, object]]:
return [event for event in sock.sent if event.get("type") == "session.event_ack"]
async def _drain(predicate, *, timeout_s: float = 2.0) -> None:
deadline = asyncio.get_event_loop().time() + timeout_s
while not predicate():
if asyncio.get_event_loop().time() > deadline:
raise AssertionError("condition not reached")
await asyncio.sleep(0.01)
# ---------------------------------------------------------------------------
# Probe/benchmark helpers (ported from the retired MiniCPM demo client tests)
def test_build_realtime_url_with_model_extra_query():
url = build_realtime_url(
"ws://localhost:8099/v1/realtime?custom=1",
"openbmb/MiniCPM-o-4_5",
session_id="session-a",
extra_query={"native_duplex": "1"},
)
query = parse_qs(urlsplit(url).query)
assert query == {
"custom": ["1"],
"duplex": ["1"],
"model": ["openbmb/MiniCPM-o-4_5"],
"native_duplex": ["1"],
"session_id": ["session-a"],
}
def test_build_realtime_url_resume_only_when_autostart_disabled():
url = build_realtime_url(
"ws://localhost:8099/v1/realtime?duplex=1",
"openbmb/MiniCPM-o-4_5",
autostart=False,
extra_query={"native_duplex": "1"},
)
query = parse_qs(urlsplit(url).query)
assert query["autostart"] == ["0"]
assert query["native_duplex"] == ["1"]
def test_event_collector_partitions_audio_by_response():
collector = EventCollector()
collector.add({"type": "response.created", "response": {"id": "resp-a"}})
collector.add(
{
"type": "response.audio.delta",
"response_id": "resp-a",
"delta": base64.b64encode(b"audio-a").decode("ascii"),
"sample_rate_hz": 16_000,
}
)
assert collector.response_ids == ["resp-a"]
assert collector.audio_bytes("resp-a") == b"audio-a"
assert collector.output_sample_rate_hz == 16_000
assert collector.first_received_at("response.created") is not None
assert collector.last_received_at("response.audio.delta") is not None
def test_event_collector_reports_engine_token_and_audio_intervals():
collector = EventCollector()
collector.add(
{"type": "response.created", "response": {"id": "resp-a"}},
received_at_s=10.0,
)
stage_metrics = {
"0": {
"num_tokens_out": 4,
"vllm_ttft_ms": 120.0,
"vllm_tpot_ms": 15.0,
"vllm_itl_ms": 14.0,
"vllm_itls_ms": [10.0, 14.0, 18.0],
}
}
for received_at_s, cumulative_audio_ms in ((10.2, 80), (10.25, 160), (10.36, 240)):
collector.add(
{
"type": "response.audio.delta",
"response_id": "resp-a",
"delta": base64.b64encode(b"audio").decode("ascii"),
"sample_rate_hz": 16_000,
"metadata": {
"audio_duration_ms": cumulative_audio_ms,
"vllm_omni": {"stage_metrics": stage_metrics},
},
},
received_at_s=received_at_s,
)
collector.add(
{"type": "response.audio_transcript.delta", "response_id": "resp-a", "delta": ""},
received_at_s=10.1,
)
collector.add(
{"type": "response.audio_transcript.delta", "response_id": "resp-a", "delta": "hello"},
received_at_s=10.15,
)
collector.add(
{"type": "response.done", "response": {"id": "resp-a"}},
received_at_s=10.4,
)
timing = collector.timing_summary(
after_s=10.0,
input_committed_at_s=9.9,
response_id="resp-a",
)
assert timing["stage0_tokens"] == {
"source": "engine_stage_metrics",
"output_token_count": 4,
"ttft_ms": 120.0,
"tpot_ms": 15.0,
"itls_ms": [10.0, 14.0, 18.0],
"inter_token_interval_ms": {
"count": 3,
"mean": 14.0,
"p50": 14.0,
"p95": 18.0,
"max": 18.0,
},
}
assert timing["audio_output"] == {
"source": "client_monotonic_receive",
"chunk_count": 3,
"response_created_to_first_audio_ms": 200.0,
"commit_to_first_audio_ms": 300.0,
"inter_chunk_interval_ms": {
"count": 2,
"mean": 80.0,
"p50": 50.0,
"p95": 110.0,
"max": 110.0,
},
"chunk_duration_ms": {
"count": 3,
"mean": 80.0,
"p50": 80.0,
"p95": 80.0,
"max": 80.0,
},
"max_chunk_gap_ms": 110.0,
}
# Raw data only: derived metrics such as the RTF are the caller's job
# (e.g. vllm_omni.metrics.definitions.compute_audio_rtf).
assert timing["request_metrics"] == {
"source": "client_monotonic_receive",
"measurement_origin": {
"ttft": "input_audio_buffer.commit client send to first non-empty text delta",
"ttfp": "input_audio_buffer.commit client send to first audio packet",
},
"ttft_ms": 250.0,
"ttfp_ms": 300.0,
"audio_generation_ms": 460.0,
"audio_duration_ms": 240.0,
}
def test_response_timing_ignores_unowned_session_level_metrics():
collector = EventCollector()
collector.add(
{"type": "response.created", "response": {"id": "resp-a"}},
received_at_s=10.0,
)
collector.add(
{
"type": "response.audio.delta",
"response_id": "resp-a",
"delta": base64.b64encode(b"audio").decode("ascii"),
"metadata": {
"vllm_omni": {
"stage_metrics": {
"0": {
"num_tokens_out": 20,
"vllm_ttft_ms": 157.0,
"vllm_tpot_ms": 16.0,
"vllm_itls_ms": [15.0, 17.0],
}
}
}
},
},
received_at_s=10.2,
)
collector.add(
{
"type": "response.listen",
"metadata": {
"vllm_omni": {
"stage_metrics": {
"0": {
"num_tokens_out": 2,
"vllm_ttft_ms": 106.0,
"vllm_tpot_ms": 0.0,
"vllm_itls_ms": [],
}
}
}
},
},
received_at_s=10.3,
)
timing = collector.timing_summary(after_s=10.0, response_id="resp-a")
assert timing["stage0_tokens"]["output_token_count"] == 20
assert timing["stage0_tokens"]["ttft_ms"] == 157.0
def test_summarize_session_request_metrics_averages_audio_turns():
summary = summarize_session_request_metrics(
[
{"ttft_ms": 100.0, "ttfp_ms": 200.0, "rtf": 0.5},
{"ttft_ms": 300.0, "ttfp_ms": 400.0, "rtf": 0.7},
],
session_id="sess-1",
)
assert summary == {
"session_id": "sess-1",
"audio_turn_count": 2,
"mean_ttft_ms": 200.0,
"mean_ttfp_ms": 300.0,
"mean_rtf": 0.6,
}
def test_pcm16_wav_round_trip(tmp_path):
path = tmp_path / "audio.wav"
pcm16 = b"\x01\x00\x02\x00"
write_pcm16_wav(path, pcm16, sample_rate_hz=16_000)
with wave.open(str(path), "rb") as wav_file:
assert wav_file.getnchannels() == 1
assert wav_file.getframerate() == 16_000
assert read_pcm16_wav(path) == pcm16
# ---------------------------------------------------------------------------
# Model-unit helpers and camera-frame interleaving
def test_duplex_unit_boundary_and_residual_math():
assert duplex_unit_boundary_ms(0) == 1030
assert duplex_unit_boundary_ms(2) == 3030
assert has_residual_model_unit(b"\x00" * 32_000, chunk_period_ms=1000) is False
assert has_residual_model_unit(b"\x00" * 32_002, chunk_period_ms=1000) is True
created = {"type": "session.created", "session": {"capabilities": {"chunk_period_ms": 500}}}
assert chunk_period_ms([created]) == 500
assert chunk_period_ms([{"type": "session.created", "session": {}}]) == 1000
async def test_stream_pcm_sends_each_units_composite_beside_its_base_frame():
sock = FakeSocket()
sock.feed(SESSION_CREATED)
client, _ = make_client(sock)
async with client:
frames_sent = await client.stream_pcm(
b"\x01\x00" * (16_000 * 3),
chunk_ms=200,
realtime=False,
video_frames=["f0", "f1"],
stacked_video_frames=["s0", None],
)
appends = [event for event in sock.sent if event.get("type") == "input_audio_buffer.append"]
# A composite belongs to the unit it was captured in, so it rides the
# same append as that unit's base frame; a unit without one sends the
# base alone. Frame k rides the append that closes model unit k.
assert [event["video_frames"] for event in appends if "video_frames" in event] == [["f0", "s0"], ["f1"]]
assert [event["audio_end_ms"] for event in appends if "video_frames" in event] == [1200, 2200]
assert frames_sent == 2
sock.feed(SESSION_CLOSED)
def test_build_realtime_url_native_duplex_flag_and_http_scheme():
url = build_realtime_url("http://localhost:8099/v1/realtime", None, native_duplex=None)
parts = urlsplit(url)
assert parts.scheme == "ws"
assert parse_qs(parts.query) == {"duplex": ["1"]}
url = build_realtime_url("https://host/v1/realtime", "m", native_duplex=False)
parts = urlsplit(url)
assert parts.scheme == "wss"
assert parse_qs(parts.query)["native_duplex"] == ["0"]
with pytest.raises(ValueError):
build_realtime_url("ftp://host/v1/realtime", "m")
def test_reference_audio_data_url(tmp_path):
assert reference_audio_data_url(None) is None
path = tmp_path / "ref.wav"
path.write_bytes(b"RIFF")
assert reference_audio_data_url(str(path)) == "data:audio/wav;base64," + base64.b64encode(b"RIFF").decode("ascii")
with pytest.raises(FileNotFoundError):
reference_audio_data_url(str(tmp_path / "missing.wav"))
def test_event_collector_response_text_joins_deltas_per_response():
collector = EventCollector()
collector.add({"type": "response.created", "response_id": "r1"})
collector.add({"type": "response.audio_transcript.delta", "response_id": "r1", "delta": "he"})
collector.add({"type": "response.output_text.delta", "response_id": "r2", "delta": "other"})
collector.add({"type": "response.text.delta", "response_id": "r1", "delta": "llo"})
assert collector.response_text("r1") == "hello"
assert collector.response_text("r2") == "other"
assert collector.response_text("r3") == ""
tests/clients/test_model_session_configs.py
+70
View File
@@ -0,0 +1,70 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Model duplex session presets under ``vllm_omni.clients.<model>``."""
import subprocess
import sys
import pytest
from vllm_omni.clients.duplex import AudioFormat
from vllm_omni.clients.minicpmo_4_5 import (
create_duplex_session_config as create_minicpmo45_session_config,
)
from vllm_omni.clients.personaplex import (
create_duplex_session_config as create_personaplex_session_config,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def test_minicpmo45_session_config_matches_native_duplex_deployment():
config = create_minicpmo45_session_config(ref_audio="data:audio/wav;base64,AAA=", temperature=0.0)
payload = config.to_session_payload(model="openbmb/MiniCPM-o-4_5")
assert payload["ref_audio"] == "data:audio/wav;base64,AAA="
assert payload["overlap_policy"] == "listen_only"
assert payload["playback_commit_policy"] == "ack_only"
assert payload["extra_body"]["native_duplex"] is True
assert payload["extra_body"]["auto_response"] is True
assert payload["temperature"] == 0.0
def test_personaplex_session_config_matches_deployment():
config = create_personaplex_session_config(voice="NATF2.pt", persona="You are calm.")
assert config.input_audio == AudioFormat("pcm_f32le", 24_000)
payload = config.to_session_payload(model="nvidia/personaplex-7b-v1")
assert payload["input_audio_format"] == "pcm_f32le"
assert payload["voice"] == "NATF2.pt"
assert payload["instructions"] == "You are calm."
@pytest.mark.parametrize(
"duplex_package",
[
"vllm_omni.model_executor.models.minicpmo_4_5.duplex",
"vllm_omni.model_executor.models.personaplex.duplex",
],
)
def test_model_duplex_packages_stay_clear_of_client_library(duplex_package: str):
"""Model duplex plugin packages are server-side: no vllm_omni.clients."""
script = f"""
import sys
import {duplex_package}
loaded = [
name
for name in sys.modules
if name == "vllm_omni.clients" or name.startswith("vllm_omni.clients.")
]
if loaded:
raise SystemExit("model duplex package loaded the client library: " + ", ".join(loaded))
"""
result = subprocess.run(
[sys.executable, "-c", script],
capture_output=True,
text=True,
timeout=120,
check=False,
)
assert result.returncode == 0, result.stderr
tests/e2e/features/fullduplex/personaplex/test_import_boundary.py
-30
View File
@@ -1,30 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import subprocess
import sys
def test_batched_runtime_import_does_not_require_optional_opus_dependency() -> None:
script = """
import builtins
real_import = builtins.__import__
def guarded_import(name, *args, **kwargs):
if name == "sphn":
raise ModuleNotFoundError("blocked optional dependency: sphn")
return real_import(name, *args, **kwargs)
builtins.__import__ = guarded_import
from vllm_omni.experimental.fullduplex.personaplex.serving.batched import BatchedSessionManager
assert BatchedSessionManager is not None
"""
result = subprocess.run(
[sys.executable, "-c", script],
check=False,
capture_output=True,
text=True,
)
assert result.returncode == 0, result.stderr
tests/e2e/features/fullduplex/test_client.py
+7 -7
View File
@@ -15,20 +15,20 @@ from vllm_omni.experimental.fullduplex.client import (
read_pcm16_wav,
write_pcm16_wav,
)
from vllm_omni.experimental.fullduplex.minicpmo45.policy import (
MiniCPMO45DuplexPolicy,
)
from vllm_omni.experimental.fullduplex.video_stacking import (
concat_frames,
unit_subframe_offsets,
)
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.policy import (
MiniCPMO45DuplexPolicy,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def test_realtime_client_builds_explicit_native_duplex_url():
url = build_realtime_url(
"ws://localhost:8099/v1/realtime?custom=1&duplex=0&model=stale&minicpmo45_native_duplex=0&session_id=stale",
"ws://localhost:8099/v1/realtime?custom=1&duplex=0&model=stale&native_duplex=0&session_id=stale",
"openbmb/MiniCPM-o-4_5",
session_id="session-a",
)
@@ -38,7 +38,7 @@ def test_realtime_client_builds_explicit_native_duplex_url():
"custom": ["1"],
"duplex": ["1"],
"model": ["openbmb/MiniCPM-o-4_5"],
"minicpmo45_native_duplex": ["1"],
"native_duplex": ["1"],
"session_id": ["session-a"],
}
@@ -65,7 +65,7 @@ def test_realtime_client_builds_resume_only_url_when_autostart_disabled():
query = parse_qs(urlsplit(url).query)
assert query["autostart"] == ["0"]
assert query["minicpmo45_native_duplex"] == ["1"]
assert query["native_duplex"] == ["1"]
@pytest.mark.asyncio
@@ -232,7 +232,7 @@ async def test_realtime_client_configure_explicit_tts_opts_out_of_native_duplex(
session_extra_body = client.sent[0]["session"]["extra_body"]
assert session_extra_body == {
"ref_audio": "data:audio/wav;base64,AAAA",
"minicpmo45_native_duplex": False,
"native_duplex": False,
}
tests/e2e/features/fullduplex/test_core_contracts.py
+2 -2
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Contract tests for the fullduplex core (PR #3907 alignment).
Covers the DuplexFence identity value, the session identity rules (epoch only
@@ -14,6 +14,7 @@ import dataclasses
import pytest
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.experimental.fullduplex.core import protocol as ev
from vllm_omni.experimental.fullduplex.core.adapter import (
DuplexAdapter,
@@ -26,7 +27,6 @@ from vllm_omni.experimental.fullduplex.core.session import (
DuplexSessionConfig,
DuplexState,
)
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/e2e/features/fullduplex/test_personaplex_adapter.py
-262
View File
@@ -1,262 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""PersonaPlex adapter driven through the model-owned lockstep runtime.
PersonaPlexDuplexRuntime starts ONE response for the whole session, streams one
agent frame per user frame, and drains on close (core stays verbatim upstream;
the lockstep lifecycle is model policy) — all with a GPU-free stub stepper.
"""
import asyncio
import numpy as np
import pytest
from vllm_omni.experimental.fullduplex.core import protocol as ev
from vllm_omni.experimental.fullduplex.core.session import (
DuplexSession,
DuplexSessionConfig,
DuplexState,
)
from vllm_omni.experimental.fullduplex.personaplex.adapter import (
PersonaPlexDuplexAdapter,
PersonaPlexDuplexRuntime,
)
from vllm_omni.experimental.fullduplex.personaplex.config import FRAME_SIZE
from vllm_omni.experimental.fullduplex.personaplex.engine import FrameOutput
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
class StubStepper:
"""Deterministic FrameStepper: frame i -> audio filled with i, text 't{i}' on odds."""
sample_rate = 24000
frame_size = FRAME_SIZE
def __init__(self) -> None:
self.opened: tuple[str | None, str | None] | None = None
self.steps = 0
def open_session(self, voice_prompt=None, persona=None) -> None:
self.opened = (voice_prompt, persona)
self.steps = 0
def step(self, user_pcm):
assert user_pcm.shape[0] == self.frame_size
self.steps += 1
i = self.steps
return FrameOutput(
audio=np.full(self.frame_size, float(i), dtype=np.float32),
text=(f"t{i}" if i % 2 else None),
)
class RecordingAdapter(PersonaPlexDuplexAdapter):
def __init__(self, stepper: StubStepper) -> None:
super().__init__(stepper)
self.closed = False
async def on_close(self, session: DuplexSession) -> None:
self.closed = True
await super().on_close(session)
def _collector():
out: list[dict] = []
async def emit(event: dict) -> None:
out.append(event)
return out, emit
async def _feed(events):
for e in events:
yield e
async def _raising_feed(frame):
yield {"type": ev.INPUT_APPEND, "modality": "audio", "data": frame}
await asyncio.sleep(0)
raise RuntimeError("input failed")
def _respond_tasks() -> list[asyncio.Task]:
current = asyncio.current_task()
return [
task
for task in asyncio.all_tasks()
if task is not current and "_respond_once" in getattr(task.get_coro(), "__qualname__", "")
]
def _session():
cfg = DuplexSessionConfig(
input_modalities=("audio",),
output_modalities=("audio", "text"),
)
return DuplexSession("s", cfg)
@pytest.mark.asyncio
async def test_lockstep_one_agent_frame_per_user_frame():
stub = StubStepper()
adapter = PersonaPlexDuplexAdapter(stub, voice_prompt="NATF2.pt", persona="be terse")
rt = PersonaPlexDuplexRuntime(_session(), adapter)
out, emit = _collector()
frame = np.zeros(FRAME_SIZE, dtype=np.float32)
events = [{"type": ev.INPUT_APPEND, "modality": "audio", "data": frame} for _ in range(3)]
events.append({"type": ev.CLOSE})
await rt.run(_feed(events), emit)
types = [e["type"] for e in out]
# exactly one response for the whole session, drained cleanly on close
assert types.count(ev.RESPONSE_CREATED) == 1
assert types.count(ev.RESPONSE_DONE) == 1
audio = [e for e in out if e["type"] == ev.RESPONSE_DELTA and e["modality"] == "audio"]
text = [e["data"] for e in out if e["type"] == ev.RESPONSE_DELTA and e["modality"] == "text"]
assert len(audio) == 3 # one agent frame per user frame
assert [int(e["data"][0]) for e in audio] == [1, 2, 3] # lockstep order
assert text == ["t1", "t3"]
assert stub.steps == 3
assert stub.opened == ("NATF2.pt", "be terse")
@pytest.mark.asyncio
async def test_unaligned_chunks_are_reframed_to_80ms():
stub = StubStepper()
adapter = PersonaPlexDuplexAdapter(stub)
rt = PersonaPlexDuplexRuntime(_session(), adapter)
out, emit = _collector()
# 2.5 frames in one chunk + 0.5 frame in another => 3 whole frames
events = [
{"type": ev.INPUT_APPEND, "modality": "audio", "data": np.zeros(FRAME_SIZE * 5 // 2, dtype=np.float32)},
{"type": ev.INPUT_APPEND, "modality": "audio", "data": np.zeros(FRAME_SIZE // 2, dtype=np.float32)},
{"type": ev.CLOSE},
]
await rt.run(_feed(events), emit)
audio = [e for e in out if e["type"] == ev.RESPONSE_DELTA and e["modality"] == "audio"]
assert len(audio) == 3
assert stub.steps == 3
@pytest.mark.asyncio
async def test_close_without_input_is_clean():
adapter = PersonaPlexDuplexAdapter(StubStepper())
rt = PersonaPlexDuplexRuntime(_session(), adapter)
out, emit = _collector()
await rt.run(_feed([{"type": ev.CLOSE}]), emit)
# no input -> response never started -> no created/done, no error
assert not any(e["type"] == ev.ERROR for e in out)
assert not any(e["type"] == ev.RESPONSE_CREATED for e in out)
@pytest.mark.asyncio
async def test_iterator_eof_without_close_drains_and_stops():
# Transport drop: the input iterator ends WITHOUT an explicit CLOSE event.
# The runtime must still signal on_close so the eternal response drains
# instead of blocking on its inbox forever.
stub = StubStepper()
adapter = PersonaPlexDuplexAdapter(stub)
rt = PersonaPlexDuplexRuntime(_session(), adapter)
out, emit = _collector()
frame = np.zeros(FRAME_SIZE, dtype=np.float32)
events = [{"type": ev.INPUT_APPEND, "modality": "audio", "data": frame}]
await asyncio.wait_for(rt.run(_feed(events), emit), timeout=5.0)
types = [e["type"] for e in out]
assert types.count(ev.RESPONSE_DONE) == 1
assert stub.steps == 1
@pytest.mark.asyncio
async def test_input_exception_closes_response_and_runtime() -> None:
stub = StubStepper()
adapter = RecordingAdapter(stub)
rt = PersonaPlexDuplexRuntime(_session(), adapter)
out, emit = _collector()
frame = np.zeros(FRAME_SIZE, dtype=np.float32)
with pytest.raises(RuntimeError, match="input failed"):
await rt.run(_raising_feed(frame), emit)
leaked = _respond_tasks()
for task in leaked:
task.cancel()
if leaked:
await asyncio.gather(*leaked, return_exceptions=True)
assert adapter.closed is True
assert rt.session.state is DuplexState.CLOSED
assert not leaked
@pytest.mark.asyncio
async def test_default_session_config_gets_lockstep_lifecycle():
# The lifecycle comes from the model-owned runtime class, not a session
# flag: a plain default-constructed session config runs lockstep.
stub = StubStepper()
adapter = PersonaPlexDuplexAdapter(stub)
session = DuplexSession(
"s",
DuplexSessionConfig(input_modalities=("audio",), output_modalities=("audio", "text")),
)
rt = PersonaPlexDuplexRuntime(session, adapter)
out, emit = _collector()
frame = np.zeros(FRAME_SIZE, dtype=np.float32)
events = [{"type": ev.INPUT_APPEND, "modality": "audio", "data": frame}, {"type": ev.CLOSE}]
await rt.run(_feed(events), emit)
types = [e["type"] for e in out]
assert types.count(ev.RESPONSE_CREATED) == 1
assert types.count(ev.RESPONSE_DONE) == 1
assert stub.steps == 1
@pytest.mark.asyncio
async def test_partial_tail_is_padded_and_stepped_on_close():
# A trailing sub-frame chunk is zero-padded and stepped at close (same
# contract as PersonaPlexSession.flush) rather than silently dropped.
stub = StubStepper()
adapter = PersonaPlexDuplexAdapter(stub)
rt = PersonaPlexDuplexRuntime(_session(), adapter)
out, emit = _collector()
events = [
{"type": ev.INPUT_APPEND, "modality": "audio", "data": np.ones(FRAME_SIZE // 2, dtype=np.float32)},
{"type": ev.CLOSE},
]
await rt.run(_feed(events), emit)
audio = [e for e in out if e["type"] == ev.RESPONSE_DELTA and e["modality"] == "audio"]
assert len(audio) == 1
assert stub.steps == 1
@pytest.mark.asyncio
async def test_single_eternal_response_across_barge_in():
# Moved from the core contract tests when the lockstep lifecycle became
# model-owned: ONE response for the whole session, RESPONSE_CANCEL bumps the
# epoch fence (playback cursor) but the stream keeps flowing, and close
# drains via on_close.
stub = StubStepper()
adapter = PersonaPlexDuplexAdapter(stub)
session = _session()
rt = PersonaPlexDuplexRuntime(session, adapter)
out, emit = _collector()
frame = np.zeros(FRAME_SIZE, dtype=np.float32)
events = [
{"type": ev.INPUT_APPEND, "modality": "audio", "data": frame},
{"type": ev.RESPONSE_CANCEL}, # barge-in: epoch bumps, stream continues
{"type": ev.INPUT_APPEND, "modality": "audio", "data": frame},
{"type": ev.CLOSE},
]
await rt.run(_feed(events), emit)
types = [e["type"] for e in out]
assert types.count(ev.RESPONSE_CREATED) == 1, "exactly one eternal response"
assert types.count(ev.RESPONSE_DONE) == 1
audio = [e for e in out if e["type"] == ev.RESPONSE_DELTA and e["modality"] == "audio"]
assert len(audio) == 2, "stream keeps flowing across barge-in"
assert session.epoch == 1, "cancel still advanced the epoch fence"
assert stub.steps == 2
tests/e2e/features/fullduplex/test_personaplex_batched.py
-178
View File
@@ -1,178 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""GPU-free tests for the batched PersonaPlex slot manager (stub engine)."""
from types import SimpleNamespace
import numpy as np
import pytest
from vllm_omni.experimental.fullduplex.personaplex.engine import FrameOutput, PrefillStep
from vllm_omni.experimental.fullduplex.personaplex.serving.batched import (
BatchedSessionManager,
SlotPhase,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
FRAME = 4
class StubEngine:
"""Records the slot-API call sequence; echoes each row's pcm as its output."""
frame_size = FRAME
def __init__(self, batch_size: int = 4, prefill_len: int = 2) -> None:
self.config = SimpleNamespace(batch_size=batch_size)
self.prefill_len = prefill_len
self.recycled: list[int] = []
self.voice_ended: list[int] = []
self.prefill_done: list[int] = []
self.steps: list[tuple[np.ndarray, dict | None]] = []
def prefill_steps(self, persona=None):
steps = [PrefillStep("sacrifice"), PrefillStep("voice", embedding=None), PrefillStep("voice_end")]
steps += [PrefillStep("tokens", moshi_tokens=None, text_token=3, user_sine=True)] * self.prefill_len
return steps
def recycle_slot(self, b):
self.recycled.append(b)
def voice_end(self, b):
self.voice_ended.append(b)
def end_prefill(self, b):
self.prefill_done.append(b)
def step_batch(self, pcm, prefill=None):
self.steps.append((pcm.copy(), dict(prefill) if prefill else None))
return [FrameOutput(audio=pcm[b].copy(), text=None) for b in range(self.config.batch_size)]
def test_all_idle_does_not_step():
eng = StubEngine()
mgr = BatchedSessionManager(eng)
assert mgr.tick() is False
assert eng.steps == []
def test_pristine_slot_is_live_immediately():
eng = StubEngine()
mgr = BatchedSessionManager(eng)
slot, ready_now = mgr.acquire()
assert ready_now is True
assert slot.phase is SlotPhase.LIVE
assert eng.recycled == []
def test_custom_persona_prefills_and_recycles_on_tick_thread():
eng = StubEngine(prefill_len=2)
mgr = BatchedSessionManager(eng)
slot, ready_now = mgr.acquire(persona="custom")
assert ready_now is False
assert slot.phase is SlotPhase.PREFILL
assert eng.recycled == [] # deferred: reset must run between engine steps
ready = []
slot.on_ready = lambda: ready.append(True)
mgr.tick() # recycle + sacrifice
assert eng.recycled == [slot.index]
assert eng.steps[-1][1][slot.index].kind == "sacrifice"
mgr.tick() # voice frame
assert eng.steps[-1][1][slot.index].kind == "voice"
assert eng.voice_ended == []
mgr.tick() # voice_end executed BETWEEN ticks, then first tokens frame
assert eng.voice_ended == [slot.index]
assert eng.steps[-1][1][slot.index].kind == "tokens"
mgr.tick() # second tokens frame
assert not ready
mgr.tick() # prefill exhausted -> live
assert eng.prefill_done == [slot.index]
assert slot.phase is SlotPhase.LIVE
assert ready == [True]
def test_pristine_slot_after_ticking_requires_recycle():
# step_batch advances EVERY row each tick, so a pristine idle row that has
# lived through ticks has drifted from its boot prefill and must be
# recycled before a client may join it.
eng = StubEngine()
mgr = BatchedSessionManager(eng)
first, ready_now = mgr.acquire()
assert ready_now is True # claimed before any tick: truly pristine
mgr.tick()
second, ready_now2 = mgr.acquire()
assert second.index != first.index
assert ready_now2 is False
assert second.phase is SlotPhase.PREFILL
def test_used_slot_requires_recycle_even_with_default_persona():
eng = StubEngine()
mgr = BatchedSessionManager(eng)
slot, ready_now = mgr.acquire()
assert ready_now
mgr.release(slot)
slot2, ready_now2 = mgr.acquire()
assert slot2.index == slot.index
assert ready_now2 is False
assert slot2.phase is SlotPhase.PREFILL
def test_acquire_returns_none_when_full():
eng = StubEngine(batch_size=2)
mgr = BatchedSessionManager(eng)
assert mgr.acquire() is not None
assert mgr.acquire() is not None
assert mgr.acquire() is None
def test_feed_frames_epoch_guard_and_backlog_cap():
eng = StubEngine()
mgr = BatchedSessionManager(eng, max_buffered_frames=3)
slot, _ = mgr.acquire()
mgr.feed(slot, np.arange(FRAME * 2, dtype=np.float32), slot.epoch)
assert len(slot.inq) == 2
mgr.feed(slot, np.zeros(3, dtype=np.float32), slot.epoch) # partial tail buffered
assert len(slot.inq) == 2 and slot.pending.shape[0] == 3
mgr.feed(slot, np.zeros(FRAME * 5, dtype=np.float32), slot.epoch)
assert len(slot.inq) == 3 # oldest dropped beyond the cap
mgr.feed(slot, np.zeros(FRAME, dtype=np.float32), slot.epoch - 1) # stale epoch ignored
assert len(slot.inq) == 3
def test_live_slot_consumes_input_and_emits_output():
eng = StubEngine()
mgr = BatchedSessionManager(eng)
slot, _ = mgr.acquire()
got = []
slot.on_output = got.append
frame = np.full(FRAME, 0.5, dtype=np.float32)
mgr.feed(slot, frame, slot.epoch)
mgr.tick()
assert len(got) == 1
np.testing.assert_array_equal(got[0].audio, frame)
# queue drained: next tick feeds zeros
mgr.tick()
np.testing.assert_array_equal(got[1].audio, np.zeros(FRAME, dtype=np.float32))
def test_release_mid_flight_stops_emission():
eng = StubEngine()
mgr = BatchedSessionManager(eng)
slot, _ = mgr.acquire()
got = []
slot.on_output = got.append
mgr.tick()
assert len(got) == 1
mgr.release(slot)
# another slot keeps the batch non-idle
other, _ = mgr.acquire() # same slot index, now PREFILL (used)
assert other.phase is SlotPhase.PREFILL
mgr.tick()
assert len(got) == 1 # no further output for the released epoch
tests/e2e/features/fullduplex/test_personaplex_elastic.py
-245
View File
@@ -1,245 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""GPU e2e for PersonaPlex elastic batching (bit-exact isolation + recycle).
Greedy decoding makes every cross-slot leak observable as a bit difference:
1. Isolation (bit-exact): recycling one slot mid-flight leaves every other
slot's output bit-identical to a run that never recycled.
2. Replay exactness: a recycled slot's opening live ticks match a pristine boot
at numerical-noise level (a broken KV mask / staging roll garbles frame 0).
3. Coherence (behavioral): a recycled slot fed a real spoken question produces
a real reply in its inner-monologue text stream.
Run (moshi-free native stack; ~30 GB GPU + HF access to the gated repo; set
PPLEX_QUESTION_WAV to a spoken-question wav to enable the coherence test):
pytest tests/e2e/features/fullduplex/test_personaplex_elastic.py -v -s
Bench (B sweep, ms/tick vs the 80 ms budget):
python tests/e2e/features/fullduplex/test_personaplex_elastic.py --bench 2 4 8 16
"""
from __future__ import annotations
import numpy as np
import pytest
from tests.helpers.mark import hardware_marks
torch = pytest.importorskip("torch")
# slow + H100 + omni + cards_1 routes this module into weekly
# "Omni · H100 · Single-GPU" (test-weekly.yml).
pytestmark = [
pytest.mark.omni,
pytest.mark.slow,
*hardware_marks(res={"cuda": "H100"}),
]
if not torch.cuda.is_available(): # noqa: E402
pytest.skip("needs CUDA", allow_module_level=True)
from vllm_omni.experimental.fullduplex.personaplex.config import PersonaPlexConfig # noqa: E402
from vllm_omni.experimental.fullduplex.personaplex.runtime import ( # noqa: E402
PersonaPlexEngine,
)
B = 4
LIVE_TICKS = 40 # live frames compared per claim
RECYCLE_AT = 10
def _make_engine() -> PersonaPlexEngine:
cfg = PersonaPlexConfig(batch_size=B) # native stepper is greedy-only
try:
return PersonaPlexEngine(cfg).load()
except Exception as exc: # gated-repo access is environment-dependent
name = type(exc).__name__
if "Gated" in name or "401" in str(exc) or "403" in str(exc):
pytest.skip(f"no access to the gated nvidia/personaplex-7b-v1 repo: {name}")
raise
@pytest.fixture(scope="module")
def engine() -> PersonaPlexEngine:
return _make_engine()
def _slot_stream(slot: int, tick: int, frame: int) -> np.ndarray:
"""Deterministic per-slot input: distinct low-amplitude tones."""
sr = 24000
freq = 180.0 + 90.0 * slot
t0 = tick * frame
t = (np.arange(frame) + t0) / sr
return (0.12 * np.sin(2 * np.pi * freq * t)).astype(np.float32)
def _run(engine, total_ticks: int, recycle: bool):
"""Run the lockstep loop; optionally recycle slot 1 at RECYCLE_AT.
Returns per-slot lists of (audio | None) per live tick index, where the
recycled slot's list restarts its input stream at its post-prefill tick.
"""
frame = engine.frame_size
outs: list[list[np.ndarray | None]] = [[] for _ in range(B)]
stream_tick = [0] * B # per-slot conversation clock driving its input
prefill = None
tick = 0
while tick < total_ticks:
if recycle and tick == RECYCLE_AT and prefill is None:
engine.recycle_slot(1)
prefill = list(engine.prefill_steps(None))
pcm = np.zeros((B, frame), dtype=np.float32)
step_for_1 = None
if prefill:
while prefill and prefill[0].kind == "voice_end":
engine.voice_end(1)
prefill.pop(0)
if prefill:
step_for_1 = prefill.pop(0)
else:
engine.end_prefill(1)
prefill = None
stream_tick[1] = 0 # restart slot 1's input stream from its own t=0
for b in range(B):
if b == 1 and step_for_1 is not None:
continue # engine overrides the user channel for prefill rows
pcm[b] = _slot_stream(b, stream_tick[b], frame)
stream_tick[b] += 1
frame_outs = engine.step_batch(pcm, {1: step_for_1} if step_for_1 is not None else None)
for b in range(B):
if b == 1 and step_for_1 is not None:
continue # prefill tick: nothing conversational to record
frame_out = frame_outs[b]
outs[b].append(None if frame_out is None else frame_out.audio)
tick += 1
return outs
def test_recycle_isolation_and_fresh_equivalence(engine):
engine.open_batch()
prefill_len = sum(1 for s in engine.prefill_steps(None) if s.kind != "voice_end")
total = RECYCLE_AT + prefill_len + LIVE_TICKS
base = _run(engine, total, recycle=False)
engine.open_batch() # deterministic re-boot (greedy, fixed voice/persona)
rec = _run(engine, total, recycle=True)
# Claim 1 -- isolation: untouched slots bit-identical to the no-recycle run.
for b in (0, 2, 3):
assert len(base[b]) == total and len(rec[b]) == total
for i, (x, y) in enumerate(zip(base[b], rec[b])):
assert (x is None) == (y is None), f"slot {b} tick {i} validity diverged"
if x is not None:
assert np.array_equal(x, y), f"slot {b} tick {i} audio diverged (recycle leaked)"
# Claim 2 -- replay exactness: the recycled slot's opening live ticks match a
# pristine boot at numerical-noise level. A broken KV mask / staging-ring
# roll / Mimi reset garbles the audio from the very first frame (diffs of
# 0.1-1.0); a correct replay differs only by decoder-state rounding (~1e-4).
#
# Full-length bitwise identity is NOT expected, for two structural reasons:
# the replayed system prompt sits at a different ABSOLUTE RoPE offset (bf16
# rounds differently there), and the recycled Mimi ENCODER cannot restore a
# fresh stream's start padding (a batch-shared scalar), so from the first
# non-silent user frame it emits different-but-equally-valid codes for the
# same audio. Once any near-tie flips a token, the two equally valid
# trajectories legitimately diverge; coherence of the recycled slot is
# asserted behaviorally in test_recycled_slot_converses.
fresh = [a for a in base[1] if a is not None][:LIVE_TICKS]
reborn = [a for a in rec[1][RECYCLE_AT:] if a is not None][:LIVE_TICKS]
assert len(reborn) >= LIVE_TICKS // 2, "recycled slot produced too few live frames"
diffs = [float(np.max(np.abs(x - y))) for x, y in zip(fresh, reborn)]
first_flip = next((i for i, d in enumerate(diffs) if d > 5e-2), len(diffs))
print(
f"[elastic] replay exactness: {first_flip}/{len(diffs)} opening ticks at noise level "
f"(head diffs: {[f'{d:.1e}' for d in diffs[:6]]})"
)
assert first_flip >= 4, (
f"recycled slot diverged from fresh behavior at tick {first_flip} "
f"(diffs {diffs[: first_flip + 2]}); the system-prompt replay is broken"
)
assert max(diffs[:4]) < 2e-2, f"reborn audio not at noise level: {max(diffs[:4])}"
def test_recycled_slot_converses(engine):
"""Behavioral guarantee: a recycled slot holds a coherent conversation.
Feed a real spoken question into a slot that was recycled mid-flight (while
other slots keep running); the model's own inner-monologue text stream must
produce a real reply. This is the end-user property the encoder-state
caveats above cannot break.
"""
import os
import sphn
wav_path = os.environ.get("PPLEX_QUESTION_WAV")
if not wav_path:
pytest.skip("set PPLEX_QUESTION_WAV to a spoken-question wav to run")
wav, _ = sphn.read(wav_path, sample_rate=24000)
wav = wav[0].astype(np.float32)
frame = engine.frame_size
pad = (-len(wav)) % frame
wav = np.concatenate([wav, np.zeros(pad, dtype=np.float32)])
q_frames = [wav[i : i + frame] for i in range(0, len(wav), frame)]
engine.open_batch()
pcm = np.zeros((B, frame), dtype=np.float32)
for _ in range(5): # everyone live briefly
engine.step_batch(pcm)
engine.recycle_slot(1)
prefill = list(engine.prefill_steps(None))
while prefill:
while prefill and prefill[0].kind == "voice_end":
engine.voice_end(1)
prefill.pop(0)
if not prefill:
break
engine.step_batch(pcm, {1: prefill.pop(0)})
engine.end_prefill(1)
text = ""
total = len(q_frames) + 120 # question + ~10 s to answer
for i in range(total):
pcm = np.zeros((B, frame), dtype=np.float32)
if i < len(q_frames):
pcm[1] = q_frames[i]
outs = engine.step_batch(pcm)
if outs[1] is not None and outs[1].text:
text += outs[1].text
alpha = sum(c.isalpha() for c in text)
print(f"[elastic] recycled slot inner monologue: {text!r}")
assert alpha >= 15, f"recycled slot produced no coherent reply: {text!r}"
def _bench(batch_sizes: list[int]) -> None:
import time
for bs in batch_sizes:
cfg = PersonaPlexConfig(batch_size=max(bs, 2))
eng = PersonaPlexEngine(cfg).load()
eng.open_batch()
frame = eng.frame_size
pcm = np.zeros((cfg.batch_size, frame), dtype=np.float32)
for _ in range(10): # warm
eng.step_batch(pcm)
torch.cuda.synchronize()
t0 = time.monotonic()
n = 100
for _ in range(n):
eng.step_batch(pcm)
torch.cuda.synchronize()
ms = (time.monotonic() - t0) / n * 1000
mem = torch.cuda.max_memory_allocated() / 2**30
print(f"B={cfg.batch_size:3d} {ms:6.1f} ms/tick RTF/stream {ms / 80:.3f} {mem:.1f} GiB")
del eng
torch.cuda.empty_cache()
if __name__ == "__main__":
import sys
sizes = [int(x) for x in sys.argv[2:]] if len(sys.argv) > 2 and sys.argv[1] == "--bench" else [2, 4, 8]
_bench(sizes)
tests/e2e/features/fullduplex/test_personaplex_server_lease.py
-101
View File
@@ -1,101 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Single-session lease teardown: a cancelled engine call must be drained.
Cancelling an asyncio task blocked on a worker-thread engine call returns
immediately while the thread keeps running (executor futures cannot be
interrupted). The server therefore tracks the in-flight future on the session
state and waits it out before releasing the capacity-one lease; otherwise a
fast reconnect could call ``open()`` on the shared engine while the previous
connection's final ``step()`` is still mutating it.
"""
import asyncio
import contextlib
import sys
import threading
import types
import pytest
try:
import sphn # noqa: F401
except ImportError:
# These lease tests never encode/decode Opus. Keep the optional serving
# dependency from hiding dependency-free teardown regressions.
sys.modules["sphn"] = types.ModuleType("sphn")
from vllm_omni.experimental.fullduplex.personaplex.config import PersonaPlexConfig
from vllm_omni.experimental.fullduplex.personaplex.serving.server import DuplexServer
from vllm_omni.experimental.fullduplex.personaplex.session import PersonaPlexServingSessionState
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@pytest.mark.asyncio
async def test_cancelled_engine_call_is_drained_before_lease_release():
server = DuplexServer(PersonaPlexConfig()) # engine never loaded: stub calls only
state = PersonaPlexServingSessionState()
started = threading.Event()
release = threading.Event()
finished = threading.Event()
def slow_engine_op():
started.set()
release.wait(timeout=5.0)
finished.set()
return []
task = asyncio.create_task(server._engine_call(state, slow_engine_op))
await asyncio.to_thread(started.wait, 5.0)
# Cancel while the worker thread is mid-call: the await returns at once,
# the thread keeps running, and its completion event stays tracked on the
# state (the asyncio future itself reads as cancelled/done here, which is
# exactly why the drain must not rely on it).
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await task
assert state.inflight is not None
assert not state.inflight.is_set()
# The drain must genuinely wait for the worker, not just return.
drain = asyncio.create_task(server._drain_inflight(state))
await asyncio.sleep(0.05)
assert not drain.done()
release.set()
await asyncio.wait_for(drain, timeout=5.0)
assert finished.is_set()
assert state.inflight.is_set()
@pytest.mark.asyncio
async def test_completed_engine_call_clears_inflight():
server = DuplexServer(PersonaPlexConfig())
state = PersonaPlexServingSessionState()
result = await server._engine_call(state, lambda: "ok")
assert result == "ok"
assert state.inflight is None
await server._drain_inflight(state) # no-op on a clean state
@pytest.mark.asyncio
async def test_drain_timeout_keeps_lease_until_worker_finishes():
server = DuplexServer(PersonaPlexConfig())
server._active = True
state = PersonaPlexServingSessionState()
state.inflight = threading.Event()
drained = await server._release_lease_after_drain(state, timeout=0.01)
assert drained is False
assert server._active is True
state.inflight.set()
for _ in range(100):
if not server._active:
break
await asyncio.sleep(0.01)
assert server._active is False
tests/e2e/features/fullduplex/test_personaplex_session.py
-97
View File
@@ -1,97 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Lockstep-contract tests for the PersonaPlex session driver (no GPU / no moshi).
A stub :class:`FrameStepper` stands in for the real Moshi backend so we can assert
the framing + 1-frame-in / 1-frame-out invariants without weights.
"""
import numpy as np
import pytest
from vllm_omni.experimental.fullduplex.personaplex.config import FRAME_SIZE, PersonaPlexConfig
from vllm_omni.experimental.fullduplex.personaplex.engine import FrameOutput, FrameStepper
from vllm_omni.experimental.fullduplex.personaplex.session import PersonaPlexSession
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
class StubStepper:
"""Deterministic FrameStepper: frame i -> audio filled with i, text 't{i}' on odds."""
sample_rate = 24000
frame_size = FRAME_SIZE
def __init__(self) -> None:
self.opened: tuple[str | None, str | None] | None = None
self.steps = 0
def open_session(self, voice_prompt=None, persona=None) -> None:
self.opened = (voice_prompt, persona)
self.steps = 0
def step(self, user_pcm):
assert user_pcm.shape[0] == self.frame_size
self.steps += 1
i = self.steps
return FrameOutput(
audio=np.full(self.frame_size, float(i), dtype=np.float32),
text=(f"t{i}" if i % 2 else None),
)
def test_stub_satisfies_protocol():
assert isinstance(StubStepper(), FrameStepper)
def test_feed_emits_one_output_per_complete_frame():
s = PersonaPlexSession(StubStepper(), PersonaPlexConfig())
s.open(voice_prompt="NATF2.pt", persona="be terse")
out = s.feed(np.zeros(FRAME_SIZE * 3, dtype=np.float32))
assert len(out) == 3
# lockstep order preserved: frame i -> audio filled with i
assert [int(o.audio[0]) for o in out] == [1, 2, 3]
assert [o.text for o in out] == ["t1", None, "t3"]
def test_partial_frames_are_buffered_until_complete():
stub = StubStepper()
s = PersonaPlexSession(stub, PersonaPlexConfig())
s.open()
assert s.feed(np.zeros(FRAME_SIZE // 2, dtype=np.float32)) == [] # nothing yet
assert stub.steps == 0
out = s.feed(np.zeros(FRAME_SIZE, dtype=np.float32)) # 1.5 frames total -> 1 frame
assert len(out) == 1
assert stub.steps == 1
def test_flush_pads_the_tail():
stub = StubStepper()
s = PersonaPlexSession(stub, PersonaPlexConfig())
s.open()
s.feed(np.ones(FRAME_SIZE + 10, dtype=np.float32)) # 1 frame + 10-sample tail
assert stub.steps == 1
tail = s.flush()
assert len(tail) == 1 and stub.steps == 2
assert s.flush() == [] # nothing left
def test_run_concatenates_audio_and_joins_text():
s = PersonaPlexSession(StubStepper(), PersonaPlexConfig())
s.open()
audio, text = s.run(np.zeros(FRAME_SIZE * 4, dtype=np.float32))
assert audio.shape[0] == FRAME_SIZE * 4
assert text == "t1t3" # frames 1..4, text on odds
def test_open_forwards_voice_and_persona():
stub = StubStepper()
s = PersonaPlexSession(stub, PersonaPlexConfig())
s.open(voice_prompt="NATM1.pt", persona="pirate")
assert stub.opened == ("NATM1.pt", "pirate")
def test_feed_before_open_raises():
s = PersonaPlexSession(StubStepper(), PersonaPlexConfig())
with pytest.raises(RuntimeError):
s.feed(np.zeros(FRAME_SIZE, dtype=np.float32))
tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
+28 -1
View File
@@ -15,11 +15,26 @@ import pytest
from huggingface_hub import snapshot_download
from tests.helpers.runtime import OmniServerParams, get_model_prefix
from tests.helpers.stage_config import get_deploy_config_path, get_deploy_duplex_max_sessions
from tests.helpers.stage_config import (
get_deploy_config_path,
get_deploy_duplex_max_sessions,
modify_stage_config,
)
MODEL = "openbmb/MiniCPM-o-4_5"
DEPLOY_CONFIG_REL = "minicpmo_4_5.yaml"
DEPLOY_CONFIG = get_deploy_config_path(DEPLOY_CONFIG_REL)
# Eager-execution variant for fast-startup core-tier probes (e.g. the duplex
# client live test): skips CUDA-graph capture on the LLM and Talker stages.
CORE_DEPLOY_CONFIG = modify_stage_config(
DEPLOY_CONFIG,
updates={
"stages": {
0: {"enforce_eager": True},
1: {"enforce_eager": True},
}
},
)
ASSET_DIR = Path(__file__).resolve().parents[3] / "assets" / "minicpmo_4_5"
RESPONSE_REQUIRED_WAV = ASSET_DIR / "response_required_16k.wav"
RESPONSE_REQUIRED_SHA256 = "2e5fd4eb3ee434ce107ee3a0591fa624a33f7683c7462f45fe651c443c9af941"
@@ -39,6 +54,18 @@ SERVER_PARAMS = [
)
]
CORE_SERVER_PARAMS = [
pytest.param(
OmniServerParams(
model=MODEL,
stage_config_path=CORE_DEPLOY_CONFIG,
use_stage_cli=False,
server_args=["--trust-remote-code"],
),
id="three-stage-single-gpu",
)
]
def deploy_max_sessions() -> int:
"""Concurrent duplex sessions the deploy config under test admits.
tests/e2e/online_serving/helpers/minicpmo_realtime_duplex_scenarios.py
+40 -14
View File
@@ -52,28 +52,49 @@ try:
except ImportError as exc: # pragma: no cover - demo dependency.
raise SystemExit("Install websockets first: pip install websockets") from exc
from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
from vllm_omni.clients.duplex import ( # noqa: E402
PCM16_BYTES_PER_SAMPLE,
PCM16_SAMPLE_RATE,
RealtimeEventCollector,
EventCollector,
build_realtime_url,
duplex_unit_boundary_ms,
read_pcm16_wav,
summarize_session_request_metrics,
)
from vllm_omni.experimental.fullduplex.client import ( # noqa: E402
from vllm_omni.clients.duplex import ( # noqa: E402
reference_audio_data_url as _ref_audio_data_url,
)
from vllm_omni.experimental.fullduplex.video_stacking import ( # noqa: E402
concat_frames_b64,
unit_subframe_offsets,
)
from vllm_omni.metrics.definitions import compute_audio_rtf # noqa: E402
def _url_with_model(url, model, *, autostart=None, session_id=None):
# This driver speaks the MiniCPM-o native duplex wire contract.
return build_realtime_url(
url,
model,
autostart=autostart,
session_id=session_id,
extra_query={"native_duplex": "1"},
)
_url_with_model = build_realtime_url
_read_wav_pcm16 = read_pcm16_wav
DemoArgs = argparse.Namespace | SimpleNamespace
def _audio_rtf_from_raw_metric(raw_metric: dict[str, object]) -> float | None:
"""Derive audio RTF from the raw timings reported by the client."""
generation_ms = raw_metric.get("audio_generation_ms")
duration_ms = raw_metric.get("audio_duration_ms")
if not isinstance(generation_ms, int | float) or not isinstance(duration_ms, int | float) or duration_ms <= 0:
return None
return round(compute_audio_rtf(float(generation_ms) / 1000.0, float(duration_ms) / 1000.0), 6)
def _video_frames_from_file(
path: Path,
*,
@@ -156,7 +177,7 @@ def _resolve_video_frames(args: DemoArgs) -> tuple[list[str], list[str | None]]:
@dataclass
class DemoState:
events: list[dict[str, object]] = field(default_factory=list)
timing_events: RealtimeEventCollector = field(default_factory=RealtimeEventCollector)
timing_events: EventCollector = field(default_factory=EventCollector)
audio_deltas: list[bytes] = field(default_factory=list)
response_audio_deltas: dict[str, list[bytes]] = field(default_factory=dict)
response_ids: list[str] = field(default_factory=list)
@@ -273,6 +294,11 @@ class DemoState:
after_s=created_at_s,
input_committed_at_s=input_committed_at_s,
response_id=response_id,
measurement_origin={
"ttft": "input_audio_buffer.commit client send to first non-empty text delta",
"ttfp": "input_audio_buffer.commit client send to first audio packet",
"rtf": "commit-to-last-audio receive time divided by emitted audio duration",
},
)
timing["measurement_origin"] = {
"response": "response.created client receive",
@@ -292,14 +318,14 @@ class DemoState:
metrics = timing.get("request_metrics")
if not isinstance(metrics, dict):
continue
requests.append(
{
"session_id": session_id,
"request_index": request_index,
"response_id": response_id,
**metrics,
}
)
request_metrics = {
"session_id": session_id,
"request_index": request_index,
"response_id": response_id,
**metrics,
"rtf": _audio_rtf_from_raw_metric(metrics),
}
requests.append(request_metrics)
return requests
def session_metric_summary(
@@ -602,7 +628,7 @@ def _session_update_event(args: DemoArgs) -> dict[str, object]:
"playback_commit_policy": "ack_only",
"extra_body": {
"auto_response": True,
"minicpmo45_native_duplex": True,
"native_duplex": True,
"force_listen_count": 0,
},
}
tests/e2e/online_serving/personaplex_realtime_duplex.py
+67 -12
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Strict PersonaPlex validation for the unified OpenAI Realtime duplex path."""
from __future__ import annotations
@@ -18,17 +18,72 @@ from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
import numpy as np
from vllm_omni.experimental.fullduplex.client import (
RealtimeDuplexClient,
wait_for,
try:
import websockets
from websockets.exceptions import ConnectionClosed
except ImportError as exc: # pragma: no cover - driver dependency.
raise SystemExit("Install websockets first: pip install websockets") from exc
from vllm_omni.clients.duplex import (
EventCollector,
write_pcm16_wav,
)
from vllm_omni.clients.duplex import (
wait_for_condition as wait_for,
)
SAMPLE_RATE_HZ = 24_000
FRAME_SAMPLES = 1_920
FRAME_PERIOD_S = FRAME_SAMPLES / SAMPLE_RATE_HZ
class RawRealtimeProbe:
"""Raw-event Realtime probe: one websocket plus an :class:`EventCollector`.
This driver speaks the wire protocol directly (it validates handshake
errors such as admission limits, which the public
:class:`vllm_omni.clients.duplex.DuplexClient` treats as failures), so it
keeps a transport-only client and reads everything from ``.events``.
"""
def __init__(self, url: str, *, max_size: int = 64 * 1024 * 1024) -> None:
self.url = url
self.max_size = max_size
self.events = EventCollector()
self._ws = None
self._reader_task: asyncio.Task[None] | None = None
async def __aenter__(self) -> RawRealtimeProbe:
self._ws = await websockets.connect(self.url, max_size=self.max_size)
self._reader_task = asyncio.create_task(self._read_events())
return self
async def __aexit__(self, exc_type, exc, traceback) -> None:
if self._ws is not None:
await self._ws.close()
if self._reader_task is not None:
self._reader_task.cancel()
try:
await self._reader_task
except asyncio.CancelledError:
pass
async def _read_events(self) -> None:
try:
while True:
raw = await self._ws.recv()
if not isinstance(raw, str):
continue
event = json.loads(raw)
if isinstance(event, dict):
self.events.add(event)
except ConnectionClosed:
return
async def send(self, event: dict[str, object]) -> None:
await self._ws.send(json.dumps(event))
def _input_identity(
path: Path,
*,
@@ -83,7 +138,7 @@ def _read_wav_as_float32(path: Path) -> np.ndarray:
return pcm
def _events(client: RealtimeDuplexClient, event_type: str) -> list[dict[str, object]]:
def _events(client: RawRealtimeProbe, event_type: str) -> list[dict[str, object]]:
return [event for event in client.events.events if event.get("type") == event_type]
@@ -93,8 +148,8 @@ async def _open_session(
session_id: str,
persona: str,
expect_error: bool = False,
) -> tuple[RealtimeDuplexClient, dict[str, object]]:
client = RealtimeDuplexClient(_realtime_url(args.url, args.model, session_id))
) -> tuple[RawRealtimeProbe, dict[str, object]]:
client = RawRealtimeProbe(_realtime_url(args.url, args.model, session_id))
await client.__aenter__()
await client.send(
{
@@ -121,7 +176,7 @@ async def _open_session(
return client, _events(client, event_type)[-1]
async def _close_session(client: RealtimeDuplexClient, *, timeout_s: float) -> None:
async def _close_session(client: RawRealtimeProbe, *, timeout_s: float) -> None:
await client.send({"type": "session.close"})
await wait_for(
lambda: client.events.count("session.closed") > 0,
@@ -132,7 +187,7 @@ async def _close_session(client: RealtimeDuplexClient, *, timeout_s: float) -> N
async def _stream_frames(
client: RealtimeDuplexClient,
client: RawRealtimeProbe,
pcm: np.ndarray,
*,
max_frames: int | None = None,
@@ -186,7 +241,7 @@ def _audio_frame_stats(
}
def _response_ids(client: RealtimeDuplexClient) -> set[str]:
def _response_ids(client: RawRealtimeProbe) -> set[str]:
return {
str(event["response_id"])
for event in _events(client, "response.audio.delta")
@@ -195,7 +250,7 @@ def _response_ids(client: RealtimeDuplexClient) -> set[str]:
def _session_result(
client: RealtimeDuplexClient,
client: RawRealtimeProbe,
*,
input_frames: int,
args: argparse.Namespace,
@@ -268,7 +323,7 @@ def _capabilities(created: dict[str, object]) -> dict[str, object]:
return capabilities
def _save(output_dir: Path, name: str, client: RealtimeDuplexClient, raw: bytes) -> None:
def _save(output_dir: Path, name: str, client: RawRealtimeProbe, raw: bytes) -> None:
(output_dir / f"{name}-events.jsonl").write_text(
"".join(json.dumps(event, ensure_ascii=False) + "\n" for event in client.events.events),
encoding="utf-8",
tests/e2e/online_serving/run_minicpmo_realtime_duplex_multi_session.py
+4 -4
View File
@@ -228,7 +228,7 @@ async def _open_admission_session(
"session_id": session_id,
"model": args.model,
"modalities": ["audio", "text"],
"extra_body": {"minicpmo45_native_duplex": True},
"extra_body": {"native_duplex": True},
**({"ref_audio": _ref_audio_data_url(args.ref_audio)} if getattr(args, "ref_audio", None) else {}),
},
}
@@ -271,7 +271,7 @@ async def _admission_probe(args: argparse.Namespace, *, limit: int) -> dict[str,
"session_id": overflow_id,
"model": args.model,
"modalities": ["audio", "text"],
"extra_body": {"minicpmo45_native_duplex": True},
"extra_body": {"native_duplex": True},
**(
{"ref_audio": _ref_audio_data_url(args.ref_audio)}
if getattr(args, "ref_audio", None)
@@ -337,7 +337,7 @@ async def _resume_probe(
"session_id": session_id,
"model": args.model,
"modalities": ["audio", "text"],
"extra_body": {"minicpmo45_native_duplex": True},
"extra_body": {"native_duplex": True},
**(
{"ref_audio": _ref_audio_data_url(args.ref_audio)}
if getattr(args, "ref_audio", None)
@@ -442,7 +442,7 @@ async def _takeover_probe(
"session_id": session_id,
"model": args.model,
"modalities": ["audio", "text"],
"extra_body": {"minicpmo45_native_duplex": True},
"extra_body": {"native_duplex": True},
**(
{"ref_audio": _ref_audio_data_url(args.ref_audio)}
if getattr(args, "ref_audio", None)
tests/e2e/online_serving/test_duplex_client_live.py
+139
View File
@@ -0,0 +1,139 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Live E2E coverage for the public duplex client (``vllm_omni.clients.duplex``).
Drives a real MiniCPM-o 4.5 duplex session end to end through
:class:`DuplexClient`: session handshake with reference audio, paced PCM
streaming, a speak response consumed via :class:`ResponseHandle` with
incremental playback acks, a forced transport drop recovered by automatic
``session.resume``, and a clean close.
"""
from __future__ import annotations
import asyncio
import uuid
from pathlib import Path
import pytest
from tests.e2e.online_serving.helpers.minicpmo_4_5_duplex import (
CORE_SERVER_PARAMS,
realtime_url,
resolve_ref_audio,
validated_input_wav,
)
from tests.helpers.mark import hardware_test
from vllm_omni.clients.duplex import (
DuplexClient,
EventCollector,
ReconnectPolicy,
audio_data_url,
read_pcm16_wav,
)
from vllm_omni.clients.minicpmo_4_5 import create_duplex_session_config
pytestmark = pytest.mark.omni
_SESSION_TIMEOUT_S = 900.0
_MAX_LISTEN_ROUNDS = 30
async def _run_live_session(*, url: str, model: str, ref_audio: Path, input_wav: Path) -> dict[str, object]:
# temperature 0.0 keeps the response-required reply short and
# deterministic (same as the validated demo driver).
config = create_duplex_session_config(ref_audio=audio_data_url(ref_audio), temperature=0.0)
client = DuplexClient(
url,
model=model,
config=config,
session_id=f"duplex-client-live-{uuid.uuid4().hex}",
reconnect=ReconnectPolicy(max_attempts=5, backoff_s=(0.25, 1.0)),
# Short interval so the periodic heartbeat path runs during the
# session — including across the forced transport drop below.
heartbeat_interval_s=5.0,
)
collector = EventCollector()
pcm = read_pcm16_wav(input_wav)
summary: dict[str, object] = {}
async with client:
collector_task = asyncio.create_task(collector.consume(client))
summary["session_id"] = client.session_id
summary["resume_token_issued"] = client.resume_token is not None
# Phase 1: stream the validated utterance and consume one speak response.
await client.stream_pcm(pcm, chunk_ms=200)
await client.commit()
listens = 0
async for response in client.responses():
if response.decision == "listen":
listens += 1
if listens > _MAX_LISTEN_ROUNDS:
raise AssertionError(f"no speak response after {listens} listen decisions")
continue
chunk_count = 0
audio_byte_count = 0
async for chunk in response.audio():
chunk_count += 1
audio_byte_count += len(chunk)
await client.ack_playback(response.played_ms)
summary.update(
{
"decision": response.decision,
"listen_rounds": listens,
"audio_chunks": chunk_count,
"audio_bytes": audio_byte_count,
"played_ms": response.played_ms,
"transcript": response.transcript,
}
)
break
# Phase 2: force-drop the transport underneath the client and require
# the automatic session.resume path to bring the session back.
resumed_waiter = asyncio.create_task(client.wait_for("session.resumed", timeout_s=90.0))
await asyncio.sleep(0.1) # let the waiter subscribe before the drop
assert client._ws is not None
await client._ws.close()
resumed = await resumed_waiter
summary["resumed_session_id"] = resumed.session_id or client.session_id
# The resumed transport must be live: a heartbeat gets acknowledged.
ack_waiter = asyncio.create_task(client.wait_for("session.heartbeat_ack", timeout_s=30.0))
await asyncio.sleep(0.1)
await client.send({"type": "session.heartbeat"})
await ack_waiter
summary["post_resume_heartbeat_ok"] = True
await client.close()
await asyncio.wait_for(collector_task, timeout=10.0)
summary["error_events"] = collector.errors()
summary["closed_cleanly"] = collector.count("session.closed") > 0
summary["timing"] = collector.timing_summary(after_s=0.0)
return summary
@pytest.mark.core_model
@hardware_test(res={"cuda": "H100"}, num_cards=1)
@pytest.mark.parametrize("omni_server", CORE_SERVER_PARAMS, indirect=True)
def test_duplex_client_live_session(omni_server) -> None:
summary = asyncio.run(
asyncio.wait_for(
_run_live_session(
url=realtime_url(omni_server),
model=omni_server.model,
ref_audio=resolve_ref_audio(),
input_wav=validated_input_wav(),
),
timeout=_SESSION_TIMEOUT_S,
)
)
assert summary["decision"] == "speak", summary
assert summary["audio_chunks"] > 0, summary
assert summary["played_ms"] > 0, summary
assert summary["transcript"], summary
assert summary["resume_token_issued"] is True, summary
assert summary["post_resume_heartbeat_ok"] is True, summary
assert summary["closed_cleanly"] is True, summary
assert summary["error_events"] == [], summary
tests/e2e/online_serving/test_minicpmo_4_5_duplex.py
+9 -3
View File
@@ -30,7 +30,7 @@ from tests.e2e.online_serving.run_minicpmo_realtime_duplex_multi_session import
run_multi_session,
)
from tests.helpers.mark import hardware_test
from vllm_omni.experimental.fullduplex.client import build_realtime_url
from vllm_omni.clients.duplex import build_realtime_url
from vllm_omni.experimental.fullduplex.video_stacking import concat_frames_b64
pytestmark = pytest.mark.omni
@@ -88,7 +88,13 @@ async def _receive_protocol_events(ws, required_types: set[str], *, timeout_s: f
async def _run_protocol_smoke(*, url: str, model: str, ref_audio: Path) -> list[dict[str, object]]:
session_id = f"duplex-ci-protocol-{uuid.uuid4().hex}"
websocket_url = build_realtime_url(url, model, autostart=False, session_id=session_id)
websocket_url = build_realtime_url(
url,
model,
autostart=False,
session_id=session_id,
extra_query={"native_duplex": "1"},
)
async with websockets.connect(websocket_url, max_size=64 * 1024 * 1024) as ws:
await ws.send(
json.dumps(
@@ -99,7 +105,7 @@ async def _run_protocol_smoke(*, url: str, model: str, ref_audio: Path) -> list[
"model": model,
"modalities": ["audio", "text"],
"ref_audio": _ref_audio_data_url(str(ref_audio)),
"extra_body": {"minicpmo45_native_duplex": True},
"extra_body": {"native_duplex": True},
},
}
)
tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py
+2 -2
View File
@@ -1001,7 +1001,7 @@ def test_realtime_duplex_demo_explicitly_enables_native_runtime_before_connect()
)
query = parse_qs(urlsplit(url).query)
assert query["minicpmo45_native_duplex"] == ["1"]
assert query["native_duplex"] == ["1"]
def test_realtime_duplex_demo_explicit_session_id_reaches_autostart_query():
@@ -1032,7 +1032,7 @@ def test_realtime_duplex_demo_session_update_uses_explicit_session_id():
assert event["type"] == "session.update"
assert "session_id" not in event
assert event["session"]["session_id"] == "reopen-e2e"
assert event["session"]["extra_body"]["minicpmo45_native_duplex"] is True
assert event["session"]["extra_body"]["native_duplex"] is True
def test_realtime_duplex_demo_response_required_uses_deterministic_sampling():
tests/e2e/features/fullduplex/minicpmo45/__init__.py → tests/engine/duplex/__init__.py
View File
File renamed without changes.
tests/e2e/features/fullduplex/engine/test_duplex_control_client.py → tests/engine/duplex/test_duplex_control_client.py
+4 -4
View File
@@ -1,16 +1,16 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import pytest
from vllm_omni.experimental.fullduplex.engine.duplex_control_client import (
from vllm_omni.engine.duplex.control_client import (
DuplexControlClient,
DuplexControlRequestError,
)
from vllm_omni.experimental.fullduplex.engine.lease import DuplexLeaseActivity
from vllm_omni.experimental.fullduplex.engine.messages import (
from vllm_omni.engine.duplex.lease import DuplexLeaseActivity
from vllm_omni.engine.duplex.messages import (
DuplexControlError,
DuplexControlResultMessage,
DuplexFence,
tests/e2e/features/fullduplex/engine/test_duplex_control_plane.py → tests/engine/duplex/test_duplex_control_plane.py
+9 -9
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
@@ -10,7 +10,7 @@ from dataclasses import FrozenInstanceError
import msgspec
import pytest
from vllm_omni.experimental.fullduplex.engine.duplex_control_plane import (
from vllm_omni.engine.duplex.control_plane import (
DuplexControlPlane,
DuplexOutputContext,
DuplexRequestIdentity,
@@ -18,13 +18,8 @@ from vllm_omni.experimental.fullduplex.engine.duplex_control_plane import (
DuplexStageSubmission,
DuplexStageSubmissionResult,
)
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import (
DuplexAppendPlan,
DuplexInputMode,
DuplexRuntimeCapabilities,
)
from vllm_omni.experimental.fullduplex.engine.lease import DuplexLeaseActivity, DuplexLeaseConfig
from vllm_omni.experimental.fullduplex.engine.messages import (
from vllm_omni.engine.duplex.lease import DuplexLeaseActivity, DuplexLeaseConfig
from vllm_omni.engine.duplex.messages import (
AppendDuplexInputMessage,
CloseDuplexSessionMessage,
DuplexFence,
@@ -34,6 +29,11 @@ from vllm_omni.experimental.fullduplex.engine.messages import (
SignalDuplexTurnMessage,
TouchDuplexSessionMessage,
)
from vllm_omni.engine.duplex.runtime import (
DuplexAppendPlan,
DuplexInputMode,
DuplexRuntimeCapabilities,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/e2e/features/fullduplex/engine/test_duplex_deploy_config.py → tests/engine/duplex/test_duplex_deploy_config.py
+1 -1
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Focused tests for typed deployment-only runtime configuration."""
from __future__ import annotations
tests/e2e/features/fullduplex/engine/test_duplex_intermediate.py → tests/engine/duplex/test_duplex_intermediate.py
+4 -1
View File
@@ -1,6 +1,9 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
from vllm_omni.experimental.fullduplex.engine.intermediate import get_stream_request_key
from vllm_omni.engine.duplex.intermediate import get_stream_request_key
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/e2e/features/fullduplex/engine/test_duplex_lease.py → tests/engine/duplex/test_duplex_lease.py
+5 -5
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
@@ -7,13 +7,13 @@ from dataclasses import FrozenInstanceError
import pytest
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import DuplexInputMode
from vllm_omni.experimental.fullduplex.engine.duplex_session import DuplexSessionRuntimeManager
from vllm_omni.experimental.fullduplex.engine.lease import (
from vllm_omni.engine.duplex.lease import (
DuplexLeaseActivity,
DuplexLeaseConfig,
)
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.duplex.runtime import DuplexInputMode
from vllm_omni.engine.duplex.session import DuplexSessionRuntimeManager
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/e2e/features/fullduplex/engine/test_duplex_runtime.py → tests/engine/duplex/test_duplex_runtime.py
+8 -5
View File
@@ -1,24 +1,27 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from dataclasses import FrozenInstanceError
from types import SimpleNamespace
import pytest
from vllm.sampling_params import SamplingParams
from vllm_omni.experimental.fullduplex.engine import duplex_runtime
from vllm_omni.experimental.fullduplex.engine.contracts import (
from vllm_omni.engine.duplex import runtime as duplex_runtime
from vllm_omni.engine.duplex.contracts import (
duplex_data_plane_request_info,
duplex_resource_request_belongs_to_session,
duplex_resource_request_id,
)
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import (
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.duplex.runtime import (
DuplexInputMode,
DuplexOutputAction,
DuplexOutputDecision,
DuplexRuntimeCapabilities,
DuplexSessionRuntimeManager,
)
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.experimental.fullduplex.minicpmo45.runtime import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.runtime import (
MiniCPMO45DuplexRuntimeExtension,
build_duplex_data_plane_prompt,
duplex_scheduler_token_budget,
tests/engine/test_async_omni_engine_outputs.py
+13 -10
View File
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Tests for AsyncOmniEngine.try_get_output and try_get_output_async.
Focuses on the critical behavior: when the orchestrator thread dies,
@@ -15,22 +18,22 @@ from pytest_mock import MockerFixture
from vllm_omni.engine.async_engine_utils import weak_shutdown_async_omni_engine
from vllm_omni.engine.async_omni_engine import AsyncOmniEngine
from vllm_omni.engine.duplex.control_client import (
DuplexControlClient,
DuplexControlRequestError,
)
from vllm_omni.engine.duplex.messages import (
AppendDuplexInputMessage,
DuplexControlResultMessage,
DuplexFence,
SignalDuplexTurnMessage,
)
from vllm_omni.engine.messages import (
CollectiveRPCResultMessage,
ErrorMessage,
OutputMessage,
)
from vllm_omni.engine.rpc_result_router import CorrelatedRpcClient
from vllm_omni.experimental.fullduplex.engine.duplex_control_client import (
DuplexControlClient,
DuplexControlRequestError,
)
from vllm_omni.experimental.fullduplex.engine.messages import (
AppendDuplexInputMessage,
DuplexControlResultMessage,
DuplexFence,
SignalDuplexTurnMessage,
)
from vllm_omni.outputs import OmniRequestOutput
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/engine/test_duplex_import_boundary.py
+107 -14
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
@@ -29,7 +29,9 @@ def _assert_isolated_import_succeeds(script: str) -> None:
env=env,
capture_output=True,
text=True,
timeout=60,
# The CLI probe imports the whole vllm + benchmark graph; on a loaded
# shared host that alone can take over a minute.
timeout=180,
check=False,
)
assert result.returncode == 0, result.stderr
@@ -43,7 +45,11 @@ def test_isolated_import_ignores_conflicting_gpu_visibility(monkeypatch: pytest.
_assert_isolated_import_succeeds("import vllm_omni.outputs")
def test_stable_engine_imports_do_not_load_experimental_duplex() -> None:
def test_stable_engine_imports_load_duplex_kernel_eagerly() -> None:
# The duplex kernel is imported eagerly by the stable engine modules.
# Model-specific duplex adapters must still load only via the
# dotted-string plugin paths, and the client-side vllm_omni.clients
# package must never enter the stable runtime import graph.
_assert_isolated_import_succeeds("""
import sys
@@ -51,14 +57,84 @@ import vllm_omni.engine.async_omni_engine
import vllm_omni.engine.orchestrator
import vllm_omni.entrypoints.async_omni
expected_eager = (
"vllm_omni.engine.duplex.contracts",
"vllm_omni.engine.duplex.control_plane",
"vllm_omni.entrypoints.duplex_request_client",
"vllm_omni.outputs.duplex",
)
missing = sorted(name for name in expected_eager if name not in sys.modules)
if missing:
raise SystemExit("duplex kernel modules not imported eagerly: " + ", ".join(missing))
forbidden_prefixes = (
"vllm_omni.clients",
"vllm_omni.model_executor.models.minicpmo_4_5.duplex",
"vllm_omni.model_executor.models.nemotron_voicechat.duplex",
"vllm_omni.model_executor.models.personaplex.duplex",
)
loaded = sorted(
name
for name in sys.modules
if name == "vllm_omni.experimental.fullduplex"
or name.startswith("vllm_omni.experimental.fullduplex.")
if any(name == prefix or name.startswith(prefix + ".") for prefix in forbidden_prefixes)
)
if loaded:
raise SystemExit("stable imports loaded experimental duplex: " + ", ".join(loaded))
raise SystemExit("stable imports loaded plugin-only duplex modules: " + ", ".join(loaded))
""")
def test_cli_import_graph_stays_clear_of_the_client_package() -> None:
# `vllm-omni serve` imports the CLI package, whose __init__ eagerly pulls
# in the benchmark patch module and every benchmark subcommand. None of
# that may drag the client-side vllm_omni.clients package into the server
# process; the benchmark runtime imports it lazily at execution time.
_assert_isolated_import_succeeds("""
import sys
import vllm_omni.entrypoints.cli.main
loaded = sorted(
name
for name in sys.modules
if name == "vllm_omni.clients" or name.startswith("vllm_omni.clients.")
)
if loaded:
raise SystemExit("CLI import graph loaded the client package: " + ", ".join(loaded))
""")
def test_client_package_stays_clear_of_the_server_import_graph() -> None:
# vllm_omni.clients.duplex advertises a lightweight dependency set
# (pybase64 + websockets). Importing it and exercising the code paths
# that previously reached into the server tree (timing_summary once
# imported vllm_omni.metrics, dragging in prometheus_client) must not
# load any vllm_omni module outside the client package.
_assert_isolated_import_succeeds("""
import base64
import sys
import vllm_omni # the package __init__ loads its own baseline graph
baseline = set(sys.modules)
from vllm_omni.clients.duplex import EventCollector
collector = EventCollector()
chunk = base64.b64encode(b"\\x00\\x00" * 2400).decode("ascii")
collector.add({"type": "response.created", "response_id": "r1"}, received_at_s=1.0)
collector.add({"type": "response.audio.delta", "response_id": "r1", "delta": chunk}, received_at_s=1.1)
collector.add({"type": "response.audio.delta", "response_id": "r1", "delta": chunk}, received_at_s=1.2)
summary = collector.timing_summary(after_s=0.0, response_id="r1")
if "audio_output" not in summary or "request_metrics" not in summary:
raise SystemExit("timing_summary produced no audio_output/request_metrics sections")
added = sorted(
name
for name in set(sys.modules) - baseline
if name.startswith("vllm_omni.") and not name.startswith("vllm_omni.clients")
)
if added:
raise SystemExit("the client package pulled in server modules: " + ", ".join(added))
""")
@@ -95,26 +171,43 @@ if "duplex_output_decision" in fields:
""")
def test_stable_model_executor_does_not_expose_duplex_helper_module() -> None:
def test_duplex_sampling_helper_lives_in_model_executor() -> None:
_assert_isolated_import_succeeds("""
import importlib.util
if importlib.util.find_spec("vllm_omni.model_executor.duplex") is not None:
raise SystemExit("stable model_executor still exposes duplex helper module")
if importlib.util.find_spec("vllm_omni.model_executor.duplex_sampling") is None:
raise SystemExit("model_executor duplex_sampling helper module is missing")
""")
assert not (REPO_ROOT / "vllm_omni" / "experimental" / "fullduplex" / "model_executor.py").exists()
def test_runtime_package_does_not_bundle_the_browser_demo() -> None:
assert not (REPO_ROOT / "vllm_omni" / "experimental" / "fullduplex" / "web").exists()
def test_experimental_engine_uses_canonical_contract_module_names() -> None:
engine_dir = REPO_ROOT / "vllm_omni" / "experimental" / "fullduplex" / "engine"
def test_engine_duplex_uses_canonical_contract_module_names() -> None:
engine_dir = REPO_ROOT / "vllm_omni" / "engine" / "duplex"
core_dir = REPO_ROOT / "vllm_omni" / "experimental" / "fullduplex" / "core"
assert (engine_dir / "contracts.py").is_file()
assert (engine_dir / "lease.py").is_file()
assert (engine_dir / "messages.py").is_file()
for name in (
"contracts.py",
"lease.py",
"messages.py",
"session.py",
"control_plane.py",
"control_client.py",
"runtime.py",
"intermediate.py",
):
assert (engine_dir / name).is_file()
# the duplex_ prefix is dropped inside the duplex package
assert not (engine_dir / "duplex_session.py").exists()
assert not (engine_dir / "duplex_lease.py").exists()
assert not (engine_dir / "duplex_types.py").exists()
# the experimental package retains only the joyvl framework; stale
# __pycache__ leftovers may exist, so assert on source files, not the dir
old_engine_dir = REPO_ROOT / "vllm_omni" / "experimental" / "fullduplex" / "engine"
assert not list(old_engine_dir.glob("*.py"))
old_personaplex_dir = REPO_ROOT / "vllm_omni" / "experimental" / "fullduplex" / "personaplex"
assert not list(old_personaplex_dir.glob("**/*.py"))
assert not (core_dir / "identity.py").exists()
tests/engine/test_orchestrator.py
+15 -15
View File
@@ -21,6 +21,20 @@ from vllm.v1.engine import EngineCoreOutput, EngineCoreOutputs, FinishReason
from vllm.v1.engine.exceptions import EngineDeadError
from vllm.v1.metrics.stats import IterationStats
from vllm_omni.engine.duplex.control_plane import DuplexControlPlane
from vllm_omni.engine.duplex.messages import (
AppendDuplexInputMessage,
CloseDuplexSessionMessage,
DuplexFence,
OpenDuplexSessionMessage,
SignalDuplexTurnMessage,
)
from vllm_omni.engine.duplex.runtime import (
DuplexInputMode,
DuplexRuntimeCapabilities,
DuplexSessionRuntimeState,
duplex_resource_request_id,
)
from vllm_omni.engine.messages import (
AbortRequestMessage,
AbortResultMessage,
@@ -39,22 +53,8 @@ from vllm_omni.engine.orchestrator import (
_build_terminal_empty_output,
)
from vllm_omni.engine.stage_pool import StagePool
from vllm_omni.experimental.fullduplex.engine.duplex_control_plane import DuplexControlPlane
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import (
DuplexInputMode,
DuplexRuntimeCapabilities,
DuplexSessionRuntimeState,
duplex_resource_request_id,
)
from vllm_omni.experimental.fullduplex.engine.messages import (
AppendDuplexInputMessage,
CloseDuplexSessionMessage,
DuplexFence,
OpenDuplexSessionMessage,
SignalDuplexTurnMessage,
)
from vllm_omni.experimental.fullduplex.minicpmo45.runtime import MiniCPMO45DuplexRuntimeExtension
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.runtime import MiniCPMO45DuplexRuntimeExtension
from vllm_omni.outputs import OmniRequestOutput
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/engine/test_orchestrator_segment_stage_keying.py
+3 -3
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Streaming segment-boundary state must be keyed by stage.
``Orchestrator._orchestration_loop`` polls every stage into the same
@@ -15,13 +15,13 @@ from dataclasses import dataclass, field
import pytest
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.contracts import DuplexRequestIdentity
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.orchestrator import (
Orchestrator,
OrchestratorRequestState,
StreamingSegmentState,
)
from vllm_omni.experimental.fullduplex.engine.contracts import DuplexRequestIdentity
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/engine/test_orchestrator_stage_input_bridge.py
+6 -6
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
@@ -14,6 +14,11 @@ import pytest
from vllm.outputs import CompletionOutput, RequestOutput
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.contracts import (
DuplexStageRequestContext,
DuplexStageSubmission,
)
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.orchestrator import (
Orchestrator,
OrchestratorRequestState,
@@ -21,11 +26,6 @@ from vllm_omni.engine.orchestrator import (
_OrchestratorDuplexStagePort,
)
from vllm_omni.engine.stage_pool import StagePool
from vllm_omni.experimental.fullduplex.engine.contracts import (
DuplexStageRequestContext,
DuplexStageSubmission,
)
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/e2e/features/fullduplex/openai/test_duplex_audio.py → tests/entrypoints/duplex/test_duplex_audio.py
+2 -2
View File
@@ -1,12 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
import numpy as np
import pytest
from vllm_omni.experimental.fullduplex.openai.audio import convert_input_audio_with_rate
from vllm_omni.entrypoints.duplex.audio import convert_input_audio_with_rate
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/e2e/features/fullduplex/openai/test_runtime_adapter_boundary.py → tests/entrypoints/duplex/test_runtime_adapter_boundary.py
+17 -10
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
@@ -10,7 +10,7 @@ from types import SimpleNamespace
import pytest
from vllm_omni.experimental.fullduplex.openai.runtime_adapter import (
from vllm_omni.entrypoints.duplex.runtime_adapter import (
validate_serving_runtime_adapter,
)
@@ -20,26 +20,33 @@ pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@pytest.mark.parametrize(
"module_name",
[
"vllm_omni.experimental.fullduplex.openai.session_runner",
"vllm_omni.experimental.fullduplex.openai.runtime_bridge",
"vllm_omni.experimental.fullduplex.openai.serving",
"vllm_omni.entrypoints.duplex.session_runner",
"vllm_omni.entrypoints.duplex.runtime_bridge",
"vllm_omni.entrypoints.duplex.serving",
],
)
def test_generic_openai_runtime_import_does_not_load_minicpmo45(module_name: str) -> None:
def test_generic_openai_runtime_import_does_not_load_model_adapters(module_name: str) -> None:
repo_root = Path(__file__).resolve().parents[3]
script = f"""
import sys
import {module_name}
model_duplex_prefixes = (
"vllm_omni.model_executor.models.minicpmo_4_5.duplex",
"vllm_omni.model_executor.models.nemotron_voicechat.duplex",
"vllm_omni.model_executor.models.personaplex.duplex",
)
loaded = sorted(
name
for name in sys.modules
if name == "vllm_omni.experimental.fullduplex.minicpmo45"
or name.startswith("vllm_omni.experimental.fullduplex.minicpmo45.")
if any(
name == prefix or name.startswith(prefix + ".")
for prefix in model_duplex_prefixes
)
)
if loaded:
raise SystemExit("generic OpenAI runtime loaded MiniCPM-o: " + ", ".join(loaded))
raise SystemExit("generic OpenAI runtime loaded model duplex modules: " + ", ".join(loaded))
"""
result = subprocess.run(
@@ -59,7 +66,7 @@ def test_generic_handler_requires_explicit_serving_runtime_adapter() -> None:
script = """
from types import SimpleNamespace
from vllm_omni.experimental.fullduplex.openai.serving import OmniDuplexSessionHandler
from vllm_omni.entrypoints.duplex.serving import OmniDuplexSessionHandler
chat_service = SimpleNamespace(engine_client=SimpleNamespace())
try:
tests/e2e/features/fullduplex/openai/test_runtime_control_redaction.py → tests/entrypoints/duplex/test_runtime_control_redaction.py
+3 -3
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Regression tests for duplex ``runtime_control`` payload redaction.
@@ -19,8 +19,8 @@ from enum import Enum
import pytest
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.experimental.fullduplex.openai.runtime_bridge import NativeRuntimeBridgeMixin
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.entrypoints.duplex.runtime_bridge import NativeRuntimeBridgeMixin
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/e2e/features/fullduplex/openai/test_websocket_actor.py → tests/entrypoints/duplex/test_websocket_actor.py
+2 -2
View File
@@ -1,11 +1,11 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import asyncio
import pytest
from vllm_omni.experimental.fullduplex.openai.websocket import (
from vllm_omni.entrypoints.duplex.websocket import (
DuplexWebSocketActor,
normalize_duplex_input_event,
)
tests/entrypoints/openai/test_duplex_protocol.py
+81 -11
View File
@@ -1,10 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import asyncio
import json
import pytest
from vllm_omni.experimental.fullduplex.openai.protocol import (
DuplexCapabilities,
from vllm_omni.entrypoints.duplex.protocol import (
DuplexOverlapPolicy,
DuplexSession,
DuplexSessionConfig,
@@ -14,10 +16,13 @@ from vllm_omni.experimental.fullduplex.openai.protocol import (
DuplexTurnState,
ResponseCreateOptions,
)
from vllm_omni.experimental.fullduplex.openai.realtime_session import (
from vllm_omni.entrypoints.duplex.realtime_session import (
NativeRealtimeSessionProtocol,
)
from vllm_omni.experimental.fullduplex.openai.realtime_state import RealtimeStateOwner
from vllm_omni.entrypoints.duplex.realtime_state import RealtimeStateOwner
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.capabilities import (
minicpmo45_native_capabilities,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@@ -116,12 +121,28 @@ def test_realtime_model_name_does_not_implicitly_enable_native_duplex():
event = protocol._session_create_from_realtime({"model": "openbmb/MiniCPM-o-4_5"})
assert "native_duplex" not in event["session"]["extra_body"]
assert "minicpmo45_native_duplex" not in event["session"]["extra_body"]
def test_realtime_explicit_native_duplex_flag_is_preserved():
protocol = NativeRealtimeSessionProtocol({})
event = protocol._session_create_from_realtime(
{
"model": "openbmb/MiniCPM-o-4_5",
"extra_body": {"native_duplex": True},
}
)
assert event["session"]["extra_body"]["native_duplex"] is True
def test_realtime_legacy_native_duplex_alias_is_normalized():
# The deprecated model-prefixed spelling still opts in, but everything
# downstream (and every session echo) sees only the canonical key.
protocol = NativeRealtimeSessionProtocol({})
event = protocol._session_create_from_realtime(
{
"model": "openbmb/MiniCPM-o-4_5",
@@ -129,10 +150,25 @@ def test_realtime_explicit_native_duplex_flag_is_preserved():
}
)
assert event["session"]["extra_body"]["minicpmo45_native_duplex"] is True
assert event["session"]["extra_body"]["native_duplex"] is True
assert "minicpmo45_native_duplex" not in event["session"]["extra_body"]
def test_realtime_explicit_query_native_duplex_flag_is_available_before_autostart():
protocol = NativeRealtimeSessionProtocol(
{
"model": "openbmb/MiniCPM-o-4_5",
"native_duplex": "1",
}
)
event = json.loads(asyncio.run(protocol.receive_internal_event_text(None)))
assert event["type"] == "session.create"
assert event["session"]["extra_body"]["native_duplex"] is True
def test_realtime_legacy_query_native_duplex_alias_seeds_canonical_key():
protocol = NativeRealtimeSessionProtocol(
{
"model": "openbmb/MiniCPM-o-4_5",
@@ -143,7 +179,8 @@ def test_realtime_explicit_query_native_duplex_flag_is_available_before_autostar
event = json.loads(asyncio.run(protocol.receive_internal_event_text(None)))
assert event["type"] == "session.create"
assert event["session"]["extra_body"]["minicpmo45_native_duplex"] is True
assert event["session"]["extra_body"]["native_duplex"] is True
assert "minicpmo45_native_duplex" not in event["session"]["extra_body"]
@pytest.mark.asyncio
@@ -151,7 +188,7 @@ async def test_realtime_resume_heartbeat_and_event_ack_translate_without_session
protocol = NativeRealtimeSessionProtocol(
{
"model": "openbmb/MiniCPM-o-4_5",
"minicpmo45_native_duplex": "1",
"native_duplex": "1",
}
)
@@ -217,7 +254,7 @@ async def test_realtime_resume_query_suppresses_model_autostart_on_same_url():
protocol = NativeRealtimeSessionProtocol(
{
"model": "openbmb/MiniCPM-o-4_5",
"minicpmo45_native_duplex": "1",
"native_duplex": "1",
"resume": "1",
}
)
@@ -293,6 +330,39 @@ def test_realtime_projects_resume_lifecycle_events_without_duplex_prefix():
}
def test_realtime_projects_listen_response_identity():
protocol = NativeRealtimeSessionProtocol({})
protocol.encode_outbound_event({"type": "session.created", "session": {"id": "sid-listen"}})
bound = protocol.encode_outbound_event(
{
"type": "response.listen",
"session_id": "sid-listen",
"epoch": 0,
"reason": "model_listen",
"model_listen": True,
"response_id": "resp_1",
}
)[0]
# A listen that terminates a precreated response keeps the response
# identity on the Realtime dialect; clients demultiplex on response_id.
assert bound["type"] == "response.listen"
assert bound["response_id"] == "resp_1"
assert bound["response"]["id"] == "resp_1"
assert bound["response"]["status"] == "listening"
anonymous = protocol.encode_outbound_event(
{
"type": "response.listen",
"session_id": "sid-listen",
"epoch": 0,
"reason": "silence_or_noise",
}
)[0]
assert "response_id" not in anonymous
assert "id" not in anonymous["response"]
def test_duplex_session_registry_advances_incarnation_when_id_is_reused():
registry = DuplexSessionRegistry()
@@ -436,7 +506,7 @@ def test_duplex_capabilities_do_not_claim_core_kv_or_input_append():
def test_minicpmo_native_capabilities_separate_model_state_from_core_kv_lease():
caps = DuplexCapabilities.minicpmo45_native(max_sessions=2).as_dict()
caps = minicpmo45_native_capabilities(max_sessions=2).as_dict()
assert caps["implementation_level"] == "model_native_duplex"
assert caps["supports_input_append"] is True
@@ -463,7 +533,7 @@ def test_minicpmo_native_capabilities_separate_model_state_from_core_kv_lease():
def test_duplex_overlap_policy_auto_falls_back_to_listen_only():
from vllm_omni.experimental.fullduplex.openai.protocol import (
from vllm_omni.entrypoints.duplex.protocol import (
DuplexOverlapPolicy,
DuplexSessionConfig,
)
@@ -472,7 +542,7 @@ def test_duplex_overlap_policy_auto_falls_back_to_listen_only():
def test_minicpmo_native_capabilities_do_not_overclaim_single_session_deployment():
caps = DuplexCapabilities.minicpmo45_native(max_sessions=1).as_dict()
caps = minicpmo45_native_capabilities(max_sessions=1).as_dict()
assert caps["supports_multi_session"] is False
assert caps["supports_multi_session_same_replica"] is False
tests/entrypoints/openai/test_duplex_session_attachment.py
+3 -3
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
@@ -9,7 +9,7 @@ import json
import pytest
from vllm_omni.experimental.fullduplex.openai.session_attachment import (
from vllm_omni.entrypoints.duplex.session_attachment import (
DuplexEventJournal,
DuplexJournalGapError,
DuplexJournalOverflowError,
@@ -43,7 +43,7 @@ def test_resume_credential_uses_256_bit_token_digest_and_redacted_repr(monkeypat
credential = DuplexResumeCredential.from_token(token)
calls = []
import vllm_omni.experimental.fullduplex.openai.session_attachment as attachment_module
import vllm_omni.entrypoints.duplex.session_attachment as attachment_module
original_compare = attachment_module.hmac.compare_digest
tests/entrypoints/openai_api/test_duplex_capability.py
+34 -7
View File
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import ast
@@ -7,13 +10,37 @@ import pytest
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
_REPO_ROOT = Path(__file__).parents[3]
_CLIENTS_PACKAGE = "vllm_omni.clients"
def test_api_server_does_not_import_experimental_duplex_at_module_load() -> None:
api_server = Path(__file__).parents[3] / "vllm_omni/entrypoints/openai/api_server.py"
module = ast.parse(api_server.read_text())
top_level_imports = {
node.module for node in module.body if isinstance(node, ast.ImportFrom) and isinstance(node.module, str)
}
def _imports_clients_package(path: Path) -> bool:
"""True if the module imports the client-side package at module level.
assert not any(module.startswith("vllm_omni.experimental.fullduplex") for module in top_level_imports)
``vllm_omni.clients`` is client-side by construction: the serving stack
(imported eagerly by the API server) must never depend on it.
"""
module = ast.parse(path.read_text(encoding="utf-8"))
for node in module.body:
if isinstance(node, ast.Import):
if any(
alias.name == _CLIENTS_PACKAGE or alias.name.startswith(_CLIENTS_PACKAGE + ".") for alias in node.names
):
return True
elif isinstance(node, ast.ImportFrom):
if node.module and (node.module == _CLIENTS_PACKAGE or node.module.startswith(_CLIENTS_PACKAGE + ".")):
return True
if node.module == "vllm_omni" and any(alias.name == "clients" for alias in node.names):
return True
return False
def test_api_server_does_not_import_client_package() -> None:
api_server = _REPO_ROOT / "vllm_omni/entrypoints/openai/api_server.py"
assert not _imports_clients_package(api_server)
def test_duplex_serving_stack_does_not_import_client_package() -> None:
duplex_pkg = _REPO_ROOT / "vllm_omni/entrypoints/duplex"
offenders = [path.name for path in sorted(duplex_pkg.glob("*.py")) if _imports_clients_package(path)]
assert not offenders, f"duplex serving modules import the client package: {offenders}"
tests/entrypoints/openai_api/test_duplex_handler.py
+107 -89
View File
@@ -14,46 +14,59 @@ import numpy as np
import pytest
from starlette.websockets import WebSocketDisconnect
# Import the OpenAI entrypoints package at collection time, before pytest
# attaches its caplog handlers to the root logger. vllm's api_server
# transitively imports model_hosting_container_standards, whose import-time
# configure_root_logger() raises the level of every handler then attached to
# the root logger to ERROR. The serving runtime under test reaches this chain
# lazily (runtime_bridge -> entrypoints.openai.protocol.audio), so without
# this eager import the side effect fires mid-test and silences caplog for
# the remainder of the session. Before the capability predicate moved to
# vllm_omni.entrypoints.duplex.capability, this file imported the package at
# module scope implicitly.
import vllm_omni.entrypoints.openai # noqa: F401
from vllm_omni.config.stage_config import DuplexSessionRuntimeConfig
from vllm_omni.experimental.fullduplex.engine.duplex_control_client import DuplexControlRequestError
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import duplex_resource_request_id
from vllm_omni.experimental.fullduplex.engine.lease import DuplexLeaseActivity
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence, DuplexSessionLifecycleMessage
from vllm_omni.experimental.fullduplex.minicpmo45 import (
MiniCPMO45NativeDuplexServingAdapter,
MiniCPMO45PcmAppendBuffer,
)
from vllm_omni.experimental.fullduplex.minicpmo45.data_plane import (
MiniCPMO45DataPlaneContext,
MiniCPMO45DataPlaneSession,
)
from vllm_omni.experimental.fullduplex.minicpmo45.runtime import (
MiniCPMO45DuplexRuntimeExtension,
)
from vllm_omni.experimental.fullduplex.minicpmo45.session import (
MiniCPMO45ServingSessionState,
)
from vllm_omni.experimental.fullduplex.openai import vad as realtime_vad
from vllm_omni.experimental.fullduplex.openai.protocol import (
DuplexCapabilities,
from vllm_omni.engine.duplex.control_client import DuplexControlRequestError
from vllm_omni.engine.duplex.lease import DuplexLeaseActivity
from vllm_omni.engine.duplex.messages import DuplexFence, DuplexSessionLifecycleMessage
from vllm_omni.engine.duplex.runtime import duplex_resource_request_id
from vllm_omni.entrypoints.duplex import vad as realtime_vad
from vllm_omni.entrypoints.duplex.protocol import (
DuplexOverlapPolicy,
DuplexPlaybackCommitPolicy,
DuplexSession,
DuplexSessionConfig,
ResponseCreateOptions,
)
from vllm_omni.experimental.fullduplex.openai.realtime_session import NativeRealtimeSessionProtocol
from vllm_omni.experimental.fullduplex.openai.runtime_adapter import ServingRuntimeConfigError
from vllm_omni.experimental.fullduplex.openai.serving import (
from vllm_omni.entrypoints.duplex.realtime_session import NativeRealtimeSessionProtocol
from vllm_omni.entrypoints.duplex.runtime_adapter import ServingRuntimeConfigError
from vllm_omni.entrypoints.duplex.serving import (
OmniDuplexSessionHandler,
should_enable_duplex_endpoint,
)
from vllm_omni.experimental.fullduplex.openai.websocket import DuplexWebSocketActor
from vllm_omni.experimental.fullduplex.output import attach_duplex_output_decision
from vllm_omni.experimental.fullduplex.personaplex.serving_adapter import (
from vllm_omni.entrypoints.duplex.websocket import DuplexWebSocketActor
from vllm_omni.model_executor.models.minicpmo_4_5.duplex import (
MiniCPMO45NativeDuplexServingAdapter,
MiniCPMO45PcmAppendBuffer,
)
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.capabilities import (
minicpmo45_native_capabilities,
)
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.data_plane import (
MiniCPMO45DataPlaneContext,
MiniCPMO45DataPlaneSession,
)
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.runtime import (
MiniCPMO45DuplexRuntimeExtension,
)
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.session import (
MiniCPMO45ServingSessionState,
)
from vllm_omni.model_executor.models.personaplex.duplex.serving_adapter import (
PersonaPlexServingRuntimeAdapter,
)
from vllm_omni.outputs import OmniRequestOutput
from vllm_omni.outputs.duplex import attach_duplex_output_decision
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@@ -62,12 +75,12 @@ def test_native_input_append_supports_explicit_session_opt_out():
session = DuplexSession(
"sid-explicit-tts",
DuplexSessionConfig(model="test-model"),
capabilities=DuplexCapabilities.minicpmo45_native(),
capabilities=minicpmo45_native_capabilities(),
)
assert OmniDuplexSessionHandler._uses_native_input_append(session) is True
session.config.extra_body["minicpmo45_native_duplex"] = False
session.config.extra_body["native_duplex"] = False
assert OmniDuplexSessionHandler._uses_native_input_append(session) is False
@@ -293,7 +306,7 @@ class FakeEngineClient:
class FakeChatService:
duplex_serving_adapter_path = (
"vllm_omni.experimental.fullduplex.minicpmo45.serving_adapter.MiniCPMO45ServingRuntimeAdapter"
"vllm_omni.model_executor.models.minicpmo_4_5.duplex.serving_adapter.MiniCPMO45ServingRuntimeAdapter"
)
def __init__(self, engine_client: FakeEngineClient) -> None:
@@ -844,7 +857,7 @@ def test_native_response_options_ignore_private_runtime_config():
session = DuplexSession(
session_id="sid-response-private-config",
config=DuplexSessionConfig(),
capabilities=DuplexCapabilities.minicpmo45_native(),
capabilities=minicpmo45_native_capabilities(),
)
handler = OmniDuplexSessionHandler(chat_service=FakeChatService(FakeEngineClient()))
@@ -878,7 +891,7 @@ def test_native_response_create_rejects_unwired_generation_options(payload):
session = DuplexSession(
session_id="sid-response-unwired-options",
config=DuplexSessionConfig(),
capabilities=DuplexCapabilities.minicpmo45_native(),
capabilities=minicpmo45_native_capabilities(),
)
handler = OmniDuplexSessionHandler(chat_service=FakeChatService(FakeEngineClient()))
@@ -1244,7 +1257,7 @@ async def test_minicpmo_native_session_update_rejects_native_duplex_mode_flip():
{
"type": "turn.signal",
"event": "session.update",
"payload": {"extra_body": {"minicpmo45_native_duplex": False}},
"payload": {"extra_body": {"native_duplex": False}},
}
)
ws.put(
@@ -1581,7 +1594,7 @@ def _native_session_create(
event["session"]["model"] = "openbmb/MiniCPM-o-4_5"
event["session"]["modalities"] = list(modalities or ["text"])
event["session"]["instructions"] = "You are a concise assistant."
event["session"]["extra_body"] = {"minicpmo45_native_duplex": True}
event["session"]["extra_body"] = {"native_duplex": True}
return event
@@ -1598,7 +1611,7 @@ def _native_realtime_session_update(
"modalities": list(modalities or ["text"]),
"instructions": "You are a concise assistant.",
"idle_timeout_s": 1,
"extra_body": {"minicpmo45_native_duplex": True},
"extra_body": {"native_duplex": True},
},
}
@@ -1674,7 +1687,7 @@ def _install_direct_silence_scheduler(
handler: OmniDuplexSessionHandler,
session: DuplexSession,
) -> None:
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
async def _schedule(payload: object, **kwargs: Any) -> bool:
if handler._native_silence_continuation_is_stale(
@@ -1819,7 +1832,7 @@ def test_legacy_auto_overlap_policy_falls_back_to_listen_only():
overlap_policy=DuplexSessionConfig._normalize_overlap_policy("auto"),
),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
decision = handler._overlap_decision(
session,
@@ -2066,11 +2079,11 @@ def test_auto_response_post_response_silence_remains_model_owned_input():
@pytest.mark.asyncio
async def test_auto_response_terminal_advances_open_realtime_input_without_commit():
handler, session = _auto_response_context("sid-auto-realtime-open-input", playback_active=True)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
response_id = session.active_response_id
assert response_id is not None
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
native.input_since_commit = True
native.speech_since_commit = True
session.accumulate_overlap_speech(800)
@@ -2113,11 +2126,11 @@ async def test_auto_response_terminal_allows_next_model_unit_while_playback_drai
"sid-auto-realtime-open-before-overlap",
playback_active=True,
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
response_id = session.active_response_id
assert response_id is not None
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
native.input_since_commit = True
native.speech_since_commit = True
session.end_response(commit_text=False, preserve_request=True)
@@ -2147,11 +2160,11 @@ async def test_auto_response_terminal_allows_next_model_unit_while_playback_drai
@pytest.mark.asyncio
async def test_realtime_committed_overlap_promotes_after_response_done():
handler, session = _auto_response_context("sid-auto-realtime-committed-input", playback_active=True)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
response_id = session.active_response_id
assert response_id is not None
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
native.committed_audio_payload = _native_audio_payload()
native.deferred_response_create = True
session.accumulate_overlap_speech(1000)
@@ -3061,7 +3074,7 @@ async def test_native_append_propagates_current_turn_fence_to_engine():
session_id="sid-fenced-append",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.turn_id = 2
session.begin_response(turn_id=2)
ws = TimedWebSocket()
@@ -3091,7 +3104,7 @@ async def test_minicpmo_auto_response_tts_segment_boundary_appends_silence_unit(
session_id="sid-segment-boundary",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.begin_response(turn_id=0)
session.turn_id = 1
session.bind_request(request_id)
@@ -3140,7 +3153,7 @@ async def test_minicpmo_auto_response_pre_response_tts_boundary_continues_model_
session_id="sid-pre-response-boundary",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.bind_request(request_id)
_install_direct_silence_scheduler(handler, session)
ws = TimedWebSocket()
@@ -3187,11 +3200,11 @@ async def test_minicpmo_pre_response_continuation_drops_after_model_turn_ends():
session_id="sid-stale-pre-response-boundary",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.bind_request(request_id)
model_turn_id = session.turn_id
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
async def _stale_before_append(payload: object, **kwargs: Any) -> bool:
session.complete_model_turn(model_turn_id)
@@ -3237,7 +3250,7 @@ async def test_minicpmo_auto_response_continuation_stops_at_large_safety_boundar
session_id="sid-long-response",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.begin_response(turn_id=0)
session.turn_id = 1
session.bind_request(request_id)
@@ -3283,10 +3296,10 @@ async def test_minicpmo_auto_response_boundary_listen_closes_response():
session_id="sid-boundary-listen",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
response_id = session.begin_response(turn_id=0)
session.bind_request(request_id)
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
native.continuation_owner_id = f"response:{response_id}"
native.continuation_units = handler._NATIVE_AUTO_RESPONSE_MAX_CONTINUATION_UNITS
_install_direct_silence_scheduler(handler, session)
@@ -3313,6 +3326,11 @@ async def test_minicpmo_auto_response_boundary_listen_closes_response():
assert emitted is True
assert engine.appended == []
assert ws.sent_types() == ["response.listen", "response.done"]
# The listen terminates the precreated response: clients demultiplex the
# decision by its id, so the payload must carry the response identity.
listen_payload, done_payload = ws.sent[-2], ws.sent[-1]
assert listen_payload["response_id"] == response_id
assert done_payload["response_id"] == response_id
assert session.active_response_id is None
assert session.active_request_id == request_id
assert session.turn_id == 1
@@ -3327,10 +3345,10 @@ async def test_minicpmo_auto_response_boundary_allows_next_model_turn_response()
session_id="sid-boundary-next-turn",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.begin_response(turn_id=0)
session.bind_request(request_id)
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
native.continuation_owner_id = f"response:{session.active_response_id}"
native.continuation_units = handler._NATIVE_AUTO_RESPONSE_MAX_CONTINUATION_UNITS
ws = TimedWebSocket()
@@ -3382,10 +3400,10 @@ async def test_minicpmo_auto_response_boundary_does_not_close_new_epoch_response
session_id="sid-boundary-race",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
old_response_id = session.begin_response(turn_id=0)
session.bind_request(request_id)
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
native.continuation_owner_id = f"response:{old_response_id}"
native.continuation_units = handler._NATIVE_AUTO_RESPONSE_MAX_CONTINUATION_UNITS
sent: list[dict[str, Any]] = []
@@ -3418,7 +3436,7 @@ async def test_minicpmo_auto_response_pre_speak_listen_continues_same_response()
session_id="sid-pre-speak-listen",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
response_id = session.begin_response(turn_id=0)
session.turn_id = 1
session.bind_request(request_id)
@@ -3458,7 +3476,7 @@ async def test_minicpmo_auto_response_listen_before_response_keeps_resumable_req
session_id="sid-listen-before-response",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.bind_request(request_id)
ws = TimedWebSocket()
@@ -3482,7 +3500,7 @@ async def test_minicpmo_auto_response_listen_before_response_keeps_resumable_req
assert close_reason is None
assert emitted is True
assert session.active_request_id == request_id
assert not handler._minicpmo_data_plane.is_terminal(request_id)
assert not handler._serving_runtime_adapter.data_plane.is_terminal(request_id)
assert ws.sent_types().count("response.listen") == 1
@@ -3666,7 +3684,7 @@ async def test_minicpmo_auto_response_post_speak_listen_continues_same_response(
session_id="sid-post-speak-listen",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
response_id = session.begin_response(turn_id=0)
session.turn_id = 1
session.bind_request(request_id)
@@ -3707,7 +3725,7 @@ async def test_minicpmo_auto_response_turn_end_preserves_resumable_request():
session_id="sid-turn-end-preserve-request",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.begin_response(turn_id=0)
session.bind_request(request_id)
ws = TimedWebSocket()
@@ -3746,7 +3764,7 @@ async def test_minicpmo_auto_response_empty_turn_end_emits_model_listen():
session_id="sid-empty-turn-end",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.bind_request(request_id)
ws = TimedWebSocket()
@@ -3789,7 +3807,7 @@ async def test_minicpmo_auto_response_drops_late_audio_from_completed_model_turn
session_id="sid-late-completed-turn",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.bind_request(request_id)
session.complete_model_turn(0)
ws = TimedWebSocket()
@@ -4015,7 +4033,7 @@ async def test_duplex_auto_response_empty_terminal_does_not_create_empty_respons
)
native_results = list(
handler._minicpmo_data_plane.project(
handler._serving_runtime_adapter.data_plane.project(
{"data_plane_outputs": [output]},
context=_data_plane_context(session),
)
@@ -4030,7 +4048,7 @@ async def test_duplex_auto_response_empty_terminal_does_not_create_empty_respons
assert "response.created" not in [m.get("type") for m in sent]
assert "response.done" not in [m.get("type") for m in sent]
assert session.turn_id == 1
assert not handler._minicpmo_data_plane.is_terminal(request_id)
assert not handler._serving_runtime_adapter.data_plane.is_terminal(request_id)
@pytest.mark.asyncio
@@ -4040,11 +4058,11 @@ async def test_minicpmo_native_duplex_session_close_cleans_auto_response_data_pl
engine = FakeEngineClient()
chat_service = FakeChatService(engine)
handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1)
handler._minicpmo_data_plane.slice_cumulative_audio(request_id, np.zeros(24000, dtype=np.float32))
handler._minicpmo_data_plane.segment_text_delta(request_id, "hello")
handler._minicpmo_data_plane.mark_terminal(request_id)
handler._serving_runtime_adapter.data_plane.slice_cumulative_audio(request_id, np.zeros(24000, dtype=np.float32))
handler._serving_runtime_adapter.data_plane.segment_text_delta(request_id, "hello")
handler._serving_runtime_adapter.data_plane.mark_terminal(request_id)
native = MiniCPMO45ServingSessionState()
handler._minicpmo_sessions[session_id] = native
handler._serving_runtime_adapter.session_states[session_id] = native
event = _native_session_create(session_id)
event["session"]["extra_body"]["auto_response"] = True
@@ -4054,8 +4072,8 @@ async def test_minicpmo_native_duplex_session_close_cleans_auto_response_data_pl
await handler.handle_session(ws)
assert not handler._minicpmo_data_plane.has_request(request_id)
assert session_id not in handler._minicpmo_sessions
assert not handler._serving_runtime_adapter.data_plane.has_request(request_id)
assert session_id not in handler._serving_runtime_adapter.session_states
@pytest.mark.asyncio
@@ -4923,7 +4941,7 @@ async def test_realtime_disconnect_detaches_resumable_session_without_runtime_cl
("sid-resumable-disconnect", DuplexLeaseActivity.DETACH),
]
assert handler._registry.get("sid-resumable-disconnect") is not None
assert "sid-resumable-disconnect" in handler._minicpmo_sessions
assert "sid-resumable-disconnect" in handler._serving_runtime_adapter.session_states
@pytest.mark.asyncio
@@ -4947,7 +4965,7 @@ async def test_realtime_initial_resume_token_delivery_failure_closes_unrecoverab
assert engine.closed == [("sid-created-delivery-failure", "disconnect")]
assert handler._registry.get("sid-created-delivery-failure") is None
assert "sid-created-delivery-failure" not in handler._minicpmo_sessions
assert "sid-created-delivery-failure" not in handler._serving_runtime_adapter.session_states
@pytest.mark.asyncio
@@ -4983,7 +5001,7 @@ async def test_realtime_send_after_created_disconnect_preserves_resumable_sessio
("sid-created-then-disconnected", DuplexLeaseActivity.DETACH),
]
assert handler._registry.get("sid-created-then-disconnected") is not None
assert "sid-created-then-disconnected" in handler._minicpmo_sessions
assert "sid-created-then-disconnected" in handler._serving_runtime_adapter.session_states
@pytest.mark.asyncio
@@ -5007,7 +5025,7 @@ async def test_realtime_resume_rotates_token_replays_and_preserves_runtime_ident
second = TimedWebSocket(receive_timeout_s=0.1)
second.query_params = {
"model": "openbmb/MiniCPM-o-4_5",
"minicpmo45_native_duplex": "1",
"native_duplex": "1",
"resume": "1",
}
second.put(
@@ -5083,7 +5101,7 @@ async def test_realtime_resume_preserves_append_tail_order_across_connections():
assert created is not None
session = handler._registry.get("sid-resume-append-order")
assert session is not None
native_state = handler._minicpmo_session_state(session)
native_state = handler._runtime_session_state(session)
first_scheduler = native_state.silence_continuation_scheduler
assert first_scheduler is not None
first.put(
@@ -5101,7 +5119,7 @@ async def test_realtime_resume_preserves_append_tail_order_across_connections():
second = TimedWebSocket(receive_timeout_s=0.5)
second.query_params = {
"model": "openbmb/MiniCPM-o-4_5",
"minicpmo45_native_duplex": "1",
"native_duplex": "1",
"resume": "1",
}
second.put(
@@ -5402,7 +5420,7 @@ async def test_realtime_disconnect_grace_cancels_only_orphan_response():
assert engine.aborted == ["orphan-request"]
assert orphan_task.cancelled()
assert session.active_response_id is None
assert "sid-grace-response" in handler._minicpmo_sessions
assert "sid-grace-response" in handler._serving_runtime_adapter.session_states
@pytest.mark.asyncio
@@ -5453,7 +5471,7 @@ async def test_realtime_idle_ttl_lifecycle_removes_detached_serving_projection()
await asyncio.sleep(0.05)
assert handler._registry.get("sid-idle-expired") is None
assert "sid-idle-expired" not in handler._minicpmo_sessions
assert "sid-idle-expired" not in handler._serving_runtime_adapter.session_states
assert "sid-idle-expired" not in handler._session_tasks
assert engine.closed == []
@@ -6428,7 +6446,7 @@ async def test_minicpmo_native_duplex_separates_public_and_runtime_config(monkey
return [0.25] * 1600, 16000
monkeypatch.setattr(
"vllm_omni.experimental.fullduplex.minicpmo45.MiniCPMO45NativeDuplexServingAdapter.resolve_ref_audio",
"vllm_omni.model_executor.models.minicpmo_4_5.duplex.MiniCPMO45NativeDuplexServingAdapter.resolve_ref_audio",
fake_resolve_ref_audio,
)
event = _native_session_create("sid-native-ref-audio", modalities=["text", "audio"])
@@ -6554,7 +6572,7 @@ async def test_minicpmo_native_duplex_preserves_ref_audio_channels_until_normali
return np.tile(np.array([[0.25, -0.25], [0.5, -0.5]], dtype=np.float32), (800, 1)), 16000
monkeypatch.setattr(
"vllm_omni.experimental.fullduplex.minicpmo45.adapter.MediaConnector",
"vllm_omni.model_executor.models.minicpmo_4_5.duplex.adapter.MediaConnector",
FakeMediaConnector,
)
@@ -6580,7 +6598,7 @@ async def test_minicpmo_native_duplex_rejects_ref_audio_path():
handler = OmniDuplexSessionHandler(chat_service=chat_service, config_timeout_s=0.1, idle_timeout_s=1)
event = _native_session_create("sid-native-ref-path")
event["session"]["extra_body"] = {
"minicpmo45_native_duplex": True,
"native_duplex": True,
"ref_audio_path": "/tmp/ref.wav",
}
ws = TimedWebSocket()
@@ -6603,7 +6621,7 @@ def test_minicpmo_native_duplex_explicit_barge_in_request_interrupts():
session = DuplexSession(
session_id="sid-native-explicit-barge-disabled",
config=DuplexSessionConfig(overlap_policy=DuplexOverlapPolicy.BARGE_IN_ON_SPEECH.value),
capabilities=DuplexCapabilities.minicpmo45_native(),
capabilities=minicpmo45_native_capabilities(),
)
payload = {
"type": "audio",
@@ -6767,9 +6785,9 @@ async def test_minicpmo_auto_response_restarts_drain_when_append_races_idle_exit
session_id="sid-native-late-output",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
session.bind_request(request_id)
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
projected_batches: list[object] = []
projected = asyncio.Event()
@@ -7521,7 +7539,7 @@ async def test_minicpmo_native_auto_response_preserves_silence_continuations_acr
session.capabilities.chunk_period_ms = 50
session.bind_request(request_id)
response_id = session.begin_response(turn_id=session.turn_id)
native = handler._minicpmo_session_state(session)
native = handler._runtime_session_state(session)
scheduler = native.silence_continuation_scheduler
assert scheduler is not None
@@ -7598,7 +7616,7 @@ async def test_minicpmo_native_duplex_uses_segment_text_metadata_for_transcript_
async def emit(output: object) -> None:
result = {"data_plane_outputs": [output]}
for native_result in handler._minicpmo_data_plane.project(
for native_result in handler._serving_runtime_adapter.data_plane.project(
result,
context=_data_plane_context(session),
):
@@ -7682,7 +7700,7 @@ async def test_minicpmo_native_duplex_continuous_speak_reuses_active_response_un
async def emit(output: object) -> None:
result = {"data_plane_outputs": [output]}
for native_result in handler._minicpmo_data_plane.project(
for native_result in handler._serving_runtime_adapter.data_plane.project(
result,
context=_data_plane_context(session),
):
@@ -7932,7 +7950,7 @@ async def test_minicpmo_auto_response_segment_complete_continues_until_model_lis
session_id="sid-native-segment-complete",
config=DuplexSessionConfig(extra_body={"auto_response": True}),
)
session.capabilities = DuplexCapabilities.minicpmo45_native()
session.capabilities = minicpmo45_native_capabilities()
response_id = session.begin_response()
request_id = "duplex-sid-native-segment-complete-stage0"
session.bind_request(request_id)
@@ -7968,7 +7986,7 @@ async def test_minicpmo_auto_response_segment_complete_continues_until_model_lis
assert emitted is False
assert session.active_response_id == response_id
assert session.active_request_id == request_id
assert not handler._minicpmo_data_plane.is_terminal(request_id)
assert not handler._serving_runtime_adapter.data_plane.is_terminal(request_id)
assert sent == []
assert len(engine.appended) == 1
_, mode, payload, final = engine.appended[0]
tests/entrypoints/test_async_omni_duplex.py
+11 -8
View File
@@ -1,21 +1,24 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from types import SimpleNamespace
import pytest
from vllm_omni.engine.messages import OutputMessage
from vllm_omni.entrypoints.async_omni import AsyncOmni
from vllm_omni.entrypoints.client_request_state import ClientRequestState
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import (
from vllm_omni.engine.duplex.lease import DuplexLeaseActivity
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.duplex.runtime import (
DuplexOutputAction,
DuplexOutputDecision,
duplex_resource_request_id,
)
from vllm_omni.experimental.fullduplex.engine.lease import DuplexLeaseActivity
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.experimental.fullduplex.output import attach_duplex_output_decision
from vllm_omni.experimental.fullduplex.request_client import DuplexRequestClient
from vllm_omni.engine.messages import OutputMessage
from vllm_omni.entrypoints.async_omni import AsyncOmni
from vllm_omni.entrypoints.client_request_state import ClientRequestState
from vllm_omni.entrypoints.duplex_request_client import DuplexRequestClient
from vllm_omni.metrics.stats import OrchestratorAggregator, StageRequestStats, StageStats
from vllm_omni.outputs import OmniRequestOutput
from vllm_omni.outputs.duplex import attach_duplex_output_decision
def test_async_omni_uses_extracted_duplex_request_client():
tests/entrypoints/test_duplex_fence_propagation.py
+6 -3
View File
@@ -1,10 +1,13 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
from tests.entrypoints.openai_api.test_duplex_handler import FakeChatService, TimedWebSocket
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.entrypoints.async_omni import AsyncOmni
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.experimental.fullduplex.openai.protocol import DuplexSession, DuplexSessionConfig
from vllm_omni.experimental.fullduplex.openai.serving import OmniDuplexSessionHandler
from vllm_omni.entrypoints.duplex.protocol import DuplexSession, DuplexSessionConfig
from vllm_omni.entrypoints.duplex.serving import OmniDuplexSessionHandler
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/examples/test_minicpmo_realtime_duplex_simple_demo.py
+12 -7
View File
@@ -175,21 +175,26 @@ def test_open_streaming_response_requires_post_commit_drain():
def test_final_input_commit_checkpoints_playback_first():
demo = _load_demo_module()
client = demo.RealtimeDuplexClient("ws://unused")
client.events.add(
collector = demo.EventCollector()
collector.add(
{
"type": "response.created",
"response": {"id": "resp-zero-audio"},
}
)
calls = []
calls: list[dict[str, object]] = []
async def send(event):
calls.append(event)
class _Client:
async def ack_playback(self, played_ms, *, response_id=None, item_id=None, committed_ms=None):
calls.append({"type": "playback.ack", "response_id": response_id, "played_ms": int(played_ms)})
client.send = send
commit_sent_at_s = asyncio.run(demo._commit_input_after_playback_checkpoint(client))
async def commit(self):
calls.append({"type": "input_audio_buffer.commit"})
commit_sent_at_s = asyncio.run(demo._commit_input_after_playback_checkpoint(_Client(), collector))
# The zero-audio response is still acked (0 ms) before the commit: the
# ack checkpoints its history position on the server.
assert [event["type"] for event in calls] == ["playback.ack", "input_audio_buffer.commit"]
assert calls[0]["response_id"] == "resp-zero-audio"
assert calls[0]["played_ms"] == 0
tests/examples/test_minicpmo_realtime_web_static.py
+4 -1
View File
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import re
import shutil
import subprocess
@@ -33,7 +36,7 @@ def test_client_uses_proxy_relative_realtime_url_and_model_policy_session():
assert "new URL(config.realtimePath, window.location.href)" in source
assert "url.protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:'" in source
assert "url.searchParams.set('autostart', '0')" in source
assert "url.searchParams.set('minicpmo45_native_duplex', '1')" in source
assert "url.searchParams.set('native_duplex', '1')" in source
assert "auto_response: true" in source
assert "input_audio_buffer.append" in source
assert "input_audio_buffer.commit" not in source
tests/model_executor/models/minicpmo_4_5/duplex/__init__.py
View File
Whitespace-only changes.
tests/e2e/features/fullduplex/minicpmo45/test_commit_policy.py → tests/model_executor/models/minicpmo_4_5/duplex/test_commit_policy.py
+4 -1
View File
@@ -1,8 +1,11 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import pytest
from vllm_omni.experimental.fullduplex.openai.commit_policy import (
from vllm_omni.entrypoints.duplex.commit_policy import (
CommitAction,
CommitSnapshot,
decide_commit_action,
tests/e2e/features/fullduplex/minicpmo45/test_input.py → tests/model_executor/models/minicpmo_4_5/duplex/test_input.py
+2 -2
View File
@@ -1,12 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
import numpy as np
import pytest
from vllm_omni.experimental.fullduplex.minicpmo45.input import MiniCPMO45PcmAppendBuffer
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.input import MiniCPMO45PcmAppendBuffer
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
tests/model_executor/models/minicpmo_4_5/duplex/test_policy.py
+22
View File
@@ -0,0 +1,22 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.policy import (
MiniCPMO45DuplexPolicy,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def test_seed_tts_initial_text_is_part_of_native_duplex_context():
prefix, suffix = MiniCPMO45DuplexPolicy.session_context_texts(
"Speak exactly.",
True,
"The quick brown fox.",
)
assert prefix == "<|im_start|>system\nSpeak exactly.\n<|audio_start|>"
assert suffix == (
"<|audio_end|><|im_end|>\n<|im_start|>user\nThe quick brown fox.<|im_end|>\n<|im_start|>assistant\n"
)
tests/model_executor/models/minicpmo_4_5/test_pipeline.py
+1 -1
View File
@@ -61,7 +61,7 @@ class TestRegistryDeclaration:
pipeline = OMNI_PIPELINES[_PIPELINE_KEY]
assert pipeline.duplex_control_enabled is True
assert pipeline.duplex_serving_adapter == (
"vllm_omni.experimental.fullduplex.minicpmo45.serving_adapter.MiniCPMO45ServingRuntimeAdapter"
"vllm_omni.model_executor.models.minicpmo_4_5.duplex.serving_adapter.MiniCPMO45ServingRuntimeAdapter"
)
assert not hasattr(pipeline, "max_native_duplex_sessions")
tests/model_executor/models/nemotron_voicechat/__init__.py
+2
View File
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
tests/model_executor/models/nemotron_voicechat/duplex/__init__.py
+2
View File
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
tests/e2e/features/fullduplex/nemotron_voicechat/test_data_plane.py → tests/model_executor/models/nemotron_voicechat/duplex/test_data_plane.py
+6 -3
View File
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
from types import SimpleNamespace
@@ -5,9 +8,9 @@ from types import SimpleNamespace
import numpy as np
import pytest
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import duplex_resource_request_id
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.experimental.fullduplex.nemotron_voicechat.data_plane import (
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.duplex.runtime import duplex_resource_request_id
from vllm_omni.model_executor.models.nemotron_voicechat.duplex.data_plane import (
NemotronVoiceChatDataPlaneContext,
NemotronVoiceChatDataPlaneSession,
)
tests/e2e/features/fullduplex/nemotron_voicechat/test_input.py → tests/model_executor/models/nemotron_voicechat/duplex/test_input.py
+4 -1
View File
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import base64
@@ -5,7 +8,7 @@ import base64
import numpy as np
import pytest
from vllm_omni.experimental.fullduplex.nemotron_voicechat.input import (
from vllm_omni.model_executor.models.nemotron_voicechat.duplex.input import (
NemotronVoiceChatPcmAppendBuffer,
)
tests/e2e/features/fullduplex/nemotron_voicechat/test_runtime_contract.py → tests/model_executor/models/nemotron_voicechat/duplex/test_runtime_contract.py
+4 -4
View File
@@ -10,12 +10,12 @@ import numpy as np
import pytest
from vllm.sampling_params import SamplingParams
from vllm_omni.experimental.fullduplex.engine.contracts import DuplexInputMode
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.experimental.fullduplex.nemotron_voicechat.runtime import (
from vllm_omni.engine.duplex.contracts import DuplexInputMode
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.model_executor.models.nemotron_voicechat.duplex.runtime import (
NemotronVoiceChatDuplexRuntimeExtension,
)
from vllm_omni.experimental.fullduplex.nemotron_voicechat.serving_adapter import (
from vllm_omni.model_executor.models.nemotron_voicechat.duplex.serving_adapter import (
NemotronVoiceChatServingRuntimeAdapter,
_render_tool_response,
)
tests/model_executor/models/personaplex/__init__.py
+2
View File
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
tests/model_executor/models/personaplex/duplex/__init__.py
+2
View File
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
tests/e2e/features/fullduplex/personaplex/test_stage0_runtime.py → tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py
+3 -3
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
from types import SimpleNamespace
@@ -8,11 +8,11 @@ import numpy as np
import pytest
import torch
from vllm_omni.experimental.fullduplex.personaplex.policy import (
from vllm_omni.model_executor.models.personaplex.duplex.policy import (
SILENCE_TOKENS,
SINE_TOKENS,
)
from vllm_omni.experimental.fullduplex.personaplex.stage0 import (
from vllm_omni.model_executor.models.personaplex.duplex.stage0 import (
PersonaPlexStage0DuplexRuntime,
)
from vllm_omni.model_executor.models.personaplex.personaplex_talker import (
tests/e2e/features/fullduplex/personaplex/test_unified_runtime.py → tests/model_executor/models/personaplex/duplex/test_unified_runtime.py
+17 -12
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
from pathlib import Path
@@ -14,25 +14,25 @@ from vllm_omni.config.stage_config import (
load_deploy_config,
merge_pipeline_deploy,
)
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import (
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.duplex.runtime import (
DuplexInputMode,
)
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
from vllm_omni.experimental.fullduplex.openai.runtime_adapter import (
from vllm_omni.entrypoints.duplex.runtime_adapter import (
ServingRuntimeConfigError,
)
from vllm_omni.experimental.fullduplex.personaplex.config import DEFAULT_PERSONA
from vllm_omni.experimental.fullduplex.personaplex.data_plane import (
from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import (
PersonaPlexDataPlaneContext,
PersonaPlexDataPlaneSession,
)
from vllm_omni.experimental.fullduplex.personaplex.input import (
from vllm_omni.model_executor.models.personaplex.duplex.input import (
PersonaPlexPcmAppendBuffer,
)
from vllm_omni.experimental.fullduplex.personaplex.runtime_extension import (
from vllm_omni.model_executor.models.personaplex.duplex.runtime_extension import (
PersonaPlexDuplexRuntimeExtension,
)
from vllm_omni.experimental.fullduplex.personaplex.serving_adapter import (
from vllm_omni.model_executor.models.personaplex.duplex.serving_adapter import (
PersonaPlexServingRuntimeAdapter,
)
from vllm_omni.model_executor.models.personaplex.pipeline import (
@@ -184,11 +184,11 @@ def test_e2e_driver_rejects_unexpected_input_sha256(tmp_path: Path) -> None:
[
(
"duplex_runtime_extension",
"vllm_omni.experimental.fullduplex.personaplex.runtime_extension.PersonaPlexDuplexRuntimeExtension",
"vllm_omni.model_executor.models.personaplex.duplex.runtime_extension.PersonaPlexDuplexRuntimeExtension",
),
(
"duplex_serving_adapter",
"vllm_omni.experimental.fullduplex.personaplex.serving_adapter.PersonaPlexServingRuntimeAdapter",
"vllm_omni.model_executor.models.personaplex.duplex.serving_adapter.PersonaPlexServingRuntimeAdapter",
),
],
)
@@ -286,9 +286,14 @@ def test_personaplex_runtime_config_update_rejects_changed_voice() -> None:
assert unchanged["personaplex_voice_prompt"] == "NATF2.pt"
def test_personaplex_rejects_minicpmo_only_opt_out_flag() -> None:
def test_personaplex_rejects_native_duplex_opt_out_flag() -> None:
# PersonaPlex is always model-native; the per-session opt-in knob is
# server-owned there, under both the canonical name and its deprecated
# model-prefixed alias.
adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
with pytest.raises(ServingRuntimeConfigError, match="native_duplex"):
adapter.validate_client_extra_body({"native_duplex": False})
with pytest.raises(ServingRuntimeConfigError, match="minicpmo45_native_duplex"):
adapter.validate_client_extra_body({"minicpmo45_native_duplex": False})
tests/e2e/features/fullduplex/personaplex/test_streaming_code2wav.py → tests/model_executor/models/personaplex/test_streaming_code2wav.py
+1 -1
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from types import SimpleNamespace
tests/worker/test_native_duplex_hooks.py
+32 -32
View File
@@ -18,7 +18,7 @@ def _minicpmo_duplex_policy_case(
state: SimpleNamespace,
payload: dict[str, object],
):
from vllm_omni.experimental.fullduplex.model_executor import DuplexSamplingRow
from vllm_omni.model_executor.duplex_sampling import DuplexSamplingRow
from vllm_omni.model_executor.models.minicpmo_4_5.minicpmo_4_5_omni import (
MiniCPMO45OmniForConditionalGeneration,
)
@@ -45,7 +45,7 @@ def _minicpmo_duplex_policy_case(
def test_minicpmo_model_hook_owns_duplex_sampling_rows_and_force_listen():
from vllm_omni.experimental.fullduplex.model_executor import DuplexSamplingRow
from vllm_omni.model_executor.duplex_sampling import DuplexSamplingRow
from vllm_omni.model_executor.models.minicpmo_4_5.minicpmo_4_5_omni import (
MiniCPMO45OmniForConditionalGeneration,
)
@@ -329,7 +329,7 @@ def test_minicpmo_model_hook_ignores_serving_new_user_turn_marker():
def test_generic_ar_runner_builds_typed_duplex_sampling_rows():
from vllm_omni.experimental.fullduplex.model_executor import DuplexSamplingHelper
from vllm_omni.model_executor.duplex_sampling import DuplexSamplingHelper
from vllm_omni.worker.gpu_ar_model_runner import GPUARModelRunner
runner = GPUARModelRunner.__new__(GPUARModelRunner)
@@ -532,7 +532,7 @@ def test_minicpmo_stage0_routes_duplex_metadata_per_batched_request():
def test_minicpmo_stage0_rejects_invalid_resolved_ref_audio():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -546,7 +546,7 @@ def test_minicpmo_stage0_rejects_invalid_resolved_ref_audio():
def test_minicpmo_stage0_special_token_ids_are_tokenizer_derived():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -583,7 +583,7 @@ def test_minicpmo_stage0_special_token_ids_are_tokenizer_derived():
def test_minicpmo_stage0_rejects_unknown_special_token_fallbacks():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -659,7 +659,7 @@ def _transformers_processor_stub(auto_processor):
def test_minicpmo_stage0_loads_processor_from_hf_id(monkeypatch):
import sys
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -701,7 +701,7 @@ def test_minicpmo_stage0_loads_processor_from_hf_id(monkeypatch):
def test_minicpmo_stage0_processor_load_preserves_original_exception(monkeypatch):
import sys
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -728,7 +728,7 @@ def test_minicpmo_stage0_processor_load_preserves_original_exception(monkeypatch
def test_minicpmo_stage0_processor_requires_tokenizer(monkeypatch):
import sys
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -751,7 +751,7 @@ def test_minicpmo_stage0_processor_requires_tokenizer(monkeypatch):
def test_minicpmo_stage0_loaded_processor_validates_special_tokens(monkeypatch):
import sys
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -793,7 +793,7 @@ def test_minicpmo_stage0_loaded_processor_validates_special_tokens(monkeypatch):
def test_minicpmo_stage0_data_plane_prefill_matches_official_unit_format():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -855,7 +855,7 @@ def test_minicpmo_stage0_data_plane_prefill_matches_official_unit_format():
def _stage0_vision_runtime():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -912,7 +912,7 @@ def test_minicpmo_stage0_puts_every_frame_of_an_append_in_one_unit():
that closed a single unit left a frame over, failed the prefill, and still
advanced the session state.
"""
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
_MiniCPMO45Stage0SessionState,
)
@@ -936,7 +936,7 @@ def test_minicpmo_stage0_wraps_hd_slices_after_the_source_image():
"""Official stacked pair is max_slice_nums=[2, 1]: source + slices, then audio."""
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
_MiniCPMO45Stage0SessionState,
)
@@ -985,7 +985,7 @@ def test_minicpmo_stage0_reports_frames_that_close_no_unit_instead_of_dropping_t
Frames are not carried across appends, so one arriving early used to vanish
under a generic "audio not enough" buffering result.
"""
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
_MiniCPMO45Stage0SessionState,
)
@@ -1010,7 +1010,7 @@ def test_minicpmo_stage0_reports_frames_that_close_no_unit_instead_of_dropping_t
def test_minicpmo_stage0_speech_append_sets_pending_context_once():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1118,7 +1118,7 @@ def test_minicpmo_stage0_speech_append_sets_pending_context_once():
def test_minicpmo_stage0_failed_append_does_not_set_pending_speech_context():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1173,7 +1173,7 @@ def test_minicpmo_stage0_failed_append_does_not_set_pending_speech_context():
def test_minicpmo_stage0_streaming_processor_is_isolated_per_session():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1220,7 +1220,7 @@ def test_minicpmo_stage0_streaming_processor_is_isolated_per_session():
def test_minicpmo_stage0_data_plane_next_append_reinjects_previous_listen():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1283,7 +1283,7 @@ def test_minicpmo_stage0_data_plane_next_append_reinjects_previous_listen():
def test_minicpmo_stage0_data_plane_turn_eos_closes_previous_unit():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1346,7 +1346,7 @@ def test_minicpmo_stage0_data_plane_turn_eos_closes_previous_unit():
def test_minicpmo_stage0_data_plane_model_owned_turn_boundary_preserves_audio_cache():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1411,7 +1411,7 @@ def test_minicpmo_stage0_data_plane_model_owned_turn_boundary_preserves_audio_ca
def test_minicpmo_stage0_data_plane_final_first_chunk_does_not_add_silence_unit():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1485,7 +1485,7 @@ def test_minicpmo_stage0_data_plane_final_first_chunk_does_not_add_silence_unit(
def test_minicpmo_stage0_final_does_not_promote_first_chunk_alignment_tail_to_unit():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1567,7 +1567,7 @@ def test_minicpmo_stage0_final_does_not_promote_first_chunk_alignment_tail_to_un
def test_minicpmo_stage0_runtime_uses_loaded_vllm_embed_tokens_when_get_input_embeddings_is_broken():
import torch
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
)
@@ -1603,7 +1603,7 @@ def test_minicpmo_stage0_runtime_uses_loaded_vllm_embed_tokens_when_get_input_em
def test_minicpmo_remote_config_patch_handles_nested_and_dict_configs():
from vllm_omni.experimental.fullduplex.minicpmo45.compat import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.compat import (
patch_minicpmo_remote_config,
)
@@ -1625,7 +1625,7 @@ def test_minicpmo_remote_config_patch_handles_nested_and_dict_configs():
def test_minicpmo_stage0_short_audio_buffers_without_context_mutation():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
@@ -1701,7 +1701,7 @@ def test_minicpmo_stage0_native_sampler_penalizes_repeated_text_token():
def test_minicpmo_stage0_native_sampler_penalizes_text_from_prior_chunk():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
_MiniCPMO45Stage0SessionState,
)
from vllm_omni.model_executor.models.minicpmo_4_5.minicpmo_4_5_omni import (
@@ -1761,7 +1761,7 @@ def test_minicpmo_stage0_native_sampler_penalizes_text_from_prior_chunk():
def test_minicpmo_stage0_native_sampler_matches_official_negative_logit_penalty():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
_MiniCPMO45Stage0SessionState,
)
from vllm_omni.model_executor.models.minicpmo_4_5.minicpmo_4_5_omni import (
@@ -1820,7 +1820,7 @@ def test_minicpmo_stage0_native_sampler_matches_official_negative_logit_penalty(
def test_minicpmo_stage0_records_bounded_model_policy_history():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
_MiniCPMO45Stage0SessionState,
)
from vllm_omni.model_executor.models.minicpmo_4_5.minicpmo_4_5_omni import (
@@ -2248,7 +2248,7 @@ def test_minicpmo_stage0_native_sampler_preserves_model_chunk_eos():
def test_minicpmo_stage0_native_sampler_keeps_hard_chunk_cap():
from vllm_omni.experimental.fullduplex.minicpmo45.policy import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.policy import (
MiniCPMO45DuplexPolicy,
)
from vllm_omni.model_executor.models.minicpmo_4_5.minicpmo_4_5_omni import (
@@ -2360,7 +2360,7 @@ def test_minicpmo_stage0_native_sampler_cuts_before_request_length_cap():
def test_minicpmo_stage0_native_sampler_cuts_on_decoded_text_length():
from vllm_omni.experimental.fullduplex.minicpmo45.policy import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.policy import (
MiniCPMO45DuplexPolicy,
)
from vllm_omni.model_executor.models.minicpmo_4_5.minicpmo_4_5_omni import (
@@ -2598,7 +2598,7 @@ def test_minicpmo_stage0_native_sampler_uses_runner_duplex_rows():
def test_minicpmo_stage0_session_context_includes_resolved_ref_audio():
from vllm_omni.experimental.fullduplex.minicpmo45.stage0 import (
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.stage0 import (
MiniCPMO45Stage0DuplexRuntime,
_MiniCPMO45Stage0SessionState,
)
tools/pre_commit/check_forbidden_imports.py
+12 -14
View File
@@ -79,19 +79,19 @@ CHECK_IMPORTS = {
"vllm_omni/entrypoints/openai/video_stream_base.py",
"vllm_omni/entrypoints/openai/video_stream_session.py",
"vllm_omni/experimental/fullduplex/client.py",
"vllm_omni/experimental/fullduplex/engine/contracts.py",
"vllm_omni/engine/duplex/contracts.py",
"vllm_omni/experimental/fullduplex/joyvl/memory/memory.py",
"vllm_omni/experimental/fullduplex/minicpmo45/adapter.py",
"vllm_omni/experimental/fullduplex/minicpmo45/input.py",
"vllm_omni/experimental/fullduplex/minicpmo45/runtime.py",
"vllm_omni/experimental/fullduplex/minicpmo45/stage0.py",
"vllm_omni/experimental/fullduplex/openai/audio.py",
"vllm_omni/experimental/fullduplex/openai/realtime_input.py",
"vllm_omni/experimental/fullduplex/openai/runtime_bridge.py",
"vllm_omni/experimental/fullduplex/openai/serving.py",
"vllm_omni/experimental/fullduplex/personaplex/input.py",
"vllm_omni/experimental/fullduplex/personaplex/runtime_extension.py",
"vllm_omni/experimental/fullduplex/personaplex/stage0.py",
"vllm_omni/model_executor/models/minicpmo_4_5/duplex/adapter.py",
"vllm_omni/model_executor/models/minicpmo_4_5/duplex/input.py",
"vllm_omni/model_executor/models/minicpmo_4_5/duplex/runtime.py",
"vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py",
"vllm_omni/entrypoints/duplex/audio.py",
"vllm_omni/entrypoints/duplex/realtime_input.py",
"vllm_omni/entrypoints/duplex/runtime_bridge.py",
"vllm_omni/entrypoints/duplex/serving.py",
"vllm_omni/model_executor/models/personaplex/duplex/input.py",
"vllm_omni/model_executor/models/personaplex/duplex/runtime_extension.py",
"vllm_omni/model_executor/models/personaplex/duplex/stage0.py",
"vllm_omni/model_executor/models/indextts2/tokenizer_v2_5.py",
"vllm_omni/model_executor/models/qwen3_tts/prompt_embeds_builder.py",
"vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py",
@@ -197,8 +197,6 @@ CHECK_IMPORTS = {
"vllm_omni/entrypoints/openai/api_server.py",
"vllm_omni/entrypoints/openai/serving_speech.py",
"vllm_omni/entrypoints/openai/tts_adapters/cosyvoice3.py",
"vllm_omni/experimental/fullduplex/personaplex/runtime.py",
"vllm_omni/experimental/fullduplex/personaplex/serving/server.py",
"vllm_omni/model_executor/model_loader/weight_utils.py",
"vllm_omni/model_executor/models/audex/checkpoint.py",
"vllm_omni/model_executor/models/bagel/bagel.py",
vllm_omni/benchmarks/omniinteract.py
+224 -48
View File
@@ -22,7 +22,7 @@ from collections import Counter
from collections.abc import Callable, Iterator, Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, cast
from typing import TYPE_CHECKING, Any, cast
from urllib.parse import urljoin, urlsplit
import pybase64 as base64
@@ -33,24 +33,27 @@ from vllm_omni.benchmarks.data_modules.omniinteract_dataset import (
OmniInteractPreparedInput,
case_manifest,
)
from vllm_omni.experimental.fullduplex.client import (
PCM16_BYTES_PER_SAMPLE,
PCM16_SAMPLE_RATE,
RealtimeDuplexClient,
RealtimeEventCollector,
build_realtime_url,
chunk_period_ms,
has_residual_model_unit,
reference_audio_data_url,
summarize_session_request_metrics,
write_pcm16_wav,
)
# The duplex client library (vllm_omni.clients) is imported lazily inside the
# functions that need it: this module reaches the CLI import graph through the
# benchmark patch package (`vllm-omni serve` included), and the client package
# must stay out of that graph
# (tests/engine/test_duplex_import_boundary.py enforces the boundary).
if TYPE_CHECKING:
from vllm_omni.clients.duplex import EventCollector, WebSocketTransport
# The duplex input wire format (16 kHz mono PCM16); mirrors
# vllm_omni.clients.duplex without importing it at module scope.
PCM16_SAMPLE_RATE = 16_000
PCM16_BYTES_PER_SAMPLE = 2
OUTPUT_SAMPLE_RATE = 24_000
SUCCESS_ARTIFACTS = (".done", "output.wav", "wav_transcript.json", "events.json", "result.json")
BATCH_ARTIFACTS = ("batch_summary.json", "official_eval_manifest.jsonl")
ARTIFACT_LOCK_FILE = ".omniinteract.lock"
_INPUT_CHUNK_MS = 200
# Minimum new played audio between two incremental playback.ack sends for one
# response; completion acks always flush regardless of the step.
_ACK_STEP_MS = 200
VIDEO_FPS = 1.0
_COMPLETION_SETTLE_S = 2.0
logger = logging.getLogger(__name__)
@@ -215,12 +218,13 @@ class _Playback:
self.warnings: list[str] = []
self._total_samples: dict[str, int] = {}
self._response_end_s: dict[str, float] = {}
self._acked_ms: dict[str, int] = {}
def _warn_once(self, warning: str) -> None:
if warning not in self.warnings:
self.warnings.append(warning)
def ingest(self, events: RealtimeEventCollector) -> None:
def ingest(self, events: EventCollector) -> None:
while self.cursor < len(events.events):
index, self.cursor = self.cursor, self.cursor + 1
event = events.events[index]
@@ -260,21 +264,52 @@ class _Playback:
self._total_samples[response_id] = self._total_samples.get(response_id, 0) + samples
self._response_end_s[response_id] = segment.end_s
async def acknowledge(self, client: RealtimeDuplexClient, now: float | None = None) -> None:
async def acknowledge(self, client: _RealtimeSession, now: float | None = None) -> None:
"""Ack playback progress incrementally along the serialized clock.
A real listener reports cumulative played audio while it plays, not
once after the fact. The first ack for a response goes out as soon as
its audio starts arriving, whatever the played amount: it checkpoints
the response's history position on the server, so a user input
committed later updates that slot in place instead of displacing it.
Later acks follow ``_ACK_STEP_MS``, with the exact total at drain.
"""
events = client.events
self.ingest(events)
now = time.monotonic() if now is None else now
for response_id in self.completed - self.completion_acked:
total_samples = self._total_samples.get(response_id, 0)
if not total_samples or now < self._response_end_s[response_id]:
for response_id, total_samples in self._total_samples.items():
if response_id in self.completion_acked:
continue
acked_ms = self._acked_ms.get(response_id)
if response_id in self.completed and now >= self._response_end_s[response_id]:
played_ms = total_samples * 1000 // OUTPUT_SAMPLE_RATE
if acked_ms is None or played_ms > acked_ms:
await client.send_playback_ack(response_id, played_ms)
self._acked_ms[response_id] = played_ms
self.completion_acked.add(response_id)
continue
played_ms = self._played_ms(response_id, now)
if acked_ms is not None and played_ms - acked_ms < _ACK_STEP_MS:
continue
played_ms = total_samples * 1000 // OUTPUT_SAMPLE_RATE
await client.send_playback_ack(response_id, played_ms)
self.completion_acked.add(response_id)
self._acked_ms[response_id] = played_ms
def _played_ms(self, response_id: str, now: float) -> int:
"""Cumulative audio of one response played by ``now`` on the serial clock."""
played_samples = 0
for segment in self.segments:
if segment.response_id != response_id or now <= segment.start_s:
continue
samples = len(segment.pcm16) // PCM16_BYTES_PER_SAMPLE
if now >= segment.end_s:
played_samples += samples
else:
played_samples += min(samples, int((now - segment.start_s) * OUTPUT_SAMPLE_RATE + 0.5))
return played_samples * 1000 // OUTPUT_SAMPLE_RATE
async def stream_inputs(
client: RealtimeDuplexClient,
client: _RealtimeSession,
pcm: bytes,
frames: Sequence[str | None],
playback: _Playback,
@@ -319,7 +354,7 @@ def _response_status(event: dict[str, object]) -> str | None:
def response_ledger(
collector: RealtimeEventCollector,
collector: EventCollector,
*,
end: int | None = None,
) -> tuple[set[str], set[str]]:
@@ -357,7 +392,7 @@ def _is_model_listen_decision(event: dict[str, object]) -> bool:
def _has_post_commit_decision(
collector: RealtimeEventCollector,
collector: EventCollector,
events: list[dict[str, object]],
*,
prior_response_ids: set[str],
@@ -377,15 +412,34 @@ def _has_post_commit_decision(
return False
# An ack computed just before a commit event was delivered can still lose the
# race server-side; the response it acks was already superseded, so the
# refusal is harmless — record it instead of failing the case.
_TOLERATED_ERROR_CODES = frozenset({"playback_ack_too_late"})
def _error_code(event: dict[str, object]) -> str | None:
error = event.get("error")
code = error.get("code") if isinstance(error, dict) else event.get("code")
return code if isinstance(code, str) else None
def _raise_if_session_terminated(
collector: RealtimeEventCollector,
collector: EventCollector,
from_index: int,
*,
explicit_close_from: int | None = None,
warnings: list[str] | None = None,
) -> None:
errors = collector.errors()
if errors:
raise RuntimeError(str(errors[-1]))
fatal = [event for event in errors if _error_code(event) not in _TOLERATED_ERROR_CODES]
if fatal:
raise RuntimeError(str(fatal[-1]))
if warnings is not None:
for event in errors:
warning = f"tolerated in-flight server rejection: {_error_code(event)}"
if warning not in warnings:
warnings.append(warning)
for index, event in enumerate(collector.events[from_index:], start=from_index):
event_type = event.get("type")
if event_type not in {"session.expired", "session.closed"}:
@@ -416,6 +470,8 @@ def _ensure_final_commit_tail(pcm: bytes, events: list[dict[str, object]]) -> by
that correlation; the server pads the missing sample back to a full unit.
"""
from vllm_omni.clients.duplex import chunk_period_ms, has_residual_model_unit
period_ms = chunk_period_ms(events)
if len(pcm) >= PCM16_BYTES_PER_SAMPLE and not has_residual_model_unit(pcm, chunk_period_ms=period_ms):
return pcm[:-PCM16_BYTES_PER_SAMPLE]
@@ -423,7 +479,7 @@ def _ensure_final_commit_tail(pcm: bytes, events: list[dict[str, object]]) -> by
async def wait_for_session_completion(
client: RealtimeDuplexClient,
client: _RealtimeSession,
playback: _Playback,
*,
commit_from: int,
@@ -438,7 +494,11 @@ async def wait_for_session_completion(
stable_since = time.monotonic()
while time.monotonic() < deadline:
client.raise_if_reader_stopped()
_raise_if_session_terminated(client.events, commit_from if session_from is None else session_from)
_raise_if_session_terminated(
client.events,
commit_from if session_from is None else session_from,
warnings=playback.warnings,
)
await playback.acknowledge(client)
if len(client.events.events) != last_event_count:
last_event_count = len(client.events.events)
@@ -522,6 +582,8 @@ def _atomic_write_json(path: Path, value: object) -> None:
def _atomic_write_wav(path: Path, pcm: bytes | bytearray, rate: int) -> None:
from vllm_omni.clients.duplex import write_pcm16_wav
_atomic_replace(
path,
lambda temporary: write_pcm16_wav(
@@ -578,7 +640,7 @@ def _publish_success_artifacts(
def _collect_output(
collector: RealtimeEventCollector,
collector: EventCollector,
playback: _Playback,
*,
stream_start: float,
@@ -677,7 +739,7 @@ def _collect_output(
def prepare_success_artifacts(
root: Path,
case: OmniInteractCase,
collector: RealtimeEventCollector,
collector: EventCollector,
*,
playback: _Playback | None = None,
stream_start: float,
@@ -770,6 +832,8 @@ def _websocket_url(config: OmniInteractBenchmarkConfig, session_id: str) -> str:
if urlsplit(config.endpoint).scheme
else urljoin(config.base_url.rstrip("/") + "/", config.endpoint.lstrip("/"))
)
from vllm_omni.clients.duplex import build_realtime_url
return build_realtime_url(
endpoint,
config.model,
@@ -779,9 +843,121 @@ def _websocket_url(config: OmniInteractBenchmarkConfig, session_id: str) -> str:
)
class _RealtimeSession:
"""The public :class:`DuplexClient` behind the benchmark's session surface.
The case runner works in event-cursor style over one growing collector,
so this wrapper pairs a ``DuplexClient`` with an :class:`EventCollector`
fed by a background consumer task and exposes the few calls the runner
needs: raw sends, the final commit, cumulative playback acks, reader
liveness checks, and a confirmed close.
"""
_MAX_FRAME_BYTES = 64 * 1024 * 1024
def __init__(self, config: OmniInteractBenchmarkConfig, session_id: str, reference_audio: str) -> None:
from vllm_omni.clients.duplex import DuplexClient, EventCollector, SessionConfig
self.session_config = SessionConfig(
ref_audio=reference_audio,
overlap_policy="listen_only",
playback_commit_policy="ack_only",
idle_timeout_s=float(config.timeout_s),
extra_body={
"native_duplex": True,
"force_listen_count": 0,
**(config.extra_body or {}),
},
)
self.url = _websocket_url(config, session_id)
headers = dict(config.extra_headers or {})
async def connect(url: str) -> WebSocketTransport:
import websockets
return await websockets.connect(
url,
max_size=self._MAX_FRAME_BYTES,
additional_headers=headers or None,
)
self.events = EventCollector()
self._client = DuplexClient(
self.url,
model=config.model,
config=self.session_config,
session_id=session_id,
reconnect=None,
heartbeat_interval_s=None,
handshake_timeout_s=min(config.timeout_s, 20.0),
connect=connect,
)
self._consume_task: asyncio.Task[None] | None = None
async def __aenter__(self) -> _RealtimeSession:
await self._client.__aenter__()
self._consume_task = asyncio.create_task(self.events.consume(self._client))
# The handshake consumed session.created before the collector
# subscribed; seed it so capability lookups and cursors still work.
self.events.add({"type": "session.created", "session": dict(self._client.session_info)})
return self
async def __aexit__(self, exc_type, exc, traceback) -> None:
try:
await self._client.__aexit__(exc_type, exc, traceback)
finally:
if self._consume_task is not None:
self._consume_task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await self._consume_task
async def send(self, event: dict[str, object]) -> None:
await self._client.send(event)
async def commit(self) -> None:
await self._client.commit()
async def send_playback_ack(self, response_id: str, played_ms: int) -> None:
await self._client.ack_playback(
played_ms,
response_id=response_id,
item_id=f"item_{response_id}",
)
def raise_if_reader_stopped(self) -> None:
"""Fail a caller waiting on events after the event consumer exits."""
task = self._consume_task
if task is None or not task.done():
return
if task.cancelled():
raise ConnectionError("Realtime event consumer was cancelled")
error = task.exception()
if error is not None:
raise ConnectionError("Realtime session failed") from error
raise ConnectionError("Realtime session closed before the requested event arrived")
async def close_session(self, *, timeout_s: float = 20.0) -> None:
await self._client.close(timeout_s=timeout_s)
if self._consume_task is not None:
# Let the collector drain session.closed before callers check it.
with contextlib.suppress(asyncio.TimeoutError, asyncio.CancelledError):
await asyncio.wait_for(asyncio.shield(self._consume_task), timeout=5.0)
def _audio_rtf_from_raw_metric(raw_metric: dict[str, object]) -> float | None:
"""Derive the audio RTF from a client-reported raw request metric."""
from vllm_omni.metrics.definitions import compute_audio_rtf
generation_ms = raw_metric.get("audio_generation_ms")
duration_ms = raw_metric.get("audio_duration_ms")
if not isinstance(generation_ms, int | float) or not isinstance(duration_ms, int | float) or duration_ms <= 0:
return None
return round(compute_audio_rtf(float(generation_ms) / 1000.0, float(duration_ms) / 1000.0), 6)
def _populate_response_metrics(
result: OmniInteractCaseResult,
collector: RealtimeEventCollector,
collector: EventCollector,
*,
stream_start: float,
) -> None:
@@ -807,6 +983,10 @@ def _populate_response_metrics(
"response_id": response_id,
**(raw_metric if isinstance(raw_metric, dict) else {}),
}
if isinstance(raw_metric, dict):
# The client reports raw data only; the RTF is derived here with
# the canonical server-side metric definition.
metric["rtf"] = _audio_rtf_from_raw_metric(raw_metric)
if isinstance(stage0, dict):
metric["stage0_tokens"] = dict(stage0)
output_tokens += int(stage0.get("output_token_count") or 0)
@@ -814,6 +994,8 @@ def _populate_response_metrics(
request_metrics.append(metric)
result.output_tokens = output_tokens
result.duplex_request_metrics = request_metrics
from vllm_omni.clients.duplex import summarize_session_request_metrics
result.duplex_session_metrics = summarize_session_request_metrics(
request_metrics,
session_id=result.session_id,
@@ -847,6 +1029,8 @@ async def run_omniinteract_case(
started_at = time.monotonic()
try:
if prepared_input is None:
from vllm_omni.clients.duplex import reference_audio_data_url
reference_audio = reference_audio_data_url(config.ref_audio)
duration, pcm, frames = await asyncio.to_thread(
prepare_media,
@@ -862,18 +1046,8 @@ async def run_omniinteract_case(
frames = prepared_input.video_frames
if not any(frames):
raise ValueError(f"No video frames were decoded from {case.video_path}")
async with RealtimeDuplexClient(
_websocket_url(config, session_id), additional_headers=config.extra_headers
) as client:
session_from = len(client.events.events)
await client.configure(
config.model,
ref_audio=reference_audio,
session_id=session_id,
extra_body=config.extra_body,
idle_timeout_s=config.timeout_s,
timeout_s=min(config.timeout_s, 20.0),
)
async with _RealtimeSession(config, session_id, reference_audio) as client:
session_from = 0 # the collector holds only this session's events
pcm = _ensure_final_commit_tail(pcm, client.events.events)
playback = _Playback()
stream_start = time.monotonic()
@@ -897,17 +1071,19 @@ async def run_omniinteract_case(
settle_s=_COMPLETION_SETTLE_S,
)
await playback.acknowledge(client, playback.end_s)
_raise_if_session_terminated(client.events, session_from)
_raise_if_session_terminated(client.events, session_from, warnings=playback.warnings)
close_from = len(client.events.events)
await client.close_session(timeout_s=min(config.timeout_s, 20.0))
except Exception:
with contextlib.suppress(Exception):
await client.close_session(timeout_s=min(config.timeout_s, 20.0))
raise
errors = client.events.errors()
if errors:
raise RuntimeError(str(errors[-1]))
_raise_if_session_terminated(client.events, session_from, explicit_close_from=close_from)
_raise_if_session_terminated(
client.events,
session_from,
explicit_close_from=close_from,
warnings=playback.warnings,
)
result.latency_s = time.monotonic() - started_at
result.input_audio_chunks, result.input_video_frames = chunks, frame_count
result.pacing_mean_lag_s, result.pacing_max_lag_s = mean_lag, max_lag
vllm_omni/benchmarks/patch/patch.py
+102 -10
View File
@@ -18,7 +18,7 @@ from collections.abc import Iterable
from dataclasses import dataclass, replace
from datetime import datetime
from pathlib import Path
from typing import Any, Literal
from typing import TYPE_CHECKING, Any, Literal
import aiohttp
import numpy as np
@@ -70,6 +70,7 @@ from vllm_omni.benchmarks.omniinteract import (
VIDEO_FPS,
OmniInteractBenchmarkConfig,
OmniInteractCaseResult,
_audio_rtf_from_raw_metric,
clear_batch_artifacts,
clear_case_artifacts,
prepare_media,
@@ -83,16 +84,18 @@ from vllm_omni.benchmarks.omniinteract import (
from vllm_omni.benchmarks.omniinteract import (
write_batch_artifacts as write_omniinteract_batch_artifacts,
)
from vllm_omni.experimental.fullduplex.client import (
RealtimeDuplexClient,
build_realtime_url,
reference_audio_data_url,
summarize_session_request_metrics,
wait_for,
)
from vllm_omni.metrics import definitions as defs
from vllm_omni.metrics.utils import coerce_positive_int_scalar
if TYPE_CHECKING:
from vllm_omni.clients.duplex import DuplexClient
# The duplex client library (vllm_omni.clients) is imported lazily inside the
# functions that drive Realtime sessions: this module is imported for its
# patching side effects by the CLI package (`vllm-omni serve` included), and
# the client package must stay out of that import graph
# (tests/engine/test_duplex_import_boundary.py enforces the boundary).
logger = init_logger(__name__)
_AUDIO_CONTINUITY_THRESHOLD_ENV = "VLLM_OMNI_BENCH_AUDIO_CONTINUITY_THRESHOLD_S"
@@ -423,6 +426,8 @@ def get_samples(args, tokenizer):
)
if not requests:
raise ValueError("No OmniInteract sessions were selected")
from vllm_omni.clients.duplex import reference_audio_data_url
encoded_ref_audio = reference_audio_data_url(options.ref_audio)
assert encoded_ref_audio is not None
for request in requests:
@@ -1531,6 +1536,8 @@ async def async_request_openai_audio_speech(
def _realtime_websocket_url(api_url: str) -> str:
from vllm_omni.clients.duplex import build_realtime_url
return build_realtime_url(api_url, None, native_duplex=None)
@@ -1685,11 +1692,93 @@ async def _async_request_omniinteract(
return output
class _RealtimeTTSProbe:
"""Explicit-session Realtime TTS driver over the public duplex client.
Owns an :class:`EventCollector` (``.events``) so turn metrics can be read
with controlled timestamps; the session handshake happens in
:meth:`configure` because the benchmark decides the session shape at
request time.
"""
def __init__(self, url: str) -> None:
from vllm_omni.clients.duplex import EventCollector
self._url = url
self.events: EventCollector = EventCollector()
self._client: DuplexClient | None = None
self._consume_task: asyncio.Task[None] | None = None
async def __aenter__(self) -> "_RealtimeTTSProbe":
return self
async def __aexit__(self, exc_type, exc, tb) -> None:
if self._client is not None:
await self._client.__aexit__(exc_type, exc, tb)
if self._consume_task is not None:
try:
await asyncio.wait_for(self._consume_task, timeout=5.0)
except (asyncio.TimeoutError, asyncio.CancelledError):
self._consume_task.cancel()
async def configure(
self,
model: str,
*,
output_audio_format: str = "pcm16",
instructions: str | None = None,
native_duplex: bool = False,
auto_response: bool = False,
extra_body: dict[str, object] | None = None,
session_id: str | None = None,
timeout_s: float = 120.0,
) -> None:
from vllm_omni.clients.duplex import AudioFormat, DuplexClient, SessionConfig
session_extra_body: dict[str, object] = dict(extra_body or {})
session_extra_body["native_duplex"] = bool(native_duplex)
config = SessionConfig(
output_audio=AudioFormat(output_audio_format, 24_000),
instructions=instructions,
auto_response=auto_response,
overlap_policy="listen_only",
playback_commit_policy="ack_only",
extra_body=session_extra_body,
)
self._client = DuplexClient(
self._url,
model=model,
config=config,
session_id=session_id,
reconnect=None,
heartbeat_interval_s=None,
handshake_timeout_s=timeout_s,
)
await self._client.__aenter__()
self._consume_task = asyncio.create_task(self.events.consume(self._client))
async def send(self, event: dict[str, object]) -> None:
assert self._client is not None
await self._client.send(event)
async def acknowledge_playback(self) -> None:
from vllm_omni.clients.duplex import acknowledge_collected_playback
assert self._client is not None
await acknowledge_collected_playback(self._client, self.events)
async def close_session(self, *, timeout_s: float = 20.0) -> None:
assert self._client is not None
await self._client.close(timeout_s=timeout_s)
async def async_request_openai_realtime_duplex(
request_func_input: RequestFuncInput,
session: aiohttp.ClientSession,
pbar: tqdm | None = None,
) -> MixRequestFuncOutput:
from vllm_omni.clients.duplex import summarize_session_request_metrics, wait_for_condition
del session
if getattr(request_func_input, "omniinteract_case", None) is not None:
return await _async_request_omniinteract(request_func_input, pbar=pbar)
@@ -1715,7 +1804,7 @@ async def async_request_openai_realtime_duplex(
turn_prompts = [("", request_func_input.prompt)]
session_id = f"seed-tts-{request_func_input.request_id or uuid.uuid4().hex}"
try:
async with RealtimeDuplexClient(_realtime_websocket_url(request_func_input.api_url)) as client:
async with _RealtimeTTSProbe(_realtime_websocket_url(request_func_input.api_url)) as client:
await client.configure(
request_func_input.model_name or request_func_input.model,
output_audio_format="pcm16",
@@ -1755,7 +1844,7 @@ async def async_request_openai_realtime_duplex(
}
)
await client.send({"type": "response.create"})
await wait_for(
await wait_for_condition(
lambda: (
client.events.count("response.done") > done_before
or len(client.events.errors()) > errors_before
@@ -1794,6 +1883,9 @@ async def async_request_openai_realtime_duplex(
"utterance_id": utterance_id or None,
"response_id": response_id,
**request_metrics,
# The client reports raw data only; the RTF is derived
# here with the canonical server-side definition.
"rtf": _audio_rtf_from_raw_metric(request_metrics),
}
)
response_audio = client.events.audio_bytes(response_id)
vllm_omni/clients/__init__.py
+9
View File
@@ -0,0 +1,9 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Client-side libraries for vLLM-Omni serving APIs.
Modules under this package speak to a running vLLM-Omni server over the
network. They are for applications and tests; server runtime code
(``vllm_omni.engine``, ``vllm_omni.entrypoints``, ``vllm_omni.model_executor``)
must never import from here.
"""
vllm_omni/clients/duplex.py
+1688
View File
File diff suppressed because it is too large. Load diff
vllm_omni/clients/minicpmo_4_5.py
+24
View File
@@ -0,0 +1,24 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Client-side duplex session preset for MiniCPM-o 4.5 native duplex."""
from __future__ import annotations
from dataclasses import replace
from vllm_omni.clients.duplex import SessionConfig
__all__ = ["create_duplex_session_config"]
def create_duplex_session_config(*, ref_audio: str | None = None, **overrides: object) -> SessionConfig:
"""Session preset matching the MiniCPM-o 4.5 native duplex deployment."""
extra_body: dict[str, object] = {"native_duplex": True, "force_listen_count": 0}
extra_body.update(overrides.pop("extra_body", {}))
config = SessionConfig(
ref_audio=ref_audio,
overlap_policy="listen_only",
playback_commit_policy="ack_only",
extra_body=extra_body,
)
return replace(config, **overrides) if overrides else config
vllm_omni/clients/personaplex.py
+22
View File
@@ -0,0 +1,22 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Client-side duplex session preset for PersonaPlex duplex serving."""
from __future__ import annotations
from dataclasses import replace
from vllm_omni.clients.duplex import AudioFormat, SessionConfig
__all__ = ["create_duplex_session_config"]
def create_duplex_session_config(*, voice: str = "NATF2.pt", persona: str = "", **overrides: object) -> SessionConfig:
"""Session preset matching the PersonaPlex duplex deployment (24 kHz float32 in)."""
config = SessionConfig(
input_audio=AudioFormat("pcm_f32le", 24_000),
output_audio=AudioFormat("pcm16", 24_000),
voice=voice,
instructions=persona,
)
return replace(config, **overrides) if overrides else config
vllm_omni/engine/async_omni_engine.py
+8 -13
View File
@@ -22,7 +22,7 @@ import uuid
import weakref
from collections.abc import Mapping, Sequence
from dataclasses import asdict
from typing import TYPE_CHECKING, Any, Literal, cast
from typing import Any, Literal, cast
import janus
import torch
@@ -58,6 +58,13 @@ from vllm_omni.engine.async_engine_utils import (
upgrade_to_omni_request,
weak_shutdown_async_omni_engine,
)
from vllm_omni.engine.duplex.control_client import DuplexControlClient
from vllm_omni.engine.duplex.lease import DuplexLeaseActivity
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.duplex.runtime import (
load_duplex_runtime_extension,
validate_duplex_runtime_extension,
)
from vllm_omni.engine.messages import (
AbortRequestMessage,
AbortResultMessage,
@@ -90,11 +97,6 @@ from vllm_omni.metrics.prometheus import OmniRequestCounter
logger = init_logger(__name__)
if TYPE_CHECKING:
from vllm_omni.experimental.fullduplex.engine.duplex_control_client import DuplexControlClient
from vllm_omni.experimental.fullduplex.engine.lease import DuplexLeaseActivity
from vllm_omni.experimental.fullduplex.engine.messages import DuplexFence
_STARTUP_POLL_INTERVAL_S = 1.0
_REQUEST_QUEUE_MAXSIZE = 256
@@ -389,11 +391,6 @@ class AsyncOmniEngine:
membership_controller = self._runtime.create_membership_controller()
duplex_runtime_extension = None
if self._duplex_control_enabled:
from vllm_omni.experimental.fullduplex.engine.duplex_runtime import (
load_duplex_runtime_extension,
validate_duplex_runtime_extension,
)
duplex_runtime_extension = load_duplex_runtime_extension(
getattr(self, "_duplex_runtime_extension_path", None)
)
@@ -1812,8 +1809,6 @@ class AsyncOmniEngine:
)
def _get_duplex_control_client(self) -> DuplexControlClient:
from vllm_omni.experimental.fullduplex.engine.duplex_control_client import DuplexControlClient
client = getattr(self, "_duplex_control_client", None)
if client is None:
transport = getattr(self, "_correlated_rpc_client", None)
vllm_omni/engine/duplex/__init__.py
+4
View File
@@ -0,0 +1,4 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Engine-side helpers for the full-duplex runtime."""
Loaded 100 of 179 files, more files were not shown because too many files have changed in this diff. Show more