[Core][Model] Fit PersonaPlex into the Unified Full-duplex Framework (RFC #7181 PR 2) (#7695)

Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
authored and GitHub committed 2026-09-27 03:02:24 +00:00
1 parent 7ab582ae25
commit 798d52ce55
49 files changed
+3056 -1715

No files matched your search

+179 -190
View File
@@ -1,142 +1,137 @@
# PersonaPlex Unified Full-Duplex Design
# PersonaPlex on the Unified Full-Duplex Framework
## Status and target
This design adapts PersonaPlex PR #4771 to the unified full-duplex runtime on
top of:
- vLLM-Omni `origin/main`: `67c54777bb22e9e7e08fdf7c47a64f06b566fc47`
- PersonaPlex PR head: `477fb7c225f0c06991bc8aa55eadbd908ba282e4`
The target is the engine-native path:
PersonaPlex (`nvidia/personaplex-7b-v1`, a Moshi finetune) is served by the
[Unified Full-Duplex Framework](fullduplex.md) through one model plugin,
`PersonaPlexDuplexPlugin` (`vllm_omni/model_executor/models/personaplex/duplex/plugin.py`),
selected by `PipelineConfig.duplex_plugin`. The serving path is:
```text
/v1/realtime?duplex=1
-> OpenAI Realtime session actor
-> DuplexRequestClient
-> AsyncOmni correlated RPC
-> DuplexControlPlane
-> resumable Stage 0 request
-> PersonaPlex Talker
-> streaming PersonaPlex Code2Wav
-> PersonaPlex data-plane projector
-> OmniDuplexSessionHandler (thin: websocket I/O, Realtime <-> DuplexCommand/DuplexEvent)
-> DuplexOmni / DuplexOmniEngine
-> DuplexSessionManager -> DuplexSessionRunner (one per session)
plugin.plan_append: one 80 ms frame -> one resumable Stage 0 append
-> PersonaPlex Talker (Stage 0, lockstep temporal transformer + depformer)
-> streaming PersonaPlex Code2Wav (Stage 1, Mimi)
-> plugin.data_plane: cumulative audio/text -> deltas
-> response.output_audio.delta + response.output_audio_transcript.delta
```
The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
original PR was demo-only and has been removed from the tree; the unified
engine path above is the only serving surface. It was never evidence that the
unified engine path works.
## Why configuration-only enablement was invalid
The staged pipeline as shipped by the original PR was explicitly turn based.
Its Talker read `pplex_user_codes`, `pplex_prefill_text`, and
`pplex_silence_codes`, but no production staged input path wrote those fields.
The voice prompt, persona prefill, and streaming Mimi state lived only in the
standalone `PersonaPlexEngine` (demo-only, since removed).
Setting only the following fields would therefore advertise an endpoint whose
model never receives the live microphone stream:
```python
duplex_control_enabled = True
duplex_runtime_extension = "..."
duplex_serving_adapter = "..."
```
The adapter must supply a real scheduler data plane, not just endpoint
capabilities.
The pre-framework pair (engine runtime extension plus serving adapter) and
the standalone `/api/chat` / `/v1/audio/duplex` server are gone; the plugin is
the only integration surface, and this document describes it.
## Supported scope
The unified implementation supports:
The integration supports:
- up to two engine-owned sessions on one replica;
- 24 kHz mono float PCM input;
- one 1920-sample, 80 ms model frame per physical append unit;
- continuous user input while assistant audio is generated or played;
- bundled `.pt` voice prompts and a session persona;
- greedy text and depformer sampling, matching the current PersonaPlex port;
- `/v1/realtime?duplex=1` (the wire vocabulary is catalogued in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide);
- up to `duplex_session.max_sessions` engine-owned sessions on one replica
(the shipped deploy sets two);
- 24 kHz mono float PCM input, one 1920-sample (80 ms) model frame per
physical append unit; clients may send any chunking, the session buffers
whole frames;
- continuous user input while assistant audio is generated or played
(pure lockstep: the model listens while it speaks);
- bundled `.pt` voice prompts (`voice`) and a session persona (`instructions`);
- greedy text and depformer sampling (one temporal token per frame);
- the public client preset
`vllm_omni.clients.personaplex.create_duplex_session_config()` (24 kHz
`pcm_f32le` input format, voice prompt, persona) as the canonical
session-config source for `DuplexClient` consumers;
- engine lease close, disconnect cleanup, reconnect after cleanup, and explicit
response cancellation without cross-session state reuse.
`pcm_f32le` input, voice, persona) as the canonical session-config source
for `DuplexClient` / `InlineDuplexClient` consumers;
- engine lease close, disconnect cleanup and explicit response cancellation
without cross-session state reuse.
The implementation does not claim:
It does not claim:
- more than two simultaneous PersonaPlex sessions on one replica;
- arbitrary WAV voice cloning;
- turn-based `response.create` semantics for an otherwise continuous model;
- scheduler migration of live codec state between replicas;
- exact output equality with the standalone engine after different scheduling
boundaries.
- arbitrary WAV voice cloning (voices are bundled basenames, resolved by the
worker under the checkpoint);
- turn-based `response.create` semantics, client commits or external turn
signals (`supports_client_commit=false`, `supports_external_turn_signal=false`);
- text seeding, hence no `/v1/chat/completions` route
(`supports_chat_completions=false`);
- session resume across a transport drop (`supports_session_resume=false`);
- destructive output interruption or model-state rewind at a playback cursor
(`supports_barge_in=false`, `supports_audio_truncate=false`): overlapping
speech is native model behaviour, not a barge-in contract.
The capability payload derives multi-session support from the configured
session limit. The shipped two-session deployment reports
`supports_multi_session=true` and `supports_multi_session_same_replica=true`.
`duplex_session.max_sessions` is the only capacity source: config resolution
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools read
that model-config value. Connector extras do not carry a second model-specific
capacity knob that could drift from engine admission.
It still reports `supports_barge_in=false`: the generic epoch fence can suppress
stale transport output, but neither PersonaPlex nor the current MiniCPM-o 4.5
adapter proves that model-owned streaming state can be destructively rewound or
restarted at a playback cursor. Continuous overlapping speech is model-native
duplex behavior, not by itself a barge-in contract.
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools
(Stage 0 encoders, Stage 1 decoders) read that model-config value. The
capability payload derives `supports_multi_session` /
`supports_multi_session_same_replica` from it.
## Components and ownership
### Serving adapter
### The plugin (engine side)
`PersonaPlexServingRuntimeAdapter` owns only serving-side state:
`PersonaPlexDuplexPlugin` owns both halves of the contract.
- a transactional PCM append buffer;
- validation of `voice_prompt` and `instructions`;
- public capabilities;
- PersonaPlex data-plane output projection.
Engine policy:
It accepts 24 kHz `pcm_f32le`, groups client packets into whole 1920-sample
frames, zero-pads only the final residual, and rolls a reservation back when an
engine append fails. It never loads CUDA weights and never encodes user audio.
- `configure_sampling_params`: Stage 0 greedy (`temperature=0`, `top_k=1`,
`max_tokens=1`), other stages untouched.
- `plan_append`: validates exactly one 1920-sample 24 kHz `pcm_f32le` frame
(through `model_executor/common/duplex/payload.py`) and reserves one
scheduler slot per frame plus, on the first append of an epoch (`seq == 1`),
the `personaplex_prefill_slots` of the voice/persona prefill. The prompt
carries the fence, `seq`, the payload and the runtime config under
`model_intermediate_buffer["duplex"]`.
- `decide_output`: never decides. PersonaPlex is always-clocked; what the
client hears comes from the final-stage data plane.
- `silence_unit_payload`: one frame of zeros at 24 kHz
(`silence_continuation_samples=1920`,
`silence_continuation_sample_rate_hz=24000`). The runner uses it to keep a
model turn clocked when the client pauses; the startup warmup sends it.
Client input cannot provide local filesystem paths. A voice is a bundled
basename such as `NATF2.pt`; the worker resolves it under the local model
checkpoint. `instructions` is the persona string.
Session policy:
### Runtime extension
- `capabilities`: `personaplex_capabilities(max_sessions)` (80 ms units,
append-only, no commits, no barge-in, no resume, no chat route).
- `prepare_runtime_config`: `voice` must be a bundled `.pt` basename,
`instructions` defaults to the shipped persona; the prefill slot count is
computed once per `(model, voice, persona)` off the orchestrator loop
(`personaplex_prefill_slots`, itself caching the tokenizer and the voice
bundle row count). The result is server-owned runtime config
(`personaplex_model_path`, `personaplex_voice_prompt`,
`personaplex_persona`, `personaplex_prefill_slots`); the same keys are
refused in a client's `extra_body`.
- `runtime_config_for_update`: persona and voice are immutable for the
session (`persona_update_unsupported`, `voice_update_unsupported`).
- session state, extra-body validation and data-plane context are the
framework defaults (`DefaultDuplexModelSessionState` with the 80 ms
`PersonaPlexPcmAppendBuffer`, `DuplexDataPlaneContext`).
`PersonaPlexDuplexRuntimeExtension` is pure model policy. It:
Because `supports_client_commit` is off, the session auto-responds without
`extra_body.auto_response`: a stock Realtime client streams audio and hears
the model without any vendor flag.
- configures greedy Stage 0 sampling and bounded segment lengths;
- maps each accepted PCM append to a scheduler prompt;
- places immutable session identity, append sequence, PCM payload, voice, and
persona under `model_intermediate_buffer["duplex"]`;
- reserves exactly one scheduler prompt slot per encoded Mimi frame, plus the
first-append voice/persona prefill length;
- never performs model inference or owns session state.
### Input framing
The extension returns no turn/listen decision. PersonaPlex is an always-clocked
model, so visible audio/text comes from the final stage data plane.
`PersonaPlexPcmAppendBuffer` is the `FixedFramePcmAppendBuffer` of
`model_executor/common/duplex/pcm_buffer.py` at 24 kHz / 1920 samples /
80 ms. It accepts 24 kHz `pcm_f32le` only, groups client packets into whole
frames, takes one frame out per append as a reservation (committed when the
stage accepted the append, rolled back to the front of the buffer when it did
not), zero-pads only a final residual on commit, and never encodes audio.
### Stage 0 streaming runtime
### Stage 0 streaming runtime (worker side)
The Talker owns a `PersonaPlexStage0DuplexRuntime` helper, analogous to
MiniCPM-o's Stage 0 helper but with PersonaPlex lockstep semantics.
The Talker owns a `PersonaPlexStage0DuplexRuntime`
(`model_executor/models/personaplex/duplex/stage0.py`), created lazily in
`_duplex_stage0_runtime()` and released through `on_requests_finished`.
For each admitted session it owns:
For each live `(session_id, epoch)` it owns:
- streaming Mimi encoder convolution and transformer state;
- the selected voice embedding bundle;
- persona tokenization and prefill embeddings;
- the prior user code frame needed by the one-frame acoustic delay;
- append identity used to make a retried scheduler update idempotent.
- the streaming Mimi encoder convolution and transformer state;
- the selected voice embedding bundle and the persona prefill embeddings;
- the prior user code frames needed by the one- and two-frame acoustic
delays;
- the append identity `(epoch, seq)` that makes a retried scheduler update
idempotent.
The first append builds this ordered prefill:
The first append of an epoch builds this ordered prefill:
```text
voice embeddings
@@ -146,45 +141,47 @@ voice embeddings
-> first live user frame
```
Later appends encode only new 1920-sample frames. The helper returns the
Later appends encode only new 1920-sample frames. The runtime returns the
per-frame user codes and prompt embeddings through the request's
`model_intermediate_buffer`. The normal vLLM runner remains authoritative for
attention metadata, block tables, KV allocation, scheduling, and sampling.
`model_intermediate_buffer`; the normal vLLM runner remains authoritative for
attention metadata, block tables, KV allocation, scheduling and sampling.
The Talker must distinguish resumable prompt prefill from decode positions.
Prompt rows consume the exact prepared embeddings; sampled decode rows continue
to use the existing delayed agent/user frame construction. No code path may
fall back to an all-initial user stream for a duplex request.
**Epochs.** The framework identifies a Stage 0 request by
`(session_id, epoch)`; `response.cancel` and `output_audio_buffer.clear`
advance the epoch, abort the current request and start the next append at
`seq == 1` on a fresh request with fresh KV. The Stage 0 runtime therefore
keys its state by `(session_id, epoch)`: when an append of a newer epoch
arrives, any older-epoch state of the same session is closed first (its Mimi
encoder returns to the pool, so the codec budget never counts a superseded
epoch), and the voice/persona prefill is replayed because the plan reserved
the slots again. The user-visible consequence is that a cancel restarts the
model's conversation context. A late `on_requests_finished` for the aborted
request finds nothing to close.
Cleanup is keyed by the full `(session_id, incarnation)` identity. Every live
session has an independent Mimi encoder instance; encoder convolution/KV state
is never shared between asynchronously scheduled sessions. A finished or
Every live session has an independent Mimi encoder instance; encoder state is
never shared between asynchronously scheduled sessions. A finished or
aborted scheduler request resets and returns only that session's encoder.
### Stage 1 streaming decoder
The current `PersonaPlexCode2Wav` calls one-shot `MimiModel.decode` and is not
CUDA-graph safe. Unified duplex uses eager Stage 1 and maintains an independent
streaming Mimi decoder for every active request. Decoder ownership is keyed by
the stable Stage 1 request id and released by `on_requests_finished`; a mixed
batch must never advance another request's convolution or transformer state.
Each Stage 0 segment emits de-delayed agent codebooks. Stage 1 decodes only the
new code frames, emits only the new PCM suffix, and resets state when the
request is closed. Connector chunk boundaries retain the final raw code frame
needed to de-delay the next chunk.
The deploy default sets Stage 1 `enforce_eager: true`; a default configuration
that fails during CUDA graph capture is not an acceptable deployment profile.
`PersonaPlexCode2Wav` runs eager (`enforce_eager: true` in the deploy) and
maintains an independent streaming Mimi decoder for every active request,
keyed by the Stage 1 request id and released by `on_requests_finished`. Each
Stage 0 segment emits de-delayed agent codebooks; Stage 1 decodes only the new
code frames and emits only the new PCM suffix. Connector chunk boundaries
retain the final raw code frame needed to de-delay the next chunk.
### Data-plane projector
`PersonaPlexDataPlaneSession` converts cumulative or delta Stage 1 output into
model-neutral native results:
`PersonaPlexDataPlaneSession` is the `CumulativeAudioTextDataPlane` of
`model_executor/common/duplex/data_plane.py` with a 24 kHz default rate. It
keeps one audio/text cursor per request so cumulative Stage 1 output cannot
replay old audio, and yields one internal result per new suffix:
```python
{
"stage_role": "tts",
"is_listen": False,
"data_plane_request_id": request_id,
"text": text_delta,
"audio_data": encoded_audio_delta,
@@ -192,82 +189,74 @@ model-neutral native results:
"sample_rate_hz": 24000,
"audio_duration_ms": delta_duration,
"end_of_turn": False,
"runtime_impl": "scheduler_data_plane",
...
}
```
It owns per-request audio and text cursors so a cumulative output cannot replay
old audio. The generic projector turns each native result into the Realtime
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair (and
`response.output_text.delta` for text) under one `response_id`; the full
mapping is the name map in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
PersonaPlex keeps one visible response open while continuous output arrives.
Session close or cancellation terminates that response through the generic
Realtime lifecycle; a codec segment finishing is not a conversational turn
boundary. PersonaPlex advertises `supports_barge_in=false` and
`supports_session_resume=false`, so `barge_in`, `turn_detection.server_vad`,
and `session.resume` are rejected on this model.
The runner turns each result into the Realtime
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair
under one `response_id`. PersonaPlex keeps one visible response open while
continuous output arrives; session close or cancellation terminates it through
the generic Realtime lifecycle. A codec segment finishing is not a
conversational turn boundary.
## Error and lifecycle contracts
- Unsupported sample rate, malformed base64, non-finite PCM, invalid voice
basename, or changed format fails before scheduler submission.
- Unsupported sample rate, malformed base64, non-finite PCM, a partial frame
at the plan, an invalid voice basename or a changed format fails before
scheduler submission, as a typed `error` event.
- Append is prepare/submit/commit. Failure rolls back the exact reserved PCM
bytes and does not advance the model frame cursor.
- A repeated `operation_id` must not encode or submit the same frame twice.
- Input iterator exceptions execute the same cleanup as explicit close.
- Cancellation does not release the engine lease until the stage request and
any in-flight codec operation are actually finished.
- A bounded cleanup timeout returns a cleanup error and keeps the session in
the closing admission set; it must not make the slot available while work
still mutates shared state.
- New sessions cannot observe the previous voice, persona, PCM tail, Mimi
convolution state, or Talker delayed code frame.
(The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path,
which followed the same exception-safe rule, have been removed from the tree.)
- A repeated `(epoch, seq)` identity does not encode or submit the same frame
twice.
- Cancellation advances the epoch; the aborted request's Stage 0 state is
released by the next-epoch append or by `on_requests_finished`, whichever
comes first.
- Close releases the admission slot only once stage cleanup succeeded; the
reaper retries a failed cleanup while the slot stays held.
- New sessions cannot observe a previous voice, persona, PCM tail, Mimi
convolution state or Talker delayed code frame.
## Testing and acceptance
### Contract tests
### CPU contract tests
Tests first cover:
- `tests/model_executor/models/personaplex/duplex/test_plugin.py`: pipeline
binding and plugin load, honest capabilities, private keys, voice/persona
resolution and caching, immutable updates, greedy sampling, one-slot-plus-
prefill planning, frame validation, the 24 kHz silence unit, and the
CPU-checked helpers of the E2E driver.
- `tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py`:
first-append prefill, depformer teacher forcing, causal user-frame delays,
idempotent retries, independent encoders per session, capacity, epoch
restart and late finish of a superseded request.
- `tests/engine/duplex/test_session_runner_personaplex.py`: the session
runner with the real plugin -- one submission per frame, prefill on the
first append only, half frames buffered, wrong rate refused, cumulative
Code2Wav output projected as 24 kHz deltas, cancel restarting the epoch,
close aborting the request.
- `tests/model_executor/executor_common/`: the shared toolbox (PCM helpers, payload
validation, fixed-frame buffer, cumulative data plane, request-output
readers).
- pipeline registration enables the control plane and selects both PersonaPlex
adapters;
- the capability payload is two-session, 80 ms, append-only, and honest;
- PCM reservation commit/rollback, partial-frame flush, invalid input, and
operation idempotency;
- runtime prompt fields and exact token budgeting;
- first-append voice/persona prefill followed by live user codes;
- later appends retain per-session Mimi state and do not replay prefill;
- interleaved Stage 0 and Stage 1 work preserves independent codec histories;
- output projection emits only audio/text deltas;
- close, exception, timeout, and reconnect cleanup;
- ordinary non-duplex imports do not load PersonaPlex modules.
### GPU validation
### Remote H20 validation
`tests/e2e/online_serving/personaplex_realtime_duplex.py` (wrapped by
`tests/e2e/online_serving/test_personaplex_duplex.py` when
`PERSONAPLEX_MODEL_PATH` is set) drives the default `personaplex.yaml`
deployment and requires:
Validation runs in an isolated remote worktree using the ModelScope
`nv-community/personaplex-7b-v1` checkpoint and its Mimi dependency.
The required evidence is:
1. default `personaplex.yaml` reaches ready without a local eager override;
2. `/health` returns 200;
3. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`,
and two-session admission;
4. paced 24 kHz PCM appends produce finite, non-silent 24 kHz audio deltas and
text deltas;
5. microphone input continues during assistant output without cancelling the
scheduler request;
6. two paced sessions simultaneously produce independent non-empty audio, and
a third session is rejected with `resource_exhausted`;
7. closing either session frees only its scheduler request and GPU codec state,
after which a replacement session can use a different persona without state
leakage;
8. malformed input returns one typed error and does not poison the next append;
9. all focused unit tests and `git diff --check` pass.
1. the server reaches ready without a local eager override; `/health` is 200;
2. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`
and two-session admission, with server-allocated session ids;
3. paced 24 kHz PCM appends produce finite, non-silent, whole-frame 24 kHz
audio deltas and text deltas;
4. two paced sessions simultaneously produce independent non-empty audio, and
a third session is refused with `resource_exhausted`;
5. closing one session frees only its scheduler request and codec state, after
which a replacement session with a different persona is admitted with a
fresh id and no state leakage.
Audio that is empty, all zero, non-finite, or only a protocol `listen` event is
not a successful end-to-end result.
+1
View File
@@ -107,6 +107,7 @@ th {
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
| `PersonaPlexTalkerForConditionalGeneration` | PersonaPlex (full duplex, 24 kHz speech in/out over `/v1/realtime?duplex=1`) | `nvidia/personaplex-7b-v1` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/PersonaPlex.md) |
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
+8 -4
View File
@@ -384,9 +384,9 @@ with no OpenAI counterpart.
The event vocabulary is uniform, but several surfaces are gated by the
`capabilities` object the server returns in `session.created`; a client must
branch on those flags rather than on the model name. MiniCPM-o 4.5 is the
only model on the plugin contract today; the other two columns record what
their integrations advertise once the follow-up PRs port them:
branch on those flags rather than on the model name. MiniCPM-o 4.5 and
PersonaPlex are on the plugin contract today; the Nemotron VoiceChat column
records what its integration advertises once the follow-up PR ports it:
| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
| --- | --- | --- | --- | --- |
@@ -401,7 +401,11 @@ their integrations advertise once the follow-up PRs port them:
Everything else in the catalogue — session lifecycle, heartbeat and event
acknowledgement, append/commit/clear, the response envelope, playback
acknowledgement, and the error envelope — behaves identically for every
model.
model. Two PersonaPlex specifics follow from its capabilities rather than from
special-casing: a model with `supports_client_commit=false` auto-responds
without `extra_body.auto_response`, and `response.cancel` /
`output_audio_buffer.clear` restart its conversation context (a new Stage 0
request replays the voice/persona prefill).
### Compatibility with the OpenAI Realtime protocol
+36 -20
View File
@@ -2,15 +2,15 @@
Serve [`nvidia/personaplex-7b-v1`](https://huggingface.co/nvidia/personaplex-7b-v1)
(a Moshi-based full-duplex speech-to-speech model) with the native vLLM-Omni engine
through the unified duplex serving stack (`/v1/duplex` and `/v1/realtime?duplex=1`).
through the unified full-duplex framework (`/v1/realtime?duplex=1`, alias `/v1/duplex`).
> Requires a GPU and Hugging Face access to the gated repo
> (`HF_TOKEN` with access to `nvidia/personaplex-7b-v1`).
## Start the server
The default `vllm_omni/deploy/personaplex.yaml` enables the engine-owned
full-duplex control plane (`session_mode: duplex`):
The default `vllm_omni/deploy/personaplex.yaml` is a duplex deployment
(`session_mode: duplex`, two sessions per replica):
```bash
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
@@ -19,18 +19,32 @@ HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main ser
--deploy-config vllm_omni/deploy/personaplex.yaml
```
This exposes:
This exposes `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`): the OpenAI
Realtime session protocol projected onto vLLM-Omni duplex sessions (client API and
wire protocol: [`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md)).
There is no `/v1/chat/completions` route: PersonaPlex answers speech only.
- `WS /v1/duplex` — the native duplex session dialect
(`session.create` / `input_audio_buffer.append` / `response.output_audio.delta` ...);
- `WS /v1/realtime?duplex=1` — the same sessions projected onto the
OpenAI Realtime event vocabulary (client API and wire protocol:
[`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md)).
PersonaPlex is a pure-lockstep model: every session is native duplex, audio flows
continuously in both directions in 80 ms frames, the model decides when to speak,
and there are no client commits or external turn signals
(`supports_client_commit=false`, `supports_external_turn_signal=false`). A session
therefore auto-responds without any vendor flag.
PersonaPlex is a pure-lockstep model: every session on a PersonaPlex deployment is
native duplex (`is_enabled()` is unconditionally true), audio flows continuously in
both directions, and there are no client commits or external turn signals
(`supports_client_commit=false`, `supports_external_turn_signal=false`).
## Talk to it
With the client library and the PersonaPlex preset (24 kHz `pcm_f32le` in, bundled
voice prompt, persona text):
```python
from vllm_omni.clients.duplex import DuplexClient
from vllm_omni.clients.personaplex import create_duplex_session_config
cfg = create_duplex_session_config(voice="NATF2.pt", persona="You are a concise assistant.")
async with DuplexClient("ws://127.0.0.1:8000/v1/realtime?duplex=1", model="/path/to/personaplex-7b-v1", config=cfg) as c:
await c.stream_pcm(pcm_f32le_24k) # keep streaming; the model speaks while it listens
```
Voice and persona are fixed for the session (`session.update` cannot change them).
## Validate the serving path
@@ -45,9 +59,11 @@ python tests/e2e/online_serving/personaplex_realtime_duplex.py \
--output-dir /tmp/personaplex-realtime-duplex
```
The unified endpoint advertises `supports_barge_in=false`: overlapping speech
is native model behavior, but destructive output interruption and model-state
rewind have not been validated for PersonaPlex.
The endpoint advertises `supports_barge_in=false`: overlapping speech is native
model behaviour, but destructive output interruption and model-state rewind have
not been validated for PersonaPlex. `response.cancel` and
`output_audio_buffer.clear` restart the model's conversation context (a fresh
Stage 0 request replays the voice/persona prefill).
## Notes
@@ -56,8 +72,8 @@ rewind have not been validated for PersonaPlex.
regardless of engine speed. On localhost it is smooth.
- The earlier standalone Moshi-web compatibility server (browser client at `/`,
binary WS protocol at `/api/chat`, raw-PCM `/v1/audio/duplex`) was demo-only
and has been removed; use the unified endpoints above.
- Session config (voice / persona / sampling) is passed per session via
`extra_body`; see
`vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py`.
and has been removed; use the unified endpoint above.
- The model plugin, worker-side lockstep runtime and input framing live in
`vllm_omni/model_executor/models/personaplex/duplex/`; design notes in
[`docs/design/fullduplex-personaplex.md`](../../../docs/design/fullduplex-personaplex.md).
- Full runbook: [`recipes/NVIDIA/PersonaPlex.md`](../../../recipes/NVIDIA/PersonaPlex.md).
+12 -10
View File
@@ -25,10 +25,10 @@ matching the reference implementation frame for frame on the golden replays used
as the acceptance gate.
PersonaPlex is the first Moshi-class (pure-lockstep) model on the vLLM-Omni
full-duplex serving stack: it plugs into the generic `/v1/duplex` handler
through the standard plugin seams (`duplex_serving_adapter` /
`duplex_runtime_extension` in its `pipeline.py`) with a model-specific
package at `vllm_omni/model_executor/models/personaplex/duplex/`.
unified full-duplex framework: its `pipeline.py` declares one
`duplex_plugin` (`PersonaPlexDuplexPlugin`), and the model-specific package at
`vllm_omni/model_executor/models/personaplex/duplex/` holds the plugin, the
worker-side lockstep Stage 0 runtime and the 80 ms input framing.
## References
@@ -114,16 +114,18 @@ HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main ser
--deploy-config vllm_omni/deploy/personaplex.yaml
```
This exposes `WS /v1/duplex` (native duplex dialect) and
`WS /v1/realtime?duplex=1` (OpenAI Realtime projection; client API and wire
protocol in [`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md)).
Voice and persona are set per session via `extra_body`.
This exposes `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`): the OpenAI
Realtime session protocol, client API and wire vocabulary in
[`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md).
Voice (`voice`, a bundled `.pt` basename) and persona (`instructions`) are set
in `session.update`; the model takes no client commits and serves no
`/v1/chat/completions` route.
#### Verification
```bash
# GPU-free contract tests (stage0 runtime + unified serving adapter)
pytest tests/model_executor/models/personaplex/duplex/ -q
# GPU-free contract tests (plugin, stage0 runtime, runner scenario)
pytest tests/model_executor/models/personaplex/duplex/ tests/engine/duplex/test_session_runner_personaplex.py -q
# GPU e2e: paced 24 kHz PCM over /v1/realtime?duplex=1, two concurrent
# sessions, overflow admission, slot recycling, non-silent output
@@ -11,7 +11,6 @@ import hashlib
import json
import math
import time
import uuid
import wave
from collections.abc import Awaitable, Callable, Sequence
from pathlib import Path
@@ -125,10 +124,11 @@ def _input_identity(
return {"path": str(resolved), "sha256": actual}
def _realtime_url(base_url: str, model: str, session_id: str) -> str:
def _realtime_url(base_url: str, model: str) -> str:
"""The duplex Realtime URL; the session id is allocated by the server, never chosen here."""
parts = urlsplit(base_url)
query = dict(parse_qsl(parts.query, keep_blank_values=True))
query.update(duplex="1", model=model, autostart="0", session_id=session_id)
query.update(duplex="1", model=model, autostart="0")
return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment))
@@ -197,16 +197,27 @@ def _audio_bytes(client: RawRealtimeProbe) -> bytes:
return b"".join(chunk for _, chunk in _validated_audio_chunks(client) if chunk)
def _session_id(created: dict[str, object]) -> str | None:
"""The server-allocated id announced in ``session.created``."""
session = created.get("session")
if isinstance(session, dict):
for key in ("id", "session_id"):
value = session.get(key)
if isinstance(value, str) and value:
return value
value = created.get("session_id")
return value if isinstance(value, str) and value else None
async def _open_session(
args: argparse.Namespace,
*,
session_id: str,
persona: str,
expect_error: bool = False,
) -> tuple[RawRealtimeProbe, dict[str, object]]:
client = RawRealtimeProbe(_realtime_url(args.url, args.model, session_id))
client = RawRealtimeProbe(_realtime_url(args.url, args.model))
await client.__aenter__()
await client.send(_session_update(args, session_id=session_id, persona=persona))
await client.send(_session_update(args, persona=persona))
event_type = "error" if expect_error else "session.created"
await wait_for(
lambda: client.events.count(event_type) > 0,
@@ -216,11 +227,10 @@ async def _open_session(
return client, _events(client, event_type)[-1]
def _session_update(args: argparse.Namespace, *, session_id: str, persona: str) -> dict[str, object]:
def _session_update(args: argparse.Namespace, *, persona: str) -> dict[str, object]:
return {
"type": "session.update",
"session": {
"session_id": session_id,
"model": args.model,
"modalities": ["audio", "text"],
"input_audio_format": "pcm_f32le",
@@ -426,13 +436,11 @@ async def run(args: argparse.Namespace) -> dict[str, object]:
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
ids = {name: f"personaplex-{name}-{uuid.uuid4().hex}" for name in ("primary", "secondary", "replacement")}
primary, created = await _open_session(args, session_id=ids["primary"], persona=args.persona)
secondary, secondary_created = await _open_session(
args,
session_id=ids["secondary"],
persona=args.secondary_persona,
)
primary, created = await _open_session(args, persona=args.persona)
secondary, secondary_created = await _open_session(args, persona=args.secondary_persona)
ids = {"primary": _session_id(created), "secondary": _session_id(secondary_created)}
if not ids["primary"] or not ids["secondary"] or ids["primary"] == ids["secondary"]:
raise AssertionError(f"server did not allocate distinct session ids: {ids}")
capabilities = _capabilities(created)
if _capabilities(secondary_created) != capabilities:
raise AssertionError("concurrent sessions returned different capabilities")
@@ -446,12 +454,7 @@ async def run(args: argparse.Namespace) -> dict[str, object]:
if any(capabilities.get(key) != value for key, value in expected_capabilities.items()):
raise AssertionError(f"unexpected PersonaPlex capabilities: {capabilities}")
overflow, error = await _open_session(
args,
session_id=f"personaplex-overflow-{uuid.uuid4().hex}",
persona=args.persona,
expect_error=True,
)
overflow, error = await _open_session(args, persona=args.persona, expect_error=True)
error_body = error.get("error")
overflow_code = error_body.get("code") if isinstance(error_body, dict) else error.get("code")
await overflow.__aexit__(None, None, None)
@@ -492,11 +495,10 @@ async def run(args: argparse.Namespace) -> dict[str, object]:
_save(output_dir, "primary", primary, primary_audio)
await _close_session(primary, timeout_s=args.timeout_s)
replacement, replacement_created = await _open_session(
args,
session_id=ids["replacement"],
persona=args.replacement_persona,
)
replacement, replacement_created = await _open_session(args, persona=args.replacement_persona)
ids["replacement"] = _session_id(replacement_created)
if not ids["replacement"] or ids["replacement"] in {ids["primary"], ids["secondary"]}:
raise AssertionError(f"replacement session id was not freshly allocated: {ids}")
if _capabilities(replacement_created) != capabilities:
raise AssertionError("replacement session returned different capabilities")
continuation_frames, replacement_frames = await asyncio.gather(
@@ -694,8 +696,9 @@ async def _run_load_session(
ready: asyncio.Future[None],
start: asyncio.Future[float],
) -> dict[str, object]:
session_id = f"personaplex-load-{index}-{uuid.uuid4().hex}"
client = RawRealtimeProbe(_realtime_url(args.url, args.model, session_id), close_timeout_s=args.cleanup_timeout_s)
# Label only: the server allocates the real session id (read back from session.created).
session_id = f"personaplex-load-{index}"
client = RawRealtimeProbe(_realtime_url(args.url, args.model), close_timeout_s=args.cleanup_timeout_s)
sends: list[tuple[float, float, float]] = []
error: str | None = None
cleanup_error: str | None = None
@@ -706,10 +709,9 @@ async def _run_load_session(
try:
async with client:
try:
await asyncio.wait_for(
client.send(_session_update(args, session_id=session_id, persona=args.persona)), args.timeout_s
)
await asyncio.wait_for(client.send(_session_update(args, persona=args.persona)), args.timeout_s)
await _wait_load_event(client, "session.created", args.timeout_s)
session_id = _session_id(_events(client, "session.created")[-1]) or session_id
capabilities = _capabilities(_events(client, "session.created")[-1])
if (
capabilities.get("chunk_period_ms") != 80
@@ -0,0 +1,84 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""GPU coverage for PersonaPlex on the unified full-duplex framework.
Boots the PersonaPlex deploy (``vllm_omni/deploy/personaplex.yaml``) and runs
the strict Realtime driver (``personaplex_realtime_duplex.py``): two paced
24 kHz sessions, admission overflow, per-session slot recycling, audible
whole-frame output. The checkpoint is gated, so the test runs only when
``PERSONAPLEX_MODEL_PATH`` points at a local copy of ``nvidia/personaplex-7b-v1``.
"""
from __future__ import annotations
import asyncio
import os
from pathlib import Path
import pytest
from tests.e2e.online_serving import personaplex_realtime_duplex as driver
from tests.helpers.mark import hardware_test
from tests.helpers.runtime import OmniServerParams
from tests.helpers.stage_config import get_deploy_config_path
pytestmark = pytest.mark.omni
MODEL_PATH = os.environ.get("PERSONAPLEX_MODEL_PATH", "")
DEPLOY_CONFIG = get_deploy_config_path("personaplex.yaml")
SERVER_PARAMS = [
pytest.param(
OmniServerParams(
model=MODEL_PATH or "nvidia/personaplex-7b-v1",
stage_config_path=DEPLOY_CONFIG,
use_stage_cli=False,
server_args=["--trust-remote-code"],
),
id="two-stage-single-gpu",
)
]
requires_checkpoint = pytest.mark.skipif(
not MODEL_PATH or not Path(MODEL_PATH).is_dir(),
reason="set PERSONAPLEX_MODEL_PATH to a local nvidia/personaplex-7b-v1 checkout",
)
def _speech_wav() -> Path:
"""Real speech (the MiniCPM-o asset, 16 kHz; the driver resamples to 24 kHz).
PersonaPlex answers speech, not tones: a synthetic signal yields a silent
reply and the driver's audibility floor rightly fails it.
"""
return Path(__file__).resolve().parents[2] / "assets" / "minicpmo_4_5" / "response_required_16k.wav"
@requires_checkpoint
@pytest.mark.advanced_model
@hardware_test(res={"cuda": "H100"}, num_cards=1)
@pytest.mark.parametrize("omni_server", SERVER_PARAMS, indirect=True)
def test_personaplex_realtime_duplex_sessions(omni_server, tmp_path: Path) -> None:
input_wav = _speech_wav()
args = driver.parse_args(
[
"--url",
f"ws://{omni_server.host}:{omni_server.port}/v1/realtime?duplex=1",
"--model",
MODEL_PATH,
"--input-wav",
str(input_wav),
"--output-dir",
str(tmp_path / "out"),
]
)
result = asyncio.run(driver.run(args))
sessions = {name: result[name] for name in ("primary", "secondary", "replacement")}
assert all(isinstance(session, dict) for session in sessions.values())
ids = {name: session["session_id"] for name, session in sessions.items() if isinstance(session, dict)}
assert ids["primary"]
assert ids["secondary"] != ids["primary"]
assert ids["replacement"] not in {ids["primary"], ids["secondary"]}
+120
View File
@@ -5,6 +5,7 @@
from __future__ import annotations
import base64
from dataclasses import fields
from pathlib import Path
from typing import Any
@@ -16,10 +17,13 @@ from vllm_omni.config.stage_config import DuplexSessionRuntimeConfig, PipelineCo
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.contracts import DuplexAppendPlan
from vllm_omni.engine.duplex.plugin import (
DefaultDuplexModelSessionState,
DuplexDataPlane,
DuplexDataPlaneContext,
DuplexModelPlugin,
DuplexModelSessionState,
DuplexRuntimeConfigError,
PcmAppendBuffer,
coerce_int,
load_duplex_plugin,
payload_turn_id,
@@ -322,3 +326,119 @@ def test_pipeline_config_binds_one_duplex_plugin_path() -> None:
for source in framework_sources
if "completed_append_cache_size" in source.read_text(encoding="utf-8")
] == []
# --------------------------------------------------------------------------- #
# Plugin defaults shared by every model #
# --------------------------------------------------------------------------- #
def test_default_silence_unit_is_16k_zeros_and_follows_the_plugin_attributes() -> None:
plugin = FakePlugin(_encode_audio)
unit = plugin.silence_unit_payload()
assert unit["type"] == "audio"
assert unit["format"] == "pcm_f32le"
assert unit["sample_rate_hz"] == 16000
assert base64.b64decode(unit["audio"]) == bytes(16000 * 4)
class FramePlugin(FakePlugin):
silence_continuation_samples = 1920
silence_continuation_sample_rate_hz = 24000
frame_unit = FramePlugin(_encode_audio).silence_unit_payload()
assert frame_unit["sample_rate_hz"] == 24000
assert len(base64.b64decode(frame_unit["audio"])) == 1920 * 4
def test_default_extra_body_validation_rejects_the_private_keys_naming_the_plugin() -> None:
class PrivateKeysPlugin(FakePlugin):
plugin_id = "keyed"
private_runtime_config_keys = frozenset({"secret_a", "secret_b"})
def validate_client_extra_body(self, extra_body: object) -> None:
DuplexModelPlugin.validate_client_extra_body(self, extra_body)
plugin = PrivateKeysPlugin(_encode_audio)
plugin.validate_client_extra_body(None)
plugin.validate_client_extra_body({"auto_response": True})
with pytest.raises(
DuplexRuntimeConfigError, match="keyed runtime configuration is server-owned: secret_a, secret_b"
):
plugin.validate_client_extra_body({"secret_b": 1, "secret_a": 2})
def test_default_data_plane_context_is_the_framework_dataclass() -> None:
class DefaultContextPlugin(FakePlugin):
def data_plane_context(self, **kwargs):
return DuplexModelPlugin.data_plane_context(self, **kwargs)
context = DefaultContextPlugin(_encode_audio).data_plane_context(
epoch=2,
turn_id=3,
active_response_turn_id=1,
active_response_id="resp",
auto_responds=True,
response_format="pcm16",
speed=1.5,
modalities=("audio",),
)
assert isinstance(context, DuplexDataPlaneContext)
assert context == DuplexDataPlaneContext(
epoch=2,
turn_id=3,
active_response_turn_id=1,
active_response_id="resp",
auto_responds=True,
response_format="pcm16",
speed=1.5,
modalities=("audio",),
)
def test_default_session_state_implements_the_shared_transitions() -> None:
class Buffer(PcmAppendBuffer):
pending_byte_count = 0
def clear(self) -> None: ...
def clear_force_listen(self) -> None: ...
def has_pending(self) -> bool:
return False
def has_reserved(self) -> bool:
return False
def prepare_append(self, payload, *, operation_id, chunk_period_ms, allow_emit):
return None
def prepare_commit(self, *, operation_id, chunk_period_ms):
raise NotImplementedError
def flush(self, *, chunk_period_ms):
return None
state = DefaultDuplexModelSessionState(audio_buffer=Buffer())
assert isinstance(state, DuplexModelSessionState)
assert state.committed_audio_reserved_bytes == 0 and state.continuation_units == 0
state.retain_committed_audio({"audio": "a"}, operation_id="op-1", reserved_bytes=10)
state.retain_committed_audio({"audio": "b"}, operation_id="op-2", reserved_bytes=5)
state.deferred_response_create = True
assert state.committed_audio_payload == {"audio": "b"}
assert state.committed_audio_operation_id == "op-2"
assert state.clear_committed_audio() == 15
assert state.committed_audio_payload is None
assert state.committed_audio_operation_id is None
assert state.deferred_response_create is False
state.continuation_owner_id = "response:r"
state.continuation_units = 3
state.pending_silence_owner_id = "response:r"
state.clear_continuation()
assert state.continuation_owner_id is None
assert state.continuation_units == 0
assert state.pending_silence_task is None and state.pending_silence_owner_id is None
@@ -0,0 +1,245 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""The session runner driven by the real PersonaPlex plugin.
Same harness as ``test_session_runner.py`` (recording stage port, typed
commands through the manager, stage outputs pushed by hand), plugin swapped:
this pins the lockstep contract -- one Stage 0 submission per 80 ms frame,
prefill slots only on the first append of an epoch, auto-response implied by
``supports_client_commit=False``, cumulative Code2Wav output projected as
24 kHz deltas, and a cancel restarting the epoch.
"""
from __future__ import annotations
import asyncio
import base64
import struct
from types import SimpleNamespace
from typing import Any
import numpy as np
import pytest
from tests.engine.duplex.test_session_runner import (
SESSION_ID,
Harness,
RecordingStagePort,
_fake_encode_audio,
close_harness,
find,
types,
)
from vllm_omni.config.stage_config import DuplexSessionRuntimeConfig
from vllm_omni.engine.duplex import commands
from vllm_omni.engine.duplex.config import DuplexSessionConfig
from vllm_omni.engine.duplex.messages import DuplexControlResultMessage, OpenDuplexSessionMessage
from vllm_omni.engine.duplex.session.manager import DuplexSessionManager
from vllm_omni.model_executor.models.personaplex.duplex import stage0
from vllm_omni.model_executor.models.personaplex.duplex.config import FRAME_SIZE, SAMPLE_RATE
from vllm_omni.model_executor.models.personaplex.duplex.plugin import PersonaPlexDuplexPlugin
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
PREFILL_SLOTS = 4
@pytest.fixture(autouse=True)
def _fake_prefill(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(stage0, "personaplex_prefill_slots", lambda model_path, voice, persona: PREFILL_SLOTS)
async def open_personaplex_harness(*, extra_body: dict[str, object] | None = None) -> Harness:
plugin = PersonaPlexDuplexPlugin(_fake_encode_audio)
port = RecordingStagePort(stage_count=2)
output: asyncio.Queue[Any] = asyncio.Queue()
results: asyncio.Queue[Any] = asyncio.Queue()
manager = DuplexSessionManager(
plugin=plugin,
stage_port=port,
output_sink=output,
result_sink=results,
runtime_config=DuplexSessionRuntimeConfig(),
model_config=SimpleNamespace(model="/models/personaplex-7b-v1"),
)
config = DuplexSessionConfig(
model="nvidia/personaplex-7b-v1",
modalities=["audio", "text"],
instructions="Be concise.",
voice="NATF2.pt",
extra_body=dict(extra_body or {}),
)
await manager.handle(OpenDuplexSessionMessage(control_id="c-open", session_id=SESSION_ID, session_config=config))
result = await asyncio.wait_for(results.get(), timeout=2.0)
assert isinstance(result, DuplexControlResultMessage) and result.ok, result
harness = Harness(manager=manager, port=port, output=output, results=results, runner=manager.runners[SESSION_ID])
await harness.settle()
return harness
def frame(samples: int = FRAME_SIZE, *, value: float = 0.05) -> commands.AppendAudio:
return commands.AppendAudio(
audio=struct.pack(f"<{samples}f", *([value] * samples)),
format="pcm_f32le",
sample_rate_hz=SAMPLE_RATE,
is_speech=True,
)
def code2wav_output(request_id: str, *, samples: int, text: str) -> SimpleNamespace:
"""A cumulative Stage 1 (Code2Wav) output the way the orchestrator hands it to the runner."""
return SimpleNamespace(
request_id=request_id,
finished=False,
outputs=[SimpleNamespace(text=text, token_ids=[], multimodal_output={})],
multimodal_output={"audio": np.zeros(samples, dtype=np.float32), "sr": SAMPLE_RATE},
)
def submitted_duplex(port: RecordingStagePort) -> list[dict[str, object]]:
return [dict(sub.prompt["model_intermediate_buffer"]["duplex"]) for sub in port.submissions]
@pytest.mark.asyncio
async def test_open_advertises_the_lockstep_capabilities_and_reserves_stage0() -> None:
h = await open_personaplex_harness()
try:
created = find(h.events, "session.created")
capabilities = created.session["capabilities"]
assert capabilities["chunk_period_ms"] == 80
assert capabilities["supports_client_commit"] is False
assert capabilities["supports_barge_in"] is False
# Not on the wire: the api server reads it from the plugin to decide the chat route.
assert h.session.capabilities.supports_chat_completions is False
assert [context.request_id for context in h.port.ensured] == [h.stage0_request_id(epoch=0)]
assert h.session.runtime_config["personaplex_prefill_slots"] == PREFILL_SLOTS
assert h.session.runtime_config["personaplex_voice_prompt"] == "NATF2.pt"
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_a_session_without_the_auto_response_flag_still_auto_responds() -> None:
h = await open_personaplex_harness(extra_body={})
try:
assert h.runner.out.auto_responds() is True
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_each_frame_is_one_stage0_submission_with_prefill_only_on_the_first() -> None:
h = await open_personaplex_harness()
try:
await h.run(frame())
await h.run(frame())
await h.run(frame())
assert len(h.port.submissions) == 3
assert [len(sub.prompt["prompt_token_ids"]) for sub in h.port.submissions] == [1 + PREFILL_SLOTS, 1, 1]
duplex = submitted_duplex(h.port)
assert [item["seq"] for item in duplex] == [1, 2, 3]
assert all(item["epoch"] == 0 for item in duplex)
assert all(item["session_id"] == SESSION_ID for item in duplex)
assert all("incarnation" not in item for item in duplex)
assert all(sub.context.request_id == h.stage0_request_id(epoch=0) for sub in h.port.submissions)
assert [sub.already_submitted for sub in h.port.submissions] == [False, True, True]
assert h.port.submissions[0].context.stage_sampling_params.max_tokens == 1
assert not [event for event in h.events if event.type == "error"]
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_half_frames_are_buffered_until_a_whole_frame_exists() -> None:
h = await open_personaplex_harness()
try:
await h.run(frame(FRAME_SIZE // 2))
assert h.port.submissions == []
assert h.runner.model_state.audio_buffer.pending_byte_count == FRAME_SIZE * 2
await h.run(frame(FRAME_SIZE // 2))
assert len(h.port.submissions) == 1
assert h.runner.model_state.audio_buffer.pending_byte_count == 0
payload = submitted_duplex(h.port)[0]["payload"]
assert isinstance(payload, dict)
assert len(base64.b64decode(payload["audio"])) == FRAME_SIZE * 4
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_a_wrong_rate_append_is_refused_without_a_submission() -> None:
h = await open_personaplex_harness()
try:
events = await h.run(
commands.AppendAudio(audio=b"\x00" * 64, format="pcm_f32le", sample_rate_hz=16000, is_speech=None)
)
error = find(events, "error")
assert error.code == "bad_event"
assert "24000" in error.message
assert h.port.submissions == []
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_cumulative_code2wav_output_becomes_24k_audio_and_text_deltas() -> None:
h = await open_personaplex_harness()
try:
await h.run(frame())
request_id = h.stage0_request_id(epoch=0)
first = await h.deliver_and_settle(code2wav_output(request_id, samples=FRAME_SIZE, text="he"), stage_id=1)
second = await h.deliver_and_settle(
code2wav_output(request_id, samples=2 * FRAME_SIZE, text="hello"), stage_id=1
)
assert types(first)[0] == "response.created"
delta = find(first, "response.output_audio.delta")
assert delta.delta == f"wav-{FRAME_SIZE}"
assert delta.sample_rate_hz == SAMPLE_RATE
assert find(first, "response.output_audio_transcript.delta").delta == "he"
assert h.session.active_response_id is not None
[delta_2] = [event for event in second if event.type == "response.output_audio.delta"]
assert delta_2.delta == f"wav-{FRAME_SIZE}"
assert find(second, "response.output_audio_transcript.delta").delta == "llo"
assert not [event for event in second if event.type == "response.created"]
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_cancel_restarts_the_epoch_and_the_next_frame_replays_the_prefill() -> None:
h = await open_personaplex_harness()
try:
await h.run(frame())
old_request_id = h.stage0_request_id(epoch=0)
await h.deliver_and_settle(code2wav_output(old_request_id, samples=FRAME_SIZE, text="he"), stage_id=1)
cancelled = await h.run(commands.CancelResponse())
await h.run(frame())
assert h.session.epoch == 1
assert "audio.cancelled" in types(cancelled) or "response.done" in types(cancelled)
assert ([old_request_id], True) in h.port.cleanups
assert h.port.submissions[-1].context.request_id == h.stage0_request_id(epoch=1)
assert submitted_duplex(h.port)[-1]["seq"] == 1
assert len(h.port.submissions[-1].prompt["prompt_token_ids"]) == 1 + PREFILL_SLOTS
finally:
await close_harness(h)
@pytest.mark.asyncio
async def test_close_aborts_the_stage0_request() -> None:
h = await open_personaplex_harness()
await h.run(frame())
request_id = h.stage0_request_id(epoch=0)
await h.run(commands.CloseSession(reason="client_close"))
assert h.port.cleanups == [([request_id], True)]
await close_harness(h)
@@ -65,6 +65,7 @@ forbidden_prefixes = (
"vllm_omni.entrypoints.duplex",
"vllm_omni.entrypoints.duplex_omni",
"vllm_omni.clients",
"vllm_omni.model_executor.common.duplex",
"vllm_omni.model_executor.models.minicpmo_4_5.duplex",
"vllm_omni.model_executor.models.nemotron_voicechat.duplex",
"vllm_omni.model_executor.models.personaplex.duplex",
@@ -104,6 +105,7 @@ if missing:
forbidden_prefixes = (
"vllm_omni.clients",
"vllm_omni.model_executor.common.duplex",
"vllm_omni.model_executor.models.minicpmo_4_5.duplex",
"vllm_omni.model_executor.models.nemotron_voicechat.duplex",
"vllm_omni.model_executor.models.personaplex.duplex",
@@ -0,0 +1,53 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""The startup warmup sends the model's own silence unit, not a fixed 16 kHz one."""
from __future__ import annotations
import base64
import pytest
from vllm_omni.entrypoints.duplex.warmup import warmup_silence_unit
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
class _Plugin:
silence_continuation_samples = 1920
def silence_unit_payload(self) -> dict[str, object]:
return {
"type": "audio",
"audio": base64.b64encode(bytes(1920 * 4)).decode("ascii"),
"format": "pcm_f32le",
"sample_rate_hz": 24000,
}
class _LegacyPlugin:
silence_continuation_samples = 8000
def test_the_plugin_unit_is_used_verbatim() -> None:
unit = warmup_silence_unit(_Plugin())
assert unit["format"] == "pcm_f32le"
assert unit["sample_rate_hz"] == 24000
assert len(base64.b64decode(str(unit["audio"]))) == 1920 * 4
def test_a_plugin_without_the_hook_falls_back_to_16k_zeros_of_its_sample_count() -> None:
unit = warmup_silence_unit(_LegacyPlugin())
assert unit["sample_rate_hz"] == 16000
assert unit["format"] == "pcm_f32le"
assert len(base64.b64decode(str(unit["audio"]))) == 8000 * 4
def test_no_plugin_means_the_minicpm_sized_default() -> None:
unit = warmup_silence_unit(None)
assert unit["sample_rate_hz"] == 16000
assert len(base64.b64decode(str(unit["audio"]))) == 16000 * 4
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
@@ -0,0 +1,63 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
import numpy as np
import pytest
from vllm_omni.model_executor.common.audio.pcm import (
PCM_F32LE_BYTES_PER_SAMPLE,
decode_pcm_f32le_base64,
pcm_f32le_sample_count,
pcm_f32le_samples,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _b64(samples: np.ndarray) -> str:
return base64.b64encode(np.ascontiguousarray(samples, dtype="<f4").tobytes()).decode("ascii")
def test_decode_returns_the_raw_bytes_of_whole_finite_samples() -> None:
samples = np.arange(4, dtype=np.float32)
raw = decode_pcm_f32le_base64(_b64(samples))
assert raw == samples.tobytes()
assert pcm_f32le_sample_count(raw) == 4
assert PCM_F32LE_BYTES_PER_SAMPLE == 4
def test_decode_accepts_an_empty_chunk() -> None:
assert decode_pcm_f32le_base64("") == b""
assert pcm_f32le_sample_count(b"") == 0
@pytest.mark.parametrize(
("encoded", "match"),
[
(None, "must be base64 pcm_f32le"),
(b"bytes", "must be base64 pcm_f32le"),
("not*base64", "not valid base64"),
(base64.b64encode(b"\x00\x00\x00").decode("ascii"), "divisible by four"),
(_b64(np.array([0.0, np.nan], dtype=np.float32)), "must be finite"),
(_b64(np.array([np.inf], dtype=np.float32)), "must be finite"),
],
)
def test_decode_rejects_malformed_audio_naming_the_model(encoded: object, match: str) -> None:
with pytest.raises(ValueError, match=match) as excinfo:
decode_pcm_f32le_base64(encoded, model="TestModel")
assert str(excinfo.value).startswith("TestModel ")
def test_samples_are_a_writable_contiguous_float32_copy() -> None:
raw = np.arange(3, dtype="<f4").tobytes()
samples = pcm_f32le_samples(raw)
assert samples.dtype == np.float32
assert samples.flags.writeable and samples.flags.c_contiguous
assert samples.tolist() == [0.0, 1.0, 2.0]
@@ -0,0 +1,2 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
@@ -0,0 +1,104 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from types import SimpleNamespace
import numpy as np
import pytest
from vllm_omni.engine.duplex.contracts import DuplexFence, duplex_resource_request_id
from vllm_omni.engine.duplex.plugin import DuplexDataPlane, DuplexDataPlaneContext
from vllm_omni.model_executor.common.duplex.data_plane import CumulativeAudioTextDataPlane
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _encode_audio(audio, _sample_rate, _response_format, _speed):
if audio is None:
return None
size = int(np.asarray(audio, dtype=np.float32).size)
return f"audio-{size}" if size else None
def _output(request_id: str, *, text: str, samples: int, sr: int = 24000) -> SimpleNamespace:
return SimpleNamespace(
request_id=request_id,
outputs=[SimpleNamespace(text=text, multimodal_output={})],
multimodal_output={"model_outputs": np.arange(samples, dtype=np.float32), "sr": sr},
finished=False,
)
class _TestDataPlane(CumulativeAudioTextDataPlane):
stage_role = "codec"
default_sample_rate_hz = 16000
def test_is_a_duplex_data_plane_and_projects_cumulative_output_as_deltas() -> None:
plane = CumulativeAudioTextDataPlane(_encode_audio)
plane.begin_request("req")
context = DuplexDataPlaneContext(response_format="wav", modalities=("audio", "text"))
projected = list(
plane.project(
{"data_plane_outputs": [_output("req", text="he", samples=4), _output("req", text="hello", samples=6)]},
context=context,
)
)
assert isinstance(plane, DuplexDataPlane)
assert [item["text"] for item in projected] == ["he", "llo"]
assert [item["audio_data"] for item in projected] == ["audio-4", "audio-2"]
assert [item["audio_duration_ms"] for item in projected] == [0, 0]
assert all(item["sample_rate_hz"] == 24000 for item in projected)
assert all(item["data_plane_request_id"] == "req" for item in projected)
assert all(item["is_listen"] is False and item["end_of_turn"] is False for item in projected)
assert projected[0]["stage_role"] == "tts"
assert projected[0]["runtime_impl"] == "scheduler_data_plane"
assert projected[0]["audio_format"] == "wav"
def test_subclass_constants_and_default_sample_rate_reach_the_projection() -> None:
plane = _TestDataPlane(_encode_audio)
output = SimpleNamespace(
request_id="req",
outputs=[SimpleNamespace(text="", multimodal_output={})],
multimodal_output={"audio": np.zeros(160, dtype=np.float32)},
finished=False,
)
[item] = list(plane.project({"data_plane_outputs": [output]}))
assert item["stage_role"] == "codec"
assert item["sample_rate_hz"] == 16000
assert item["audio_duration_ms"] == 10
def test_nothing_new_yields_nothing_and_non_results_are_ignored() -> None:
plane = CumulativeAudioTextDataPlane(_encode_audio)
same = _output("req", text="hi", samples=4)
assert len(list(plane.project({"data_plane_outputs": [same, same]}))) == 1
assert list(plane.project("not a dict")) == []
assert list(plane.project({"data_plane_outputs": None})) == []
def test_terminal_and_close_bookkeeping() -> None:
plane = CumulativeAudioTextDataPlane(_encode_audio)
session_request = duplex_resource_request_id(DuplexFence("sid", epoch=0), "stage0")
plane.begin_request(session_request)
plane.begin_request("other-req")
assert plane.is_terminal(None) is False
assert plane.is_terminal(session_request) is False
plane.mark_terminal(session_request)
assert plane.is_terminal(session_request) is True
plane.begin_request(session_request)
assert plane.is_terminal(session_request) is False
plane.close_session("sid", active_request_id="other-req")
assert plane._requests == {}
plane.mark_terminal("late")
plane.close_stream("late")
assert plane.is_terminal("late") is False
@@ -0,0 +1,60 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
import numpy as np
import pytest
from vllm_omni.model_executor.common.duplex.payload import (
decode_pcm_f32le_payload,
payload_audio,
payload_sample_count,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _payload(samples: int, *, sample_rate_hz: int = 24000, fmt: str = "pcm_f32le", key: str = "audio") -> dict:
audio = base64.b64encode(np.zeros(samples, dtype="<f4").tobytes()).decode("ascii")
return {"type": "audio", "format": fmt, "sample_rate_hz": sample_rate_hz, key: audio}
def test_decode_returns_the_pcm_of_a_matching_payload() -> None:
raw = decode_pcm_f32le_payload(_payload(1920), sample_rate_hz=24000, exact_samples=1920)
assert len(raw) == 1920 * 4
def test_decode_accepts_the_older_data_key() -> None:
assert len(decode_pcm_f32le_payload(_payload(8, key="data"), sample_rate_hz=24000)) == 32
assert payload_audio({"data": "x"}) == "x"
assert payload_audio("not a mapping") is None
@pytest.mark.parametrize(
("payload", "match"),
[
("frame", "must be a mapping"),
(_payload(1920, fmt="pcm16"), "format must be pcm_f32le"),
(_payload(1920, sample_rate_hz=16000), "sample_rate_hz must be 24000"),
(_payload(1919), "exactly 1920 samples"),
({"format": "pcm_f32le", "sample_rate_hz": 24000, "audio": 12}, "must be base64"),
],
)
def test_decode_rejects_payloads_that_do_not_match_the_model_unit(payload: object, match: str) -> None:
with pytest.raises(ValueError, match=match) as excinfo:
decode_pcm_f32le_payload(payload, sample_rate_hz=24000, exact_samples=1920, model="TestModel")
assert str(excinfo.value).startswith("TestModel ")
def test_decode_without_exact_samples_accepts_any_whole_number_of_samples() -> None:
assert len(decode_pcm_f32le_payload(_payload(7), sample_rate_hz=24000)) == 28
def test_payload_sample_count_is_lenient() -> None:
assert payload_sample_count(_payload(16000, sample_rate_hz=16000)) == 16000
assert payload_sample_count(_payload(16000, fmt="pcm16")) is None
assert payload_sample_count({"format": "pcm_f32le", "audio": "not*base64"}) is None
assert payload_sample_count(None) is None
@@ -0,0 +1,185 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
import numpy as np
import pytest
from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation
from vllm_omni.model_executor.common.duplex.pcm_buffer import (
FixedFramePcmAppendBuffer,
FixedFramePcmAppendReservation,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
FRAME = 1920
RATE = 24000
PERIOD = 80
def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = RATE, fmt: str = "pcm_f32le") -> dict[str, object]:
samples = np.ascontiguousarray(samples, dtype="<f4")
return {
"type": "audio",
"format": fmt,
"sample_rate_hz": sample_rate_hz,
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
}
def _buffer() -> FixedFramePcmAppendBuffer:
return FixedFramePcmAppendBuffer(sample_rate_hz=RATE, frame_samples=FRAME, chunk_period_ms=PERIOD, model="Test")
def _decode(payload: dict[str, object]) -> np.ndarray:
return np.frombuffer(base64.b64decode(str(payload["audio"])), dtype="<f4")
def test_buffer_implements_the_framework_contracts() -> None:
buffer = _buffer()
assert isinstance(buffer, PcmAppendBuffer)
assert buffer.frame_bytes == FRAME * 4
reservation = buffer.prepare_append(
_pcm_payload(np.arange(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True
)
assert isinstance(reservation, FixedFramePcmAppendReservation)
assert isinstance(reservation, PcmAppendReservation)
def test_one_whole_frame_is_reserved_transactionally() -> None:
buffer = _buffer()
reservation = buffer.prepare_append(
_pcm_payload(np.arange(FRAME)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True
)
assert reservation is not None
assert reservation.active
assert reservation.byte_count == FRAME * 4
assert reservation.payload is not None and reservation.payload["sample_rate_hz"] == RATE
assert _decode(reservation.payload).tolist() == list(range(FRAME))
assert buffer.pending_byte_count == 0
assert buffer.has_reserved() and not buffer.has_pending()
reservation.rollback()
assert not reservation.active
assert buffer.pending_byte_count == FRAME * 4
assert buffer.has_pending() and not buffer.has_reserved()
def test_partial_chunks_accumulate_until_a_whole_frame_exists() -> None:
buffer = _buffer()
half = _pcm_payload(np.zeros(FRAME // 2))
assert buffer.prepare_append(half, operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True) is None
assert buffer.pending_byte_count == FRAME * 2
reservation = buffer.prepare_append(half, operation_id="op-2", chunk_period_ms=PERIOD, allow_emit=True)
assert reservation is not None
assert reservation.byte_count == FRAME * 4
assert buffer.pending_byte_count == 0
def test_appends_are_not_emitted_while_emission_is_disallowed() -> None:
buffer = _buffer()
assert (
buffer.prepare_append(
_pcm_payload(np.zeros(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=False
)
is None
)
assert buffer.pending_byte_count == FRAME * 4
def test_commit_flushes_a_padded_final_frame_and_an_empty_buffer_reserves_nothing() -> None:
buffer = _buffer()
buffer.prepare_append(_pcm_payload(np.ones(10)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=False)
reservation = buffer.prepare_commit(operation_id="commit", chunk_period_ms=PERIOD)
assert reservation.payload is not None
assert reservation.payload["final"] is True
decoded = _decode(reservation.payload)
assert decoded.size == FRAME
assert decoded[:10].tolist() == [1.0] * 10 and not decoded[10:].any()
assert reservation.byte_count == 40
reservation.commit()
assert not buffer.has_reserved()
empty = buffer.prepare_commit(operation_id="commit-2", chunk_period_ms=PERIOD)
assert empty.payload is None and empty.byte_count == 0
empty.commit()
assert buffer.flush(chunk_period_ms=PERIOD) is None
def test_rolling_back_an_earlier_reservation_restores_later_ones_in_order() -> None:
buffer = _buffer()
first = buffer.prepare_append(
_pcm_payload(np.full(FRAME, 1.0)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True
)
second = buffer.prepare_append(
_pcm_payload(np.full(FRAME, 2.0)), operation_id="op-2", chunk_period_ms=PERIOD, allow_emit=True
)
assert first is not None and second is not None
first.rollback()
assert not first.active and not second.active
assert buffer.pending_byte_count == 2 * FRAME * 4
replay = buffer.prepare_append(
_pcm_payload(np.zeros(0)), operation_id="op-3", chunk_period_ms=PERIOD, allow_emit=True
)
assert replay is not None and _decode(replay.payload)[0] == 1.0
def test_clear_drops_pending_audio_and_deactivates_reservations() -> None:
buffer = _buffer()
reservation = buffer.prepare_append(
_pcm_payload(np.zeros(FRAME + 8)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True
)
assert reservation is not None and buffer.pending_byte_count == 32
buffer.clear()
buffer.clear_force_listen()
assert not reservation.active
assert buffer.pending_byte_count == 0
reservation.rollback() # no-op after clear
assert buffer.pending_byte_count == 0
@pytest.mark.parametrize(
("payload", "match"),
[
(_pcm_payload(np.zeros(8), sample_rate_hz=16000), "sample_rate_hz must be 24000"),
(_pcm_payload(np.zeros(8), fmt="pcm16"), "format must be pcm_f32le"),
(_pcm_payload(np.array([0.0, np.nan])), "finite"),
],
)
def test_malformed_audio_is_rejected_and_nothing_is_buffered(payload: dict[str, object], match: str) -> None:
buffer = _buffer()
with pytest.raises(ValueError, match=match):
buffer.prepare_append(payload, operation_id="op", chunk_period_ms=PERIOD, allow_emit=True)
assert buffer.pending_byte_count == 0
def test_chunk_period_and_duplicate_operations_are_refused() -> None:
buffer = _buffer()
with pytest.raises(ValueError, match="chunk_period_ms must be 80"):
buffer.prepare_append(_pcm_payload(np.zeros(8)), operation_id="op", chunk_period_ms=1000, allow_emit=True)
buffer.prepare_append(_pcm_payload(np.zeros(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True)
with pytest.raises(ValueError, match="duplicate active operation_id"):
buffer.prepare_append(_pcm_payload(np.zeros(8)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True)
def test_constructor_validates_its_parameters() -> None:
with pytest.raises(ValueError, match="positive"):
FixedFramePcmAppendBuffer(sample_rate_hz=0, frame_samples=FRAME, chunk_period_ms=PERIOD)
@@ -0,0 +1,139 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from types import SimpleNamespace
import numpy as np
import pytest
import torch
from vllm_omni.model_executor.common.request_outputs import (
audio_sample_count,
audio_value,
coerce_int,
coerce_int_list,
first_completion,
multimodal_output,
sample_rate_hz,
slice_audio_delta,
text_delta,
text_value,
unwrap_request_output,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def test_unwrap_prefers_the_wrapped_request_output_and_its_first_completion() -> None:
completion = SimpleNamespace(text="hi", multimodal_output={})
inner = SimpleNamespace(request_id="req", outputs=[completion, SimpleNamespace(text="second")])
wrapper = SimpleNamespace(request_output=inner)
output, first = unwrap_request_output(wrapper)
assert output is inner
assert first is completion
assert first_completion(SimpleNamespace(outputs=[])) is None
assert first_completion(SimpleNamespace()) is None
def test_multimodal_output_takes_the_first_non_empty_mapping_as_a_copy() -> None:
completion = SimpleNamespace(multimodal_output={"audio": [1.0]})
output = SimpleNamespace(multimodal_output={})
result = multimodal_output(output, completion)
assert result == {"audio": [1.0]}
assert result is not completion.multimodal_output
assert multimodal_output(SimpleNamespace(multimodal_output=None), None) == {}
@pytest.mark.parametrize(
("value", "expected"),
[
(4, 4),
("11", 11),
(2.9, 2),
(True, 1),
(None, None),
("x", None),
(object(), None),
([1], None),
(torch.tensor([7]), 7),
(torch.tensor(9), 9),
(torch.tensor([], dtype=torch.long), None),
(np.array([5], dtype=np.int32), 5),
(np.array([], dtype=np.int32), None),
],
)
def test_coerce_int_handles_scalars_strings_tensors_and_arrays(value: object, expected: int | None) -> None:
assert coerce_int(value) == expected
def test_coerce_int_list_flattens_tensors_and_drops_non_integers() -> None:
assert coerce_int_list(torch.tensor([[1, 2], [3, 4]])) == [1, 2, 3, 4]
assert coerce_int_list(np.array([5, 6])) == [5, 6]
assert coerce_int_list([1, "2", "x", None, torch.tensor(3)]) == [1, 2, 3]
assert coerce_int_list(None) == []
assert coerce_int_list("12") == []
def test_audio_value_reads_the_known_keys_and_unwraps_a_single_item_list() -> None:
assert audio_value({"model_outputs": [np.zeros(2)]}).shape == (2,)
assert audio_value({"latent": "x"}) == "x"
assert audio_value({"audio": [1, 2]}) == [1, 2]
assert audio_value({}) is None
def test_text_value_prefers_multimodal_text_then_llm_text_then_the_completion() -> None:
completion = SimpleNamespace(text="from completion")
assert text_value({"text": "mm"}, completion) == "mm"
assert text_value({"llm_output_text": "llm"}, completion) == "llm"
assert text_value({}, completion) == "from completion"
assert text_value({"text": ""}, None) == ""
@pytest.mark.parametrize(
("text", "previous", "expected"),
[("", "he", ""), ("he", "he", ""), ("hello", "he", "llo"), ("new", "old", "new"), ("abc", "", "abc")],
)
def test_text_delta(text: str, previous: str, expected: str) -> None:
assert text_delta(text, previous) == expected
def test_slice_audio_delta_returns_only_new_samples_for_cumulative_audio() -> None:
cumulative = np.arange(6, dtype=np.float32)
assert slice_audio_delta(cumulative, 4).tolist() == [4.0, 5.0]
assert slice_audio_delta(cumulative, 6) is None
assert slice_audio_delta(cumulative, 0).tolist() == cumulative.tolist()
# A shorter waveform than the cursor means the stream restarted: send it whole.
assert slice_audio_delta(np.arange(3, dtype=np.float32), 4).tolist() == [0.0, 1.0, 2.0]
assert slice_audio_delta(None, 2) is None
assert slice_audio_delta(np.zeros(0, dtype=np.float32), 0) is None
def test_slice_audio_delta_keeps_tensors_as_contiguous_tensors() -> None:
delta = slice_audio_delta(torch.arange(6, dtype=torch.float32).reshape(2, 3), 4)
assert isinstance(delta, torch.Tensor)
assert delta.is_contiguous()
assert delta.tolist() == [4.0, 5.0]
assert audio_sample_count(torch.zeros(2, 3)) == 6
assert audio_sample_count("not audio") is None
@pytest.mark.parametrize(
("multimodal", "expected"),
[
({"sr": 16000}, 16000),
({"sample_rate_hz": 22050}, 22050),
({"sr": [48000]}, 48000),
({"sr": np.array([8000])}, 8000),
({"sr": "bad"}, 24000),
({"sr": 0}, 24000),
({}, 24000),
],
)
def test_sample_rate_hz_falls_back_to_the_default(multimodal: dict, expected: int) -> None:
assert sample_rate_hz(multimodal, default=24000) == expected
@@ -1,21 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""PersonaPlex duplex is not on the duplex plugin framework yet.
RFC vllm-omni#7181 splits the port: this PR lands the unified framework with
MiniCPM-o 4.5, and the follow-up PR moves PersonaPlex onto ``DuplexModelPlugin``.
Until then the modules under test still import the pre-framework duplex
runtime / serving adapter that this PR removed, so collecting them would fail
at import time and abort the whole pytest session. The tests themselves are
unchanged and come back with the port.
Listed per file rather than globbed, so the coverage loss is visible and shrinks
file by file as the port lands.
"""
#: ``test_unified_runtime`` imports ``engine.duplex.runtime`` and
#: ``entrypoints.duplex.runtime_adapter`` directly. ``test_stage0_runtime`` fails
#: transitively: ``personaplex.duplex.runtime_extension`` imports ``DuplexFence``
#: from ``engine.duplex.messages``, which no longer exports it.
collect_ignore = ["test_stage0_runtime.py", "test_unified_runtime.py"]
@@ -0,0 +1,489 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""PersonaPlex on the unified full-duplex framework: the plugin contract and the E2E driver helpers."""
from __future__ import annotations
import argparse
import base64
from pathlib import Path
from types import SimpleNamespace
from typing import cast
import numpy as np
import pytest
from vllm.sampling_params import SamplingParams
from tests.e2e.online_serving import personaplex_realtime_duplex as e2e_driver
from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy
from vllm_omni.engine.duplex.config import DuplexSessionConfig
from vllm_omni.engine.duplex.contracts import DuplexFence
from vllm_omni.engine.duplex.plugin import (
DefaultDuplexModelSessionState,
DuplexDataPlaneContext,
DuplexRuntimeConfigError,
load_duplex_plugin,
validate_duplex_plugin_sampling,
)
from vllm_omni.model_executor.models.personaplex.duplex import stage0
from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA, FRAME_SIZE, SAMPLE_RATE
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import PersonaPlexDataPlaneSession
from vllm_omni.model_executor.models.personaplex.duplex.input import PersonaPlexPcmAppendBuffer
from vllm_omni.model_executor.models.personaplex.duplex.plugin import (
PRIVATE_RUNTIME_CONFIG_KEYS,
PersonaPlexDuplexPlugin,
PersonaPlexSessionState,
)
from vllm_omni.model_executor.models.personaplex.pipeline import PERSONAPLEX_PIPELINE
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
DEPLOY_PATH = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml"
def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = SAMPLE_RATE) -> dict[str, object]:
samples = np.ascontiguousarray(samples, dtype="<f4")
return {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": sample_rate_hz,
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
}
def _encode_audio(audio, _sample_rate, _response_format, _speed):
if audio is None:
return None
size = int(np.asarray(audio, dtype=np.float32).size)
return f"audio-{size}" if size else None
def _plugin() -> PersonaPlexDuplexPlugin:
return PersonaPlexDuplexPlugin(_encode_audio)
# --------------------------------------------------------------------------- #
# Pipeline binding #
# --------------------------------------------------------------------------- #
def test_pipeline_declares_the_duplex_plugin_and_no_pre_framework_wiring() -> None:
assert PERSONAPLEX_PIPELINE.duplex_plugin == (
"vllm_omni.model_executor.models.personaplex.duplex.plugin.PersonaPlexDuplexPlugin"
)
assert PERSONAPLEX_PIPELINE.duplex_runtime_extension is None
assert PERSONAPLEX_PIPELINE.duplex_serving_adapter is None
assert PERSONAPLEX_PIPELINE.duplex_control_enabled is False
plugin = load_duplex_plugin(PERSONAPLEX_PIPELINE.duplex_plugin, _encode_audio)
assert isinstance(plugin, PersonaPlexDuplexPlugin)
assert plugin.plugin_id == "personaplex"
assert isinstance(plugin.data_plane, PersonaPlexDataPlaneSession)
validate_duplex_plugin_sampling(plugin, sampling_defaults=(SamplingParams(), SamplingParams()))
@pytest.mark.parametrize(
("engine_arg", "expected"),
[("skip_tokenizer_init", True), ("enable_prefix_caching", False)],
)
def test_personaplex_stage_engine_args(engine_arg: str, expected: bool) -> None:
stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, load_deploy_config(DEPLOY_PATH))
assert [stage.yaml_engine_args.get(engine_arg) for stage in stages] == [expected, expected]
def test_personaplex_deploy_is_duplex_and_propagates_capacity_to_all_model_stages() -> None:
deploy = load_deploy_config(DEPLOY_PATH)
stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, deploy)
assert deploy.session_mode == "duplex"
assert deploy.duplex_session.max_sessions == 2
assert [stage.yaml_engine_args.get("duplex_max_sessions") for stage in stages] == [2, 2]
assert "personaplex_codec_max_sessions" not in deploy.connectors["connector_of_shared_memory"]["extra"]
# --------------------------------------------------------------------------- #
# Session policy #
# --------------------------------------------------------------------------- #
def test_capabilities_are_honest() -> None:
single = _plugin().capabilities(max_sessions=1)
multi = _plugin().capabilities(max_sessions=2)
assert multi.as_dict()["implementation_level"] == "model_native_duplex"
assert multi.as_dict()["input_modes"] == ["append_audio_chunk"]
assert multi.chunk_period_ms == 80
assert single.supports_multi_session is False
assert multi.supports_multi_session is True
assert multi.supports_multi_session_same_replica is True
assert multi.supports_barge_in is False
assert multi.supports_client_commit is False
assert multi.supports_external_turn_signal is False
assert multi.supports_session_resume is False
assert multi.supports_audio_truncate is False
assert multi.supports_chat_completions is False
assert multi.session_admission_mode == "engine_managed"
def test_session_state_is_the_framework_default_with_the_personaplex_buffer() -> None:
state = _plugin().create_session_state()
assert isinstance(state, PersonaPlexSessionState)
assert isinstance(state, DefaultDuplexModelSessionState)
assert isinstance(state.audio_buffer, PersonaPlexPcmAppendBuffer)
state.retain_committed_audio({"audio": ""}, operation_id="op", reserved_bytes=8)
assert state.clear_committed_audio() == 8
assert state.committed_audio_payload is None
def test_private_runtime_keys_are_rejected_in_extra_body() -> None:
plugin = _plugin()
assert PRIVATE_RUNTIME_CONFIG_KEYS == {
"personaplex_prefill_slots",
"personaplex_model_path",
"personaplex_voice_prompt",
"personaplex_persona",
}
plugin.validate_client_extra_body({"auto_response": True})
plugin.validate_client_extra_body(None)
with pytest.raises(DuplexRuntimeConfigError, match="personaplex_persona") as excinfo:
plugin.validate_client_extra_body({"personaplex_persona": "x", "auto_response": True})
assert excinfo.value.code == "invalid_duplex_runtime_config"
@pytest.mark.asyncio
async def test_prepare_runtime_config_resolves_voice_and_prefill_slots_once(monkeypatch) -> None:
calls: list[tuple[str, str, str]] = []
def fake_prefill_slots(model_path: str, voice: str, persona: str) -> int:
calls.append((model_path, voice, persona))
return 42
monkeypatch.setattr(stage0, "personaplex_prefill_slots", fake_prefill_slots)
plugin = _plugin()
model_config = SimpleNamespace(model="/models/personaplex")
runtime_config = await plugin.prepare_runtime_config(
DuplexSessionConfig(voice="NATM1.pt", instructions="Be brief."),
model_config=model_config,
)
again = await plugin.prepare_runtime_config(
DuplexSessionConfig(voice="NATM1.pt", instructions="Be brief."),
model_config=model_config,
)
defaults = await plugin.prepare_runtime_config(DuplexSessionConfig(), model_config=model_config)
assert runtime_config == {
"personaplex_model_path": "/models/personaplex",
"personaplex_voice_prompt": "NATM1.pt",
"personaplex_persona": "Be brief.",
"personaplex_prefill_slots": 42,
}
assert again == runtime_config
assert defaults["personaplex_voice_prompt"] == "NATF2.pt"
assert defaults["personaplex_persona"] == DEFAULT_PERSONA
# The voice bundle and tokenizer are read once per (model, voice, persona).
assert calls == [
("/models/personaplex", "NATM1.pt", "Be brief."),
("/models/personaplex", "NATF2.pt", DEFAULT_PERSONA),
]
@pytest.mark.asyncio
async def test_prepare_runtime_config_reports_prefill_failures_as_runtime_config_errors(monkeypatch) -> None:
def failing_prefill_slots(model_path: str, voice: str, persona: str) -> int:
raise FileNotFoundError("voices.tgz missing")
monkeypatch.setattr(stage0, "personaplex_prefill_slots", failing_prefill_slots)
with pytest.raises(DuplexRuntimeConfigError, match="voices.tgz missing") as excinfo:
await _plugin().prepare_runtime_config(DuplexSessionConfig(), model_config=SimpleNamespace(model="/m"))
assert excinfo.value.code == "prefill_unavailable"
with pytest.raises(DuplexRuntimeConfigError, match="model path") as excinfo:
await _plugin().prepare_runtime_config(DuplexSessionConfig(), model_config=None)
assert excinfo.value.code == "model_path_unavailable"
@pytest.mark.parametrize("voice", ["../NATF2.pt", "voices/NATF2.pt", "NATF2.wav", "/abs/NATF2.pt"])
@pytest.mark.asyncio
async def test_prepare_runtime_config_rejects_voices_that_are_not_bundled_basenames(voice: str) -> None:
with pytest.raises(DuplexRuntimeConfigError, match="bundled .pt basename") as excinfo:
await _plugin().prepare_runtime_config(
DuplexSessionConfig(voice=voice), model_config=SimpleNamespace(model="/m")
)
assert excinfo.value.code == "invalid_voice"
def test_runtime_config_update_rejects_changed_persona() -> None:
plugin = _plugin()
short_persona = "You are a helpful assistant."
current = {"personaplex_persona": short_persona, "personaplex_voice_prompt": "NATF2.pt"}
with pytest.raises(DuplexRuntimeConfigError, match="persona") as excinfo:
plugin.runtime_config_for_update(DuplexSessionConfig(instructions="You are a pirate."), current)
assert excinfo.value.code == "persona_update_unsupported"
unchanged = plugin.runtime_config_for_update(DuplexSessionConfig(instructions=short_persona), current)
assert unchanged["personaplex_persona"] == short_persona
assert unchanged is not current
def test_runtime_config_update_rejects_changed_voice() -> None:
plugin = _plugin()
current = {"personaplex_persona": DEFAULT_PERSONA, "personaplex_voice_prompt": "NATF2.pt"}
with pytest.raises(DuplexRuntimeConfigError, match="voice") as excinfo:
plugin.runtime_config_for_update(DuplexSessionConfig(voice="NATM1.pt"), current)
assert excinfo.value.code == "voice_update_unsupported"
unchanged = plugin.runtime_config_for_update(DuplexSessionConfig(voice="NATF2.pt"), current)
assert unchanged["personaplex_voice_prompt"] == "NATF2.pt"
def test_data_plane_context_is_the_framework_default() -> None:
context = _plugin().data_plane_context(
epoch=1,
turn_id=2,
active_response_turn_id=None,
active_response_id="resp",
auto_responds=True,
response_format="pcm16",
speed=None,
modalities=("audio",),
)
assert isinstance(context, DuplexDataPlaneContext)
assert context.response_format == "pcm16" and context.epoch == 1
# --------------------------------------------------------------------------- #
# Engine policy #
# --------------------------------------------------------------------------- #
def test_sampling_params_are_greedy_one_token_on_stage0_only() -> None:
defaults = (SamplingParams(temperature=0.8, top_k=10, max_tokens=10), SamplingParams(max_tokens=1024))
configured = _plugin().configure_sampling_params(runtime_config={}, defaults=defaults)
assert configured[0].temperature == 0.0
assert configured[0].top_k == 1
assert configured[0].max_tokens == 1
assert configured[1] is defaults[1]
assert defaults[0].max_tokens == 10
assert _plugin().configure_sampling_params(runtime_config={}, defaults=()) == ()
def test_plan_append_reserves_one_slot_plus_prefill_on_the_first_seq() -> None:
plugin = _plugin()
fence = DuplexFence("session", epoch=3, turn_id=1)
common = {
"request_id": "req",
"fence": fence,
"session_config": {"instructions": "Be concise."},
"runtime_config": {"personaplex_prefill_slots": 4},
"turn_seq": 1,
"payload": _pcm_payload(np.zeros(FRAME_SIZE, np.float32)),
"final": False,
"sampling_params": SamplingParams(),
}
first = plugin.plan_append(seq=1, **common)
second = plugin.plan_append(seq=2, **common)
assert len(first.prompt["prompt_token_ids"]) == 5
assert len(second.prompt["prompt_token_ids"]) == 1
duplex = first.prompt["model_intermediate_buffer"]["duplex"]
assert duplex["session_id"] == "session"
assert duplex["epoch"] == 3
assert duplex["turn_id"] == 1
assert duplex["seq"] == 1
assert duplex["data_plane"] is True
assert duplex["fence"] == fence
assert duplex["scheduler_token_budget"] == 5
assert duplex["runtime_config"] == {"personaplex_prefill_slots": 4}
assert "incarnation" not in duplex
assert first.prompt["model_intermediate_buffer"]["global_request_id"] == ["session"]
@pytest.mark.parametrize(
("payload", "match"),
[
(_pcm_payload(np.zeros(FRAME_SIZE - 1, np.float32)), "exactly 1920 samples"),
(_pcm_payload(np.zeros(FRAME_SIZE, np.float32), sample_rate_hz=16000), "sample_rate_hz must be 24000"),
({"format": "pcm16", "sample_rate_hz": SAMPLE_RATE, "audio": ""}, "format must be pcm_f32le"),
("frame", "must be a mapping"),
],
)
def test_plan_append_rejects_anything_but_one_24k_frame(payload: object, match: str) -> None:
with pytest.raises(ValueError, match=match):
_plugin().plan_append(
request_id="req",
fence=DuplexFence("session"),
session_config={},
runtime_config={},
seq=1,
turn_seq=1,
payload=payload,
final=False,
sampling_params=None,
)
def test_plan_append_rejects_a_malformed_prefill_slot_count() -> None:
with pytest.raises(ValueError, match="personaplex_prefill_slots"):
_plugin().plan_append(
request_id="req",
fence=DuplexFence("session"),
session_config={},
runtime_config={"personaplex_prefill_slots": "many"},
seq=1,
turn_seq=1,
payload=_pcm_payload(np.zeros(FRAME_SIZE, np.float32)),
final=False,
sampling_params=None,
)
def test_decide_output_never_decides() -> None:
assert (
_plugin().decide_output(
stage_id=0,
final_stage_id=1,
segment_finished=True,
segment_token_ids=(1,),
segment_output_metadata={},
output=SimpleNamespace(outputs=[]),
)
is None
)
def test_silence_unit_is_one_24k_frame_that_plan_append_accepts() -> None:
plugin = _plugin()
unit = plugin.silence_unit_payload()
assert unit["format"] == "pcm_f32le"
assert unit["sample_rate_hz"] == SAMPLE_RATE
assert len(base64.b64decode(unit["audio"])) == FRAME_SIZE * 4
plan = plugin.plan_append(
request_id="req",
fence=DuplexFence("session"),
session_config={},
runtime_config={},
seq=7,
turn_seq=7,
payload=unit,
final=False,
sampling_params=None,
)
assert len(plan.prompt["prompt_token_ids"]) == 1
# --------------------------------------------------------------------------- #
# E2E driver helpers (CPU-checked parts of the GPU driver) #
# --------------------------------------------------------------------------- #
def _audio_client(frame_count: int = 10, *, voiced_frames: int = 0) -> e2e_driver.RawRealtimeProbe:
"""A probe double carrying only the collected events the driver's result builder reads."""
frames = np.full((frame_count, e2e_driver.FRAME_SAMPLES), 7, dtype="<i2")
frames[:voiced_frames] = 1000
raw = frames.tobytes()
event = {
"type": "response.output_audio.delta",
"response_id": "response-1",
"sample_rate_hz": e2e_driver.SAMPLE_RATE_HZ,
"metadata": {
"vllm_omni": {
"runtime_impl": "scheduler_data_plane",
"uses_model_runner_scheduler": True,
"runner_kv_backed": True,
}
},
}
events = SimpleNamespace(events=[event], response_audio={"response-1": [raw]}, audio_bytes=lambda: raw)
return cast(e2e_driver.RawRealtimeProbe, SimpleNamespace(events=events))
def test_realtime_audio_frame_stats_separate_voiced_and_silent_frames() -> None:
silent = np.zeros(e2e_driver.FRAME_SAMPLES, dtype="<i2")
voiced = np.full(e2e_driver.FRAME_SAMPLES, 1000, dtype="<i2")
stats = e2e_driver._audio_frame_stats(
silent.tobytes() + voiced.tobytes(),
input_frames=3,
voiced_frame_rms_threshold=1e-4,
)
assert stats == {
"output_frames": 2,
"voiced_frames": 1,
"silent_frames": 1,
"frame_deficit": 1,
"frame_coverage_ratio": pytest.approx(2 / 3),
}
def test_realtime_audio_frame_stats_reject_partial_codec_frame() -> None:
partial_frame = np.zeros(e2e_driver.FRAME_SAMPLES - 1, dtype="<i2")
with pytest.raises(AssertionError, match="whole 80 ms codec frames"):
e2e_driver._audio_frame_stats(partial_frame.tobytes(), input_frames=1, voiced_frame_rms_threshold=1e-4)
@pytest.mark.parametrize("frame_count", [10, 40])
def test_e2e_driver_rejects_inaudible_sessions(frame_count: int) -> None:
args = argparse.Namespace(max_frame_deficit=4, voiced_frame_rms_threshold=1e-3, min_voiced_frames=5)
with pytest.raises(AssertionError, match="audible speech"):
e2e_driver._session_result(_audio_client(frame_count), input_frames=frame_count, args=args, minimum_chunks=1)
def test_e2e_driver_uses_absolute_audible_floor() -> None:
args = argparse.Namespace(max_frame_deficit=4, voiced_frame_rms_threshold=1e-3, min_voiced_frames=5)
e2e_driver._session_result(_audio_client(200, voiced_frames=5), input_frames=200, args=args, minimum_chunks=1)
def test_e2e_driver_defaults_replacement_to_full_input() -> None:
args = e2e_driver.parse_args(["--model", "/model", "--input-wav", "/input.wav"])
assert args.replacement_frames == 0
assert args.min_voiced_frames == 5
def test_e2e_driver_records_and_validates_input_sha256(tmp_path: Path) -> None:
input_wav = tmp_path / "input.wav"
input_wav.write_bytes(b"fixture-wav")
identity = e2e_driver._input_identity(
input_wav,
expected_sha256="bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
)
assert identity == {
"path": str(input_wav.resolve()),
"sha256": "bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
}
def test_e2e_driver_rejects_unexpected_input_sha256(tmp_path: Path) -> None:
input_wav = tmp_path / "input.wav"
input_wav.write_bytes(b"fixture-wav")
with pytest.raises(ValueError, match="input WAV SHA-256 mismatch"):
e2e_driver._input_identity(input_wav, expected_sha256="0" * 64)
def test_e2e_driver_lets_the_server_allocate_the_session_id() -> None:
url = e2e_driver._realtime_url("ws://127.0.0.1:8099/v1/realtime?duplex=1", "nvidia/personaplex-7b-v1")
assert "session_id=" not in url
assert "duplex=1" in url and "autostart=0" in url
@@ -14,6 +14,7 @@ from vllm_omni.model_executor.models.personaplex.duplex.policy import (
)
from vllm_omni.model_executor.models.personaplex.duplex.stage0 import (
PersonaPlexStage0DuplexRuntime,
PersonaPlexStage0PreparedAppend,
)
from vllm_omni.model_executor.models.personaplex.personaplex_talker import (
PersonaPlexTalkerForConditionalGeneration,
@@ -80,13 +81,12 @@ class _FakeTalker:
return torch.full((1, 4), value, device=device)
def _duplex_info(*, seq: int, session_id: str = "session", incarnation: int = 1):
def _duplex_info(*, seq: int, session_id: str = "session", epoch: int = 0):
pcm = np.zeros(1920, dtype="<f4")
return {
"data_plane": True,
"session_id": session_id,
"incarnation": incarnation,
"epoch": 0,
"epoch": epoch,
"seq": seq,
"payload": {
"format": "pcm_f32le",
@@ -103,7 +103,7 @@ def _duplex_info(*, seq: int, session_id: str = "session", incarnation: int = 1)
def _runtime(*codecs: _FakeCodec) -> PersonaPlexStage0DuplexRuntime:
voice_embeddings = torch.arange(8, dtype=torch.float32).reshape(2, 1, 1, 4)
codec_args = {"codec": codecs[0]}
codec_args: dict[str, object] = {"codec": codecs[0]}
if len(codecs) > 1:
available = iter(codecs)
codec_args = {
@@ -248,15 +248,20 @@ def test_decoded_pcm_is_writable_for_torch_zero_copy() -> None:
def _prepare_two_sessions(
runtime: PersonaPlexStage0DuplexRuntime,
) -> tuple[object, object, object, object]:
) -> tuple[
PersonaPlexStage0PreparedAppend,
PersonaPlexStage0PreparedAppend,
PersonaPlexStage0PreparedAppend,
PersonaPlexStage0PreparedAppend,
]:
first_1 = runtime.prepare_append(_duplex_info(seq=1), prompt_len=18)
second_1 = runtime.prepare_append(
_duplex_info(seq=1, session_id="other", incarnation=2),
_duplex_info(seq=1, session_id="other"),
prompt_len=18,
)
first_2 = runtime.prepare_append(_duplex_info(seq=2), prompt_len=19)
second_2 = runtime.prepare_append(
_duplex_info(seq=2, session_id="other", incarnation=2),
_duplex_info(seq=2, session_id="other"),
prompt_len=19,
)
return first_1, second_1, first_2, second_2
@@ -281,7 +286,7 @@ def test_stage0_session_capacity_fails_before_codec_state_is_shared() -> None:
with pytest.raises(RuntimeError, match="capacity 2"):
runtime.prepare_append(
_duplex_info(seq=1, session_id="third", incarnation=3),
_duplex_info(seq=1, session_id="third"),
prompt_len=18,
)
@@ -292,12 +297,55 @@ def test_close_session_resets_and_reuses_released_codec() -> None:
runtime = _runtime(first_codec, second_codec)
_prepare_two_sessions(runtime)
runtime.close_session("session", 1)
runtime.close_session("session", 0)
assert first_codec.reset_calls == 1
assert second_codec.reset_calls == 0
replacement = runtime.prepare_append(
_duplex_info(seq=1, session_id="replacement", incarnation=4),
_duplex_info(seq=1, session_id="replacement"),
prompt_len=18,
)
assert replacement.user_codes[:, 0].tolist() == [3]
def test_a_new_epoch_replays_the_prefill_and_recycles_the_codec() -> None:
codec = _FakeCodec()
runtime = _runtime(codec)
runtime.prepare_append(_duplex_info(seq=1), prompt_len=18, request_id="req-e0")
runtime.prepare_append(_duplex_info(seq=2), prompt_len=19, request_id="req-e0")
# A cancel advanced the fence: the next append is seq 1 of epoch 1 on a
# fresh Stage 0 request, so the voice/persona prefill is replayed and the
# earlier epoch's lockstep state is released first.
restarted = runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18, request_id="req-e1")
assert restarted.prefill_applied is True
assert restarted.prompt_offset == 0
assert restarted.user_codes.shape == (1, 8)
assert list(runtime.sessions) == [("session", 1)]
assert runtime.request_sessions == {"req-e1": ("session", 1)}
assert codec.reset_calls == 1
assert runtime.sessions[("session", 1)].codec is codec
def test_a_late_finish_of_the_old_epoch_request_does_not_close_the_new_state() -> None:
runtime = _runtime(_FakeCodec())
runtime.prepare_append(_duplex_info(seq=1), prompt_len=18, request_id="req-e0")
runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18, request_id="req-e1")
runtime.close_request("req-e0")
assert list(runtime.sessions) == [("session", 1)]
runtime.close_request("req-e1")
assert runtime.sessions == {}
def test_stage0_capacity_counts_live_epochs_not_superseded_ones() -> None:
runtime = _runtime(_FakeCodec(), _FakeCodec())
runtime.prepare_append(_duplex_info(seq=1), prompt_len=18)
runtime.prepare_append(_duplex_info(seq=1, session_id="other"), prompt_len=18)
# Restarting one session must not need a third codec.
runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18)
assert sorted(runtime.sessions) == [("other", 0), ("session", 1)]
@@ -1,420 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import base64
from pathlib import Path
from types import SimpleNamespace
import numpy as np
import pytest
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.runtime import (
DuplexInputMode,
)
from vllm_omni.entrypoints.duplex.runtime_adapter import (
ServingRuntimeConfigError,
)
from tests.e2e.online_serving import personaplex_realtime_duplex as e2e_driver
from vllm_omni.config.stage_config import (
load_deploy_config,
merge_pipeline_deploy,
)
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import (
PersonaPlexDataPlaneContext,
PersonaPlexDataPlaneSession,
)
from vllm_omni.model_executor.models.personaplex.duplex.input import (
PersonaPlexPcmAppendBuffer,
)
from vllm_omni.model_executor.models.personaplex.duplex.runtime_extension import (
PersonaPlexDuplexRuntimeExtension,
)
from vllm_omni.model_executor.models.personaplex.duplex.serving_adapter import (
PersonaPlexServingRuntimeAdapter,
)
from vllm_omni.model_executor.models.personaplex.pipeline import (
PERSONAPLEX_PIPELINE,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = 24000) -> dict[str, object]:
samples = np.ascontiguousarray(samples, dtype="<f4")
return {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": sample_rate_hz,
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
}
def _audio_client(
frame_count: int = 10,
*,
voiced_frames: int = 0,
) -> SimpleNamespace:
frames = np.full((frame_count, e2e_driver.FRAME_SAMPLES), 7, dtype="<i2")
frames[:voiced_frames] = 1000
raw = frames.tobytes()
event = {
"type": "response.output_audio.delta",
"response_id": "response-1",
"sample_rate_hz": e2e_driver.SAMPLE_RATE_HZ,
"metadata": {
"vllm_omni": {
"runtime_impl": "scheduler_data_plane",
"uses_model_runner_scheduler": True,
"runner_kv_backed": True,
}
},
}
events = SimpleNamespace(
events=[event],
response_audio={"response-1": [raw]},
audio_bytes=lambda: raw,
)
return SimpleNamespace(events=events)
def test_realtime_audio_frame_stats_separate_voiced_and_silent_frames() -> None:
silent = np.zeros(e2e_driver.FRAME_SAMPLES, dtype="<i2")
voiced = np.full(e2e_driver.FRAME_SAMPLES, 1000, dtype="<i2")
stats = e2e_driver._audio_frame_stats(
silent.tobytes() + voiced.tobytes(),
input_frames=3,
voiced_frame_rms_threshold=1e-4,
)
assert stats == {
"output_frames": 2,
"voiced_frames": 1,
"silent_frames": 1,
"frame_deficit": 1,
"frame_coverage_ratio": pytest.approx(2 / 3),
}
def test_realtime_audio_frame_stats_reject_partial_codec_frame() -> None:
partial_frame = np.zeros(e2e_driver.FRAME_SAMPLES - 1, dtype="<i2")
with pytest.raises(AssertionError, match="whole 80 ms codec frames"):
e2e_driver._audio_frame_stats(
partial_frame.tobytes(),
input_frames=1,
voiced_frame_rms_threshold=1e-4,
)
@pytest.mark.parametrize("frame_count", [10, 40])
def test_e2e_driver_rejects_inaudible_sessions(frame_count: int) -> None:
args = SimpleNamespace(
max_frame_deficit=4,
voiced_frame_rms_threshold=1e-3,
min_voiced_frames=5,
)
with pytest.raises(AssertionError, match="audible speech"):
e2e_driver._session_result(
_audio_client(frame_count),
input_frames=frame_count,
args=args,
minimum_chunks=1,
)
def test_e2e_driver_uses_absolute_audible_floor() -> None:
args = SimpleNamespace(
max_frame_deficit=4,
voiced_frame_rms_threshold=1e-3,
min_voiced_frames=5,
)
e2e_driver._session_result(
_audio_client(200, voiced_frames=5),
input_frames=200,
args=args,
minimum_chunks=1,
)
def test_e2e_driver_defaults_replacement_to_full_input() -> None:
args = e2e_driver.parse_args(
[
"--model",
"/model",
"--input-wav",
"/input.wav",
]
)
assert args.replacement_frames == 0
assert args.min_voiced_frames == 5
def test_e2e_driver_records_and_validates_input_sha256(tmp_path: Path) -> None:
input_wav = tmp_path / "input.wav"
input_wav.write_bytes(b"fixture-wav")
identity = e2e_driver._input_identity(
input_wav,
expected_sha256="bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
)
assert identity == {
"path": str(input_wav.resolve()),
"sha256": "bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
}
def test_e2e_driver_rejects_unexpected_input_sha256(tmp_path: Path) -> None:
input_wav = tmp_path / "input.wav"
input_wav.write_bytes(b"fixture-wav")
with pytest.raises(ValueError, match="input WAV SHA-256 mismatch"):
e2e_driver._input_identity(input_wav, expected_sha256="0" * 64)
@pytest.mark.parametrize(
("attribute", "expected"),
[
(
"duplex_runtime_extension",
"vllm_omni.model_executor.models.personaplex.duplex.runtime_extension.PersonaPlexDuplexRuntimeExtension",
),
(
"duplex_serving_adapter",
"vllm_omni.model_executor.models.personaplex.duplex.serving_adapter.PersonaPlexServingRuntimeAdapter",
),
],
)
def test_personaplex_pipeline_registers_unified_duplex_components(
attribute: str,
expected: str,
) -> None:
assert PERSONAPLEX_PIPELINE.duplex_control_enabled is True
assert getattr(PERSONAPLEX_PIPELINE, attribute) == expected
@pytest.mark.parametrize(
("engine_arg", "expected"),
[
("skip_tokenizer_init", True),
("enable_prefix_caching", False),
],
)
def test_personaplex_stage_engine_args(
engine_arg: str,
expected: bool,
) -> None:
deploy_path = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml"
stages = merge_pipeline_deploy(
PERSONAPLEX_PIPELINE,
load_deploy_config(deploy_path),
)
assert [stage.yaml_engine_args.get(engine_arg) for stage in stages] == [
expected,
expected,
]
def test_personaplex_duplex_capacity_is_propagated_to_all_model_stages() -> None:
deploy_path = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml"
deploy = load_deploy_config(deploy_path)
stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, deploy)
assert deploy.duplex_session.max_sessions == 2
assert [stage.yaml_engine_args.get("duplex_max_sessions") for stage in stages] == [2, 2]
assert "personaplex_codec_max_sessions" not in deploy.connectors["connector_of_shared_memory"]["extra"]
def test_personaplex_capabilities_are_honest() -> None:
single = PersonaPlexServingRuntimeAdapter(lambda *_: None).capabilities(max_sessions=1)
multi = PersonaPlexServingRuntimeAdapter(lambda *_: None).capabilities(max_sessions=2)
assert multi.implementation_level == "model_native_duplex"
assert multi.input_modes == ["append_audio_chunk"]
assert multi.chunk_period_ms == 80
assert single.supports_multi_session is False
assert multi.supports_multi_session is True
assert multi.supports_multi_session_same_replica is True
assert multi.supports_barge_in is False
def test_personaplex_runtime_config_update_rejects_changed_persona() -> None:
adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
short_persona = "You are a helpful assistant."
longer_persona = (
"You are now a swashbuckling pirate captain who speaks entirely in "
"nautical slang, references buried treasure constantly, and never "
"breaks character no matter what the user asks."
)
current = {"personaplex_persona": short_persona, "personaplex_voice_prompt": "NATF2.pt"}
with pytest.raises(ServingRuntimeConfigError, match="persona"):
adapter.runtime_config_for_update(
SimpleNamespace(instructions=longer_persona, extra_body={}),
current,
)
unchanged = adapter.runtime_config_for_update(
SimpleNamespace(instructions=short_persona, extra_body={}),
current,
)
assert unchanged["personaplex_persona"] == short_persona
def test_personaplex_runtime_config_update_rejects_changed_voice() -> None:
adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
current = {"personaplex_persona": DEFAULT_PERSONA, "personaplex_voice_prompt": "NATF2.pt"}
with pytest.raises(ServingRuntimeConfigError, match="voice"):
adapter.runtime_config_for_update(
SimpleNamespace(instructions=None, voice="OtherVoice.pt", extra_body={}),
current,
)
unchanged = adapter.runtime_config_for_update(
SimpleNamespace(instructions=None, voice="NATF2.pt", extra_body={}),
current,
)
assert unchanged["personaplex_voice_prompt"] == "NATF2.pt"
def test_personaplex_rejects_native_duplex_opt_out_flag() -> None:
# PersonaPlex is always model-native; the per-session opt-in knob is
# server-owned there, under both the canonical name and its deprecated
# model-prefixed alias.
adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
with pytest.raises(ServingRuntimeConfigError, match="native_duplex"):
adapter.validate_client_extra_body({"native_duplex": False})
with pytest.raises(ServingRuntimeConfigError, match="minicpmo45_native_duplex"):
adapter.validate_client_extra_body({"minicpmo45_native_duplex": False})
def test_pcm_buffer_emits_one_80ms_frame_transactionally() -> None:
buffer = PersonaPlexPcmAppendBuffer()
reservation = buffer.prepare_append(
_pcm_payload(np.arange(1920, dtype=np.float32)),
operation_id="op-1",
chunk_period_ms=80,
allow_emit=True,
)
assert reservation is not None
assert reservation.byte_count == 1920 * 4
assert buffer.pending_byte_count == 0
reservation.rollback()
assert buffer.pending_byte_count == 1920 * 4
def test_pcm_buffer_rejects_changed_sample_rate() -> None:
buffer = PersonaPlexPcmAppendBuffer()
assert (
buffer.prepare_append(
_pcm_payload(np.zeros(100, np.float32)),
operation_id="op-1",
chunk_period_ms=80,
allow_emit=False,
)
is None
)
with pytest.raises(ValueError, match="sample_rate"):
buffer.prepare_append(
_pcm_payload(np.zeros(100, np.float32), sample_rate_hz=16000),
operation_id="op-2",
chunk_period_ms=80,
allow_emit=False,
)
def test_pcm_buffer_rejects_non_finite_samples() -> None:
buffer = PersonaPlexPcmAppendBuffer()
with pytest.raises(ValueError, match="finite"):
buffer.prepare_append(
_pcm_payload(np.array([0.0, np.nan], np.float32)),
operation_id="op-1",
chunk_period_ms=80,
allow_emit=False,
)
def test_runtime_extension_builds_one_frame_scheduler_append() -> None:
extension = PersonaPlexDuplexRuntimeExtension()
defaults = (
SamplingParams(temperature=0.8, top_k=10, max_tokens=10),
SamplingParams(max_tokens=1024),
)
configured = extension.configure_sampling_params(runtime_config={}, defaults=defaults)
assert configured[0].temperature == 0.0
assert configured[0].top_k == 1
assert configured[0].max_tokens == 1
assert configured[1] is defaults[1]
plan = extension.plan_append(
request_id="req",
fence=DuplexFence("session", incarnation=2, epoch=3),
session_config={"instructions": "Be concise."},
runtime_config={"personaplex_prefill_slots": 4},
seq=1,
turn_seq=1,
mode=DuplexInputMode.APPEND_AUDIO_CHUNK,
payload=_pcm_payload(np.zeros(1920, np.float32)),
final=False,
sampling_params=configured[0],
)
assert len(plan.prompt["prompt_token_ids"]) == 5
duplex = plan.prompt["model_intermediate_buffer"]["duplex"]
assert duplex["session_id"] == "session"
assert duplex["incarnation"] == 2
assert duplex["epoch"] == 3
assert duplex["seq"] == 1
assert duplex["data_plane"] is True
def test_projector_emits_only_cumulative_audio_and_text_suffixes() -> None:
encoded_sizes: list[int] = []
def encode_audio(audio, _sample_rate, _response_format, _speed):
size = int(np.asarray(audio, dtype=np.float32).size)
encoded_sizes.append(size)
return f"audio-{size}"
projector = PersonaPlexDataPlaneSession(encode_audio)
projector.begin_request("req")
context = PersonaPlexDataPlaneContext(response_format="wav")
first = SimpleNamespace(
request_id="req",
outputs=[SimpleNamespace(text="he", multimodal_output={})],
multimodal_output={"model_outputs": np.arange(4, dtype=np.float32), "sr": 24000},
finished=False,
)
second = SimpleNamespace(
request_id="req",
outputs=[SimpleNamespace(text="hello", multimodal_output={})],
multimodal_output={"model_outputs": np.arange(6, dtype=np.float32), "sr": 24000},
finished=False,
)
projected = list(
projector.project(
{"data_plane_outputs": [first, second]},
context=context,
)
)
assert encoded_sizes == [4, 2]
assert [item["text"] for item in projected] == ["he", "llo"]
assert [item["audio_data"] for item in projected] == ["audio-4", "audio-2"]
assert all(item["sample_rate_hz"] == 24000 for item in projected)
assert all(item["end_of_turn"] is False for item in projected)
+4 -4
View File
@@ -324,10 +324,10 @@ class PipelineConfig:
duplex_plugin: str | None = None
# Preserve legacy turn deployments when adding an optional duplex plugin.
default_session_mode: str | None = None
# Legacy duplex wiring of the models that are not ported to the plugin
# framework yet (PersonaPlex, Nemotron VoiceChat). Nothing reads them: a
# pipeline that only declares these is served turn-based. Each field goes
# away with the follow-up PR that ports its model to ``duplex_plugin``.
# Legacy duplex wiring of the model that is not ported to the plugin
# framework yet (Nemotron VoiceChat, RFC vllm-omni#7181 PR 4). Nothing
# reads them: a pipeline that only declares these is served turn-based.
# The fields go away with the PR that ports it to ``duplex_plugin``.
duplex_runtime_extension: str | None = None
duplex_serving_adapter: str | None = None
duplex_control_enabled: bool = False
@@ -37,6 +37,7 @@ from vllm_omni.diffusion.models.qwen_image.qwen_image_transformer import (
QwenImageTransformer2DModel,
)
from vllm_omni.diffusion.models.qwen_image.rope_utils import txt_seq_lens_from_embeds
from vllm_omni.diffusion.offloader.config import OffloadStrategy, resolve_offload_strategy
from vllm_omni.diffusion.profiler.diffusion_pipeline_profiler import DiffusionPipelineProfilerMixin
from vllm_omni.diffusion.utils.prompt_utils import (
validate_prompt_sequence_lengths,
@@ -341,7 +342,7 @@ class QwenImagePipeline(
# do not share VRAM with DiT construction (#7555). The DiT follows the
# loader's default-device context (CUDA for online / AutoRound INT under
# offload, CPU for layerwise / unquantized HSDP defer).
cpu_offload = bool(getattr(self.od_config, "enable_cpu_offload", False))
cpu_offload = resolve_offload_strategy(self.od_config) is OffloadStrategy.MODEL_LEVEL
enc_vae_device = torch.device("cpu") if cpu_offload else self.device
self.text_encoder = self.text_encoder.to(enc_vae_device)
self.vae = from_pretrained_with_prefetch(
+61 -4
View File
@@ -16,9 +16,12 @@ import asyncio
from abc import ABC, abstractmethod
from collections.abc import Callable, Iterable, Mapping
from dataclasses import dataclass
from functools import lru_cache
from importlib import import_module
from typing import TYPE_CHECKING
import pybase64 as base64
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.contracts import (
DuplexAppendPlan,
@@ -219,6 +222,20 @@ class DefaultDuplexModelSessionState(DuplexModelSessionState):
self.silence_deadline_monotonic = None
@dataclass(frozen=True, slots=True)
class DuplexDataPlaneContext:
"""Session state the runner hands a data plane to project one stage output."""
epoch: int = 0
turn_id: int = 0
active_response_turn_id: int | None = None
active_response_id: str | None = None
auto_responds: bool = False
response_format: str = "wav"
speed: float | None = None
modalities: tuple[str, ...] = ()
class DuplexDataPlane(ABC):
"""Projects raw stage outputs of one model into internal duplex events."""
@@ -277,12 +294,29 @@ class DuplexModelPlugin(ABC):
private_runtime_config_keys: frozenset[str] = frozenset()
#: Samples per silence unit the runner appends to keep a model turn going.
silence_continuation_samples: int = 16000
#: Sample rate of that unit: the runner submits it through ``plan_append``
#: exactly like client audio, so it must be a unit the model accepts.
silence_continuation_sample_rate_hz: int = 16000
data_plane: DuplexDataPlane
def __init__(self, encode_audio: EncodeAudio) -> None:
# Constructor-only: concrete plugins hand the encoder to their data plane.
del encode_audio
def silence_unit_payload(self) -> dict[str, object]:
"""One silence unit as an append payload (``pcm_f32le`` zeros).
Used by the runner's turn continuation and by the startup warmup; a
model whose unit is not plain zero PCM overrides it.
"""
samples = int(self.silence_continuation_samples)
return {
"type": "audio",
"audio": _silence_pcm_f32le_base64(samples),
"format": "pcm_f32le",
"sample_rate_hz": int(self.silence_continuation_sample_rate_hz),
}
# ---- engine policy (was DuplexRuntimeExtension) ----
@abstractmethod
@@ -430,8 +464,15 @@ class DuplexModelPlugin(ABC):
@abstractmethod
def capabilities(self, *, max_sessions: int) -> DuplexCapabilities: ...
@abstractmethod
def validate_client_extra_body(self, extra_body: object) -> None: ...
def validate_client_extra_body(self, extra_body: object) -> None:
"""Refuse client ``extra_body`` keys the server owns (``private_runtime_config_keys``)."""
if not isinstance(extra_body, Mapping):
return
private_keys = sorted(self.private_runtime_config_keys.intersection(extra_body))
if private_keys:
raise DuplexRuntimeConfigError(
f"{self.plugin_id} runtime configuration is server-owned: " + ", ".join(private_keys)
)
@abstractmethod
async def prepare_runtime_config(
@@ -445,7 +486,6 @@ class DuplexModelPlugin(ABC):
current: Mapping[str, object],
) -> dict[str, object]: ...
@abstractmethod
def data_plane_context(
self,
*,
@@ -457,7 +497,18 @@ class DuplexModelPlugin(ABC):
response_format: str,
speed: float | None,
modalities: tuple[str, ...],
) -> object: ...
) -> object:
"""The context handed to ``data_plane.project``; the default is the generic dataclass."""
return DuplexDataPlaneContext(
epoch=epoch,
turn_id=turn_id,
active_response_turn_id=active_response_turn_id,
active_response_id=active_response_id,
auto_responds=auto_responds,
response_format=response_format,
speed=speed,
modalities=modalities,
)
# Optional hook: build the runtime config patch for a function-call output
# item. Plugins without tools keep the default (no change).
@@ -515,6 +566,11 @@ def validate_duplex_plugin_sampling(plugin: DuplexModelPlugin, *, sampling_defau
)
@lru_cache(maxsize=8)
def _silence_pcm_f32le_base64(samples: int) -> str:
return base64.b64encode(bytes(max(0, samples) * 4)).decode("ascii")
def payload_turn_id(payload: object) -> int | None:
if not isinstance(payload, Mapping):
return None
@@ -537,6 +593,7 @@ def coerce_int(value: object) -> int | None:
__all__ = [
"DefaultDuplexModelSessionState",
"DuplexDataPlane",
"DuplexDataPlaneContext",
"DuplexModelPlugin",
"DuplexModelSessionState",
"DuplexRuntimeConfigError",
+8 -1
View File
@@ -121,7 +121,14 @@ class SessionEmitter:
# ------------------------------------------------------------------ #
def auto_responds(self) -> bool:
"""Whether the session answers committed input without a ``response.create``."""
"""Whether the session answers committed input without a ``response.create``.
A model that takes no client commits (``supports_client_commit`` off, a
lockstep model) can only auto-respond; for every other model the
client opts in through ``extra_body.auto_response``.
"""
if not self._ctx.session.capabilities.supports_client_commit:
return True
extra = getattr(self._ctx.session.config, "extra_body", None)
if not isinstance(extra, dict):
return False
@@ -29,7 +29,6 @@ from collections.abc import Awaitable, Callable, Mapping
from dataclasses import replace
from typing import TYPE_CHECKING, Protocol
import pybase64 as base64
from vllm.logger import init_logger
from vllm_omni.engine.duplex.config import DuplexSessionState
@@ -83,7 +82,6 @@ class ModelChannel:
"""Appends out to the model, events back from it, for one session."""
# One MiniCPM model unit (1 s at 16 kHz) is the compatibility default.
_SILENCE_UNIT_PAYLOAD_AUDIO = base64.b64encode(bytes(16000 * 4)).decode("ascii")
_RESPONSE_MAX_CONTINUATION_UNITS = 8
_AUTO_RESPONSE_MAX_CONTINUATION_UNITS = 64
@@ -1210,13 +1208,8 @@ class ModelChannel:
# ------------------------------------------------------------------ #
def silence_unit_payload(self) -> dict[str, object]:
samples = int(self._ctx.plugin.silence_continuation_samples)
audio = (
self._SILENCE_UNIT_PAYLOAD_AUDIO
if samples == 16000
else base64.b64encode(bytes(samples * 4)).decode("ascii")
)
return {"type": "audio", "audio": audio, "format": "pcm_f32le", "sample_rate_hz": 16000}
"""One silence unit in the model's own format: the plugin defines it (rate and length)."""
return dict(self._ctx.plugin.silence_unit_payload())
def response_continuations_remaining(self, response_id: str) -> bool:
model_state = self._ctx.model_state
+43 -5
View File
@@ -94,6 +94,26 @@ def _warmup_jpeg_b64() -> str:
return base64.b64encode(buf.getvalue()).decode("ascii")
_DEFAULT_SILENCE_SAMPLES = 16000
_DEFAULT_SILENCE_SAMPLE_RATE_HZ = 16000
def warmup_silence_unit(plugin: object | None) -> dict[str, object]:
"""The append the silent-frames warmup sends: the plugin's silence unit, or the 16 kHz default."""
factory = getattr(plugin, "silence_unit_payload", None)
if callable(factory):
unit = dict(factory())
if isinstance(unit.get("audio"), str) and isinstance(unit.get("format"), str):
unit.setdefault("sample_rate_hz", _DEFAULT_SILENCE_SAMPLE_RATE_HZ)
return unit
samples = int(getattr(plugin, "silence_continuation_samples", _DEFAULT_SILENCE_SAMPLES))
return {
"audio": base64.b64encode(bytes(samples * 4)).decode("ascii"),
"format": "pcm_f32le",
"sample_rate_hz": _DEFAULT_SILENCE_SAMPLE_RATE_HZ,
}
async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
"""Run one throwaway realtime session before real clients are admitted.
@@ -121,9 +141,14 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
model_name = served
else:
model_name = args.model
# One silence unit as the engine-side plugin defines it (DuplexOmniEngine.plugin).
frame_samples = int(getattr(plugin, "silence_continuation_samples", 16000))
native_append = plugin.capabilities(max_sessions=1).supports_core_resumable_request if plugin else True
# One silence unit as the engine-side plugin defines it (DuplexOmniEngine.plugin):
# the unit's format, rate and length are the model's, not a fixed 16 kHz.
unit = warmup_silence_unit(plugin)
frame_samples = len(base64.b64decode(str(unit["audio"]))) // 4
capabilities = getattr(plugin, "capabilities", None)
native_append = (
bool(capabilities(max_sessions=1).supports_core_resumable_request) if callable(capabilities) else True
)
from vllm_omni.clients.duplex import build_realtime_url
url = (
@@ -163,6 +188,10 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
"turn_detection": None,
"extra_body": {"auto_response": native_append},
}
if kind != "video_turn":
session["input_audio_format"] = unit["format"]
session["sample_rate_hz"] = unit["sample_rate_hz"]
session["audio"] = {"input": {"sample_rate_hz": unit["sample_rate_hz"]}}
if kind == "video_turn":
# The default AURA prompt answers with <|silent|> unless the user
# asked for something. This session is thrown away; force one short
@@ -205,9 +234,17 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
)
sent = 1
else:
silence = base64.b64encode(bytes(frame_samples * 4)).decode("ascii")
while sent < warmup_frames:
await ws.send(json.dumps({"type": "input_audio_buffer.append", "audio": silence}))
await ws.send(
json.dumps(
{
"type": "input_audio_buffer.append",
"audio": unit["audio"],
"format": unit["format"],
"sample_rate_hz": unit["sample_rate_hz"],
}
)
)
sent += 1
await asyncio.sleep(0.08)
if not native_append:
@@ -251,4 +288,5 @@ __all__ = [
"_warmup_duplex_realtime",
"lookup_duplex_plugin",
"startup_warmup_kind",
"warmup_silence_unit",
]
@@ -0,0 +1,8 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Model-neutral helpers shared between model packages.
``audio`` and ``request_outputs`` carry no duplex vocabulary and may be used
from any model. ``duplex`` builds on the duplex framework contracts and is
imported only from a model's ``duplex`` package.
"""
@@ -0,0 +1,3 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Generic audio helpers shared between model packages."""
@@ -0,0 +1,54 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Little-endian float32 PCM (``pcm_f32le``) helpers.
Plain audio plumbing every model that takes raw PCM needs: decode a base64
chunk, check it is whole finite samples, count and materialise them. No
duplex vocabulary lives here.
"""
from __future__ import annotations
import binascii
import numpy as np
import pybase64 as base64
PCM_F32LE_BYTES_PER_SAMPLE = 4
def decode_pcm_f32le_base64(encoded: object, *, model: str = "audio") -> bytes:
"""Decode base64 ``pcm_f32le`` into raw bytes, refusing partial or non-finite samples.
``model`` names the caller in error messages so a client sees which model
rejected its audio.
"""
if not isinstance(encoded, str):
raise ValueError(f"{model} audio must be base64 pcm_f32le")
try:
raw = base64.b64decode(encoded, validate=True)
except (binascii.Error, ValueError) as exc:
raise ValueError(f"{model} audio is not valid base64") from exc
if len(raw) % PCM_F32LE_BYTES_PER_SAMPLE:
raise ValueError(f"{model} pcm_f32le byte length must be divisible by four")
samples = np.frombuffer(raw, dtype="<f4")
if samples.size and not bool(np.isfinite(samples).all()):
raise ValueError(f"{model} pcm_f32le samples must be finite")
return raw
def pcm_f32le_sample_count(raw: bytes) -> int:
return len(raw) // PCM_F32LE_BYTES_PER_SAMPLE
def pcm_f32le_samples(raw: bytes) -> np.ndarray:
"""The samples as a writable, contiguous float32 array (safe for zero-copy tensor views)."""
return np.ascontiguousarray(np.frombuffer(raw, dtype="<f4"), dtype=np.float32).copy()
__all__ = [
"PCM_F32LE_BYTES_PER_SAMPLE",
"decode_pcm_f32le_base64",
"pcm_f32le_sample_count",
"pcm_f32le_samples",
]
@@ -0,0 +1,8 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Shared building blocks for ``DuplexModelPlugin`` implementations.
Everything here depends on the duplex framework contracts
(``vllm_omni.engine.duplex.plugin``), so it is imported only from a model's
``duplex`` package, never from the turn-based stack.
"""
@@ -0,0 +1,135 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Data plane for models whose final stage streams cumulative audio and text.
The stage runner re-sends the whole waveform (and transcript) it has produced
so far for one request; the session needs only what is new. This data plane
keeps one cursor per request, slices the delta, encodes it for the wire and
reports it as one internal ``response.output_audio.delta``-shaped result.
Model-specific constants (stage role, runtime metadata, default rate) are
class attributes for a subclass to set.
"""
from __future__ import annotations
from collections.abc import Callable, Iterator
from dataclasses import dataclass
from vllm_omni.engine.duplex.contracts import duplex_resource_request_belongs_to_session
from vllm_omni.engine.duplex.plugin import DuplexDataPlane, DuplexDataPlaneContext
from vllm_omni.model_executor.common.request_outputs import (
audio_sample_count,
audio_value,
multimodal_output,
sample_rate_hz,
slice_audio_delta,
text_delta,
text_value,
unwrap_request_output,
)
EncodeAudio = Callable[[object, int, str, float | None], str | None]
@dataclass(slots=True)
class _RequestCursor:
audio_samples: int = 0
text: str = ""
terminal: bool = False
class CumulativeAudioTextDataPlane(DuplexDataPlane):
"""Project cumulative staged audio/text output into per-request deltas."""
stage_role: str = "tts"
runtime_impl: str = "scheduler_data_plane"
uses_model_runner_scheduler: bool = True
runner_kv_backed: bool = True
owned_runtime: bool = False
default_sample_rate_hz: int = 24000
def __init__(self, encode_audio: EncodeAudio) -> None:
self._encode_audio = encode_audio
self._requests: dict[str, _RequestCursor] = {}
# ---- request / session bookkeeping ----
def begin_request(self, request_id: str) -> None:
self._requests.setdefault(request_id, _RequestCursor()).terminal = False
def is_terminal(self, request_id: str | None) -> bool:
if request_id is None:
return False
state = self._requests.get(request_id)
return state is not None and state.terminal
def mark_terminal(self, request_id: str) -> None:
self._requests.setdefault(request_id, _RequestCursor()).terminal = True
def close_stream(self, request_id: str) -> None:
self._requests.pop(request_id, None)
def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None:
if active_request_id is not None:
self._requests.pop(active_request_id, None)
for request_id in list(self._requests):
if duplex_resource_request_belongs_to_session(request_id, session_id):
self._requests.pop(request_id, None)
# ---- projection ----
def project(self, result: object, *, context: object | None = None) -> Iterator[dict[str, object]]:
if not isinstance(result, dict):
return
outputs = result.get("data_plane_outputs")
if not isinstance(outputs, list):
return
if not isinstance(context, DuplexDataPlaneContext):
context = DuplexDataPlaneContext(response_format="wav", modalities=("audio", "text"))
for output in outputs:
projected = self._project_output(output, context=context)
if projected is not None:
yield projected
def _project_output(self, output: object, *, context: DuplexDataPlaneContext) -> dict[str, object] | None:
output, completion = unwrap_request_output(output)
request_id = getattr(output, "request_id", None)
if not isinstance(request_id, str) or not request_id:
request_id = None
state = self._requests.setdefault(request_id, _RequestCursor()) if request_id is not None else _RequestCursor()
multimodal = multimodal_output(output, completion)
audio = audio_value(multimodal)
audio_delta = slice_audio_delta(audio, state.audio_samples)
total_samples = audio_sample_count(audio)
if total_samples is not None:
state.audio_samples = total_samples
text = text_value(multimodal, completion)
delta_text = text_delta(text, state.text)
if text:
state.text = text
rate = sample_rate_hz(multimodal, default=self.default_sample_rate_hz)
encoded = self._encode_audio(audio_delta, rate, context.response_format, context.speed)
if not encoded and not delta_text:
return None
delta_samples = audio_sample_count(audio_delta) or 0
return {
"supported": True,
"stage_role": self.stage_role,
"is_listen": False,
"data_plane_request_id": request_id,
"text": delta_text,
"audio_data": encoded or "",
"audio_format": context.response_format,
"sample_rate_hz": rate,
"audio_duration_ms": round(delta_samples * 1000 / max(1, rate)),
"end_of_turn": False,
"uses_model_runner_scheduler": self.uses_model_runner_scheduler,
"runner_kv_backed": self.runner_kv_backed,
"runtime_impl": self.runtime_impl,
"owned_runtime": self.owned_runtime,
}
__all__ = ["CumulativeAudioTextDataPlane"]
@@ -0,0 +1,63 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Validation of one duplex audio append payload.
The runner hands a plugin the append as a mapping ``{format, sample_rate_hz,
audio}`` (``audio`` base64 ``pcm_f32le``). Models with a fixed input rate and
frame size validate it here; the PCM itself is decoded by
``vllm_omni.model_executor.common.audio.pcm``.
"""
from __future__ import annotations
from collections.abc import Mapping
from vllm_omni.model_executor.common.audio.pcm import (
decode_pcm_f32le_base64,
pcm_f32le_sample_count,
)
def payload_audio(payload: object) -> object | None:
"""The encoded audio field of an append payload (``audio``, or the older ``data``)."""
if not isinstance(payload, Mapping):
return None
return payload.get("audio") or payload.get("data")
def decode_pcm_f32le_payload(
payload: object,
*,
sample_rate_hz: int,
exact_samples: int | None = None,
model: str = "duplex",
) -> bytes:
"""Raw PCM of one append that must be ``pcm_f32le`` at ``sample_rate_hz``.
``exact_samples`` pins the unit length (a lockstep model takes exactly one
frame per append); ``None`` accepts any whole number of samples.
"""
if not isinstance(payload, Mapping):
raise ValueError(f"{model} duplex append payload must be a mapping")
if payload.get("format") != "pcm_f32le":
raise ValueError(f"{model} duplex append format must be pcm_f32le")
if payload.get("sample_rate_hz") != sample_rate_hz:
raise ValueError(f"{model} duplex append sample_rate_hz must be {sample_rate_hz}")
raw = decode_pcm_f32le_base64(payload_audio(payload), model=model)
if exact_samples is not None and pcm_f32le_sample_count(raw) != exact_samples:
raise ValueError(f"{model} duplex append must contain exactly {exact_samples} samples")
return raw
def payload_sample_count(payload: object) -> int | None:
"""Sample count of a ``pcm_f32le`` append, or ``None`` when it is not one (lenient, for budgeting)."""
if not isinstance(payload, Mapping) or payload.get("format") != "pcm_f32le":
return None
try:
raw = decode_pcm_f32le_base64(payload_audio(payload))
except ValueError:
return None
return pcm_f32le_sample_count(raw)
__all__ = ["decode_pcm_f32le_payload", "payload_audio", "payload_sample_count"]
@@ -0,0 +1,194 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Transactional framing of client PCM into fixed model units.
A frame-locked model (PersonaPlex: 1920 samples every 80 ms at 24 kHz) takes
exactly one frame per append. Clients send arbitrary chunks, so the session
runner buffers them here and takes one whole frame out at a time as a
*reservation*: committed once the stage accepted the append, rolled back
(the bytes go back to the front of the buffer) when it did not.
"""
from __future__ import annotations
import pybase64 as base64
from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation
from vllm_omni.model_executor.common.audio.pcm import (
PCM_F32LE_BYTES_PER_SAMPLE,
decode_pcm_f32le_base64,
)
class FixedFramePcmAppendReservation(PcmAppendReservation):
__slots__ = ("_active", "_owner", "_raw", "operation_id", "payload")
def __init__(
self,
*,
owner: FixedFramePcmAppendBuffer,
operation_id: str,
payload: dict[str, object] | None,
raw: bytes,
) -> None:
self._owner = owner
self.operation_id = operation_id
self.payload = payload
self._raw = raw
self._active = True
@property
def active(self) -> bool:
return self._active
@property
def byte_count(self) -> int:
return len(self._raw)
def commit(self) -> None:
self._owner._commit_reservation(self)
def rollback(self) -> None:
self._owner._rollback_reservation(self)
class FixedFramePcmAppendBuffer(PcmAppendBuffer):
"""Frame ``pcm_f32le`` at one sample rate into ``frame_samples`` units, transactionally."""
def __init__(
self,
*,
sample_rate_hz: int,
frame_samples: int,
chunk_period_ms: int,
model: str = "duplex",
) -> None:
if sample_rate_hz <= 0 or frame_samples <= 0 or chunk_period_ms <= 0:
raise ValueError("FixedFramePcmAppendBuffer needs positive sample_rate_hz, frame_samples, chunk_period_ms")
self.sample_rate_hz = int(sample_rate_hz)
self.frame_samples = int(frame_samples)
self.chunk_period_ms = int(chunk_period_ms)
self.model = model
self._buffer = bytearray()
self._reservations: list[FixedFramePcmAppendReservation] = []
@property
def frame_bytes(self) -> int:
return self.frame_samples * PCM_F32LE_BYTES_PER_SAMPLE
@property
def pending_byte_count(self) -> int:
return len(self._buffer)
def has_pending(self) -> bool:
return bool(self._buffer)
def has_reserved(self) -> bool:
return any(reservation.active for reservation in self._reservations)
def clear_force_listen(self) -> None:
return
def clear(self) -> None:
for reservation in self._reservations:
reservation._active = False
self._reservations.clear()
self._buffer.clear()
def prepare_append(
self,
payload: dict[str, object],
*,
operation_id: str,
chunk_period_ms: int,
allow_emit: bool,
) -> FixedFramePcmAppendReservation | None:
self._require_chunk_period(chunk_period_ms)
if any(reservation.active and reservation.operation_id == operation_id for reservation in self._reservations):
raise ValueError(f"{self.model} duplicate active operation_id: {operation_id}")
self._buffer.extend(self._decode_payload(payload))
if not allow_emit or len(self._buffer) < self.frame_bytes:
return None
return self._reserve_frame(payload, operation_id=operation_id, flush=False)
def prepare_commit(self, *, operation_id: str, chunk_period_ms: int) -> FixedFramePcmAppendReservation:
self._require_chunk_period(chunk_period_ms)
if not self._buffer:
reservation = FixedFramePcmAppendReservation(owner=self, operation_id=operation_id, payload=None, raw=b"")
self._reservations.append(reservation)
return reservation
payload: dict[str, object] = {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": self.sample_rate_hz,
"audio": "",
"final": True,
}
return self._reserve_frame(payload, operation_id=operation_id, flush=True)
def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
reservation = self.prepare_commit(operation_id=f"{self.model}-flush", chunk_period_ms=chunk_period_ms)
payload = reservation.payload
reservation.commit()
return payload
# ------------------------------------------------------------------ #
# Internals #
# ------------------------------------------------------------------ #
def _require_chunk_period(self, chunk_period_ms: int) -> None:
if chunk_period_ms != self.chunk_period_ms:
raise ValueError(f"{self.model} chunk_period_ms must be {self.chunk_period_ms}")
def _decode_payload(self, payload: dict[str, object]) -> bytes:
if payload.get("format") != "pcm_f32le":
raise ValueError(f"{self.model} input format must be pcm_f32le")
if payload.get("sample_rate_hz") != self.sample_rate_hz:
raise ValueError(f"{self.model} sample_rate_hz must be {self.sample_rate_hz}")
return decode_pcm_f32le_base64(payload.get("audio"), model=self.model)
def _reserve_frame(
self,
source_payload: dict[str, object],
*,
operation_id: str,
flush: bool,
) -> FixedFramePcmAppendReservation:
frame_bytes = self.frame_bytes
consumed_bytes = min(len(self._buffer), frame_bytes) if flush else frame_bytes
raw = bytes(self._buffer[:consumed_bytes])
del self._buffer[:consumed_bytes]
encoded_raw = raw + b"\x00" * (frame_bytes - consumed_bytes)
payload = dict(source_payload)
payload["type"] = "audio"
payload["format"] = "pcm_f32le"
payload["sample_rate_hz"] = self.sample_rate_hz
payload["audio"] = base64.b64encode(encoded_raw).decode("ascii")
reservation = FixedFramePcmAppendReservation(owner=self, operation_id=operation_id, payload=payload, raw=raw)
self._reservations.append(reservation)
return reservation
def _commit_reservation(self, reservation: FixedFramePcmAppendReservation) -> None:
if not reservation.active:
return
reservation._active = False
self._reservations.remove(reservation)
def _rollback_reservation(self, reservation: FixedFramePcmAppendReservation) -> None:
if not reservation.active:
return
try:
index = self._reservations.index(reservation)
except ValueError:
reservation._active = False
return
# Later reservations were taken after this one: they go back too, in order.
rolled_back = self._reservations[index:]
restored = b"".join(item._raw for item in rolled_back if item.active)
self._buffer[:0] = restored
for item in rolled_back:
item._active = False
del self._reservations[index:]
__all__ = ["FixedFramePcmAppendBuffer", "FixedFramePcmAppendReservation"]
@@ -0,0 +1,172 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Reading a vLLM ``RequestOutput`` the way model-side projection code needs to.
Stage outputs arrive as duck-typed objects (a ``RequestOutput``, an
``OmniRequestOutput`` wrapping one, or a test double): these helpers unwrap
them, read the ``multimodal_output`` mapping, coerce tensor scalars, and turn
cumulative audio/text into the delta since the last read. No duplex
vocabulary lives here.
"""
from __future__ import annotations
from collections.abc import Mapping
from typing import Any, cast
import numpy as np
def unwrap_request_output(output: object) -> tuple[object, object | None]:
"""Return ``(request_output, first_completion)`` for a stage output or a wrapper around one."""
inner = getattr(output, "request_output", None)
if inner is not None and inner is not output:
output = inner
return output, first_completion(output)
def first_completion(output: object) -> object | None:
outputs = getattr(output, "outputs", None)
return outputs[0] if isinstance(outputs, list) and outputs else None
def multimodal_output(output: object, completion: object | None = None) -> dict[str, object]:
"""The first non-empty ``multimodal_output`` mapping of the output or its completion, copied."""
for candidate in (
getattr(output, "multimodal_output", None),
getattr(completion, "multimodal_output", None) if completion is not None else None,
):
if isinstance(candidate, Mapping) and candidate:
return dict(candidate)
return {}
def coerce_int(value: object) -> int | None:
"""``int(value)`` for scalars, one-element tensors/arrays and numeric strings; ``None`` otherwise."""
detach = getattr(value, "detach", None)
if callable(detach):
try:
flat = detach().cpu().reshape(-1)
if flat.numel() == 0:
return None
value = flat[0].item()
except (RuntimeError, TypeError, ValueError, IndexError):
return None
elif isinstance(value, np.ndarray):
if value.size == 0:
return None
value = value.reshape(-1)[0].item()
try:
return int(cast(Any, value)) # Any: duck-typed scalar (int/float/str/tensor item)
except (TypeError, ValueError):
return None
def coerce_int_list(value: object) -> list[int]:
if value is None:
return []
if hasattr(value, "detach"):
try:
value = value.detach().cpu().reshape(-1).tolist()
except (RuntimeError, TypeError, ValueError):
return []
elif isinstance(value, np.ndarray):
value = value.reshape(-1).tolist()
if not isinstance(value, (list, tuple)):
return []
return [token_id for item in value if (token_id := coerce_int(item)) is not None]
def audio_value(multimodal: Mapping[str, object]) -> object | None:
"""The audio carried by a ``multimodal_output`` (``audio`` / ``model_outputs`` / ``latent``)."""
value = next(
(multimodal[key] for key in ("audio", "model_outputs", "latent") if key in multimodal),
None,
)
if isinstance(value, list) and len(value) == 1:
return value[0]
return value
def text_value(multimodal: Mapping[str, object], completion: object | None) -> str:
for candidate in (
multimodal.get("text"),
multimodal.get("llm_output_text"),
getattr(completion, "text", None) if completion is not None else None,
):
if isinstance(candidate, str) and candidate:
return candidate
return ""
def text_delta(text: str, previous: str) -> str:
"""What ``text`` adds over ``previous`` when outputs are cumulative; the whole text on a restart."""
if not text:
return ""
if text == previous:
return ""
if previous and text.startswith(previous):
return text[len(previous) :]
return text
def audio_sample_count(audio: object | None) -> int | None:
if audio is None:
return None
try:
import torch
if isinstance(audio, torch.Tensor):
return int(audio.numel())
except ImportError:
pass
try:
return int(np.asarray(audio, dtype=np.float32).size)
except (TypeError, ValueError):
return None
def slice_audio_delta(audio: object | None, offset: int) -> object | None:
"""The samples of a cumulative ``audio`` past ``offset``; the whole audio when it restarted."""
samples = audio_sample_count(audio)
if samples is None or samples <= 0:
return None
if offset <= 0 or samples < offset:
return audio
if samples == offset:
return None
try:
import torch
if isinstance(audio, torch.Tensor):
return audio.reshape(-1)[offset:].contiguous()
except ImportError:
pass
return np.asarray(audio, dtype=np.float32).reshape(-1)[offset:]
def sample_rate_hz(multimodal: Mapping[str, object], *, default: int) -> int:
value = multimodal.get("sr", multimodal.get("sample_rate_hz", default))
if isinstance(value, list) and value:
value = value[0]
if hasattr(value, "item"):
try:
value = value.item()
except (RuntimeError, TypeError, ValueError):
value = default
return int(value) if isinstance(value, int | float) and value > 0 else default
__all__ = [
"audio_sample_count",
"audio_value",
"coerce_int",
"coerce_int_list",
"first_completion",
"multimodal_output",
"sample_rate_hz",
"slice_audio_delta",
"text_delta",
"text_value",
"unwrap_request_output",
]
@@ -12,7 +12,7 @@ from vllm.logger import init_logger
from vllm_omni.engine.duplex.contracts import (
duplex_resource_request_belongs_to_session,
)
from vllm_omni.engine.duplex.plugin import DuplexDataPlane
from vllm_omni.engine.duplex.plugin import DuplexDataPlane, DuplexDataPlaneContext
from vllm_omni.outputs.duplex import get_duplex_output_decision
logger = init_logger(__name__)
@@ -21,17 +21,8 @@ EncodeAudio = Callable[[object, int, str, float | None], str | None]
@dataclass(frozen=True, slots=True)
class MiniCPMO45DataPlaneContext:
"""Serving state needed to project one MiniCPM data-plane output."""
epoch: int = 0
turn_id: int = 0
active_response_turn_id: int | None = None
active_response_id: str | None = None
auto_responds: bool = False
response_format: str = "wav"
speed: float | None = None
modalities: tuple[str, ...] = ()
class MiniCPMO45DataPlaneContext(DuplexDataPlaneContext):
"""Serving state needed to project one MiniCPM data-plane output (the framework fields)."""
@dataclass(slots=True)
@@ -18,7 +18,7 @@ from base64 import b64decode
from binascii import Error as BinasciiError
from collections.abc import Mapping
from copy import deepcopy
from typing import TYPE_CHECKING, Any, cast
from typing import TYPE_CHECKING
import numpy as np
from numpy.typing import NDArray
@@ -40,6 +40,11 @@ from vllm_omni.engine.duplex.plugin import (
reject_changed_runtime_value,
reject_private_runtime_keys,
)
from vllm_omni.model_executor.common.duplex.payload import payload_sample_count as _duplex_pcm_sample_count
from vllm_omni.model_executor.common.request_outputs import coerce_int as _coerce_int
from vllm_omni.model_executor.common.request_outputs import coerce_int_list as _coerce_int_list
from vllm_omni.model_executor.common.request_outputs import first_completion as _first_completion
from vllm_omni.model_executor.common.request_outputs import multimodal_output as _multimodal_output
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.capabilities import (
minicpmo45_native_capabilities,
)
@@ -56,7 +61,6 @@ from vllm_omni.model_executor.models.minicpmo_4_5.duplex.session import (
)
if TYPE_CHECKING:
import torch
from transformers import PreTrainedTokenizerBase
from vllm.config import ModelConfig
@@ -170,19 +174,6 @@ def _duplex_vision_tokens(payload: object, *, tile_pixels: int | None = None) ->
return blocks * _DUPLEX_VISION_TOKENS_PER_FRAME
def _duplex_pcm_sample_count(payload: object) -> int | None:
if not isinstance(payload, dict):
return None
audio = payload.get("audio") or payload.get("data")
if payload.get("format") != "pcm_f32le" or not isinstance(audio, str):
return None
try:
raw = b64decode(audio, validate=True)
except (BinasciiError, ValueError):
return None
return len(raw) // 4
def duplex_payload_is_exact_chunks(payload: object) -> bool:
sample_count = _duplex_pcm_sample_count(payload)
return sample_count is not None and sample_count != 0 and sample_count % _DUPLEX_CHUNK_SAMPLES == 0
@@ -288,48 +279,6 @@ def build_duplex_data_plane_prompt(
# ---- engine policy helpers: listen decision ----
def _coerce_int(value: object) -> int | None:
detach = getattr(value, "detach", None)
if callable(detach):
try:
flat: torch.Tensor = detach().cpu().reshape(-1)
if flat.numel() == 0:
return None
value = flat[0].item()
except Exception:
return None
try:
return int(cast(Any, value)) # Any: duck-typed scalar (int/float/str/tensor item)
except (TypeError, ValueError):
return None
def _coerce_int_list(value: object) -> list[int]:
if value is None:
return []
if hasattr(value, "detach"):
try:
value = value.detach().cpu().reshape(-1).tolist()
except Exception:
return []
if not isinstance(value, (list, tuple)):
return []
return [token_id for item in value if (token_id := _coerce_int(item)) is not None]
def _first_completion(output: object) -> object | None:
outputs = getattr(output, "outputs", None)
return outputs[0] if isinstance(outputs, list) and outputs else None
def _multimodal_output(output: object, completion: object | None) -> dict[str, object]:
metadata = getattr(output, "multimodal_output", None)
if isinstance(metadata, dict):
return metadata
metadata = getattr(completion, "multimodal_output", None) if completion is not None else None
return metadata if isinstance(metadata, dict) else {}
def _special_token_ids(metadata: dict[str, object]) -> dict[str, int]:
sources: list[object] = [metadata.get("special_token_ids"), metadata.get("meta")]
sources.append(
@@ -3,39 +3,22 @@
"""PersonaPlex full-duplex integration.
PersonaPlex (``nvidia/personaplex-7b-v1``) is a Moshi finetune: a pure-lockstep
speech-to-speech model. This package plugs it into the generic duplex serving
stack through the standard plugin seams (``duplex_serving_adapter`` /
``duplex_runtime_extension`` dotted strings in the model's ``pipeline.py``):
speech-to-speech model. This package plugs it into the unified full-duplex
framework through the one seam the framework has, ``PipelineConfig.duplex_plugin``:
- :class:`PersonaPlexConfig` immutable session config (voice / persona / sampling)
- :class:`PersonaPlexServingRuntimeAdapter` the ``ServingRuntimeAdapter`` impl
- :class:`PersonaPlexDuplexRuntimeExtension` the engine ``DuplexRuntimeExtension``
- :class:`PersonaPlexStage0DuplexRuntime` Stage 0 session state and prefill
- :class:`PersonaPlexPcmAppendBuffer` PCM input framing
- :class:`PersonaPlexDuplexPlugin` the ``DuplexModelPlugin`` (engine + session policy)
- :class:`PersonaPlexStage0DuplexRuntime` worker-side lockstep state and first-append prefill
- :class:`PersonaPlexPcmAppendBuffer` 80 ms PCM input framing
"""
from .config import (
PersonaPlexConfig,
)
from .input import (
PersonaPlexPcmAppendBuffer,
)
from .input import PersonaPlexPcmAppendBuffer
from .plugin import PersonaPlexDuplexPlugin
from .policy import PrefillStep
from .runtime_extension import (
PersonaPlexDuplexRuntimeExtension,
)
from .serving_adapter import (
PersonaPlexServingRuntimeAdapter,
)
from .stage0 import (
PersonaPlexStage0DuplexRuntime,
)
from .stage0 import PersonaPlexStage0DuplexRuntime
__all__ = [
"PersonaPlexConfig",
"PersonaPlexDuplexRuntimeExtension",
"PersonaPlexDuplexPlugin",
"PersonaPlexPcmAppendBuffer",
"PersonaPlexServingRuntimeAdapter",
"PersonaPlexStage0DuplexRuntime",
"PrefillStep",
]
@@ -0,0 +1,55 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
from vllm_omni.engine.duplex.config import DuplexCapabilities
from vllm_omni.model_executor.models.personaplex.duplex.config import CHUNK_PERIOD_MS
def personaplex_capabilities(*, max_sessions: int = 1) -> DuplexCapabilities:
"""What a PersonaPlex session advertises in ``session.created``.
PersonaPlex is a pure-lockstep model: audio flows both ways at the codec
frame rate, the model itself decides when to speak, and there are no
client commits or external turn signals. Barge-in is native model
behaviour but destructive output interruption and model-state rewind are
not validated, so it is not advertised. No text seeding exists, so the
model cannot serve ``/v1/chat/completions``.
"""
supports_multi_session = max_sessions > 1
return DuplexCapabilities(
supports_model_native_turn_policy=True,
supports_external_turn_signal=False,
supports_client_commit=False,
supports_barge_in=False,
supports_playback_ack=True,
supports_input_append=True,
supports_replace_latest_chunk=False,
supports_reencode_context=False,
supports_rollback_to_checkpoint=False,
supports_turn_commit_only=False,
supports_model_internal_state=True,
supports_stage_resumption=True,
supports_core_resumable_request=True,
supports_stage_connector_handoff=True,
supports_independent_io_streams=True,
supports_realtime_endpoint=True,
supports_multi_session=supports_multi_session,
supports_multi_session_same_replica=supports_multi_session,
supports_session_lease=True,
supports_session_resume=False,
session_admission_mode="engine_managed",
supports_audio_truncate=False,
supports_chat_completions=False,
requires_model_runner_kv=True,
requires_native_stage_role=True,
adapter_patterns=["scheduler_data_plane"],
signal_sources=["model_native", "client_event"],
stage_handoff_transport="scheduler_data_plane",
chunk_period_ms=CHUNK_PERIOD_MS,
target_barge_in_latency_ms=None,
)
__all__ = ["personaplex_capabilities"]
@@ -1,64 +1,24 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Configuration for the PersonaPlex full-duplex backend.
"""Constants of the PersonaPlex full-duplex integration.
PersonaPlex (``nvidia/personaplex-7b-v1``) is a Moshi finetune: a pure-lockstep
full-duplex speech-to-speech model running at the Mimi codec frame rate
(12.5 Hz / 80 ms). One config drives both the offline driver and the duplex
adapter. Defaults mirror the PersonaPlex reference loop.
(12.5 Hz / 80 ms). The values are fixed by the pretrained checkpoint; they
live here so the rest of the package never hard-codes them.
"""
from __future__ import annotations
from dataclasses import dataclass
# Mimi codec constants (loaders.py: SAMPLE_RATE / FRAME_RATE). Fixed by the
# pretrained checkpoint; exposed here only so the rest of the package never
# hard-codes them.
# Mimi codec constants (loaders.py: SAMPLE_RATE / FRAME_RATE).
SAMPLE_RATE = 24000
FRAME_RATE = 12.5
FRAME_SIZE = int(SAMPLE_RATE / FRAME_RATE) # 1920 samples per 80 ms frame
CHUNK_PERIOD_MS = int(round(1000 / FRAME_RATE)) # 80
# Default assistant persona shipped with PersonaPlex (offline.py:335).
DEFAULT_PERSONA = "You are a wise and friendly teacher. Answer questions or provide advice in a clear and engaging way."
# Bundled voice prompt used when a session names none.
DEFAULT_VOICE = "NATF2.pt"
@dataclass(frozen=True)
class PersonaPlexConfig:
"""Immutable session configuration for a PersonaPlex conversation.
Attributes:
hf_repo: HuggingFace repo holding the weights, Mimi codec and tokenizer.
voice_prompt: Voice-clone reference. Either a bundled basename
(``"NATF2.pt"`` / ``"NATM1.pt"`` from ``voices.tgz``) or a path to a
``.pt`` embedding bundle or a reference ``.wav``.
persona: System role text; injected as ``<system> ... <system>`` into the
inner-monologue stream at session start.
device: Torch device for the backend (``"cuda"`` / ``"cpu"``).
cpu_offload: Offload LM layers to CPU when GPU memory is tight (needs
``accelerate``).
batch_size: Concurrent conversation slots sharing one engine. ``1`` is
the single-session path; ``> 1`` enables elastic batching with
per-slot recycle for new callers.
Note: the native stepper decodes greedily (argmax) for both the text head
and the depformer, so there are no sampling knobs here yet. Temperature /
top-k / seed fields will be added if and when a sampling path is wired in.
"""
hf_repo: str = "nvidia/personaplex-7b-v1"
voice_prompt: str = "NATF2.pt"
persona: str = DEFAULT_PERSONA
device: str = "cuda"
cpu_offload: bool = False
batch_size: int = 1
@property
def sample_rate(self) -> int:
return SAMPLE_RATE
@property
def frame_size(self) -> int:
return FRAME_SIZE
__all__ = ["CHUNK_PERIOD_MS", "DEFAULT_PERSONA", "DEFAULT_VOICE", "FRAME_RATE", "FRAME_SIZE", "SAMPLE_RATE"]
@@ -1,228 +1,17 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""PersonaPlex output projection: cumulative Code2Wav audio and inner-monologue text into deltas."""
from __future__ import annotations
from collections.abc import Callable, Iterator, Mapping
from dataclasses import dataclass
import numpy as np
from vllm_omni.engine.duplex.contracts import (
duplex_resource_request_belongs_to_session,
)
EncodeAudio = Callable[[object, int, str, float | None], str | None]
from vllm_omni.model_executor.common.duplex.data_plane import CumulativeAudioTextDataPlane
from vllm_omni.model_executor.models.personaplex.duplex.config import SAMPLE_RATE
@dataclass(frozen=True, slots=True)
class PersonaPlexDataPlaneContext:
epoch: int = 0
turn_id: int = 0
active_response_turn_id: int | None = None
active_response_id: str | None = None
auto_responds: bool = True
response_format: str = "wav"
speed: float | None = None
modalities: tuple[str, ...] = ("audio", "text")
class PersonaPlexDataPlaneSession(CumulativeAudioTextDataPlane):
"""Project cumulative staged PersonaPlex output into Realtime deltas (24 kHz by default)."""
default_sample_rate_hz = SAMPLE_RATE
@dataclass(slots=True)
class _RequestCursor:
audio_samples: int = 0
text: str = ""
terminal: bool = False
class PersonaPlexDataPlaneSession:
"""Project cumulative staged PersonaPlex output into Realtime deltas."""
def __init__(self, encode_audio: EncodeAudio) -> None:
self._encode_audio = encode_audio
self._requests: dict[str, _RequestCursor] = {}
def begin_request(self, request_id: str) -> None:
self._requests.setdefault(request_id, _RequestCursor()).terminal = False
def is_terminal(self, request_id: str | None) -> bool:
if request_id is None:
return False
state = self._requests.get(request_id)
return state is not None and state.terminal
def mark_terminal(self, request_id: str) -> None:
self._requests.setdefault(request_id, _RequestCursor()).terminal = True
def close_stream(self, request_id: str) -> None:
self._requests.pop(request_id, None)
def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None:
if active_request_id is not None:
self._requests.pop(active_request_id, None)
for request_id in list(self._requests):
if duplex_resource_request_belongs_to_session(request_id, session_id):
self._requests.pop(request_id, None)
def project(
self,
result: object,
*,
context: PersonaPlexDataPlaneContext | None = None,
) -> Iterator[dict[str, object]]:
if not isinstance(result, dict):
return
outputs = result.get("data_plane_outputs")
if not isinstance(outputs, list):
return
context = context or PersonaPlexDataPlaneContext()
for output in outputs:
projected = self._project_output(output, context=context)
if projected is not None:
yield projected
def _project_output(
self,
output: object,
*,
context: PersonaPlexDataPlaneContext,
) -> dict[str, object] | None:
output, completion = _unwrap_output(output)
request_id = getattr(output, "request_id", None)
if not isinstance(request_id, str) or not request_id:
request_id = None
state = self._requests.setdefault(request_id, _RequestCursor()) if request_id is not None else _RequestCursor()
multimodal = _multimodal_output(output, completion)
audio = _audio_value(multimodal)
audio_delta = _slice_audio(audio, state.audio_samples)
audio_samples = _num_samples(audio)
if audio_samples is not None:
state.audio_samples = audio_samples
text = _text_value(multimodal, completion)
text_delta = _text_delta(text, state.text)
if text:
state.text = text
sample_rate_hz = _sample_rate(multimodal)
encoded = self._encode_audio(audio_delta, sample_rate_hz, context.response_format, context.speed)
if not encoded and not text_delta:
return None
delta_samples = _num_samples(audio_delta) or 0
return {
"supported": True,
"stage_role": "tts",
"is_listen": False,
"data_plane_request_id": request_id,
"text": text_delta,
"audio_data": encoded or "",
"audio_format": context.response_format,
"sample_rate_hz": sample_rate_hz,
"audio_duration_ms": round(delta_samples * 1000 / max(1, sample_rate_hz)),
"end_of_turn": False,
"uses_model_runner_scheduler": True,
"runner_kv_backed": True,
"runtime_impl": "scheduler_data_plane",
"owned_runtime": False,
}
def _unwrap_output(output: object) -> tuple[object, object | None]:
inner = getattr(output, "request_output", None)
if inner is not None and inner is not output:
output = inner
outputs = getattr(output, "outputs", None)
completion = outputs[0] if isinstance(outputs, list) and outputs else None
return output, completion
def _multimodal_output(output: object, completion: object | None) -> dict[str, object]:
for candidate in (
getattr(output, "multimodal_output", None),
getattr(completion, "multimodal_output", None) if completion is not None else None,
):
if isinstance(candidate, Mapping) and candidate:
return dict(candidate)
return {}
def _audio_value(multimodal: dict[str, object]) -> object | None:
value = next(
(multimodal[key] for key in ("audio", "model_outputs", "latent") if key in multimodal),
None,
)
if isinstance(value, list) and len(value) == 1:
return value[0]
return value
def _text_value(multimodal: dict[str, object], completion: object | None) -> str:
for candidate in (
multimodal.get("text"),
multimodal.get("llm_output_text"),
getattr(completion, "text", None) if completion is not None else None,
):
if isinstance(candidate, str) and candidate:
return candidate
return ""
def _text_delta(text: str, previous: str) -> str:
if not text:
return ""
if text == previous:
return ""
if previous and text.startswith(previous):
return text[len(previous) :]
return text
def _slice_audio(audio: object | None, offset: int) -> object | None:
samples = _num_samples(audio)
if samples is None or samples <= 0:
return None
if offset <= 0 or samples < offset:
return audio
if samples == offset:
return None
try:
import torch
if isinstance(audio, torch.Tensor):
return audio.reshape(-1)[offset:].contiguous()
except Exception:
pass
return np.asarray(audio, dtype=np.float32).reshape(-1)[offset:]
def _num_samples(audio: object | None) -> int | None:
if audio is None:
return None
try:
import torch
if isinstance(audio, torch.Tensor):
return int(audio.numel())
except Exception:
pass
try:
return int(np.asarray(audio, dtype=np.float32).size)
except Exception:
return None
def _sample_rate(multimodal: dict[str, object]) -> int:
value = multimodal.get("sr", multimodal.get("sample_rate_hz", 24000))
if isinstance(value, list) and value:
value = value[0]
if hasattr(value, "item"):
try:
value = value.item()
except Exception:
value = 24000
return int(value) if isinstance(value, int | float) else 24000
__all__ = [
"PersonaPlexDataPlaneContext",
"PersonaPlexDataPlaneSession",
]
__all__ = ["PersonaPlexDataPlaneSession"]
@@ -1,213 +1,27 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""PCM input framing for PersonaPlex: 24 kHz ``pcm_f32le`` into 1920-sample (80 ms) units."""
from __future__ import annotations
import base64
import binascii
import numpy as np
_SAMPLE_RATE_HZ = 24000
_FRAME_SAMPLES = 1920
_BYTES_PER_SAMPLE = 4
from vllm_omni.model_executor.common.duplex.pcm_buffer import FixedFramePcmAppendBuffer
from vllm_omni.model_executor.models.personaplex.duplex.config import (
CHUNK_PERIOD_MS,
FRAME_SIZE,
SAMPLE_RATE,
)
class PersonaPlexPcmAppendReservation:
__slots__ = (
"_active",
"_owner",
"_raw",
"operation_id",
"payload",
)
def __init__(
self,
*,
owner: PersonaPlexPcmAppendBuffer,
operation_id: str,
payload: dict[str, object] | None,
raw: bytes,
) -> None:
self._owner = owner
self.operation_id = operation_id
self.payload = payload
self._raw = raw
self._active = True
@property
def active(self) -> bool:
return self._active
@property
def byte_count(self) -> int:
return len(self._raw)
def commit(self) -> None:
self._owner._commit_reservation(self)
def rollback(self) -> None:
self._owner._rollback_reservation(self)
class PersonaPlexPcmAppendBuffer:
class PersonaPlexPcmAppendBuffer(FixedFramePcmAppendBuffer):
"""Transactionally frame 24 kHz float PCM into PersonaPlex 80 ms units."""
def __init__(self) -> None:
self._buffer = bytearray()
self._sample_rate_hz: int | None = None
self._reservations: list[PersonaPlexPcmAppendReservation] = []
@property
def pending_byte_count(self) -> int:
return len(self._buffer)
def has_pending(self) -> bool:
return bool(self._buffer)
def has_reserved(self) -> bool:
return any(reservation.active for reservation in self._reservations)
def clear_force_listen(self) -> None:
return
def clear(self) -> None:
for reservation in self._reservations:
reservation._active = False
self._reservations.clear()
self._buffer.clear()
self._sample_rate_hz = None
def prepare_append(
self,
payload: dict[str, object],
*,
operation_id: str,
chunk_period_ms: int,
allow_emit: bool,
) -> PersonaPlexPcmAppendReservation | None:
if chunk_period_ms != 80:
raise ValueError("PersonaPlex chunk_period_ms must be 80")
if any(reservation.active and reservation.operation_id == operation_id for reservation in self._reservations):
raise ValueError(f"PersonaPlex duplicate active operation_id: {operation_id}")
raw = self._decode_payload(payload)
self._buffer.extend(raw)
if not allow_emit or len(self._buffer) < _FRAME_SAMPLES * _BYTES_PER_SAMPLE:
return None
return self._reserve_frame(payload, operation_id=operation_id, flush=False)
def prepare_commit(
self,
*,
operation_id: str,
chunk_period_ms: int,
) -> PersonaPlexPcmAppendReservation:
if chunk_period_ms != 80:
raise ValueError("PersonaPlex chunk_period_ms must be 80")
if not self._buffer:
reservation = PersonaPlexPcmAppendReservation(
owner=self,
operation_id=operation_id,
payload=None,
raw=b"",
)
self._reservations.append(reservation)
return reservation
payload: dict[str, object] = {
"type": "audio",
"format": "pcm_f32le",
"sample_rate_hz": self._sample_rate_hz or _SAMPLE_RATE_HZ,
"audio": "",
"final": True,
}
return self._reserve_frame(payload, operation_id=operation_id, flush=True)
def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
reservation = self.prepare_commit(
operation_id="personaplex-flush",
chunk_period_ms=chunk_period_ms,
super().__init__(
sample_rate_hz=SAMPLE_RATE,
frame_samples=FRAME_SIZE,
chunk_period_ms=CHUNK_PERIOD_MS,
model="PersonaPlex",
)
payload = reservation.payload
reservation.commit()
return payload
def _decode_payload(self, payload: dict[str, object]) -> bytes:
if payload.get("format") != "pcm_f32le":
raise ValueError("PersonaPlex input format must be pcm_f32le")
sample_rate_hz = payload.get("sample_rate_hz")
if sample_rate_hz != _SAMPLE_RATE_HZ:
raise ValueError("PersonaPlex sample_rate_hz must be 24000")
if self._sample_rate_hz is not None and self._sample_rate_hz != sample_rate_hz:
raise ValueError("PersonaPlex sample_rate_hz changed within a session")
audio = payload.get("audio")
if not isinstance(audio, str):
raise ValueError("PersonaPlex audio must be base64 pcm_f32le")
try:
raw = base64.b64decode(audio, validate=True)
except (binascii.Error, ValueError) as exc:
raise ValueError("PersonaPlex audio is not valid base64") from exc
if len(raw) % _BYTES_PER_SAMPLE:
raise ValueError("PersonaPlex pcm_f32le byte length must be divisible by four")
samples = np.frombuffer(raw, dtype="<f4")
if not np.isfinite(samples).all():
raise ValueError("PersonaPlex pcm_f32le samples must be finite")
self._sample_rate_hz = int(sample_rate_hz)
return raw
def _reserve_frame(
self,
source_payload: dict[str, object],
*,
operation_id: str,
flush: bool,
) -> PersonaPlexPcmAppendReservation:
frame_bytes = _FRAME_SAMPLES * _BYTES_PER_SAMPLE
if flush:
consumed_bytes = min(len(self._buffer), frame_bytes)
else:
consumed_bytes = frame_bytes
raw = bytes(self._buffer[:consumed_bytes])
del self._buffer[:consumed_bytes]
encoded_raw = raw + b"\x00" * (frame_bytes - consumed_bytes)
payload = dict(source_payload)
payload["type"] = "audio"
payload["format"] = "pcm_f32le"
payload["sample_rate_hz"] = _SAMPLE_RATE_HZ
payload["audio"] = base64.b64encode(encoded_raw).decode("ascii")
reservation = PersonaPlexPcmAppendReservation(
owner=self,
operation_id=operation_id,
payload=payload,
raw=raw,
)
self._reservations.append(reservation)
return reservation
def _commit_reservation(self, reservation: PersonaPlexPcmAppendReservation) -> None:
if not reservation.active:
return
reservation._active = False
self._reservations.remove(reservation)
def _rollback_reservation(self, reservation: PersonaPlexPcmAppendReservation) -> None:
if not reservation.active:
return
try:
index = self._reservations.index(reservation)
except ValueError:
reservation._active = False
return
rolled_back = self._reservations[index:]
restored = b"".join(item._raw for item in rolled_back if item.active)
self._buffer[:0] = restored
for item in rolled_back:
item._active = False
del self._reservations[index:]
__all__ = [
"PersonaPlexPcmAppendBuffer",
"PersonaPlexPcmAppendReservation",
]
__all__ = ["PersonaPlexPcmAppendBuffer"]
@@ -0,0 +1,269 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""PersonaPlex full-duplex model plugin: engine policy and session policy in one class.
PersonaPlex is a pure-lockstep model. Engine policy is one 80 ms frame per
resumable Stage 0 append with greedy one-token decoding; session policy is
the bundled voice prompt and the persona text, resolved once at open into the
server-owned runtime configuration the worker replays as its prefill.
"""
from __future__ import annotations
import asyncio
from collections.abc import Mapping
from copy import deepcopy
from dataclasses import dataclass, field
from pathlib import PurePath
from typing import TYPE_CHECKING
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
from vllm_omni.engine.duplex.contracts import (
DuplexAppendPlan,
DuplexFence,
DuplexOutputDecision,
)
from vllm_omni.engine.duplex.plugin import (
DefaultDuplexModelSessionState,
DuplexModelPlugin,
DuplexRuntimeConfigError,
EncodeAudio,
reject_changed_runtime_value,
)
from vllm_omni.model_executor.common.duplex.payload import decode_pcm_f32le_payload
from vllm_omni.model_executor.models.personaplex.duplex.capabilities import personaplex_capabilities
from vllm_omni.model_executor.models.personaplex.duplex.config import (
DEFAULT_PERSONA,
DEFAULT_VOICE,
FRAME_SIZE,
SAMPLE_RATE,
)
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import PersonaPlexDataPlaneSession
from vllm_omni.model_executor.models.personaplex.duplex.input import PersonaPlexPcmAppendBuffer
if TYPE_CHECKING:
from vllm.config import ModelConfig
#: Runtime-config keys the server derives from the session fields; a client
#: cannot set them through ``extra_body``.
PRIVATE_RUNTIME_CONFIG_KEYS = frozenset(
{
"personaplex_prefill_slots",
"personaplex_model_path",
"personaplex_voice_prompt",
"personaplex_persona",
}
)
@dataclass
class PersonaPlexSessionState(DefaultDuplexModelSessionState):
"""Per-session model state: the framework flags plus the 80 ms framing buffer."""
audio_buffer: PersonaPlexPcmAppendBuffer = field(default_factory=PersonaPlexPcmAppendBuffer)
def voice_name(value: object) -> str:
"""A bundled ``.pt`` basename (``NATF2.pt``); paths are refused so the worker only reads the checkpoint."""
voice = value if isinstance(value, str) and value else DEFAULT_VOICE
path = PurePath(voice)
if path.name != voice or path.suffix != ".pt" or any(part == ".." for part in path.parts):
raise DuplexRuntimeConfigError("PersonaPlex voice must be a bundled .pt basename", code="invalid_voice")
return voice
def persona_text(value: object) -> str:
return str(value) if isinstance(value, str) and value else DEFAULT_PERSONA
def prefill_slot_count(runtime_config: Mapping[str, object]) -> int:
raw = runtime_config.get("personaplex_prefill_slots", 0)
try:
return max(0, int(raw)) # type: ignore[arg-type]
except (TypeError, ValueError) as exc:
raise ValueError("personaplex_prefill_slots must be a non-negative integer") from exc
class PersonaPlexDuplexPlugin(DuplexModelPlugin):
"""PersonaPlex-owned sampling policy, append planning, session state and output projection."""
plugin_id = "personaplex"
private_runtime_config_keys = PRIVATE_RUNTIME_CONFIG_KEYS
#: One codec frame: the runner keeps a model turn clocked with these when the client pauses.
silence_continuation_samples = FRAME_SIZE
silence_continuation_sample_rate_hz = SAMPLE_RATE
def __init__(self, encode_audio: EncodeAudio) -> None:
super().__init__(encode_audio)
self.data_plane = PersonaPlexDataPlaneSession(encode_audio)
# The prefill length depends only on (model, voice, persona); the
# voice bundle and tokenizer are read once per distinct triple.
self._prefill_slots: dict[tuple[str, str, str], int] = {}
self._prefill_lock = asyncio.Lock()
# ---- engine policy (the resumable Stage 0 request) ----
def configure_sampling_params(
self,
*,
runtime_config: dict[str, object],
defaults: tuple[object, ...],
) -> tuple[object, ...]:
del runtime_config
if not defaults:
return defaults
configured = list(defaults)
stage0 = defaults[0]
if isinstance(stage0, SamplingParams):
# Greedy, one temporal token per frame: the native stepper takes
# argmax for the text head and the depformer.
stage0 = stage0.clone()
stage0.temperature = 0.0
stage0.top_k = 1
stage0.max_tokens = 1
configured[0] = stage0
return tuple(configured)
def plan_append(
self,
*,
request_id: str,
fence: DuplexFence,
session_config: dict[str, object],
runtime_config: dict[str, object],
seq: int,
turn_seq: int,
payload: object,
final: bool,
sampling_params: object,
) -> DuplexAppendPlan:
del sampling_params
decode_pcm_f32le_payload(payload, sample_rate_hz=SAMPLE_RATE, exact_samples=FRAME_SIZE, model="PersonaPlex")
normalized_payload = dict(payload) # type: ignore[call-overload]
# One scheduler slot per frame; the first append of an epoch also
# carries the voice/persona prefill (a new epoch is a new Stage 0
# request with fresh KV, so the worker replays it).
prompt_slots = 1 + (prefill_slot_count(runtime_config) if seq <= 1 else 0)
return DuplexAppendPlan(
prompt={
"prompt_token_ids": [0] * prompt_slots,
"model_intermediate_buffer": {
"request_id": request_id,
"global_request_id": [fence.session_id],
"duplex": {
"data_plane": True,
"fence": fence,
"session_id": fence.session_id,
"epoch": fence.epoch,
"turn_id": fence.turn_id,
"seq": seq,
"turn_seq": turn_seq,
"mode": "append_audio_chunk",
"payload": normalized_payload,
"final": final,
"session_config": dict(session_config),
"runtime_config": dict(runtime_config),
"scheduler_token_budget": prompt_slots,
},
},
}
)
def decide_output(
self,
*,
stage_id: int,
final_stage_id: int,
segment_finished: bool,
segment_token_ids: tuple[int, ...],
segment_output_metadata: dict[str, object],
output: object,
) -> DuplexOutputDecision | None:
# Always-clocked model: what the client hears comes from the final
# stage data plane, never from a Stage 0 listen/speak decision.
del stage_id, final_stage_id, segment_finished, segment_token_ids, segment_output_metadata, output
return None
# ---- session policy ----
def create_session_state(self) -> PersonaPlexSessionState:
return PersonaPlexSessionState()
def capabilities(self, *, max_sessions: int) -> DuplexCapabilities:
return personaplex_capabilities(max_sessions=max_sessions)
async def prepare_runtime_config(
self, config: DuplexSessionConfig, *, model_config: ModelConfig | None
) -> dict[str, object]:
self.validate_client_extra_body(config.extra_body)
model_path = getattr(model_config, "model", None)
if not isinstance(model_path, str) or not model_path:
raise DuplexRuntimeConfigError("PersonaPlex model path is unavailable", code="model_path_unavailable")
voice = voice_name(config.voice)
persona = persona_text(config.instructions)
prefill_slots = await self._prefill_slots_for(model_path, voice, persona)
return {
"personaplex_model_path": model_path,
"personaplex_voice_prompt": voice,
"personaplex_persona": persona,
"personaplex_prefill_slots": prefill_slots,
}
async def _prefill_slots_for(self, model_path: str, voice: str, persona: str) -> int:
key = (model_path, voice, persona)
cached = self._prefill_slots.get(key)
if cached is not None:
return cached
from vllm_omni.model_executor.models.personaplex.duplex.stage0 import personaplex_prefill_slots
async with self._prefill_lock:
cached = self._prefill_slots.get(key)
if cached is not None:
return cached
try:
# Reads the voice bundle and the tokenizer: off the orchestrator loop.
slots = await asyncio.to_thread(personaplex_prefill_slots, model_path, voice, persona)
except Exception as exc:
raise DuplexRuntimeConfigError(
f"PersonaPlex voice/persona prefill could not be prepared: {exc}",
code="prefill_unavailable",
) from exc
self._prefill_slots[key] = slots
return slots
def runtime_config_for_update(
self,
config: DuplexSessionConfig,
current: Mapping[str, object],
) -> dict[str, object]:
self.validate_client_extra_body(config.extra_body)
new_persona = persona_text(config.instructions)
reject_changed_runtime_value(
new_persona,
current.get("personaplex_persona"),
message="PersonaPlex persona (instructions) cannot be changed after the session is created",
code="persona_update_unsupported",
)
new_voice = voice_name(config.voice)
reject_changed_runtime_value(
new_voice,
current.get("personaplex_voice_prompt"),
message="PersonaPlex voice cannot be changed after the session is created",
code="voice_update_unsupported",
)
runtime_config = deepcopy(dict(current))
runtime_config["personaplex_voice_prompt"] = new_voice
runtime_config["personaplex_persona"] = new_persona
return runtime_config
__all__ = [
"PRIVATE_RUNTIME_CONFIG_KEYS",
"PersonaPlexDuplexPlugin",
"PersonaPlexSessionState",
"persona_text",
"prefill_slot_count",
"voice_name",
]
@@ -1,129 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import base64
import binascii
from typing import Any
from vllm.sampling_params import SamplingParams
from vllm_omni.engine.duplex.messages import DuplexFence
from vllm_omni.engine.duplex.runtime import (
DuplexAppendPlan,
DuplexInputMode,
DuplexOutputDecision,
)
_FRAME_BYTES = 1920 * 4
def _validated_frame_payload(payload: object) -> dict[str, object]:
if not isinstance(payload, dict):
raise ValueError("PersonaPlex duplex append payload must be a mapping")
if payload.get("format") != "pcm_f32le":
raise ValueError("PersonaPlex duplex append format must be pcm_f32le")
if payload.get("sample_rate_hz") != 24000:
raise ValueError("PersonaPlex duplex append sample_rate_hz must be 24000")
audio = payload.get("audio")
if not isinstance(audio, str):
raise ValueError("PersonaPlex duplex append audio must be base64 pcm_f32le")
try:
raw = base64.b64decode(audio, validate=True)
except (binascii.Error, ValueError) as exc:
raise ValueError("PersonaPlex duplex append audio is not valid base64") from exc
if len(raw) != _FRAME_BYTES:
raise ValueError("PersonaPlex duplex append must contain exactly 1920 samples")
return dict(payload)
class PersonaPlexDuplexRuntimeExtension:
"""PersonaPlex policy for the engine-owned resumable duplex request."""
def configure_sampling_params(
self,
*,
runtime_config: dict[str, Any],
defaults: tuple[object, ...],
) -> tuple[object, ...]:
del runtime_config
if not defaults:
return defaults
configured = list(defaults)
stage0 = defaults[0]
if isinstance(stage0, SamplingParams):
stage0 = stage0.clone()
stage0.temperature = 0.0
stage0.top_k = 1
stage0.max_tokens = 1
configured[0] = stage0
return tuple(configured)
def plan_append(
self,
*,
request_id: str,
fence: DuplexFence,
session_config: dict[str, Any],
runtime_config: dict[str, Any],
seq: int,
turn_seq: int,
mode: DuplexInputMode,
payload: object,
final: bool,
sampling_params: object,
) -> DuplexAppendPlan:
del sampling_params
if mode is not DuplexInputMode.APPEND_AUDIO_CHUNK:
raise ValueError(f"PersonaPlex does not support duplex input mode {mode.value!r}")
normalized_payload = _validated_frame_payload(payload)
prefill_slots = runtime_config.get("personaplex_prefill_slots", 0)
try:
prefill_slots = max(0, int(prefill_slots))
except (TypeError, ValueError) as exc:
raise ValueError("personaplex_prefill_slots must be a non-negative integer") from exc
prompt_slots = 1 + (prefill_slots if seq <= 1 else 0)
return DuplexAppendPlan(
prompt={
"prompt_token_ids": [0] * prompt_slots,
"model_intermediate_buffer": {
"request_id": request_id,
"global_request_id": [fence.session_id],
"duplex": {
"data_plane": True,
"fence": fence,
"session_id": fence.session_id,
"incarnation": fence.incarnation,
"epoch": fence.epoch,
"turn_id": fence.turn_id,
"response_seq": fence.response_seq,
"seq": seq,
"turn_seq": turn_seq,
"mode": mode.value,
"payload": normalized_payload,
"final": final,
"session_config": dict(session_config),
"runtime_config": dict(runtime_config),
"scheduler_token_budget": prompt_slots,
},
},
}
)
def decide_output(
self,
*,
stage_id: int,
final_stage_id: int,
segment_finished: bool,
segment_token_ids: tuple[int, ...],
segment_output_metadata: dict[str, Any],
output: object,
) -> DuplexOutputDecision | None:
del stage_id, final_stage_id, segment_finished
del segment_token_ids, segment_output_metadata, output
return None
__all__ = ["PersonaPlexDuplexRuntimeExtension"]
@@ -1,253 +0,0 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import asyncio
from collections.abc import Awaitable, Callable, Mapping
from copy import deepcopy
from dataclasses import dataclass, field
from pathlib import PurePath
from typing import Any
from vllm_omni.entrypoints.duplex.protocol import (
DuplexCapabilities,
)
from vllm_omni.entrypoints.duplex.runtime_adapter import (
ServingRuntimeConfigError,
reject_changed_runtime_value,
)
from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import (
PersonaPlexDataPlaneContext,
PersonaPlexDataPlaneSession,
)
from vllm_omni.model_executor.models.personaplex.duplex.input import (
PersonaPlexPcmAppendBuffer,
)
EncodeAudio = Callable[[object, int, str, float | None], str | None]
_PRIVATE_RUNTIME_CONFIG_KEYS = frozenset(
{
"personaplex_prefill_slots",
"personaplex_model_path",
# PersonaPlex is always model-native; the per-session opt-in knob
# (canonical name and its deprecated alias) is server-owned here.
"native_duplex",
"minicpmo45_native_duplex",
}
)
@dataclass(slots=True)
class PersonaPlexServingSessionState:
audio_buffer: PersonaPlexPcmAppendBuffer = field(default_factory=PersonaPlexPcmAppendBuffer)
input_since_commit: bool = False
speech_since_commit: bool = False
native_context_locked: bool = False
committed_audio_payload: dict[str, object] | None = None
committed_audio_operation_id: str | None = None
committed_audio_reserved_bytes: int = 0
deferred_response_create: bool = False
deferred_precreate_response: bool = False
data_plane_task: asyncio.Task[None] | None = None
data_plane_restart_requested: bool = False
continuation_owner_id: str | None = None
continuation_units: int = 0
pending_silence_task: asyncio.Task[bool] | None = None
pending_silence_owner_id: str | None = None
silence_continuation_scheduler: Callable[..., Awaitable[bool]] | None = None
def retain_committed_audio(
self,
payload: dict[str, object],
*,
operation_id: str | None,
reserved_bytes: int = 0,
) -> None:
self.committed_audio_payload = payload
self.committed_audio_operation_id = operation_id
self.committed_audio_reserved_bytes += max(0, int(reserved_bytes))
def clear_committed_audio(self) -> int:
reserved_bytes = self.committed_audio_reserved_bytes
self.committed_audio_payload = None
self.committed_audio_operation_id = None
self.committed_audio_reserved_bytes = 0
self.deferred_response_create = False
self.deferred_precreate_response = False
return reserved_bytes
def clear_continuation(self) -> None:
self.continuation_owner_id = None
self.continuation_units = 0
self.pending_silence_task = None
self.pending_silence_owner_id = None
class PersonaPlexServingRuntimeAdapter:
adapter_id = "personaplex"
clean_response_done_prefix = ""
interrupted_tts_prefix = ""
private_runtime_config_keys = _PRIVATE_RUNTIME_CONFIG_KEYS
def __init__(self, encode_audio: EncodeAudio) -> None:
self.session_states: dict[str, PersonaPlexServingSessionState] = {}
self.data_plane = PersonaPlexDataPlaneSession(encode_audio)
def create_session_state(self) -> PersonaPlexServingSessionState:
return PersonaPlexServingSessionState()
def session_state(self, session_id: str) -> PersonaPlexServingSessionState:
return self.session_states.setdefault(session_id, self.create_session_state())
def remove_session_state(self, session_id: str) -> None:
self.session_states.pop(session_id, None)
@staticmethod
def is_enabled(config: object) -> bool:
del config
return True
@staticmethod
def capabilities(*, max_sessions: int) -> DuplexCapabilities:
supports_multi_session = max_sessions > 1
return DuplexCapabilities(
supports_model_native_turn_policy=True,
supports_external_turn_signal=False,
supports_client_commit=False,
supports_barge_in=False,
supports_playback_ack=True,
supports_input_append=True,
supports_replace_latest_chunk=False,
supports_reencode_context=False,
supports_rollback_to_checkpoint=False,
supports_turn_commit_only=False,
supports_model_internal_state=True,
supports_stage_resumption=True,
supports_core_resumable_request=True,
supports_stage_connector_handoff=True,
supports_independent_io_streams=True,
supports_realtime_endpoint=True,
supports_multi_session=supports_multi_session,
supports_multi_session_same_replica=supports_multi_session,
supports_session_lease=True,
supports_session_resume=False,
session_admission_mode="engine_managed",
supports_audio_truncate=False,
requires_model_runner_kv=True,
requires_native_stage_role=True,
implementation_level="model_native_duplex",
adapter_patterns=["scheduler_data_plane"],
input_modes=["append_audio_chunk"],
signal_sources=["model_native", "client_event"],
stage_handoff_transport="scheduler_data_plane",
chunk_period_ms=80,
target_barge_in_latency_ms=None,
)
@staticmethod
def validate_client_extra_body(extra_body: object) -> None:
if not isinstance(extra_body, dict):
return
private = sorted(_PRIVATE_RUNTIME_CONFIG_KEYS.intersection(extra_body))
if private:
raise ServingRuntimeConfigError("PersonaPlex runtime configuration is server-owned: " + ", ".join(private))
@classmethod
async def prepare_runtime_config(
cls,
config: object,
*,
model_config: Any,
) -> dict[str, object]:
extra_body = getattr(config, "extra_body", None)
cls.validate_client_extra_body(extra_body)
model_path = getattr(model_config, "model", None)
if not isinstance(model_path, str) or not model_path:
raise ServingRuntimeConfigError("PersonaPlex model path is unavailable")
voice = cls._voice_name(getattr(config, "voice", None))
instructions = getattr(config, "instructions", None) or DEFAULT_PERSONA
from vllm_omni.model_executor.models.personaplex.duplex.stage0 import (
personaplex_prefill_slots,
)
try:
prefill_slots = await asyncio.to_thread(
personaplex_prefill_slots,
model_path,
voice,
str(instructions),
)
except Exception as exc:
raise ServingRuntimeConfigError(f"PersonaPlex voice/persona prefill could not be prepared: {exc}") from exc
return {
"personaplex_model_path": model_path,
"personaplex_voice_prompt": voice,
"personaplex_persona": str(instructions),
"personaplex_prefill_slots": prefill_slots,
}
@classmethod
def runtime_config_for_update(
cls,
config: object,
current: Mapping[str, object],
) -> dict[str, object]:
cls.validate_client_extra_body(getattr(config, "extra_body", None))
new_persona = str(getattr(config, "instructions", None) or DEFAULT_PERSONA)
reject_changed_runtime_value(
new_persona,
current.get("personaplex_persona"),
message="PersonaPlex persona (instructions) cannot be changed after the session is created",
code="persona_update_unsupported",
)
new_voice = cls._voice_name(getattr(config, "voice", None))
reject_changed_runtime_value(
new_voice,
current.get("personaplex_voice_prompt"),
message="PersonaPlex voice cannot be changed after the session is created",
code="voice_update_unsupported",
)
runtime_config = deepcopy(dict(current))
runtime_config["personaplex_voice_prompt"] = new_voice
runtime_config["personaplex_persona"] = new_persona
return runtime_config
@staticmethod
def data_plane_context(
*,
epoch: int,
turn_id: int,
active_response_turn_id: int | None,
active_response_id: str | None,
auto_responds: bool,
response_format: str,
speed: float | None,
modalities: tuple[str, ...],
) -> PersonaPlexDataPlaneContext:
return PersonaPlexDataPlaneContext(
epoch=epoch,
turn_id=turn_id,
active_response_turn_id=active_response_turn_id,
active_response_id=active_response_id,
auto_responds=auto_responds,
response_format=response_format,
speed=speed,
modalities=modalities,
)
@staticmethod
def _voice_name(value: object) -> str:
voice = value if isinstance(value, str) and value else "NATF2.pt"
path = PurePath(voice)
if path.name != voice or path.suffix != ".pt" or any(part == ".." for part in path.parts):
raise ServingRuntimeConfigError("PersonaPlex voice must be a bundled .pt basename")
return voice
__all__ = [
"PersonaPlexServingRuntimeAdapter",
"PersonaPlexServingSessionState",
]
@@ -3,17 +3,19 @@
from __future__ import annotations
import base64
import binascii
import io
import tarfile
from collections.abc import Callable
from dataclasses import dataclass, field
from functools import lru_cache
from pathlib import Path
from typing import Any
import numpy as np
from vllm_omni.model_executor.common.audio.pcm import pcm_f32le_samples
from vllm_omni.model_executor.common.duplex.payload import decode_pcm_f32le_payload
from vllm_omni.model_executor.models.personaplex.duplex.config import FRAME_SIZE, SAMPLE_RATE
from vllm_omni.model_executor.models.personaplex.duplex.policy import (
AUDIO_SILENCE_FRAME_CNT,
SILENCE_TOKENS,
@@ -22,7 +24,7 @@ from vllm_omni.model_executor.models.personaplex.duplex.policy import (
wrap_with_system_tags,
)
_FRAME_SAMPLES = 1920
_FRAME_SAMPLES = FRAME_SIZE
@dataclass(slots=True)
@@ -37,8 +39,10 @@ class PersonaPlexStage0PreparedAppend:
@dataclass(slots=True)
class PersonaPlexStage0SessionState:
"""Lockstep state of one (session, epoch): a new epoch is a new Stage 0 request with fresh KV."""
session_id: str
incarnation: int
epoch: int
user_codes: Any | None = None
last_text_token: Any | None = None
last_agent_codes: Any | None = None
@@ -94,14 +98,29 @@ def load_personaplex_voice_state(model_path: str, voice: str) -> dict[str, Any]:
raise FileNotFoundError(f"PersonaPlex bundled voice prompt {voice!r} was not found under {model_path!r}")
def personaplex_prefill_slots(model_path: str, voice: str, persona: str) -> int:
@lru_cache(maxsize=4)
def _cached_tokenizer(model_path: str):
return load_personaplex_tokenizer(model_path)
@lru_cache(maxsize=16)
def _cached_voice_embedding_rows(model_path: str, voice: str) -> int:
state = load_personaplex_voice_state(model_path, voice)
embeddings = state.get("embeddings")
if not hasattr(embeddings, "shape") or len(embeddings.shape) < 1:
raise ValueError(f"PersonaPlex voice prompt {voice!r} has no embeddings")
tokenizer = load_personaplex_tokenizer(model_path)
persona_tokens = tokenizer(wrap_with_system_tags(persona)) if persona else []
return int(embeddings.shape[0]) + 2 * AUDIO_SILENCE_FRAME_CNT + len(persona_tokens)
return int(embeddings.shape[0])
def personaplex_prefill_slots(model_path: str, voice: str, persona: str) -> int:
"""Scheduler slots the first append of a session needs for the voice + persona prefill.
The voice bundle row count and the tokenizer are cached per model path:
they are constant, and this runs on every session open.
"""
voice_rows = _cached_voice_embedding_rows(model_path, voice)
persona_tokens = _cached_tokenizer(model_path)(wrap_with_system_tags(persona)) if persona else []
return voice_rows + 2 * AUDIO_SILENCE_FRAME_CNT + len(persona_tokens)
class PersonaPlexStage0DuplexRuntime:
@@ -147,19 +166,24 @@ class PersonaPlexStage0DuplexRuntime:
session_id = duplex.get("session_id")
if not isinstance(session_id, str) or not session_id:
raise ValueError("PersonaPlex duplex append requires session_id")
incarnation = _coerce_non_negative_int(duplex.get("incarnation"), "incarnation")
epoch = _coerce_non_negative_int(duplex.get("epoch"), "epoch")
seq = _coerce_positive_int(duplex.get("seq"), "seq")
identity = (epoch, seq)
key = (session_id, incarnation)
key = (session_id, epoch)
state = self.sessions.get(key)
if state is None:
# A newer epoch supersedes the session's earlier lockstep state:
# the engine aborted that request, but its finish notification
# may still be in flight, so release it here rather than let the
# two epochs share the codec budget.
for stale_key in [k for k in self.sessions if k[0] == session_id and k[1] < epoch]:
self.close_session(*stale_key)
if len(self.sessions) >= self.max_sessions:
raise RuntimeError(f"PersonaPlex Stage 0 session capacity {self.max_sessions} is exhausted")
state = PersonaPlexStage0SessionState(
session_id=session_id,
incarnation=incarnation,
epoch=epoch,
codec=self._acquire_codec(),
)
self.sessions[key] = state
@@ -291,7 +315,6 @@ class PersonaPlexStage0DuplexRuntime:
"stage0_prepared": True,
"prefill_applied": first_append,
"session_id": session_id,
"incarnation": incarnation,
"epoch": epoch,
"seq": seq,
},
@@ -363,8 +386,8 @@ class PersonaPlexStage0DuplexRuntime:
if not state.request_ids:
self.close_session(*key)
def close_session(self, session_id: str, incarnation: int) -> None:
key = (session_id, incarnation)
def close_session(self, session_id: str, epoch: int) -> None:
key = (session_id, epoch)
state = self.sessions.pop(key, None)
if state is None:
return
@@ -421,23 +444,13 @@ class PersonaPlexStage0DuplexRuntime:
@staticmethod
def _decode_pcm(payload: object) -> np.ndarray:
if not isinstance(payload, dict):
raise ValueError("PersonaPlex duplex payload must be a mapping")
if payload.get("format") != "pcm_f32le" or payload.get("sample_rate_hz") != 24000:
raise ValueError("PersonaPlex Stage 0 requires 24 kHz pcm_f32le")
audio = payload.get("audio")
if not isinstance(audio, str):
raise ValueError("PersonaPlex Stage 0 requires base64 audio")
try:
raw = base64.b64decode(audio, validate=True)
except (binascii.Error, ValueError) as exc:
raise ValueError("PersonaPlex Stage 0 audio is not valid base64") from exc
samples = np.frombuffer(raw, dtype="<f4")
if samples.size != _FRAME_SAMPLES:
raise ValueError(f"PersonaPlex Stage 0 requires {_FRAME_SAMPLES} samples per append")
if not np.isfinite(samples).all():
raise ValueError("PersonaPlex Stage 0 samples must be finite")
return np.ascontiguousarray(samples, dtype=np.float32).copy()
raw = decode_pcm_f32le_payload(
payload,
sample_rate_hz=SAMPLE_RATE,
exact_samples=_FRAME_SAMPLES,
model="PersonaPlex Stage 0",
)
return pcm_f32le_samples(raw)
def _coerce_non_negative_int(value: object, name: str) -> int:
@@ -2,9 +2,10 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""PersonaPlex pipeline: Talker (AR decode -> Mimi codebooks) -> Code2Wav (codebooks -> 24 kHz PCM).
PersonaPlex is a Moshi finetune (full-duplex speech-to-speech). For offline/batch
runs it is served as a 2-stage vllm-omni audio->audio pipeline, reusing the
Qwen3-TTS staged topology:
PersonaPlex is a Moshi finetune (full-duplex speech-to-speech), served as a
2-stage vllm-omni audio->audio pipeline that reuses the Qwen3-TTS staged
topology (online: one resumable Stage 0 request per duplex session, see
``duplex/plugin.py``):
* Stage 0 (``personaplex``) is the AR talker: the Helium temporal transformer
plus the depformer (both built by the lead). It emits the per-frame audio
@@ -31,13 +32,8 @@ PERSONAPLEX_PIPELINE = PipelineConfig(
# Pipeline-level default; the code2wav stage overrides per-stage below.
model_arch="PersonaPlexTalkerForConditionalGeneration",
default_deploy_config_name="personaplex.yaml",
duplex_runtime_extension=(
"vllm_omni.model_executor.models.personaplex.duplex.runtime_extension.PersonaPlexDuplexRuntimeExtension"
),
duplex_serving_adapter=(
"vllm_omni.model_executor.models.personaplex.duplex.serving_adapter.PersonaPlexServingRuntimeAdapter"
),
duplex_control_enabled=True,
# Served full duplex over /v1/realtime?duplex=1 by the unified framework.
duplex_plugin="vllm_omni.model_executor.models.personaplex.duplex.plugin.PersonaPlexDuplexPlugin",
stages=(
StagePipelineConfig(
stage_id=0,