Signed-off-by: chickeyton <ngton2014@gmail.com> Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
49 files changed
+3056
-1715
No files matched your search
@@ -1,142 +1,137 @@
|
||||
# PersonaPlex Unified Full-Duplex Design
|
||||
# PersonaPlex on the Unified Full-Duplex Framework
|
||||
|
||||
## Status and target
|
||||
|
||||
This design adapts PersonaPlex PR #4771 to the unified full-duplex runtime on
|
||||
top of:
|
||||
|
||||
- vLLM-Omni `origin/main`: `67c54777bb22e9e7e08fdf7c47a64f06b566fc47`
|
||||
- PersonaPlex PR head: `477fb7c225f0c06991bc8aa55eadbd908ba282e4`
|
||||
|
||||
The target is the engine-native path:
|
||||
PersonaPlex (`nvidia/personaplex-7b-v1`, a Moshi finetune) is served by the
|
||||
[Unified Full-Duplex Framework](fullduplex.md) through one model plugin,
|
||||
`PersonaPlexDuplexPlugin` (`vllm_omni/model_executor/models/personaplex/duplex/plugin.py`),
|
||||
selected by `PipelineConfig.duplex_plugin`. The serving path is:
|
||||
|
||||
```text
|
||||
/v1/realtime?duplex=1
|
||||
-> OpenAI Realtime session actor
|
||||
-> DuplexRequestClient
|
||||
-> AsyncOmni correlated RPC
|
||||
-> DuplexControlPlane
|
||||
-> resumable Stage 0 request
|
||||
-> PersonaPlex Talker
|
||||
-> streaming PersonaPlex Code2Wav
|
||||
-> PersonaPlex data-plane projector
|
||||
-> OmniDuplexSessionHandler (thin: websocket I/O, Realtime <-> DuplexCommand/DuplexEvent)
|
||||
-> DuplexOmni / DuplexOmniEngine
|
||||
-> DuplexSessionManager -> DuplexSessionRunner (one per session)
|
||||
plugin.plan_append: one 80 ms frame -> one resumable Stage 0 append
|
||||
-> PersonaPlex Talker (Stage 0, lockstep temporal transformer + depformer)
|
||||
-> streaming PersonaPlex Code2Wav (Stage 1, Mimi)
|
||||
-> plugin.data_plane: cumulative audio/text -> deltas
|
||||
-> response.output_audio.delta + response.output_audio_transcript.delta
|
||||
```
|
||||
|
||||
The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
|
||||
original PR was demo-only and has been removed from the tree; the unified
|
||||
engine path above is the only serving surface. It was never evidence that the
|
||||
unified engine path works.
|
||||
|
||||
## Why configuration-only enablement was invalid
|
||||
|
||||
The staged pipeline as shipped by the original PR was explicitly turn based.
|
||||
Its Talker read `pplex_user_codes`, `pplex_prefill_text`, and
|
||||
`pplex_silence_codes`, but no production staged input path wrote those fields.
|
||||
The voice prompt, persona prefill, and streaming Mimi state lived only in the
|
||||
standalone `PersonaPlexEngine` (demo-only, since removed).
|
||||
|
||||
Setting only the following fields would therefore advertise an endpoint whose
|
||||
model never receives the live microphone stream:
|
||||
|
||||
```python
|
||||
duplex_control_enabled = True
|
||||
duplex_runtime_extension = "..."
|
||||
duplex_serving_adapter = "..."
|
||||
```
|
||||
|
||||
The adapter must supply a real scheduler data plane, not just endpoint
|
||||
capabilities.
|
||||
The pre-framework pair (engine runtime extension plus serving adapter) and
|
||||
the standalone `/api/chat` / `/v1/audio/duplex` server are gone; the plugin is
|
||||
the only integration surface, and this document describes it.
|
||||
|
||||
## Supported scope
|
||||
|
||||
The unified implementation supports:
|
||||
The integration supports:
|
||||
|
||||
- up to two engine-owned sessions on one replica;
|
||||
- 24 kHz mono float PCM input;
|
||||
- one 1920-sample, 80 ms model frame per physical append unit;
|
||||
- continuous user input while assistant audio is generated or played;
|
||||
- bundled `.pt` voice prompts and a session persona;
|
||||
- greedy text and depformer sampling, matching the current PersonaPlex port;
|
||||
- `/v1/realtime?duplex=1` (the wire vocabulary is catalogued in the
|
||||
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide);
|
||||
- up to `duplex_session.max_sessions` engine-owned sessions on one replica
|
||||
(the shipped deploy sets two);
|
||||
- 24 kHz mono float PCM input, one 1920-sample (80 ms) model frame per
|
||||
physical append unit; clients may send any chunking, the session buffers
|
||||
whole frames;
|
||||
- continuous user input while assistant audio is generated or played
|
||||
(pure lockstep: the model listens while it speaks);
|
||||
- bundled `.pt` voice prompts (`voice`) and a session persona (`instructions`);
|
||||
- greedy text and depformer sampling (one temporal token per frame);
|
||||
- the public client preset
|
||||
`vllm_omni.clients.personaplex.create_duplex_session_config()` (24 kHz
|
||||
`pcm_f32le` input format, voice prompt, persona) as the canonical
|
||||
session-config source for `DuplexClient` consumers;
|
||||
- engine lease close, disconnect cleanup, reconnect after cleanup, and explicit
|
||||
response cancellation without cross-session state reuse.
|
||||
`pcm_f32le` input, voice, persona) as the canonical session-config source
|
||||
for `DuplexClient` / `InlineDuplexClient` consumers;
|
||||
- engine lease close, disconnect cleanup and explicit response cancellation
|
||||
without cross-session state reuse.
|
||||
|
||||
The implementation does not claim:
|
||||
It does not claim:
|
||||
|
||||
- more than two simultaneous PersonaPlex sessions on one replica;
|
||||
- arbitrary WAV voice cloning;
|
||||
- turn-based `response.create` semantics for an otherwise continuous model;
|
||||
- scheduler migration of live codec state between replicas;
|
||||
- exact output equality with the standalone engine after different scheduling
|
||||
boundaries.
|
||||
- arbitrary WAV voice cloning (voices are bundled basenames, resolved by the
|
||||
worker under the checkpoint);
|
||||
- turn-based `response.create` semantics, client commits or external turn
|
||||
signals (`supports_client_commit=false`, `supports_external_turn_signal=false`);
|
||||
- text seeding, hence no `/v1/chat/completions` route
|
||||
(`supports_chat_completions=false`);
|
||||
- session resume across a transport drop (`supports_session_resume=false`);
|
||||
- destructive output interruption or model-state rewind at a playback cursor
|
||||
(`supports_barge_in=false`, `supports_audio_truncate=false`): overlapping
|
||||
speech is native model behaviour, not a barge-in contract.
|
||||
|
||||
The capability payload derives multi-session support from the configured
|
||||
session limit. The shipped two-session deployment reports
|
||||
`supports_multi_session=true` and `supports_multi_session_same_replica=true`.
|
||||
`duplex_session.max_sessions` is the only capacity source: config resolution
|
||||
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools read
|
||||
that model-config value. Connector extras do not carry a second model-specific
|
||||
capacity knob that could drift from engine admission.
|
||||
It still reports `supports_barge_in=false`: the generic epoch fence can suppress
|
||||
stale transport output, but neither PersonaPlex nor the current MiniCPM-o 4.5
|
||||
adapter proves that model-owned streaming state can be destructively rewound or
|
||||
restarted at a playback cursor. Continuous overlapping speech is model-native
|
||||
duplex behavior, not by itself a barge-in contract.
|
||||
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools
|
||||
(Stage 0 encoders, Stage 1 decoders) read that model-config value. The
|
||||
capability payload derives `supports_multi_session` /
|
||||
`supports_multi_session_same_replica` from it.
|
||||
|
||||
## Components and ownership
|
||||
|
||||
### Serving adapter
|
||||
### The plugin (engine side)
|
||||
|
||||
`PersonaPlexServingRuntimeAdapter` owns only serving-side state:
|
||||
`PersonaPlexDuplexPlugin` owns both halves of the contract.
|
||||
|
||||
- a transactional PCM append buffer;
|
||||
- validation of `voice_prompt` and `instructions`;
|
||||
- public capabilities;
|
||||
- PersonaPlex data-plane output projection.
|
||||
Engine policy:
|
||||
|
||||
It accepts 24 kHz `pcm_f32le`, groups client packets into whole 1920-sample
|
||||
frames, zero-pads only the final residual, and rolls a reservation back when an
|
||||
engine append fails. It never loads CUDA weights and never encodes user audio.
|
||||
- `configure_sampling_params`: Stage 0 greedy (`temperature=0`, `top_k=1`,
|
||||
`max_tokens=1`), other stages untouched.
|
||||
- `plan_append`: validates exactly one 1920-sample 24 kHz `pcm_f32le` frame
|
||||
(through `model_executor/common/duplex/payload.py`) and reserves one
|
||||
scheduler slot per frame plus, on the first append of an epoch (`seq == 1`),
|
||||
the `personaplex_prefill_slots` of the voice/persona prefill. The prompt
|
||||
carries the fence, `seq`, the payload and the runtime config under
|
||||
`model_intermediate_buffer["duplex"]`.
|
||||
- `decide_output`: never decides. PersonaPlex is always-clocked; what the
|
||||
client hears comes from the final-stage data plane.
|
||||
- `silence_unit_payload`: one frame of zeros at 24 kHz
|
||||
(`silence_continuation_samples=1920`,
|
||||
`silence_continuation_sample_rate_hz=24000`). The runner uses it to keep a
|
||||
model turn clocked when the client pauses; the startup warmup sends it.
|
||||
|
||||
Client input cannot provide local filesystem paths. A voice is a bundled
|
||||
basename such as `NATF2.pt`; the worker resolves it under the local model
|
||||
checkpoint. `instructions` is the persona string.
|
||||
Session policy:
|
||||
|
||||
### Runtime extension
|
||||
- `capabilities`: `personaplex_capabilities(max_sessions)` (80 ms units,
|
||||
append-only, no commits, no barge-in, no resume, no chat route).
|
||||
- `prepare_runtime_config`: `voice` must be a bundled `.pt` basename,
|
||||
`instructions` defaults to the shipped persona; the prefill slot count is
|
||||
computed once per `(model, voice, persona)` off the orchestrator loop
|
||||
(`personaplex_prefill_slots`, itself caching the tokenizer and the voice
|
||||
bundle row count). The result is server-owned runtime config
|
||||
(`personaplex_model_path`, `personaplex_voice_prompt`,
|
||||
`personaplex_persona`, `personaplex_prefill_slots`); the same keys are
|
||||
refused in a client's `extra_body`.
|
||||
- `runtime_config_for_update`: persona and voice are immutable for the
|
||||
session (`persona_update_unsupported`, `voice_update_unsupported`).
|
||||
- session state, extra-body validation and data-plane context are the
|
||||
framework defaults (`DefaultDuplexModelSessionState` with the 80 ms
|
||||
`PersonaPlexPcmAppendBuffer`, `DuplexDataPlaneContext`).
|
||||
|
||||
`PersonaPlexDuplexRuntimeExtension` is pure model policy. It:
|
||||
Because `supports_client_commit` is off, the session auto-responds without
|
||||
`extra_body.auto_response`: a stock Realtime client streams audio and hears
|
||||
the model without any vendor flag.
|
||||
|
||||
- configures greedy Stage 0 sampling and bounded segment lengths;
|
||||
- maps each accepted PCM append to a scheduler prompt;
|
||||
- places immutable session identity, append sequence, PCM payload, voice, and
|
||||
persona under `model_intermediate_buffer["duplex"]`;
|
||||
- reserves exactly one scheduler prompt slot per encoded Mimi frame, plus the
|
||||
first-append voice/persona prefill length;
|
||||
- never performs model inference or owns session state.
|
||||
### Input framing
|
||||
|
||||
The extension returns no turn/listen decision. PersonaPlex is an always-clocked
|
||||
model, so visible audio/text comes from the final stage data plane.
|
||||
`PersonaPlexPcmAppendBuffer` is the `FixedFramePcmAppendBuffer` of
|
||||
`model_executor/common/duplex/pcm_buffer.py` at 24 kHz / 1920 samples /
|
||||
80 ms. It accepts 24 kHz `pcm_f32le` only, groups client packets into whole
|
||||
frames, takes one frame out per append as a reservation (committed when the
|
||||
stage accepted the append, rolled back to the front of the buffer when it did
|
||||
not), zero-pads only a final residual on commit, and never encodes audio.
|
||||
|
||||
### Stage 0 streaming runtime
|
||||
### Stage 0 streaming runtime (worker side)
|
||||
|
||||
The Talker owns a `PersonaPlexStage0DuplexRuntime` helper, analogous to
|
||||
MiniCPM-o's Stage 0 helper but with PersonaPlex lockstep semantics.
|
||||
The Talker owns a `PersonaPlexStage0DuplexRuntime`
|
||||
(`model_executor/models/personaplex/duplex/stage0.py`), created lazily in
|
||||
`_duplex_stage0_runtime()` and released through `on_requests_finished`.
|
||||
|
||||
For each admitted session it owns:
|
||||
For each live `(session_id, epoch)` it owns:
|
||||
|
||||
- streaming Mimi encoder convolution and transformer state;
|
||||
- the selected voice embedding bundle;
|
||||
- persona tokenization and prefill embeddings;
|
||||
- the prior user code frame needed by the one-frame acoustic delay;
|
||||
- append identity used to make a retried scheduler update idempotent.
|
||||
- the streaming Mimi encoder convolution and transformer state;
|
||||
- the selected voice embedding bundle and the persona prefill embeddings;
|
||||
- the prior user code frames needed by the one- and two-frame acoustic
|
||||
delays;
|
||||
- the append identity `(epoch, seq)` that makes a retried scheduler update
|
||||
idempotent.
|
||||
|
||||
The first append builds this ordered prefill:
|
||||
The first append of an epoch builds this ordered prefill:
|
||||
|
||||
```text
|
||||
voice embeddings
|
||||
@@ -146,45 +141,47 @@ voice embeddings
|
||||
-> first live user frame
|
||||
```
|
||||
|
||||
Later appends encode only new 1920-sample frames. The helper returns the
|
||||
Later appends encode only new 1920-sample frames. The runtime returns the
|
||||
per-frame user codes and prompt embeddings through the request's
|
||||
`model_intermediate_buffer`. The normal vLLM runner remains authoritative for
|
||||
attention metadata, block tables, KV allocation, scheduling, and sampling.
|
||||
`model_intermediate_buffer`; the normal vLLM runner remains authoritative for
|
||||
attention metadata, block tables, KV allocation, scheduling and sampling.
|
||||
|
||||
The Talker must distinguish resumable prompt prefill from decode positions.
|
||||
Prompt rows consume the exact prepared embeddings; sampled decode rows continue
|
||||
to use the existing delayed agent/user frame construction. No code path may
|
||||
fall back to an all-initial user stream for a duplex request.
|
||||
**Epochs.** The framework identifies a Stage 0 request by
|
||||
`(session_id, epoch)`; `response.cancel` and `output_audio_buffer.clear`
|
||||
advance the epoch, abort the current request and start the next append at
|
||||
`seq == 1` on a fresh request with fresh KV. The Stage 0 runtime therefore
|
||||
keys its state by `(session_id, epoch)`: when an append of a newer epoch
|
||||
arrives, any older-epoch state of the same session is closed first (its Mimi
|
||||
encoder returns to the pool, so the codec budget never counts a superseded
|
||||
epoch), and the voice/persona prefill is replayed because the plan reserved
|
||||
the slots again. The user-visible consequence is that a cancel restarts the
|
||||
model's conversation context. A late `on_requests_finished` for the aborted
|
||||
request finds nothing to close.
|
||||
|
||||
Cleanup is keyed by the full `(session_id, incarnation)` identity. Every live
|
||||
session has an independent Mimi encoder instance; encoder convolution/KV state
|
||||
is never shared between asynchronously scheduled sessions. A finished or
|
||||
Every live session has an independent Mimi encoder instance; encoder state is
|
||||
never shared between asynchronously scheduled sessions. A finished or
|
||||
aborted scheduler request resets and returns only that session's encoder.
|
||||
|
||||
### Stage 1 streaming decoder
|
||||
|
||||
The current `PersonaPlexCode2Wav` calls one-shot `MimiModel.decode` and is not
|
||||
CUDA-graph safe. Unified duplex uses eager Stage 1 and maintains an independent
|
||||
streaming Mimi decoder for every active request. Decoder ownership is keyed by
|
||||
the stable Stage 1 request id and released by `on_requests_finished`; a mixed
|
||||
batch must never advance another request's convolution or transformer state.
|
||||
|
||||
Each Stage 0 segment emits de-delayed agent codebooks. Stage 1 decodes only the
|
||||
new code frames, emits only the new PCM suffix, and resets state when the
|
||||
request is closed. Connector chunk boundaries retain the final raw code frame
|
||||
needed to de-delay the next chunk.
|
||||
|
||||
The deploy default sets Stage 1 `enforce_eager: true`; a default configuration
|
||||
that fails during CUDA graph capture is not an acceptable deployment profile.
|
||||
`PersonaPlexCode2Wav` runs eager (`enforce_eager: true` in the deploy) and
|
||||
maintains an independent streaming Mimi decoder for every active request,
|
||||
keyed by the Stage 1 request id and released by `on_requests_finished`. Each
|
||||
Stage 0 segment emits de-delayed agent codebooks; Stage 1 decodes only the new
|
||||
code frames and emits only the new PCM suffix. Connector chunk boundaries
|
||||
retain the final raw code frame needed to de-delay the next chunk.
|
||||
|
||||
### Data-plane projector
|
||||
|
||||
`PersonaPlexDataPlaneSession` converts cumulative or delta Stage 1 output into
|
||||
model-neutral native results:
|
||||
`PersonaPlexDataPlaneSession` is the `CumulativeAudioTextDataPlane` of
|
||||
`model_executor/common/duplex/data_plane.py` with a 24 kHz default rate. It
|
||||
keeps one audio/text cursor per request so cumulative Stage 1 output cannot
|
||||
replay old audio, and yields one internal result per new suffix:
|
||||
|
||||
```python
|
||||
{
|
||||
"stage_role": "tts",
|
||||
"is_listen": False,
|
||||
"data_plane_request_id": request_id,
|
||||
"text": text_delta,
|
||||
"audio_data": encoded_audio_delta,
|
||||
@@ -192,82 +189,74 @@ model-neutral native results:
|
||||
"sample_rate_hz": 24000,
|
||||
"audio_duration_ms": delta_duration,
|
||||
"end_of_turn": False,
|
||||
"runtime_impl": "scheduler_data_plane",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
It owns per-request audio and text cursors so a cumulative output cannot replay
|
||||
old audio. The generic projector turns each native result into the Realtime
|
||||
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair (and
|
||||
`response.output_text.delta` for text) under one `response_id`; the full
|
||||
mapping is the name map in the
|
||||
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
|
||||
PersonaPlex keeps one visible response open while continuous output arrives.
|
||||
Session close or cancellation terminates that response through the generic
|
||||
Realtime lifecycle; a codec segment finishing is not a conversational turn
|
||||
boundary. PersonaPlex advertises `supports_barge_in=false` and
|
||||
`supports_session_resume=false`, so `barge_in`, `turn_detection.server_vad`,
|
||||
and `session.resume` are rejected on this model.
|
||||
The runner turns each result into the Realtime
|
||||
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair
|
||||
under one `response_id`. PersonaPlex keeps one visible response open while
|
||||
continuous output arrives; session close or cancellation terminates it through
|
||||
the generic Realtime lifecycle. A codec segment finishing is not a
|
||||
conversational turn boundary.
|
||||
|
||||
## Error and lifecycle contracts
|
||||
|
||||
- Unsupported sample rate, malformed base64, non-finite PCM, invalid voice
|
||||
basename, or changed format fails before scheduler submission.
|
||||
- Unsupported sample rate, malformed base64, non-finite PCM, a partial frame
|
||||
at the plan, an invalid voice basename or a changed format fails before
|
||||
scheduler submission, as a typed `error` event.
|
||||
- Append is prepare/submit/commit. Failure rolls back the exact reserved PCM
|
||||
bytes and does not advance the model frame cursor.
|
||||
- A repeated `operation_id` must not encode or submit the same frame twice.
|
||||
- Input iterator exceptions execute the same cleanup as explicit close.
|
||||
- Cancellation does not release the engine lease until the stage request and
|
||||
any in-flight codec operation are actually finished.
|
||||
- A bounded cleanup timeout returns a cleanup error and keeps the session in
|
||||
the closing admission set; it must not make the slot available while work
|
||||
still mutates shared state.
|
||||
- New sessions cannot observe the previous voice, persona, PCM tail, Mimi
|
||||
convolution state, or Talker delayed code frame.
|
||||
|
||||
(The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path,
|
||||
which followed the same exception-safe rule, have been removed from the tree.)
|
||||
- A repeated `(epoch, seq)` identity does not encode or submit the same frame
|
||||
twice.
|
||||
- Cancellation advances the epoch; the aborted request's Stage 0 state is
|
||||
released by the next-epoch append or by `on_requests_finished`, whichever
|
||||
comes first.
|
||||
- Close releases the admission slot only once stage cleanup succeeded; the
|
||||
reaper retries a failed cleanup while the slot stays held.
|
||||
- New sessions cannot observe a previous voice, persona, PCM tail, Mimi
|
||||
convolution state or Talker delayed code frame.
|
||||
|
||||
## Testing and acceptance
|
||||
|
||||
### Contract tests
|
||||
### CPU contract tests
|
||||
|
||||
Tests first cover:
|
||||
- `tests/model_executor/models/personaplex/duplex/test_plugin.py`: pipeline
|
||||
binding and plugin load, honest capabilities, private keys, voice/persona
|
||||
resolution and caching, immutable updates, greedy sampling, one-slot-plus-
|
||||
prefill planning, frame validation, the 24 kHz silence unit, and the
|
||||
CPU-checked helpers of the E2E driver.
|
||||
- `tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py`:
|
||||
first-append prefill, depformer teacher forcing, causal user-frame delays,
|
||||
idempotent retries, independent encoders per session, capacity, epoch
|
||||
restart and late finish of a superseded request.
|
||||
- `tests/engine/duplex/test_session_runner_personaplex.py`: the session
|
||||
runner with the real plugin -- one submission per frame, prefill on the
|
||||
first append only, half frames buffered, wrong rate refused, cumulative
|
||||
Code2Wav output projected as 24 kHz deltas, cancel restarting the epoch,
|
||||
close aborting the request.
|
||||
- `tests/model_executor/executor_common/`: the shared toolbox (PCM helpers, payload
|
||||
validation, fixed-frame buffer, cumulative data plane, request-output
|
||||
readers).
|
||||
|
||||
- pipeline registration enables the control plane and selects both PersonaPlex
|
||||
adapters;
|
||||
- the capability payload is two-session, 80 ms, append-only, and honest;
|
||||
- PCM reservation commit/rollback, partial-frame flush, invalid input, and
|
||||
operation idempotency;
|
||||
- runtime prompt fields and exact token budgeting;
|
||||
- first-append voice/persona prefill followed by live user codes;
|
||||
- later appends retain per-session Mimi state and do not replay prefill;
|
||||
- interleaved Stage 0 and Stage 1 work preserves independent codec histories;
|
||||
- output projection emits only audio/text deltas;
|
||||
- close, exception, timeout, and reconnect cleanup;
|
||||
- ordinary non-duplex imports do not load PersonaPlex modules.
|
||||
### GPU validation
|
||||
|
||||
### Remote H20 validation
|
||||
`tests/e2e/online_serving/personaplex_realtime_duplex.py` (wrapped by
|
||||
`tests/e2e/online_serving/test_personaplex_duplex.py` when
|
||||
`PERSONAPLEX_MODEL_PATH` is set) drives the default `personaplex.yaml`
|
||||
deployment and requires:
|
||||
|
||||
Validation runs in an isolated remote worktree using the ModelScope
|
||||
`nv-community/personaplex-7b-v1` checkpoint and its Mimi dependency.
|
||||
|
||||
The required evidence is:
|
||||
|
||||
1. default `personaplex.yaml` reaches ready without a local eager override;
|
||||
2. `/health` returns 200;
|
||||
3. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`,
|
||||
and two-session admission;
|
||||
4. paced 24 kHz PCM appends produce finite, non-silent 24 kHz audio deltas and
|
||||
text deltas;
|
||||
5. microphone input continues during assistant output without cancelling the
|
||||
scheduler request;
|
||||
6. two paced sessions simultaneously produce independent non-empty audio, and
|
||||
a third session is rejected with `resource_exhausted`;
|
||||
7. closing either session frees only its scheduler request and GPU codec state,
|
||||
after which a replacement session can use a different persona without state
|
||||
leakage;
|
||||
8. malformed input returns one typed error and does not poison the next append;
|
||||
9. all focused unit tests and `git diff --check` pass.
|
||||
1. the server reaches ready without a local eager override; `/health` is 200;
|
||||
2. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`
|
||||
and two-session admission, with server-allocated session ids;
|
||||
3. paced 24 kHz PCM appends produce finite, non-silent, whole-frame 24 kHz
|
||||
audio deltas and text deltas;
|
||||
4. two paced sessions simultaneously produce independent non-empty audio, and
|
||||
a third session is refused with `resource_exhausted`;
|
||||
5. closing one session frees only its scheduler request and codec state, after
|
||||
which a replacement session with a different persona is admitted with a
|
||||
fresh id and no state leakage.
|
||||
|
||||
Audio that is empty, all zero, non-finite, or only a protocol `listen` event is
|
||||
not a successful end-to-end result.
|
||||
|
||||
@@ -107,6 +107,7 @@ th {
|
||||
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
|
||||
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
|
||||
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
|
||||
| `PersonaPlexTalkerForConditionalGeneration` | PersonaPlex (full duplex, 24 kHz speech in/out over `/v1/realtime?duplex=1`) | `nvidia/personaplex-7b-v1` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/PersonaPlex.md) |
|
||||
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
|
||||
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
|
||||
|
||||
@@ -384,9 +384,9 @@ with no OpenAI counterpart.
|
||||
|
||||
The event vocabulary is uniform, but several surfaces are gated by the
|
||||
`capabilities` object the server returns in `session.created`; a client must
|
||||
branch on those flags rather than on the model name. MiniCPM-o 4.5 is the
|
||||
only model on the plugin contract today; the other two columns record what
|
||||
their integrations advertise once the follow-up PRs port them:
|
||||
branch on those flags rather than on the model name. MiniCPM-o 4.5 and
|
||||
PersonaPlex are on the plugin contract today; the Nemotron VoiceChat column
|
||||
records what its integration advertises once the follow-up PR ports it:
|
||||
|
||||
| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -401,7 +401,11 @@ their integrations advertise once the follow-up PRs port them:
|
||||
Everything else in the catalogue — session lifecycle, heartbeat and event
|
||||
acknowledgement, append/commit/clear, the response envelope, playback
|
||||
acknowledgement, and the error envelope — behaves identically for every
|
||||
model.
|
||||
model. Two PersonaPlex specifics follow from its capabilities rather than from
|
||||
special-casing: a model with `supports_client_commit=false` auto-responds
|
||||
without `extra_body.auto_response`, and `response.cancel` /
|
||||
`output_audio_buffer.clear` restart its conversation context (a new Stage 0
|
||||
request replays the voice/persona prefill).
|
||||
|
||||
### Compatibility with the OpenAI Realtime protocol
|
||||
|
||||
|
||||
@@ -2,15 +2,15 @@
|
||||
|
||||
Serve [`nvidia/personaplex-7b-v1`](https://huggingface.co/nvidia/personaplex-7b-v1)
|
||||
(a Moshi-based full-duplex speech-to-speech model) with the native vLLM-Omni engine
|
||||
through the unified duplex serving stack (`/v1/duplex` and `/v1/realtime?duplex=1`).
|
||||
through the unified full-duplex framework (`/v1/realtime?duplex=1`, alias `/v1/duplex`).
|
||||
|
||||
> Requires a GPU and Hugging Face access to the gated repo
|
||||
> (`HF_TOKEN` with access to `nvidia/personaplex-7b-v1`).
|
||||
|
||||
## Start the server
|
||||
|
||||
The default `vllm_omni/deploy/personaplex.yaml` enables the engine-owned
|
||||
full-duplex control plane (`session_mode: duplex`):
|
||||
The default `vllm_omni/deploy/personaplex.yaml` is a duplex deployment
|
||||
(`session_mode: duplex`, two sessions per replica):
|
||||
|
||||
```bash
|
||||
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main serve \
|
||||
@@ -19,18 +19,32 @@ HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main ser
|
||||
--deploy-config vllm_omni/deploy/personaplex.yaml
|
||||
```
|
||||
|
||||
This exposes:
|
||||
This exposes `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`): the OpenAI
|
||||
Realtime session protocol projected onto vLLM-Omni duplex sessions (client API and
|
||||
wire protocol: [`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md)).
|
||||
There is no `/v1/chat/completions` route: PersonaPlex answers speech only.
|
||||
|
||||
- `WS /v1/duplex` — the native duplex session dialect
|
||||
(`session.create` / `input_audio_buffer.append` / `response.output_audio.delta` ...);
|
||||
- `WS /v1/realtime?duplex=1` — the same sessions projected onto the
|
||||
OpenAI Realtime event vocabulary (client API and wire protocol:
|
||||
[`docs/serving/realtime_duplex_api.md`](../../../docs/serving/realtime_duplex_api.md)).
|
||||
PersonaPlex is a pure-lockstep model: every session is native duplex, audio flows
|
||||
continuously in both directions in 80 ms frames, the model decides when to speak,
|
||||
and there are no client commits or external turn signals
|
||||
(`supports_client_commit=false`, `supports_external_turn_signal=false`). A session
|
||||
therefore auto-responds without any vendor flag.
|
||||
|
||||
PersonaPlex is a pure-lockstep model: every session on a PersonaPlex deployment is
|
||||
native duplex (`is_enabled()` is unconditionally true), audio flows continuously in
|
||||
both directions, and there are no client commits or external turn signals
|
||||
(`supports_client_commit=false`, `supports_external_turn_signal=false`).
|
||||
## Talk to it
|
||||
|
||||
With the client library and the PersonaPlex preset (24 kHz `pcm_f32le` in, bundled
|
||||
voice prompt, persona text):
|
||||
|
||||
```python
|
||||
from vllm_omni.clients.duplex import DuplexClient
|
||||
from vllm_omni.clients.personaplex import create_duplex_session_config
|
||||
|
||||
cfg = create_duplex_session_config(voice="NATF2.pt", persona="You are a concise assistant.")
|
||||
async with DuplexClient("ws://127.0.0.1:8000/v1/realtime?duplex=1", model="/path/to/personaplex-7b-v1", config=cfg) as c:
|
||||
await c.stream_pcm(pcm_f32le_24k) # keep streaming; the model speaks while it listens
|
||||
```
|
||||
|
||||
Voice and persona are fixed for the session (`session.update` cannot change them).
|
||||
|
||||
## Validate the serving path
|
||||
|
||||
@@ -45,9 +59,11 @@ python tests/e2e/online_serving/personaplex_realtime_duplex.py \
|
||||
--output-dir /tmp/personaplex-realtime-duplex
|
||||
```
|
||||
|
||||
The unified endpoint advertises `supports_barge_in=false`: overlapping speech
|
||||
is native model behavior, but destructive output interruption and model-state
|
||||
rewind have not been validated for PersonaPlex.
|
||||
The endpoint advertises `supports_barge_in=false`: overlapping speech is native
|
||||
model behaviour, but destructive output interruption and model-state rewind have
|
||||
not been validated for PersonaPlex. `response.cancel` and
|
||||
`output_audio_buffer.clear` restart the model's conversation context (a fresh
|
||||
Stage 0 request replays the voice/persona prefill).
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -56,8 +72,8 @@ rewind have not been validated for PersonaPlex.
|
||||
regardless of engine speed. On localhost it is smooth.
|
||||
- The earlier standalone Moshi-web compatibility server (browser client at `/`,
|
||||
binary WS protocol at `/api/chat`, raw-PCM `/v1/audio/duplex`) was demo-only
|
||||
and has been removed; use the unified endpoints above.
|
||||
- Session config (voice / persona / sampling) is passed per session via
|
||||
`extra_body`; see
|
||||
`vllm_omni/model_executor/models/personaplex/duplex/serving_adapter.py`.
|
||||
and has been removed; use the unified endpoint above.
|
||||
- The model plugin, worker-side lockstep runtime and input framing live in
|
||||
`vllm_omni/model_executor/models/personaplex/duplex/`; design notes in
|
||||
[`docs/design/fullduplex-personaplex.md`](../../../docs/design/fullduplex-personaplex.md).
|
||||
- Full runbook: [`recipes/NVIDIA/PersonaPlex.md`](../../../recipes/NVIDIA/PersonaPlex.md).
|
||||
@@ -25,10 +25,10 @@ matching the reference implementation frame for frame on the golden replays used
|
||||
as the acceptance gate.
|
||||
|
||||
PersonaPlex is the first Moshi-class (pure-lockstep) model on the vLLM-Omni
|
||||
full-duplex serving stack: it plugs into the generic `/v1/duplex` handler
|
||||
through the standard plugin seams (`duplex_serving_adapter` /
|
||||
`duplex_runtime_extension` in its `pipeline.py`) with a model-specific
|
||||
package at `vllm_omni/model_executor/models/personaplex/duplex/`.
|
||||
unified full-duplex framework: its `pipeline.py` declares one
|
||||
`duplex_plugin` (`PersonaPlexDuplexPlugin`), and the model-specific package at
|
||||
`vllm_omni/model_executor/models/personaplex/duplex/` holds the plugin, the
|
||||
worker-side lockstep Stage 0 runtime and the 80 ms input framing.
|
||||
|
||||
## References
|
||||
|
||||
@@ -114,16 +114,18 @@ HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m vllm_omni.entrypoints.cli.main ser
|
||||
--deploy-config vllm_omni/deploy/personaplex.yaml
|
||||
```
|
||||
|
||||
This exposes `WS /v1/duplex` (native duplex dialect) and
|
||||
`WS /v1/realtime?duplex=1` (OpenAI Realtime projection; client API and wire
|
||||
protocol in [`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md)).
|
||||
Voice and persona are set per session via `extra_body`.
|
||||
This exposes `WS /v1/realtime?duplex=1` (alias `WS /v1/duplex`): the OpenAI
|
||||
Realtime session protocol, client API and wire vocabulary in
|
||||
[`docs/serving/realtime_duplex_api.md`](../../docs/serving/realtime_duplex_api.md).
|
||||
Voice (`voice`, a bundled `.pt` basename) and persona (`instructions`) are set
|
||||
in `session.update`; the model takes no client commits and serves no
|
||||
`/v1/chat/completions` route.
|
||||
|
||||
#### Verification
|
||||
|
||||
```bash
|
||||
# GPU-free contract tests (stage0 runtime + unified serving adapter)
|
||||
pytest tests/model_executor/models/personaplex/duplex/ -q
|
||||
# GPU-free contract tests (plugin, stage0 runtime, runner scenario)
|
||||
pytest tests/model_executor/models/personaplex/duplex/ tests/engine/duplex/test_session_runner_personaplex.py -q
|
||||
|
||||
# GPU e2e: paced 24 kHz PCM over /v1/realtime?duplex=1, two concurrent
|
||||
# sessions, overflow admission, slot recycling, non-silent output
|
||||
|
||||
@@ -11,7 +11,6 @@ import hashlib
|
||||
import json
|
||||
import math
|
||||
import time
|
||||
import uuid
|
||||
import wave
|
||||
from collections.abc import Awaitable, Callable, Sequence
|
||||
from pathlib import Path
|
||||
@@ -125,10 +124,11 @@ def _input_identity(
|
||||
return {"path": str(resolved), "sha256": actual}
|
||||
|
||||
|
||||
def _realtime_url(base_url: str, model: str, session_id: str) -> str:
|
||||
def _realtime_url(base_url: str, model: str) -> str:
|
||||
"""The duplex Realtime URL; the session id is allocated by the server, never chosen here."""
|
||||
parts = urlsplit(base_url)
|
||||
query = dict(parse_qsl(parts.query, keep_blank_values=True))
|
||||
query.update(duplex="1", model=model, autostart="0", session_id=session_id)
|
||||
query.update(duplex="1", model=model, autostart="0")
|
||||
return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment))
|
||||
|
||||
|
||||
@@ -197,16 +197,27 @@ def _audio_bytes(client: RawRealtimeProbe) -> bytes:
|
||||
return b"".join(chunk for _, chunk in _validated_audio_chunks(client) if chunk)
|
||||
|
||||
|
||||
def _session_id(created: dict[str, object]) -> str | None:
|
||||
"""The server-allocated id announced in ``session.created``."""
|
||||
session = created.get("session")
|
||||
if isinstance(session, dict):
|
||||
for key in ("id", "session_id"):
|
||||
value = session.get(key)
|
||||
if isinstance(value, str) and value:
|
||||
return value
|
||||
value = created.get("session_id")
|
||||
return value if isinstance(value, str) and value else None
|
||||
|
||||
|
||||
async def _open_session(
|
||||
args: argparse.Namespace,
|
||||
*,
|
||||
session_id: str,
|
||||
persona: str,
|
||||
expect_error: bool = False,
|
||||
) -> tuple[RawRealtimeProbe, dict[str, object]]:
|
||||
client = RawRealtimeProbe(_realtime_url(args.url, args.model, session_id))
|
||||
client = RawRealtimeProbe(_realtime_url(args.url, args.model))
|
||||
await client.__aenter__()
|
||||
await client.send(_session_update(args, session_id=session_id, persona=persona))
|
||||
await client.send(_session_update(args, persona=persona))
|
||||
event_type = "error" if expect_error else "session.created"
|
||||
await wait_for(
|
||||
lambda: client.events.count(event_type) > 0,
|
||||
@@ -216,11 +227,10 @@ async def _open_session(
|
||||
return client, _events(client, event_type)[-1]
|
||||
|
||||
|
||||
def _session_update(args: argparse.Namespace, *, session_id: str, persona: str) -> dict[str, object]:
|
||||
def _session_update(args: argparse.Namespace, *, persona: str) -> dict[str, object]:
|
||||
return {
|
||||
"type": "session.update",
|
||||
"session": {
|
||||
"session_id": session_id,
|
||||
"model": args.model,
|
||||
"modalities": ["audio", "text"],
|
||||
"input_audio_format": "pcm_f32le",
|
||||
@@ -426,13 +436,11 @@ async def run(args: argparse.Namespace) -> dict[str, object]:
|
||||
output_dir = Path(args.output_dir)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
ids = {name: f"personaplex-{name}-{uuid.uuid4().hex}" for name in ("primary", "secondary", "replacement")}
|
||||
primary, created = await _open_session(args, session_id=ids["primary"], persona=args.persona)
|
||||
secondary, secondary_created = await _open_session(
|
||||
args,
|
||||
session_id=ids["secondary"],
|
||||
persona=args.secondary_persona,
|
||||
)
|
||||
primary, created = await _open_session(args, persona=args.persona)
|
||||
secondary, secondary_created = await _open_session(args, persona=args.secondary_persona)
|
||||
ids = {"primary": _session_id(created), "secondary": _session_id(secondary_created)}
|
||||
if not ids["primary"] or not ids["secondary"] or ids["primary"] == ids["secondary"]:
|
||||
raise AssertionError(f"server did not allocate distinct session ids: {ids}")
|
||||
capabilities = _capabilities(created)
|
||||
if _capabilities(secondary_created) != capabilities:
|
||||
raise AssertionError("concurrent sessions returned different capabilities")
|
||||
@@ -446,12 +454,7 @@ async def run(args: argparse.Namespace) -> dict[str, object]:
|
||||
if any(capabilities.get(key) != value for key, value in expected_capabilities.items()):
|
||||
raise AssertionError(f"unexpected PersonaPlex capabilities: {capabilities}")
|
||||
|
||||
overflow, error = await _open_session(
|
||||
args,
|
||||
session_id=f"personaplex-overflow-{uuid.uuid4().hex}",
|
||||
persona=args.persona,
|
||||
expect_error=True,
|
||||
)
|
||||
overflow, error = await _open_session(args, persona=args.persona, expect_error=True)
|
||||
error_body = error.get("error")
|
||||
overflow_code = error_body.get("code") if isinstance(error_body, dict) else error.get("code")
|
||||
await overflow.__aexit__(None, None, None)
|
||||
@@ -492,11 +495,10 @@ async def run(args: argparse.Namespace) -> dict[str, object]:
|
||||
_save(output_dir, "primary", primary, primary_audio)
|
||||
await _close_session(primary, timeout_s=args.timeout_s)
|
||||
|
||||
replacement, replacement_created = await _open_session(
|
||||
args,
|
||||
session_id=ids["replacement"],
|
||||
persona=args.replacement_persona,
|
||||
)
|
||||
replacement, replacement_created = await _open_session(args, persona=args.replacement_persona)
|
||||
ids["replacement"] = _session_id(replacement_created)
|
||||
if not ids["replacement"] or ids["replacement"] in {ids["primary"], ids["secondary"]}:
|
||||
raise AssertionError(f"replacement session id was not freshly allocated: {ids}")
|
||||
if _capabilities(replacement_created) != capabilities:
|
||||
raise AssertionError("replacement session returned different capabilities")
|
||||
continuation_frames, replacement_frames = await asyncio.gather(
|
||||
@@ -694,8 +696,9 @@ async def _run_load_session(
|
||||
ready: asyncio.Future[None],
|
||||
start: asyncio.Future[float],
|
||||
) -> dict[str, object]:
|
||||
session_id = f"personaplex-load-{index}-{uuid.uuid4().hex}"
|
||||
client = RawRealtimeProbe(_realtime_url(args.url, args.model, session_id), close_timeout_s=args.cleanup_timeout_s)
|
||||
# Label only: the server allocates the real session id (read back from session.created).
|
||||
session_id = f"personaplex-load-{index}"
|
||||
client = RawRealtimeProbe(_realtime_url(args.url, args.model), close_timeout_s=args.cleanup_timeout_s)
|
||||
sends: list[tuple[float, float, float]] = []
|
||||
error: str | None = None
|
||||
cleanup_error: str | None = None
|
||||
@@ -706,10 +709,9 @@ async def _run_load_session(
|
||||
try:
|
||||
async with client:
|
||||
try:
|
||||
await asyncio.wait_for(
|
||||
client.send(_session_update(args, session_id=session_id, persona=args.persona)), args.timeout_s
|
||||
)
|
||||
await asyncio.wait_for(client.send(_session_update(args, persona=args.persona)), args.timeout_s)
|
||||
await _wait_load_event(client, "session.created", args.timeout_s)
|
||||
session_id = _session_id(_events(client, "session.created")[-1]) or session_id
|
||||
capabilities = _capabilities(_events(client, "session.created")[-1])
|
||||
if (
|
||||
capabilities.get("chunk_period_ms") != 80
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""GPU coverage for PersonaPlex on the unified full-duplex framework.
|
||||
|
||||
Boots the PersonaPlex deploy (``vllm_omni/deploy/personaplex.yaml``) and runs
|
||||
the strict Realtime driver (``personaplex_realtime_duplex.py``): two paced
|
||||
24 kHz sessions, admission overflow, per-session slot recycling, audible
|
||||
whole-frame output. The checkpoint is gated, so the test runs only when
|
||||
``PERSONAPLEX_MODEL_PATH`` points at a local copy of ``nvidia/personaplex-7b-v1``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from tests.e2e.online_serving import personaplex_realtime_duplex as driver
|
||||
from tests.helpers.mark import hardware_test
|
||||
from tests.helpers.runtime import OmniServerParams
|
||||
from tests.helpers.stage_config import get_deploy_config_path
|
||||
|
||||
pytestmark = pytest.mark.omni
|
||||
|
||||
MODEL_PATH = os.environ.get("PERSONAPLEX_MODEL_PATH", "")
|
||||
DEPLOY_CONFIG = get_deploy_config_path("personaplex.yaml")
|
||||
|
||||
SERVER_PARAMS = [
|
||||
pytest.param(
|
||||
OmniServerParams(
|
||||
model=MODEL_PATH or "nvidia/personaplex-7b-v1",
|
||||
stage_config_path=DEPLOY_CONFIG,
|
||||
use_stage_cli=False,
|
||||
server_args=["--trust-remote-code"],
|
||||
),
|
||||
id="two-stage-single-gpu",
|
||||
)
|
||||
]
|
||||
|
||||
requires_checkpoint = pytest.mark.skipif(
|
||||
not MODEL_PATH or not Path(MODEL_PATH).is_dir(),
|
||||
reason="set PERSONAPLEX_MODEL_PATH to a local nvidia/personaplex-7b-v1 checkout",
|
||||
)
|
||||
|
||||
|
||||
def _speech_wav() -> Path:
|
||||
"""Real speech (the MiniCPM-o asset, 16 kHz; the driver resamples to 24 kHz).
|
||||
|
||||
PersonaPlex answers speech, not tones: a synthetic signal yields a silent
|
||||
reply and the driver's audibility floor rightly fails it.
|
||||
"""
|
||||
return Path(__file__).resolve().parents[2] / "assets" / "minicpmo_4_5" / "response_required_16k.wav"
|
||||
|
||||
|
||||
@requires_checkpoint
|
||||
@pytest.mark.advanced_model
|
||||
@hardware_test(res={"cuda": "H100"}, num_cards=1)
|
||||
@pytest.mark.parametrize("omni_server", SERVER_PARAMS, indirect=True)
|
||||
def test_personaplex_realtime_duplex_sessions(omni_server, tmp_path: Path) -> None:
|
||||
input_wav = _speech_wav()
|
||||
args = driver.parse_args(
|
||||
[
|
||||
"--url",
|
||||
f"ws://{omni_server.host}:{omni_server.port}/v1/realtime?duplex=1",
|
||||
"--model",
|
||||
MODEL_PATH,
|
||||
"--input-wav",
|
||||
str(input_wav),
|
||||
"--output-dir",
|
||||
str(tmp_path / "out"),
|
||||
]
|
||||
)
|
||||
|
||||
result = asyncio.run(driver.run(args))
|
||||
|
||||
sessions = {name: result[name] for name in ("primary", "secondary", "replacement")}
|
||||
assert all(isinstance(session, dict) for session in sessions.values())
|
||||
ids = {name: session["session_id"] for name, session in sessions.items() if isinstance(session, dict)}
|
||||
assert ids["primary"]
|
||||
assert ids["secondary"] != ids["primary"]
|
||||
assert ids["replacement"] not in {ids["primary"], ids["secondary"]}
|
||||
@@ -5,6 +5,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
from dataclasses import fields
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -16,10 +17,13 @@ from vllm_omni.config.stage_config import DuplexSessionRuntimeConfig, PipelineCo
|
||||
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
|
||||
from vllm_omni.engine.duplex.contracts import DuplexAppendPlan
|
||||
from vllm_omni.engine.duplex.plugin import (
|
||||
DefaultDuplexModelSessionState,
|
||||
DuplexDataPlane,
|
||||
DuplexDataPlaneContext,
|
||||
DuplexModelPlugin,
|
||||
DuplexModelSessionState,
|
||||
DuplexRuntimeConfigError,
|
||||
PcmAppendBuffer,
|
||||
coerce_int,
|
||||
load_duplex_plugin,
|
||||
payload_turn_id,
|
||||
@@ -322,3 +326,119 @@ def test_pipeline_config_binds_one_duplex_plugin_path() -> None:
|
||||
for source in framework_sources
|
||||
if "completed_append_cache_size" in source.read_text(encoding="utf-8")
|
||||
] == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Plugin defaults shared by every model #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_default_silence_unit_is_16k_zeros_and_follows_the_plugin_attributes() -> None:
|
||||
plugin = FakePlugin(_encode_audio)
|
||||
|
||||
unit = plugin.silence_unit_payload()
|
||||
|
||||
assert unit["type"] == "audio"
|
||||
assert unit["format"] == "pcm_f32le"
|
||||
assert unit["sample_rate_hz"] == 16000
|
||||
assert base64.b64decode(unit["audio"]) == bytes(16000 * 4)
|
||||
|
||||
class FramePlugin(FakePlugin):
|
||||
silence_continuation_samples = 1920
|
||||
silence_continuation_sample_rate_hz = 24000
|
||||
|
||||
frame_unit = FramePlugin(_encode_audio).silence_unit_payload()
|
||||
assert frame_unit["sample_rate_hz"] == 24000
|
||||
assert len(base64.b64decode(frame_unit["audio"])) == 1920 * 4
|
||||
|
||||
|
||||
def test_default_extra_body_validation_rejects_the_private_keys_naming_the_plugin() -> None:
|
||||
class PrivateKeysPlugin(FakePlugin):
|
||||
plugin_id = "keyed"
|
||||
private_runtime_config_keys = frozenset({"secret_a", "secret_b"})
|
||||
|
||||
def validate_client_extra_body(self, extra_body: object) -> None:
|
||||
DuplexModelPlugin.validate_client_extra_body(self, extra_body)
|
||||
|
||||
plugin = PrivateKeysPlugin(_encode_audio)
|
||||
plugin.validate_client_extra_body(None)
|
||||
plugin.validate_client_extra_body({"auto_response": True})
|
||||
with pytest.raises(
|
||||
DuplexRuntimeConfigError, match="keyed runtime configuration is server-owned: secret_a, secret_b"
|
||||
):
|
||||
plugin.validate_client_extra_body({"secret_b": 1, "secret_a": 2})
|
||||
|
||||
|
||||
def test_default_data_plane_context_is_the_framework_dataclass() -> None:
|
||||
class DefaultContextPlugin(FakePlugin):
|
||||
def data_plane_context(self, **kwargs):
|
||||
return DuplexModelPlugin.data_plane_context(self, **kwargs)
|
||||
|
||||
context = DefaultContextPlugin(_encode_audio).data_plane_context(
|
||||
epoch=2,
|
||||
turn_id=3,
|
||||
active_response_turn_id=1,
|
||||
active_response_id="resp",
|
||||
auto_responds=True,
|
||||
response_format="pcm16",
|
||||
speed=1.5,
|
||||
modalities=("audio",),
|
||||
)
|
||||
|
||||
assert isinstance(context, DuplexDataPlaneContext)
|
||||
assert context == DuplexDataPlaneContext(
|
||||
epoch=2,
|
||||
turn_id=3,
|
||||
active_response_turn_id=1,
|
||||
active_response_id="resp",
|
||||
auto_responds=True,
|
||||
response_format="pcm16",
|
||||
speed=1.5,
|
||||
modalities=("audio",),
|
||||
)
|
||||
|
||||
|
||||
def test_default_session_state_implements_the_shared_transitions() -> None:
|
||||
class Buffer(PcmAppendBuffer):
|
||||
pending_byte_count = 0
|
||||
|
||||
def clear(self) -> None: ...
|
||||
|
||||
def clear_force_listen(self) -> None: ...
|
||||
|
||||
def has_pending(self) -> bool:
|
||||
return False
|
||||
|
||||
def has_reserved(self) -> bool:
|
||||
return False
|
||||
|
||||
def prepare_append(self, payload, *, operation_id, chunk_period_ms, allow_emit):
|
||||
return None
|
||||
|
||||
def prepare_commit(self, *, operation_id, chunk_period_ms):
|
||||
raise NotImplementedError
|
||||
|
||||
def flush(self, *, chunk_period_ms):
|
||||
return None
|
||||
|
||||
state = DefaultDuplexModelSessionState(audio_buffer=Buffer())
|
||||
|
||||
assert isinstance(state, DuplexModelSessionState)
|
||||
assert state.committed_audio_reserved_bytes == 0 and state.continuation_units == 0
|
||||
state.retain_committed_audio({"audio": "a"}, operation_id="op-1", reserved_bytes=10)
|
||||
state.retain_committed_audio({"audio": "b"}, operation_id="op-2", reserved_bytes=5)
|
||||
state.deferred_response_create = True
|
||||
assert state.committed_audio_payload == {"audio": "b"}
|
||||
assert state.committed_audio_operation_id == "op-2"
|
||||
assert state.clear_committed_audio() == 15
|
||||
assert state.committed_audio_payload is None
|
||||
assert state.committed_audio_operation_id is None
|
||||
assert state.deferred_response_create is False
|
||||
|
||||
state.continuation_owner_id = "response:r"
|
||||
state.continuation_units = 3
|
||||
state.pending_silence_owner_id = "response:r"
|
||||
state.clear_continuation()
|
||||
assert state.continuation_owner_id is None
|
||||
assert state.continuation_units == 0
|
||||
assert state.pending_silence_task is None and state.pending_silence_owner_id is None
|
||||
@@ -0,0 +1,245 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""The session runner driven by the real PersonaPlex plugin.
|
||||
|
||||
Same harness as ``test_session_runner.py`` (recording stage port, typed
|
||||
commands through the manager, stage outputs pushed by hand), plugin swapped:
|
||||
this pins the lockstep contract -- one Stage 0 submission per 80 ms frame,
|
||||
prefill slots only on the first append of an epoch, auto-response implied by
|
||||
``supports_client_commit=False``, cumulative Code2Wav output projected as
|
||||
24 kHz deltas, and a cancel restarting the epoch.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import struct
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from tests.engine.duplex.test_session_runner import (
|
||||
SESSION_ID,
|
||||
Harness,
|
||||
RecordingStagePort,
|
||||
_fake_encode_audio,
|
||||
close_harness,
|
||||
find,
|
||||
types,
|
||||
)
|
||||
from vllm_omni.config.stage_config import DuplexSessionRuntimeConfig
|
||||
from vllm_omni.engine.duplex import commands
|
||||
from vllm_omni.engine.duplex.config import DuplexSessionConfig
|
||||
from vllm_omni.engine.duplex.messages import DuplexControlResultMessage, OpenDuplexSessionMessage
|
||||
from vllm_omni.engine.duplex.session.manager import DuplexSessionManager
|
||||
from vllm_omni.model_executor.models.personaplex.duplex import stage0
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import FRAME_SIZE, SAMPLE_RATE
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.plugin import PersonaPlexDuplexPlugin
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
PREFILL_SLOTS = 4
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _fake_prefill(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(stage0, "personaplex_prefill_slots", lambda model_path, voice, persona: PREFILL_SLOTS)
|
||||
|
||||
|
||||
async def open_personaplex_harness(*, extra_body: dict[str, object] | None = None) -> Harness:
|
||||
plugin = PersonaPlexDuplexPlugin(_fake_encode_audio)
|
||||
port = RecordingStagePort(stage_count=2)
|
||||
output: asyncio.Queue[Any] = asyncio.Queue()
|
||||
results: asyncio.Queue[Any] = asyncio.Queue()
|
||||
manager = DuplexSessionManager(
|
||||
plugin=plugin,
|
||||
stage_port=port,
|
||||
output_sink=output,
|
||||
result_sink=results,
|
||||
runtime_config=DuplexSessionRuntimeConfig(),
|
||||
model_config=SimpleNamespace(model="/models/personaplex-7b-v1"),
|
||||
)
|
||||
config = DuplexSessionConfig(
|
||||
model="nvidia/personaplex-7b-v1",
|
||||
modalities=["audio", "text"],
|
||||
instructions="Be concise.",
|
||||
voice="NATF2.pt",
|
||||
extra_body=dict(extra_body or {}),
|
||||
)
|
||||
await manager.handle(OpenDuplexSessionMessage(control_id="c-open", session_id=SESSION_ID, session_config=config))
|
||||
result = await asyncio.wait_for(results.get(), timeout=2.0)
|
||||
assert isinstance(result, DuplexControlResultMessage) and result.ok, result
|
||||
harness = Harness(manager=manager, port=port, output=output, results=results, runner=manager.runners[SESSION_ID])
|
||||
await harness.settle()
|
||||
return harness
|
||||
|
||||
|
||||
def frame(samples: int = FRAME_SIZE, *, value: float = 0.05) -> commands.AppendAudio:
|
||||
return commands.AppendAudio(
|
||||
audio=struct.pack(f"<{samples}f", *([value] * samples)),
|
||||
format="pcm_f32le",
|
||||
sample_rate_hz=SAMPLE_RATE,
|
||||
is_speech=True,
|
||||
)
|
||||
|
||||
|
||||
def code2wav_output(request_id: str, *, samples: int, text: str) -> SimpleNamespace:
|
||||
"""A cumulative Stage 1 (Code2Wav) output the way the orchestrator hands it to the runner."""
|
||||
return SimpleNamespace(
|
||||
request_id=request_id,
|
||||
finished=False,
|
||||
outputs=[SimpleNamespace(text=text, token_ids=[], multimodal_output={})],
|
||||
multimodal_output={"audio": np.zeros(samples, dtype=np.float32), "sr": SAMPLE_RATE},
|
||||
)
|
||||
|
||||
|
||||
def submitted_duplex(port: RecordingStagePort) -> list[dict[str, object]]:
|
||||
return [dict(sub.prompt["model_intermediate_buffer"]["duplex"]) for sub in port.submissions]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_open_advertises_the_lockstep_capabilities_and_reserves_stage0() -> None:
|
||||
h = await open_personaplex_harness()
|
||||
try:
|
||||
created = find(h.events, "session.created")
|
||||
capabilities = created.session["capabilities"]
|
||||
assert capabilities["chunk_period_ms"] == 80
|
||||
assert capabilities["supports_client_commit"] is False
|
||||
assert capabilities["supports_barge_in"] is False
|
||||
# Not on the wire: the api server reads it from the plugin to decide the chat route.
|
||||
assert h.session.capabilities.supports_chat_completions is False
|
||||
assert [context.request_id for context in h.port.ensured] == [h.stage0_request_id(epoch=0)]
|
||||
assert h.session.runtime_config["personaplex_prefill_slots"] == PREFILL_SLOTS
|
||||
assert h.session.runtime_config["personaplex_voice_prompt"] == "NATF2.pt"
|
||||
finally:
|
||||
await close_harness(h)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_session_without_the_auto_response_flag_still_auto_responds() -> None:
|
||||
h = await open_personaplex_harness(extra_body={})
|
||||
try:
|
||||
assert h.runner.out.auto_responds() is True
|
||||
finally:
|
||||
await close_harness(h)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_each_frame_is_one_stage0_submission_with_prefill_only_on_the_first() -> None:
|
||||
h = await open_personaplex_harness()
|
||||
try:
|
||||
await h.run(frame())
|
||||
await h.run(frame())
|
||||
await h.run(frame())
|
||||
|
||||
assert len(h.port.submissions) == 3
|
||||
assert [len(sub.prompt["prompt_token_ids"]) for sub in h.port.submissions] == [1 + PREFILL_SLOTS, 1, 1]
|
||||
duplex = submitted_duplex(h.port)
|
||||
assert [item["seq"] for item in duplex] == [1, 2, 3]
|
||||
assert all(item["epoch"] == 0 for item in duplex)
|
||||
assert all(item["session_id"] == SESSION_ID for item in duplex)
|
||||
assert all("incarnation" not in item for item in duplex)
|
||||
assert all(sub.context.request_id == h.stage0_request_id(epoch=0) for sub in h.port.submissions)
|
||||
assert [sub.already_submitted for sub in h.port.submissions] == [False, True, True]
|
||||
assert h.port.submissions[0].context.stage_sampling_params.max_tokens == 1
|
||||
assert not [event for event in h.events if event.type == "error"]
|
||||
finally:
|
||||
await close_harness(h)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_half_frames_are_buffered_until_a_whole_frame_exists() -> None:
|
||||
h = await open_personaplex_harness()
|
||||
try:
|
||||
await h.run(frame(FRAME_SIZE // 2))
|
||||
assert h.port.submissions == []
|
||||
assert h.runner.model_state.audio_buffer.pending_byte_count == FRAME_SIZE * 2
|
||||
|
||||
await h.run(frame(FRAME_SIZE // 2))
|
||||
assert len(h.port.submissions) == 1
|
||||
assert h.runner.model_state.audio_buffer.pending_byte_count == 0
|
||||
payload = submitted_duplex(h.port)[0]["payload"]
|
||||
assert isinstance(payload, dict)
|
||||
assert len(base64.b64decode(payload["audio"])) == FRAME_SIZE * 4
|
||||
finally:
|
||||
await close_harness(h)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_a_wrong_rate_append_is_refused_without_a_submission() -> None:
|
||||
h = await open_personaplex_harness()
|
||||
try:
|
||||
events = await h.run(
|
||||
commands.AppendAudio(audio=b"\x00" * 64, format="pcm_f32le", sample_rate_hz=16000, is_speech=None)
|
||||
)
|
||||
|
||||
error = find(events, "error")
|
||||
assert error.code == "bad_event"
|
||||
assert "24000" in error.message
|
||||
assert h.port.submissions == []
|
||||
finally:
|
||||
await close_harness(h)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cumulative_code2wav_output_becomes_24k_audio_and_text_deltas() -> None:
|
||||
h = await open_personaplex_harness()
|
||||
try:
|
||||
await h.run(frame())
|
||||
request_id = h.stage0_request_id(epoch=0)
|
||||
|
||||
first = await h.deliver_and_settle(code2wav_output(request_id, samples=FRAME_SIZE, text="he"), stage_id=1)
|
||||
second = await h.deliver_and_settle(
|
||||
code2wav_output(request_id, samples=2 * FRAME_SIZE, text="hello"), stage_id=1
|
||||
)
|
||||
|
||||
assert types(first)[0] == "response.created"
|
||||
delta = find(first, "response.output_audio.delta")
|
||||
assert delta.delta == f"wav-{FRAME_SIZE}"
|
||||
assert delta.sample_rate_hz == SAMPLE_RATE
|
||||
assert find(first, "response.output_audio_transcript.delta").delta == "he"
|
||||
assert h.session.active_response_id is not None
|
||||
|
||||
[delta_2] = [event for event in second if event.type == "response.output_audio.delta"]
|
||||
assert delta_2.delta == f"wav-{FRAME_SIZE}"
|
||||
assert find(second, "response.output_audio_transcript.delta").delta == "llo"
|
||||
assert not [event for event in second if event.type == "response.created"]
|
||||
finally:
|
||||
await close_harness(h)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_cancel_restarts_the_epoch_and_the_next_frame_replays_the_prefill() -> None:
|
||||
h = await open_personaplex_harness()
|
||||
try:
|
||||
await h.run(frame())
|
||||
old_request_id = h.stage0_request_id(epoch=0)
|
||||
await h.deliver_and_settle(code2wav_output(old_request_id, samples=FRAME_SIZE, text="he"), stage_id=1)
|
||||
|
||||
cancelled = await h.run(commands.CancelResponse())
|
||||
await h.run(frame())
|
||||
|
||||
assert h.session.epoch == 1
|
||||
assert "audio.cancelled" in types(cancelled) or "response.done" in types(cancelled)
|
||||
assert ([old_request_id], True) in h.port.cleanups
|
||||
assert h.port.submissions[-1].context.request_id == h.stage0_request_id(epoch=1)
|
||||
assert submitted_duplex(h.port)[-1]["seq"] == 1
|
||||
assert len(h.port.submissions[-1].prompt["prompt_token_ids"]) == 1 + PREFILL_SLOTS
|
||||
finally:
|
||||
await close_harness(h)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_close_aborts_the_stage0_request() -> None:
|
||||
h = await open_personaplex_harness()
|
||||
await h.run(frame())
|
||||
request_id = h.stage0_request_id(epoch=0)
|
||||
|
||||
await h.run(commands.CloseSession(reason="client_close"))
|
||||
|
||||
assert h.port.cleanups == [([request_id], True)]
|
||||
await close_harness(h)
|
||||
@@ -65,6 +65,7 @@ forbidden_prefixes = (
|
||||
"vllm_omni.entrypoints.duplex",
|
||||
"vllm_omni.entrypoints.duplex_omni",
|
||||
"vllm_omni.clients",
|
||||
"vllm_omni.model_executor.common.duplex",
|
||||
"vllm_omni.model_executor.models.minicpmo_4_5.duplex",
|
||||
"vllm_omni.model_executor.models.nemotron_voicechat.duplex",
|
||||
"vllm_omni.model_executor.models.personaplex.duplex",
|
||||
@@ -104,6 +105,7 @@ if missing:
|
||||
|
||||
forbidden_prefixes = (
|
||||
"vllm_omni.clients",
|
||||
"vllm_omni.model_executor.common.duplex",
|
||||
"vllm_omni.model_executor.models.minicpmo_4_5.duplex",
|
||||
"vllm_omni.model_executor.models.nemotron_voicechat.duplex",
|
||||
"vllm_omni.model_executor.models.personaplex.duplex",
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""The startup warmup sends the model's own silence unit, not a fixed 16 kHz one."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
|
||||
import pytest
|
||||
|
||||
from vllm_omni.entrypoints.duplex.warmup import warmup_silence_unit
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
class _Plugin:
|
||||
silence_continuation_samples = 1920
|
||||
|
||||
def silence_unit_payload(self) -> dict[str, object]:
|
||||
return {
|
||||
"type": "audio",
|
||||
"audio": base64.b64encode(bytes(1920 * 4)).decode("ascii"),
|
||||
"format": "pcm_f32le",
|
||||
"sample_rate_hz": 24000,
|
||||
}
|
||||
|
||||
|
||||
class _LegacyPlugin:
|
||||
silence_continuation_samples = 8000
|
||||
|
||||
|
||||
def test_the_plugin_unit_is_used_verbatim() -> None:
|
||||
unit = warmup_silence_unit(_Plugin())
|
||||
|
||||
assert unit["format"] == "pcm_f32le"
|
||||
assert unit["sample_rate_hz"] == 24000
|
||||
assert len(base64.b64decode(str(unit["audio"]))) == 1920 * 4
|
||||
|
||||
|
||||
def test_a_plugin_without_the_hook_falls_back_to_16k_zeros_of_its_sample_count() -> None:
|
||||
unit = warmup_silence_unit(_LegacyPlugin())
|
||||
|
||||
assert unit["sample_rate_hz"] == 16000
|
||||
assert unit["format"] == "pcm_f32le"
|
||||
assert len(base64.b64decode(str(unit["audio"]))) == 8000 * 4
|
||||
|
||||
|
||||
def test_no_plugin_means_the_minicpm_sized_default() -> None:
|
||||
unit = warmup_silence_unit(None)
|
||||
|
||||
assert unit["sample_rate_hz"] == 16000
|
||||
assert len(base64.b64decode(str(unit["audio"]))) == 16000 * 4
|
||||
@@ -0,0 +1,2 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
@@ -0,0 +1,2 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
@@ -0,0 +1,63 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import base64
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from vllm_omni.model_executor.common.audio.pcm import (
|
||||
PCM_F32LE_BYTES_PER_SAMPLE,
|
||||
decode_pcm_f32le_base64,
|
||||
pcm_f32le_sample_count,
|
||||
pcm_f32le_samples,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
def _b64(samples: np.ndarray) -> str:
|
||||
return base64.b64encode(np.ascontiguousarray(samples, dtype="<f4").tobytes()).decode("ascii")
|
||||
|
||||
|
||||
def test_decode_returns_the_raw_bytes_of_whole_finite_samples() -> None:
|
||||
samples = np.arange(4, dtype=np.float32)
|
||||
|
||||
raw = decode_pcm_f32le_base64(_b64(samples))
|
||||
|
||||
assert raw == samples.tobytes()
|
||||
assert pcm_f32le_sample_count(raw) == 4
|
||||
assert PCM_F32LE_BYTES_PER_SAMPLE == 4
|
||||
|
||||
|
||||
def test_decode_accepts_an_empty_chunk() -> None:
|
||||
assert decode_pcm_f32le_base64("") == b""
|
||||
assert pcm_f32le_sample_count(b"") == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("encoded", "match"),
|
||||
[
|
||||
(None, "must be base64 pcm_f32le"),
|
||||
(b"bytes", "must be base64 pcm_f32le"),
|
||||
("not*base64", "not valid base64"),
|
||||
(base64.b64encode(b"\x00\x00\x00").decode("ascii"), "divisible by four"),
|
||||
(_b64(np.array([0.0, np.nan], dtype=np.float32)), "must be finite"),
|
||||
(_b64(np.array([np.inf], dtype=np.float32)), "must be finite"),
|
||||
],
|
||||
)
|
||||
def test_decode_rejects_malformed_audio_naming_the_model(encoded: object, match: str) -> None:
|
||||
with pytest.raises(ValueError, match=match) as excinfo:
|
||||
decode_pcm_f32le_base64(encoded, model="TestModel")
|
||||
|
||||
assert str(excinfo.value).startswith("TestModel ")
|
||||
|
||||
|
||||
def test_samples_are_a_writable_contiguous_float32_copy() -> None:
|
||||
raw = np.arange(3, dtype="<f4").tobytes()
|
||||
|
||||
samples = pcm_f32le_samples(raw)
|
||||
|
||||
assert samples.dtype == np.float32
|
||||
assert samples.flags.writeable and samples.flags.c_contiguous
|
||||
assert samples.tolist() == [0.0, 1.0, 2.0]
|
||||
@@ -0,0 +1,2 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
@@ -0,0 +1,104 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from vllm_omni.engine.duplex.contracts import DuplexFence, duplex_resource_request_id
|
||||
from vllm_omni.engine.duplex.plugin import DuplexDataPlane, DuplexDataPlaneContext
|
||||
from vllm_omni.model_executor.common.duplex.data_plane import CumulativeAudioTextDataPlane
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
def _encode_audio(audio, _sample_rate, _response_format, _speed):
|
||||
if audio is None:
|
||||
return None
|
||||
size = int(np.asarray(audio, dtype=np.float32).size)
|
||||
return f"audio-{size}" if size else None
|
||||
|
||||
|
||||
def _output(request_id: str, *, text: str, samples: int, sr: int = 24000) -> SimpleNamespace:
|
||||
return SimpleNamespace(
|
||||
request_id=request_id,
|
||||
outputs=[SimpleNamespace(text=text, multimodal_output={})],
|
||||
multimodal_output={"model_outputs": np.arange(samples, dtype=np.float32), "sr": sr},
|
||||
finished=False,
|
||||
)
|
||||
|
||||
|
||||
class _TestDataPlane(CumulativeAudioTextDataPlane):
|
||||
stage_role = "codec"
|
||||
default_sample_rate_hz = 16000
|
||||
|
||||
|
||||
def test_is_a_duplex_data_plane_and_projects_cumulative_output_as_deltas() -> None:
|
||||
plane = CumulativeAudioTextDataPlane(_encode_audio)
|
||||
plane.begin_request("req")
|
||||
context = DuplexDataPlaneContext(response_format="wav", modalities=("audio", "text"))
|
||||
|
||||
projected = list(
|
||||
plane.project(
|
||||
{"data_plane_outputs": [_output("req", text="he", samples=4), _output("req", text="hello", samples=6)]},
|
||||
context=context,
|
||||
)
|
||||
)
|
||||
|
||||
assert isinstance(plane, DuplexDataPlane)
|
||||
assert [item["text"] for item in projected] == ["he", "llo"]
|
||||
assert [item["audio_data"] for item in projected] == ["audio-4", "audio-2"]
|
||||
assert [item["audio_duration_ms"] for item in projected] == [0, 0]
|
||||
assert all(item["sample_rate_hz"] == 24000 for item in projected)
|
||||
assert all(item["data_plane_request_id"] == "req" for item in projected)
|
||||
assert all(item["is_listen"] is False and item["end_of_turn"] is False for item in projected)
|
||||
assert projected[0]["stage_role"] == "tts"
|
||||
assert projected[0]["runtime_impl"] == "scheduler_data_plane"
|
||||
assert projected[0]["audio_format"] == "wav"
|
||||
|
||||
|
||||
def test_subclass_constants_and_default_sample_rate_reach_the_projection() -> None:
|
||||
plane = _TestDataPlane(_encode_audio)
|
||||
output = SimpleNamespace(
|
||||
request_id="req",
|
||||
outputs=[SimpleNamespace(text="", multimodal_output={})],
|
||||
multimodal_output={"audio": np.zeros(160, dtype=np.float32)},
|
||||
finished=False,
|
||||
)
|
||||
|
||||
[item] = list(plane.project({"data_plane_outputs": [output]}))
|
||||
|
||||
assert item["stage_role"] == "codec"
|
||||
assert item["sample_rate_hz"] == 16000
|
||||
assert item["audio_duration_ms"] == 10
|
||||
|
||||
|
||||
def test_nothing_new_yields_nothing_and_non_results_are_ignored() -> None:
|
||||
plane = CumulativeAudioTextDataPlane(_encode_audio)
|
||||
same = _output("req", text="hi", samples=4)
|
||||
|
||||
assert len(list(plane.project({"data_plane_outputs": [same, same]}))) == 1
|
||||
assert list(plane.project("not a dict")) == []
|
||||
assert list(plane.project({"data_plane_outputs": None})) == []
|
||||
|
||||
|
||||
def test_terminal_and_close_bookkeeping() -> None:
|
||||
plane = CumulativeAudioTextDataPlane(_encode_audio)
|
||||
session_request = duplex_resource_request_id(DuplexFence("sid", epoch=0), "stage0")
|
||||
plane.begin_request(session_request)
|
||||
plane.begin_request("other-req")
|
||||
|
||||
assert plane.is_terminal(None) is False
|
||||
assert plane.is_terminal(session_request) is False
|
||||
plane.mark_terminal(session_request)
|
||||
assert plane.is_terminal(session_request) is True
|
||||
plane.begin_request(session_request)
|
||||
assert plane.is_terminal(session_request) is False
|
||||
|
||||
plane.close_session("sid", active_request_id="other-req")
|
||||
assert plane._requests == {}
|
||||
|
||||
plane.mark_terminal("late")
|
||||
plane.close_stream("late")
|
||||
assert plane.is_terminal("late") is False
|
||||
@@ -0,0 +1,60 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import base64
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from vllm_omni.model_executor.common.duplex.payload import (
|
||||
decode_pcm_f32le_payload,
|
||||
payload_audio,
|
||||
payload_sample_count,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
def _payload(samples: int, *, sample_rate_hz: int = 24000, fmt: str = "pcm_f32le", key: str = "audio") -> dict:
|
||||
audio = base64.b64encode(np.zeros(samples, dtype="<f4").tobytes()).decode("ascii")
|
||||
return {"type": "audio", "format": fmt, "sample_rate_hz": sample_rate_hz, key: audio}
|
||||
|
||||
|
||||
def test_decode_returns_the_pcm_of_a_matching_payload() -> None:
|
||||
raw = decode_pcm_f32le_payload(_payload(1920), sample_rate_hz=24000, exact_samples=1920)
|
||||
|
||||
assert len(raw) == 1920 * 4
|
||||
|
||||
|
||||
def test_decode_accepts_the_older_data_key() -> None:
|
||||
assert len(decode_pcm_f32le_payload(_payload(8, key="data"), sample_rate_hz=24000)) == 32
|
||||
assert payload_audio({"data": "x"}) == "x"
|
||||
assert payload_audio("not a mapping") is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("payload", "match"),
|
||||
[
|
||||
("frame", "must be a mapping"),
|
||||
(_payload(1920, fmt="pcm16"), "format must be pcm_f32le"),
|
||||
(_payload(1920, sample_rate_hz=16000), "sample_rate_hz must be 24000"),
|
||||
(_payload(1919), "exactly 1920 samples"),
|
||||
({"format": "pcm_f32le", "sample_rate_hz": 24000, "audio": 12}, "must be base64"),
|
||||
],
|
||||
)
|
||||
def test_decode_rejects_payloads_that_do_not_match_the_model_unit(payload: object, match: str) -> None:
|
||||
with pytest.raises(ValueError, match=match) as excinfo:
|
||||
decode_pcm_f32le_payload(payload, sample_rate_hz=24000, exact_samples=1920, model="TestModel")
|
||||
|
||||
assert str(excinfo.value).startswith("TestModel ")
|
||||
|
||||
|
||||
def test_decode_without_exact_samples_accepts_any_whole_number_of_samples() -> None:
|
||||
assert len(decode_pcm_f32le_payload(_payload(7), sample_rate_hz=24000)) == 28
|
||||
|
||||
|
||||
def test_payload_sample_count_is_lenient() -> None:
|
||||
assert payload_sample_count(_payload(16000, sample_rate_hz=16000)) == 16000
|
||||
assert payload_sample_count(_payload(16000, fmt="pcm16")) is None
|
||||
assert payload_sample_count({"format": "pcm_f32le", "audio": "not*base64"}) is None
|
||||
assert payload_sample_count(None) is None
|
||||
@@ -0,0 +1,185 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import base64
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation
|
||||
from vllm_omni.model_executor.common.duplex.pcm_buffer import (
|
||||
FixedFramePcmAppendBuffer,
|
||||
FixedFramePcmAppendReservation,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
FRAME = 1920
|
||||
RATE = 24000
|
||||
PERIOD = 80
|
||||
|
||||
|
||||
def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = RATE, fmt: str = "pcm_f32le") -> dict[str, object]:
|
||||
samples = np.ascontiguousarray(samples, dtype="<f4")
|
||||
return {
|
||||
"type": "audio",
|
||||
"format": fmt,
|
||||
"sample_rate_hz": sample_rate_hz,
|
||||
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
|
||||
}
|
||||
|
||||
|
||||
def _buffer() -> FixedFramePcmAppendBuffer:
|
||||
return FixedFramePcmAppendBuffer(sample_rate_hz=RATE, frame_samples=FRAME, chunk_period_ms=PERIOD, model="Test")
|
||||
|
||||
|
||||
def _decode(payload: dict[str, object]) -> np.ndarray:
|
||||
return np.frombuffer(base64.b64decode(str(payload["audio"])), dtype="<f4")
|
||||
|
||||
|
||||
def test_buffer_implements_the_framework_contracts() -> None:
|
||||
buffer = _buffer()
|
||||
|
||||
assert isinstance(buffer, PcmAppendBuffer)
|
||||
assert buffer.frame_bytes == FRAME * 4
|
||||
reservation = buffer.prepare_append(
|
||||
_pcm_payload(np.arange(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True
|
||||
)
|
||||
assert isinstance(reservation, FixedFramePcmAppendReservation)
|
||||
assert isinstance(reservation, PcmAppendReservation)
|
||||
|
||||
|
||||
def test_one_whole_frame_is_reserved_transactionally() -> None:
|
||||
buffer = _buffer()
|
||||
|
||||
reservation = buffer.prepare_append(
|
||||
_pcm_payload(np.arange(FRAME)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True
|
||||
)
|
||||
|
||||
assert reservation is not None
|
||||
assert reservation.active
|
||||
assert reservation.byte_count == FRAME * 4
|
||||
assert reservation.payload is not None and reservation.payload["sample_rate_hz"] == RATE
|
||||
assert _decode(reservation.payload).tolist() == list(range(FRAME))
|
||||
assert buffer.pending_byte_count == 0
|
||||
assert buffer.has_reserved() and not buffer.has_pending()
|
||||
|
||||
reservation.rollback()
|
||||
|
||||
assert not reservation.active
|
||||
assert buffer.pending_byte_count == FRAME * 4
|
||||
assert buffer.has_pending() and not buffer.has_reserved()
|
||||
|
||||
|
||||
def test_partial_chunks_accumulate_until_a_whole_frame_exists() -> None:
|
||||
buffer = _buffer()
|
||||
half = _pcm_payload(np.zeros(FRAME // 2))
|
||||
|
||||
assert buffer.prepare_append(half, operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True) is None
|
||||
assert buffer.pending_byte_count == FRAME * 2
|
||||
reservation = buffer.prepare_append(half, operation_id="op-2", chunk_period_ms=PERIOD, allow_emit=True)
|
||||
|
||||
assert reservation is not None
|
||||
assert reservation.byte_count == FRAME * 4
|
||||
assert buffer.pending_byte_count == 0
|
||||
|
||||
|
||||
def test_appends_are_not_emitted_while_emission_is_disallowed() -> None:
|
||||
buffer = _buffer()
|
||||
|
||||
assert (
|
||||
buffer.prepare_append(
|
||||
_pcm_payload(np.zeros(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=False
|
||||
)
|
||||
is None
|
||||
)
|
||||
assert buffer.pending_byte_count == FRAME * 4
|
||||
|
||||
|
||||
def test_commit_flushes_a_padded_final_frame_and_an_empty_buffer_reserves_nothing() -> None:
|
||||
buffer = _buffer()
|
||||
buffer.prepare_append(_pcm_payload(np.ones(10)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=False)
|
||||
|
||||
reservation = buffer.prepare_commit(operation_id="commit", chunk_period_ms=PERIOD)
|
||||
|
||||
assert reservation.payload is not None
|
||||
assert reservation.payload["final"] is True
|
||||
decoded = _decode(reservation.payload)
|
||||
assert decoded.size == FRAME
|
||||
assert decoded[:10].tolist() == [1.0] * 10 and not decoded[10:].any()
|
||||
assert reservation.byte_count == 40
|
||||
reservation.commit()
|
||||
assert not buffer.has_reserved()
|
||||
|
||||
empty = buffer.prepare_commit(operation_id="commit-2", chunk_period_ms=PERIOD)
|
||||
assert empty.payload is None and empty.byte_count == 0
|
||||
empty.commit()
|
||||
assert buffer.flush(chunk_period_ms=PERIOD) is None
|
||||
|
||||
|
||||
def test_rolling_back_an_earlier_reservation_restores_later_ones_in_order() -> None:
|
||||
buffer = _buffer()
|
||||
first = buffer.prepare_append(
|
||||
_pcm_payload(np.full(FRAME, 1.0)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True
|
||||
)
|
||||
second = buffer.prepare_append(
|
||||
_pcm_payload(np.full(FRAME, 2.0)), operation_id="op-2", chunk_period_ms=PERIOD, allow_emit=True
|
||||
)
|
||||
assert first is not None and second is not None
|
||||
|
||||
first.rollback()
|
||||
|
||||
assert not first.active and not second.active
|
||||
assert buffer.pending_byte_count == 2 * FRAME * 4
|
||||
replay = buffer.prepare_append(
|
||||
_pcm_payload(np.zeros(0)), operation_id="op-3", chunk_period_ms=PERIOD, allow_emit=True
|
||||
)
|
||||
assert replay is not None and _decode(replay.payload)[0] == 1.0
|
||||
|
||||
|
||||
def test_clear_drops_pending_audio_and_deactivates_reservations() -> None:
|
||||
buffer = _buffer()
|
||||
reservation = buffer.prepare_append(
|
||||
_pcm_payload(np.zeros(FRAME + 8)), operation_id="op-1", chunk_period_ms=PERIOD, allow_emit=True
|
||||
)
|
||||
assert reservation is not None and buffer.pending_byte_count == 32
|
||||
|
||||
buffer.clear()
|
||||
buffer.clear_force_listen()
|
||||
|
||||
assert not reservation.active
|
||||
assert buffer.pending_byte_count == 0
|
||||
reservation.rollback() # no-op after clear
|
||||
assert buffer.pending_byte_count == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("payload", "match"),
|
||||
[
|
||||
(_pcm_payload(np.zeros(8), sample_rate_hz=16000), "sample_rate_hz must be 24000"),
|
||||
(_pcm_payload(np.zeros(8), fmt="pcm16"), "format must be pcm_f32le"),
|
||||
(_pcm_payload(np.array([0.0, np.nan])), "finite"),
|
||||
],
|
||||
)
|
||||
def test_malformed_audio_is_rejected_and_nothing_is_buffered(payload: dict[str, object], match: str) -> None:
|
||||
buffer = _buffer()
|
||||
|
||||
with pytest.raises(ValueError, match=match):
|
||||
buffer.prepare_append(payload, operation_id="op", chunk_period_ms=PERIOD, allow_emit=True)
|
||||
|
||||
assert buffer.pending_byte_count == 0
|
||||
|
||||
|
||||
def test_chunk_period_and_duplicate_operations_are_refused() -> None:
|
||||
buffer = _buffer()
|
||||
|
||||
with pytest.raises(ValueError, match="chunk_period_ms must be 80"):
|
||||
buffer.prepare_append(_pcm_payload(np.zeros(8)), operation_id="op", chunk_period_ms=1000, allow_emit=True)
|
||||
buffer.prepare_append(_pcm_payload(np.zeros(FRAME)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True)
|
||||
with pytest.raises(ValueError, match="duplicate active operation_id"):
|
||||
buffer.prepare_append(_pcm_payload(np.zeros(8)), operation_id="op", chunk_period_ms=PERIOD, allow_emit=True)
|
||||
|
||||
|
||||
def test_constructor_validates_its_parameters() -> None:
|
||||
with pytest.raises(ValueError, match="positive"):
|
||||
FixedFramePcmAppendBuffer(sample_rate_hz=0, frame_samples=FRAME, chunk_period_ms=PERIOD)
|
||||
@@ -0,0 +1,139 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from types import SimpleNamespace
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
from vllm_omni.model_executor.common.request_outputs import (
|
||||
audio_sample_count,
|
||||
audio_value,
|
||||
coerce_int,
|
||||
coerce_int_list,
|
||||
first_completion,
|
||||
multimodal_output,
|
||||
sample_rate_hz,
|
||||
slice_audio_delta,
|
||||
text_delta,
|
||||
text_value,
|
||||
unwrap_request_output,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
def test_unwrap_prefers_the_wrapped_request_output_and_its_first_completion() -> None:
|
||||
completion = SimpleNamespace(text="hi", multimodal_output={})
|
||||
inner = SimpleNamespace(request_id="req", outputs=[completion, SimpleNamespace(text="second")])
|
||||
wrapper = SimpleNamespace(request_output=inner)
|
||||
|
||||
output, first = unwrap_request_output(wrapper)
|
||||
|
||||
assert output is inner
|
||||
assert first is completion
|
||||
assert first_completion(SimpleNamespace(outputs=[])) is None
|
||||
assert first_completion(SimpleNamespace()) is None
|
||||
|
||||
|
||||
def test_multimodal_output_takes_the_first_non_empty_mapping_as_a_copy() -> None:
|
||||
completion = SimpleNamespace(multimodal_output={"audio": [1.0]})
|
||||
output = SimpleNamespace(multimodal_output={})
|
||||
|
||||
result = multimodal_output(output, completion)
|
||||
|
||||
assert result == {"audio": [1.0]}
|
||||
assert result is not completion.multimodal_output
|
||||
assert multimodal_output(SimpleNamespace(multimodal_output=None), None) == {}
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("value", "expected"),
|
||||
[
|
||||
(4, 4),
|
||||
("11", 11),
|
||||
(2.9, 2),
|
||||
(True, 1),
|
||||
(None, None),
|
||||
("x", None),
|
||||
(object(), None),
|
||||
([1], None),
|
||||
(torch.tensor([7]), 7),
|
||||
(torch.tensor(9), 9),
|
||||
(torch.tensor([], dtype=torch.long), None),
|
||||
(np.array([5], dtype=np.int32), 5),
|
||||
(np.array([], dtype=np.int32), None),
|
||||
],
|
||||
)
|
||||
def test_coerce_int_handles_scalars_strings_tensors_and_arrays(value: object, expected: int | None) -> None:
|
||||
assert coerce_int(value) == expected
|
||||
|
||||
|
||||
def test_coerce_int_list_flattens_tensors_and_drops_non_integers() -> None:
|
||||
assert coerce_int_list(torch.tensor([[1, 2], [3, 4]])) == [1, 2, 3, 4]
|
||||
assert coerce_int_list(np.array([5, 6])) == [5, 6]
|
||||
assert coerce_int_list([1, "2", "x", None, torch.tensor(3)]) == [1, 2, 3]
|
||||
assert coerce_int_list(None) == []
|
||||
assert coerce_int_list("12") == []
|
||||
|
||||
|
||||
def test_audio_value_reads_the_known_keys_and_unwraps_a_single_item_list() -> None:
|
||||
assert audio_value({"model_outputs": [np.zeros(2)]}).shape == (2,)
|
||||
assert audio_value({"latent": "x"}) == "x"
|
||||
assert audio_value({"audio": [1, 2]}) == [1, 2]
|
||||
assert audio_value({}) is None
|
||||
|
||||
|
||||
def test_text_value_prefers_multimodal_text_then_llm_text_then_the_completion() -> None:
|
||||
completion = SimpleNamespace(text="from completion")
|
||||
assert text_value({"text": "mm"}, completion) == "mm"
|
||||
assert text_value({"llm_output_text": "llm"}, completion) == "llm"
|
||||
assert text_value({}, completion) == "from completion"
|
||||
assert text_value({"text": ""}, None) == ""
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("text", "previous", "expected"),
|
||||
[("", "he", ""), ("he", "he", ""), ("hello", "he", "llo"), ("new", "old", "new"), ("abc", "", "abc")],
|
||||
)
|
||||
def test_text_delta(text: str, previous: str, expected: str) -> None:
|
||||
assert text_delta(text, previous) == expected
|
||||
|
||||
|
||||
def test_slice_audio_delta_returns_only_new_samples_for_cumulative_audio() -> None:
|
||||
cumulative = np.arange(6, dtype=np.float32)
|
||||
|
||||
assert slice_audio_delta(cumulative, 4).tolist() == [4.0, 5.0]
|
||||
assert slice_audio_delta(cumulative, 6) is None
|
||||
assert slice_audio_delta(cumulative, 0).tolist() == cumulative.tolist()
|
||||
# A shorter waveform than the cursor means the stream restarted: send it whole.
|
||||
assert slice_audio_delta(np.arange(3, dtype=np.float32), 4).tolist() == [0.0, 1.0, 2.0]
|
||||
assert slice_audio_delta(None, 2) is None
|
||||
assert slice_audio_delta(np.zeros(0, dtype=np.float32), 0) is None
|
||||
|
||||
|
||||
def test_slice_audio_delta_keeps_tensors_as_contiguous_tensors() -> None:
|
||||
delta = slice_audio_delta(torch.arange(6, dtype=torch.float32).reshape(2, 3), 4)
|
||||
|
||||
assert isinstance(delta, torch.Tensor)
|
||||
assert delta.is_contiguous()
|
||||
assert delta.tolist() == [4.0, 5.0]
|
||||
assert audio_sample_count(torch.zeros(2, 3)) == 6
|
||||
assert audio_sample_count("not audio") is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("multimodal", "expected"),
|
||||
[
|
||||
({"sr": 16000}, 16000),
|
||||
({"sample_rate_hz": 22050}, 22050),
|
||||
({"sr": [48000]}, 48000),
|
||||
({"sr": np.array([8000])}, 8000),
|
||||
({"sr": "bad"}, 24000),
|
||||
({"sr": 0}, 24000),
|
||||
({}, 24000),
|
||||
],
|
||||
)
|
||||
def test_sample_rate_hz_falls_back_to_the_default(multimodal: dict, expected: int) -> None:
|
||||
assert sample_rate_hz(multimodal, default=24000) == expected
|
||||
@@ -1,21 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""PersonaPlex duplex is not on the duplex plugin framework yet.
|
||||
|
||||
RFC vllm-omni#7181 splits the port: this PR lands the unified framework with
|
||||
MiniCPM-o 4.5, and the follow-up PR moves PersonaPlex onto ``DuplexModelPlugin``.
|
||||
Until then the modules under test still import the pre-framework duplex
|
||||
runtime / serving adapter that this PR removed, so collecting them would fail
|
||||
at import time and abort the whole pytest session. The tests themselves are
|
||||
unchanged and come back with the port.
|
||||
|
||||
Listed per file rather than globbed, so the coverage loss is visible and shrinks
|
||||
file by file as the port lands.
|
||||
"""
|
||||
|
||||
#: ``test_unified_runtime`` imports ``engine.duplex.runtime`` and
|
||||
#: ``entrypoints.duplex.runtime_adapter`` directly. ``test_stage0_runtime`` fails
|
||||
#: transitively: ``personaplex.duplex.runtime_extension`` imports ``DuplexFence``
|
||||
#: from ``engine.duplex.messages``, which no longer exports it.
|
||||
collect_ignore = ["test_stage0_runtime.py", "test_unified_runtime.py"]
|
||||
@@ -0,0 +1,489 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""PersonaPlex on the unified full-duplex framework: the plugin contract and the E2E driver helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import cast
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from vllm.sampling_params import SamplingParams
|
||||
|
||||
from tests.e2e.online_serving import personaplex_realtime_duplex as e2e_driver
|
||||
from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy
|
||||
from vllm_omni.engine.duplex.config import DuplexSessionConfig
|
||||
from vllm_omni.engine.duplex.contracts import DuplexFence
|
||||
from vllm_omni.engine.duplex.plugin import (
|
||||
DefaultDuplexModelSessionState,
|
||||
DuplexDataPlaneContext,
|
||||
DuplexRuntimeConfigError,
|
||||
load_duplex_plugin,
|
||||
validate_duplex_plugin_sampling,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex import stage0
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA, FRAME_SIZE, SAMPLE_RATE
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import PersonaPlexDataPlaneSession
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.input import PersonaPlexPcmAppendBuffer
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.plugin import (
|
||||
PRIVATE_RUNTIME_CONFIG_KEYS,
|
||||
PersonaPlexDuplexPlugin,
|
||||
PersonaPlexSessionState,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.pipeline import PERSONAPLEX_PIPELINE
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
DEPLOY_PATH = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml"
|
||||
|
||||
|
||||
def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = SAMPLE_RATE) -> dict[str, object]:
|
||||
samples = np.ascontiguousarray(samples, dtype="<f4")
|
||||
return {
|
||||
"type": "audio",
|
||||
"format": "pcm_f32le",
|
||||
"sample_rate_hz": sample_rate_hz,
|
||||
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
|
||||
}
|
||||
|
||||
|
||||
def _encode_audio(audio, _sample_rate, _response_format, _speed):
|
||||
if audio is None:
|
||||
return None
|
||||
size = int(np.asarray(audio, dtype=np.float32).size)
|
||||
return f"audio-{size}" if size else None
|
||||
|
||||
|
||||
def _plugin() -> PersonaPlexDuplexPlugin:
|
||||
return PersonaPlexDuplexPlugin(_encode_audio)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Pipeline binding #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_pipeline_declares_the_duplex_plugin_and_no_pre_framework_wiring() -> None:
|
||||
assert PERSONAPLEX_PIPELINE.duplex_plugin == (
|
||||
"vllm_omni.model_executor.models.personaplex.duplex.plugin.PersonaPlexDuplexPlugin"
|
||||
)
|
||||
assert PERSONAPLEX_PIPELINE.duplex_runtime_extension is None
|
||||
assert PERSONAPLEX_PIPELINE.duplex_serving_adapter is None
|
||||
assert PERSONAPLEX_PIPELINE.duplex_control_enabled is False
|
||||
|
||||
plugin = load_duplex_plugin(PERSONAPLEX_PIPELINE.duplex_plugin, _encode_audio)
|
||||
|
||||
assert isinstance(plugin, PersonaPlexDuplexPlugin)
|
||||
assert plugin.plugin_id == "personaplex"
|
||||
assert isinstance(plugin.data_plane, PersonaPlexDataPlaneSession)
|
||||
validate_duplex_plugin_sampling(plugin, sampling_defaults=(SamplingParams(), SamplingParams()))
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("engine_arg", "expected"),
|
||||
[("skip_tokenizer_init", True), ("enable_prefix_caching", False)],
|
||||
)
|
||||
def test_personaplex_stage_engine_args(engine_arg: str, expected: bool) -> None:
|
||||
stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, load_deploy_config(DEPLOY_PATH))
|
||||
|
||||
assert [stage.yaml_engine_args.get(engine_arg) for stage in stages] == [expected, expected]
|
||||
|
||||
|
||||
def test_personaplex_deploy_is_duplex_and_propagates_capacity_to_all_model_stages() -> None:
|
||||
deploy = load_deploy_config(DEPLOY_PATH)
|
||||
stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, deploy)
|
||||
|
||||
assert deploy.session_mode == "duplex"
|
||||
assert deploy.duplex_session.max_sessions == 2
|
||||
assert [stage.yaml_engine_args.get("duplex_max_sessions") for stage in stages] == [2, 2]
|
||||
assert "personaplex_codec_max_sessions" not in deploy.connectors["connector_of_shared_memory"]["extra"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Session policy #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_capabilities_are_honest() -> None:
|
||||
single = _plugin().capabilities(max_sessions=1)
|
||||
multi = _plugin().capabilities(max_sessions=2)
|
||||
|
||||
assert multi.as_dict()["implementation_level"] == "model_native_duplex"
|
||||
assert multi.as_dict()["input_modes"] == ["append_audio_chunk"]
|
||||
assert multi.chunk_period_ms == 80
|
||||
assert single.supports_multi_session is False
|
||||
assert multi.supports_multi_session is True
|
||||
assert multi.supports_multi_session_same_replica is True
|
||||
assert multi.supports_barge_in is False
|
||||
assert multi.supports_client_commit is False
|
||||
assert multi.supports_external_turn_signal is False
|
||||
assert multi.supports_session_resume is False
|
||||
assert multi.supports_audio_truncate is False
|
||||
assert multi.supports_chat_completions is False
|
||||
assert multi.session_admission_mode == "engine_managed"
|
||||
|
||||
|
||||
def test_session_state_is_the_framework_default_with_the_personaplex_buffer() -> None:
|
||||
state = _plugin().create_session_state()
|
||||
|
||||
assert isinstance(state, PersonaPlexSessionState)
|
||||
assert isinstance(state, DefaultDuplexModelSessionState)
|
||||
assert isinstance(state.audio_buffer, PersonaPlexPcmAppendBuffer)
|
||||
state.retain_committed_audio({"audio": ""}, operation_id="op", reserved_bytes=8)
|
||||
assert state.clear_committed_audio() == 8
|
||||
assert state.committed_audio_payload is None
|
||||
|
||||
|
||||
def test_private_runtime_keys_are_rejected_in_extra_body() -> None:
|
||||
plugin = _plugin()
|
||||
|
||||
assert PRIVATE_RUNTIME_CONFIG_KEYS == {
|
||||
"personaplex_prefill_slots",
|
||||
"personaplex_model_path",
|
||||
"personaplex_voice_prompt",
|
||||
"personaplex_persona",
|
||||
}
|
||||
plugin.validate_client_extra_body({"auto_response": True})
|
||||
plugin.validate_client_extra_body(None)
|
||||
with pytest.raises(DuplexRuntimeConfigError, match="personaplex_persona") as excinfo:
|
||||
plugin.validate_client_extra_body({"personaplex_persona": "x", "auto_response": True})
|
||||
assert excinfo.value.code == "invalid_duplex_runtime_config"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prepare_runtime_config_resolves_voice_and_prefill_slots_once(monkeypatch) -> None:
|
||||
calls: list[tuple[str, str, str]] = []
|
||||
|
||||
def fake_prefill_slots(model_path: str, voice: str, persona: str) -> int:
|
||||
calls.append((model_path, voice, persona))
|
||||
return 42
|
||||
|
||||
monkeypatch.setattr(stage0, "personaplex_prefill_slots", fake_prefill_slots)
|
||||
plugin = _plugin()
|
||||
model_config = SimpleNamespace(model="/models/personaplex")
|
||||
|
||||
runtime_config = await plugin.prepare_runtime_config(
|
||||
DuplexSessionConfig(voice="NATM1.pt", instructions="Be brief."),
|
||||
model_config=model_config,
|
||||
)
|
||||
again = await plugin.prepare_runtime_config(
|
||||
DuplexSessionConfig(voice="NATM1.pt", instructions="Be brief."),
|
||||
model_config=model_config,
|
||||
)
|
||||
defaults = await plugin.prepare_runtime_config(DuplexSessionConfig(), model_config=model_config)
|
||||
|
||||
assert runtime_config == {
|
||||
"personaplex_model_path": "/models/personaplex",
|
||||
"personaplex_voice_prompt": "NATM1.pt",
|
||||
"personaplex_persona": "Be brief.",
|
||||
"personaplex_prefill_slots": 42,
|
||||
}
|
||||
assert again == runtime_config
|
||||
assert defaults["personaplex_voice_prompt"] == "NATF2.pt"
|
||||
assert defaults["personaplex_persona"] == DEFAULT_PERSONA
|
||||
# The voice bundle and tokenizer are read once per (model, voice, persona).
|
||||
assert calls == [
|
||||
("/models/personaplex", "NATM1.pt", "Be brief."),
|
||||
("/models/personaplex", "NATF2.pt", DEFAULT_PERSONA),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_prepare_runtime_config_reports_prefill_failures_as_runtime_config_errors(monkeypatch) -> None:
|
||||
def failing_prefill_slots(model_path: str, voice: str, persona: str) -> int:
|
||||
raise FileNotFoundError("voices.tgz missing")
|
||||
|
||||
monkeypatch.setattr(stage0, "personaplex_prefill_slots", failing_prefill_slots)
|
||||
|
||||
with pytest.raises(DuplexRuntimeConfigError, match="voices.tgz missing") as excinfo:
|
||||
await _plugin().prepare_runtime_config(DuplexSessionConfig(), model_config=SimpleNamespace(model="/m"))
|
||||
assert excinfo.value.code == "prefill_unavailable"
|
||||
|
||||
with pytest.raises(DuplexRuntimeConfigError, match="model path") as excinfo:
|
||||
await _plugin().prepare_runtime_config(DuplexSessionConfig(), model_config=None)
|
||||
assert excinfo.value.code == "model_path_unavailable"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("voice", ["../NATF2.pt", "voices/NATF2.pt", "NATF2.wav", "/abs/NATF2.pt"])
|
||||
@pytest.mark.asyncio
|
||||
async def test_prepare_runtime_config_rejects_voices_that_are_not_bundled_basenames(voice: str) -> None:
|
||||
with pytest.raises(DuplexRuntimeConfigError, match="bundled .pt basename") as excinfo:
|
||||
await _plugin().prepare_runtime_config(
|
||||
DuplexSessionConfig(voice=voice), model_config=SimpleNamespace(model="/m")
|
||||
)
|
||||
assert excinfo.value.code == "invalid_voice"
|
||||
|
||||
|
||||
def test_runtime_config_update_rejects_changed_persona() -> None:
|
||||
plugin = _plugin()
|
||||
short_persona = "You are a helpful assistant."
|
||||
current = {"personaplex_persona": short_persona, "personaplex_voice_prompt": "NATF2.pt"}
|
||||
|
||||
with pytest.raises(DuplexRuntimeConfigError, match="persona") as excinfo:
|
||||
plugin.runtime_config_for_update(DuplexSessionConfig(instructions="You are a pirate."), current)
|
||||
assert excinfo.value.code == "persona_update_unsupported"
|
||||
|
||||
unchanged = plugin.runtime_config_for_update(DuplexSessionConfig(instructions=short_persona), current)
|
||||
assert unchanged["personaplex_persona"] == short_persona
|
||||
assert unchanged is not current
|
||||
|
||||
|
||||
def test_runtime_config_update_rejects_changed_voice() -> None:
|
||||
plugin = _plugin()
|
||||
current = {"personaplex_persona": DEFAULT_PERSONA, "personaplex_voice_prompt": "NATF2.pt"}
|
||||
|
||||
with pytest.raises(DuplexRuntimeConfigError, match="voice") as excinfo:
|
||||
plugin.runtime_config_for_update(DuplexSessionConfig(voice="NATM1.pt"), current)
|
||||
assert excinfo.value.code == "voice_update_unsupported"
|
||||
|
||||
unchanged = plugin.runtime_config_for_update(DuplexSessionConfig(voice="NATF2.pt"), current)
|
||||
assert unchanged["personaplex_voice_prompt"] == "NATF2.pt"
|
||||
|
||||
|
||||
def test_data_plane_context_is_the_framework_default() -> None:
|
||||
context = _plugin().data_plane_context(
|
||||
epoch=1,
|
||||
turn_id=2,
|
||||
active_response_turn_id=None,
|
||||
active_response_id="resp",
|
||||
auto_responds=True,
|
||||
response_format="pcm16",
|
||||
speed=None,
|
||||
modalities=("audio",),
|
||||
)
|
||||
|
||||
assert isinstance(context, DuplexDataPlaneContext)
|
||||
assert context.response_format == "pcm16" and context.epoch == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Engine policy #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def test_sampling_params_are_greedy_one_token_on_stage0_only() -> None:
|
||||
defaults = (SamplingParams(temperature=0.8, top_k=10, max_tokens=10), SamplingParams(max_tokens=1024))
|
||||
|
||||
configured = _plugin().configure_sampling_params(runtime_config={}, defaults=defaults)
|
||||
|
||||
assert configured[0].temperature == 0.0
|
||||
assert configured[0].top_k == 1
|
||||
assert configured[0].max_tokens == 1
|
||||
assert configured[1] is defaults[1]
|
||||
assert defaults[0].max_tokens == 10
|
||||
assert _plugin().configure_sampling_params(runtime_config={}, defaults=()) == ()
|
||||
|
||||
|
||||
def test_plan_append_reserves_one_slot_plus_prefill_on_the_first_seq() -> None:
|
||||
plugin = _plugin()
|
||||
fence = DuplexFence("session", epoch=3, turn_id=1)
|
||||
common = {
|
||||
"request_id": "req",
|
||||
"fence": fence,
|
||||
"session_config": {"instructions": "Be concise."},
|
||||
"runtime_config": {"personaplex_prefill_slots": 4},
|
||||
"turn_seq": 1,
|
||||
"payload": _pcm_payload(np.zeros(FRAME_SIZE, np.float32)),
|
||||
"final": False,
|
||||
"sampling_params": SamplingParams(),
|
||||
}
|
||||
|
||||
first = plugin.plan_append(seq=1, **common)
|
||||
second = plugin.plan_append(seq=2, **common)
|
||||
|
||||
assert len(first.prompt["prompt_token_ids"]) == 5
|
||||
assert len(second.prompt["prompt_token_ids"]) == 1
|
||||
duplex = first.prompt["model_intermediate_buffer"]["duplex"]
|
||||
assert duplex["session_id"] == "session"
|
||||
assert duplex["epoch"] == 3
|
||||
assert duplex["turn_id"] == 1
|
||||
assert duplex["seq"] == 1
|
||||
assert duplex["data_plane"] is True
|
||||
assert duplex["fence"] == fence
|
||||
assert duplex["scheduler_token_budget"] == 5
|
||||
assert duplex["runtime_config"] == {"personaplex_prefill_slots": 4}
|
||||
assert "incarnation" not in duplex
|
||||
assert first.prompt["model_intermediate_buffer"]["global_request_id"] == ["session"]
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("payload", "match"),
|
||||
[
|
||||
(_pcm_payload(np.zeros(FRAME_SIZE - 1, np.float32)), "exactly 1920 samples"),
|
||||
(_pcm_payload(np.zeros(FRAME_SIZE, np.float32), sample_rate_hz=16000), "sample_rate_hz must be 24000"),
|
||||
({"format": "pcm16", "sample_rate_hz": SAMPLE_RATE, "audio": ""}, "format must be pcm_f32le"),
|
||||
("frame", "must be a mapping"),
|
||||
],
|
||||
)
|
||||
def test_plan_append_rejects_anything_but_one_24k_frame(payload: object, match: str) -> None:
|
||||
with pytest.raises(ValueError, match=match):
|
||||
_plugin().plan_append(
|
||||
request_id="req",
|
||||
fence=DuplexFence("session"),
|
||||
session_config={},
|
||||
runtime_config={},
|
||||
seq=1,
|
||||
turn_seq=1,
|
||||
payload=payload,
|
||||
final=False,
|
||||
sampling_params=None,
|
||||
)
|
||||
|
||||
|
||||
def test_plan_append_rejects_a_malformed_prefill_slot_count() -> None:
|
||||
with pytest.raises(ValueError, match="personaplex_prefill_slots"):
|
||||
_plugin().plan_append(
|
||||
request_id="req",
|
||||
fence=DuplexFence("session"),
|
||||
session_config={},
|
||||
runtime_config={"personaplex_prefill_slots": "many"},
|
||||
seq=1,
|
||||
turn_seq=1,
|
||||
payload=_pcm_payload(np.zeros(FRAME_SIZE, np.float32)),
|
||||
final=False,
|
||||
sampling_params=None,
|
||||
)
|
||||
|
||||
|
||||
def test_decide_output_never_decides() -> None:
|
||||
assert (
|
||||
_plugin().decide_output(
|
||||
stage_id=0,
|
||||
final_stage_id=1,
|
||||
segment_finished=True,
|
||||
segment_token_ids=(1,),
|
||||
segment_output_metadata={},
|
||||
output=SimpleNamespace(outputs=[]),
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
|
||||
def test_silence_unit_is_one_24k_frame_that_plan_append_accepts() -> None:
|
||||
plugin = _plugin()
|
||||
|
||||
unit = plugin.silence_unit_payload()
|
||||
|
||||
assert unit["format"] == "pcm_f32le"
|
||||
assert unit["sample_rate_hz"] == SAMPLE_RATE
|
||||
assert len(base64.b64decode(unit["audio"])) == FRAME_SIZE * 4
|
||||
plan = plugin.plan_append(
|
||||
request_id="req",
|
||||
fence=DuplexFence("session"),
|
||||
session_config={},
|
||||
runtime_config={},
|
||||
seq=7,
|
||||
turn_seq=7,
|
||||
payload=unit,
|
||||
final=False,
|
||||
sampling_params=None,
|
||||
)
|
||||
assert len(plan.prompt["prompt_token_ids"]) == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# E2E driver helpers (CPU-checked parts of the GPU driver) #
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
|
||||
def _audio_client(frame_count: int = 10, *, voiced_frames: int = 0) -> e2e_driver.RawRealtimeProbe:
|
||||
"""A probe double carrying only the collected events the driver's result builder reads."""
|
||||
frames = np.full((frame_count, e2e_driver.FRAME_SAMPLES), 7, dtype="<i2")
|
||||
frames[:voiced_frames] = 1000
|
||||
raw = frames.tobytes()
|
||||
event = {
|
||||
"type": "response.output_audio.delta",
|
||||
"response_id": "response-1",
|
||||
"sample_rate_hz": e2e_driver.SAMPLE_RATE_HZ,
|
||||
"metadata": {
|
||||
"vllm_omni": {
|
||||
"runtime_impl": "scheduler_data_plane",
|
||||
"uses_model_runner_scheduler": True,
|
||||
"runner_kv_backed": True,
|
||||
}
|
||||
},
|
||||
}
|
||||
events = SimpleNamespace(events=[event], response_audio={"response-1": [raw]}, audio_bytes=lambda: raw)
|
||||
return cast(e2e_driver.RawRealtimeProbe, SimpleNamespace(events=events))
|
||||
|
||||
|
||||
def test_realtime_audio_frame_stats_separate_voiced_and_silent_frames() -> None:
|
||||
silent = np.zeros(e2e_driver.FRAME_SAMPLES, dtype="<i2")
|
||||
voiced = np.full(e2e_driver.FRAME_SAMPLES, 1000, dtype="<i2")
|
||||
|
||||
stats = e2e_driver._audio_frame_stats(
|
||||
silent.tobytes() + voiced.tobytes(),
|
||||
input_frames=3,
|
||||
voiced_frame_rms_threshold=1e-4,
|
||||
)
|
||||
|
||||
assert stats == {
|
||||
"output_frames": 2,
|
||||
"voiced_frames": 1,
|
||||
"silent_frames": 1,
|
||||
"frame_deficit": 1,
|
||||
"frame_coverage_ratio": pytest.approx(2 / 3),
|
||||
}
|
||||
|
||||
|
||||
def test_realtime_audio_frame_stats_reject_partial_codec_frame() -> None:
|
||||
partial_frame = np.zeros(e2e_driver.FRAME_SAMPLES - 1, dtype="<i2")
|
||||
|
||||
with pytest.raises(AssertionError, match="whole 80 ms codec frames"):
|
||||
e2e_driver._audio_frame_stats(partial_frame.tobytes(), input_frames=1, voiced_frame_rms_threshold=1e-4)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("frame_count", [10, 40])
|
||||
def test_e2e_driver_rejects_inaudible_sessions(frame_count: int) -> None:
|
||||
args = argparse.Namespace(max_frame_deficit=4, voiced_frame_rms_threshold=1e-3, min_voiced_frames=5)
|
||||
|
||||
with pytest.raises(AssertionError, match="audible speech"):
|
||||
e2e_driver._session_result(_audio_client(frame_count), input_frames=frame_count, args=args, minimum_chunks=1)
|
||||
|
||||
|
||||
def test_e2e_driver_uses_absolute_audible_floor() -> None:
|
||||
args = argparse.Namespace(max_frame_deficit=4, voiced_frame_rms_threshold=1e-3, min_voiced_frames=5)
|
||||
|
||||
e2e_driver._session_result(_audio_client(200, voiced_frames=5), input_frames=200, args=args, minimum_chunks=1)
|
||||
|
||||
|
||||
def test_e2e_driver_defaults_replacement_to_full_input() -> None:
|
||||
args = e2e_driver.parse_args(["--model", "/model", "--input-wav", "/input.wav"])
|
||||
|
||||
assert args.replacement_frames == 0
|
||||
assert args.min_voiced_frames == 5
|
||||
|
||||
|
||||
def test_e2e_driver_records_and_validates_input_sha256(tmp_path: Path) -> None:
|
||||
input_wav = tmp_path / "input.wav"
|
||||
input_wav.write_bytes(b"fixture-wav")
|
||||
|
||||
identity = e2e_driver._input_identity(
|
||||
input_wav,
|
||||
expected_sha256="bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
|
||||
)
|
||||
|
||||
assert identity == {
|
||||
"path": str(input_wav.resolve()),
|
||||
"sha256": "bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
|
||||
}
|
||||
|
||||
|
||||
def test_e2e_driver_rejects_unexpected_input_sha256(tmp_path: Path) -> None:
|
||||
input_wav = tmp_path / "input.wav"
|
||||
input_wav.write_bytes(b"fixture-wav")
|
||||
|
||||
with pytest.raises(ValueError, match="input WAV SHA-256 mismatch"):
|
||||
e2e_driver._input_identity(input_wav, expected_sha256="0" * 64)
|
||||
|
||||
|
||||
def test_e2e_driver_lets_the_server_allocate_the_session_id() -> None:
|
||||
url = e2e_driver._realtime_url("ws://127.0.0.1:8099/v1/realtime?duplex=1", "nvidia/personaplex-7b-v1")
|
||||
|
||||
assert "session_id=" not in url
|
||||
assert "duplex=1" in url and "autostart=0" in url
|
||||
@@ -14,6 +14,7 @@ from vllm_omni.model_executor.models.personaplex.duplex.policy import (
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.stage0 import (
|
||||
PersonaPlexStage0DuplexRuntime,
|
||||
PersonaPlexStage0PreparedAppend,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.personaplex_talker import (
|
||||
PersonaPlexTalkerForConditionalGeneration,
|
||||
@@ -80,13 +81,12 @@ class _FakeTalker:
|
||||
return torch.full((1, 4), value, device=device)
|
||||
|
||||
|
||||
def _duplex_info(*, seq: int, session_id: str = "session", incarnation: int = 1):
|
||||
def _duplex_info(*, seq: int, session_id: str = "session", epoch: int = 0):
|
||||
pcm = np.zeros(1920, dtype="<f4")
|
||||
return {
|
||||
"data_plane": True,
|
||||
"session_id": session_id,
|
||||
"incarnation": incarnation,
|
||||
"epoch": 0,
|
||||
"epoch": epoch,
|
||||
"seq": seq,
|
||||
"payload": {
|
||||
"format": "pcm_f32le",
|
||||
@@ -103,7 +103,7 @@ def _duplex_info(*, seq: int, session_id: str = "session", incarnation: int = 1)
|
||||
|
||||
def _runtime(*codecs: _FakeCodec) -> PersonaPlexStage0DuplexRuntime:
|
||||
voice_embeddings = torch.arange(8, dtype=torch.float32).reshape(2, 1, 1, 4)
|
||||
codec_args = {"codec": codecs[0]}
|
||||
codec_args: dict[str, object] = {"codec": codecs[0]}
|
||||
if len(codecs) > 1:
|
||||
available = iter(codecs)
|
||||
codec_args = {
|
||||
@@ -248,15 +248,20 @@ def test_decoded_pcm_is_writable_for_torch_zero_copy() -> None:
|
||||
|
||||
def _prepare_two_sessions(
|
||||
runtime: PersonaPlexStage0DuplexRuntime,
|
||||
) -> tuple[object, object, object, object]:
|
||||
) -> tuple[
|
||||
PersonaPlexStage0PreparedAppend,
|
||||
PersonaPlexStage0PreparedAppend,
|
||||
PersonaPlexStage0PreparedAppend,
|
||||
PersonaPlexStage0PreparedAppend,
|
||||
]:
|
||||
first_1 = runtime.prepare_append(_duplex_info(seq=1), prompt_len=18)
|
||||
second_1 = runtime.prepare_append(
|
||||
_duplex_info(seq=1, session_id="other", incarnation=2),
|
||||
_duplex_info(seq=1, session_id="other"),
|
||||
prompt_len=18,
|
||||
)
|
||||
first_2 = runtime.prepare_append(_duplex_info(seq=2), prompt_len=19)
|
||||
second_2 = runtime.prepare_append(
|
||||
_duplex_info(seq=2, session_id="other", incarnation=2),
|
||||
_duplex_info(seq=2, session_id="other"),
|
||||
prompt_len=19,
|
||||
)
|
||||
return first_1, second_1, first_2, second_2
|
||||
@@ -281,7 +286,7 @@ def test_stage0_session_capacity_fails_before_codec_state_is_shared() -> None:
|
||||
|
||||
with pytest.raises(RuntimeError, match="capacity 2"):
|
||||
runtime.prepare_append(
|
||||
_duplex_info(seq=1, session_id="third", incarnation=3),
|
||||
_duplex_info(seq=1, session_id="third"),
|
||||
prompt_len=18,
|
||||
)
|
||||
|
||||
@@ -292,12 +297,55 @@ def test_close_session_resets_and_reuses_released_codec() -> None:
|
||||
runtime = _runtime(first_codec, second_codec)
|
||||
_prepare_two_sessions(runtime)
|
||||
|
||||
runtime.close_session("session", 1)
|
||||
runtime.close_session("session", 0)
|
||||
|
||||
assert first_codec.reset_calls == 1
|
||||
assert second_codec.reset_calls == 0
|
||||
replacement = runtime.prepare_append(
|
||||
_duplex_info(seq=1, session_id="replacement", incarnation=4),
|
||||
_duplex_info(seq=1, session_id="replacement"),
|
||||
prompt_len=18,
|
||||
)
|
||||
assert replacement.user_codes[:, 0].tolist() == [3]
|
||||
|
||||
|
||||
def test_a_new_epoch_replays_the_prefill_and_recycles_the_codec() -> None:
|
||||
codec = _FakeCodec()
|
||||
runtime = _runtime(codec)
|
||||
runtime.prepare_append(_duplex_info(seq=1), prompt_len=18, request_id="req-e0")
|
||||
runtime.prepare_append(_duplex_info(seq=2), prompt_len=19, request_id="req-e0")
|
||||
|
||||
# A cancel advanced the fence: the next append is seq 1 of epoch 1 on a
|
||||
# fresh Stage 0 request, so the voice/persona prefill is replayed and the
|
||||
# earlier epoch's lockstep state is released first.
|
||||
restarted = runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18, request_id="req-e1")
|
||||
|
||||
assert restarted.prefill_applied is True
|
||||
assert restarted.prompt_offset == 0
|
||||
assert restarted.user_codes.shape == (1, 8)
|
||||
assert list(runtime.sessions) == [("session", 1)]
|
||||
assert runtime.request_sessions == {"req-e1": ("session", 1)}
|
||||
assert codec.reset_calls == 1
|
||||
assert runtime.sessions[("session", 1)].codec is codec
|
||||
|
||||
|
||||
def test_a_late_finish_of_the_old_epoch_request_does_not_close_the_new_state() -> None:
|
||||
runtime = _runtime(_FakeCodec())
|
||||
runtime.prepare_append(_duplex_info(seq=1), prompt_len=18, request_id="req-e0")
|
||||
runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18, request_id="req-e1")
|
||||
|
||||
runtime.close_request("req-e0")
|
||||
|
||||
assert list(runtime.sessions) == [("session", 1)]
|
||||
runtime.close_request("req-e1")
|
||||
assert runtime.sessions == {}
|
||||
|
||||
|
||||
def test_stage0_capacity_counts_live_epochs_not_superseded_ones() -> None:
|
||||
runtime = _runtime(_FakeCodec(), _FakeCodec())
|
||||
runtime.prepare_append(_duplex_info(seq=1), prompt_len=18)
|
||||
runtime.prepare_append(_duplex_info(seq=1, session_id="other"), prompt_len=18)
|
||||
|
||||
# Restarting one session must not need a third codec.
|
||||
runtime.prepare_append(_duplex_info(seq=1, epoch=1), prompt_len=18)
|
||||
|
||||
assert sorted(runtime.sessions) == [("other", 0), ("session", 1)]
|
||||
@@ -1,420 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import base64
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from vllm.sampling_params import SamplingParams
|
||||
from vllm_omni.engine.duplex.runtime import (
|
||||
DuplexInputMode,
|
||||
)
|
||||
from vllm_omni.entrypoints.duplex.runtime_adapter import (
|
||||
ServingRuntimeConfigError,
|
||||
)
|
||||
|
||||
from tests.e2e.online_serving import personaplex_realtime_duplex as e2e_driver
|
||||
from vllm_omni.config.stage_config import (
|
||||
load_deploy_config,
|
||||
merge_pipeline_deploy,
|
||||
)
|
||||
from vllm_omni.engine.duplex.messages import DuplexFence
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import (
|
||||
PersonaPlexDataPlaneContext,
|
||||
PersonaPlexDataPlaneSession,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.input import (
|
||||
PersonaPlexPcmAppendBuffer,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.runtime_extension import (
|
||||
PersonaPlexDuplexRuntimeExtension,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.serving_adapter import (
|
||||
PersonaPlexServingRuntimeAdapter,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.pipeline import (
|
||||
PERSONAPLEX_PIPELINE,
|
||||
)
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
|
||||
|
||||
def _pcm_payload(samples: np.ndarray, *, sample_rate_hz: int = 24000) -> dict[str, object]:
|
||||
samples = np.ascontiguousarray(samples, dtype="<f4")
|
||||
return {
|
||||
"type": "audio",
|
||||
"format": "pcm_f32le",
|
||||
"sample_rate_hz": sample_rate_hz,
|
||||
"audio": base64.b64encode(samples.tobytes()).decode("ascii"),
|
||||
}
|
||||
|
||||
|
||||
def _audio_client(
|
||||
frame_count: int = 10,
|
||||
*,
|
||||
voiced_frames: int = 0,
|
||||
) -> SimpleNamespace:
|
||||
frames = np.full((frame_count, e2e_driver.FRAME_SAMPLES), 7, dtype="<i2")
|
||||
frames[:voiced_frames] = 1000
|
||||
raw = frames.tobytes()
|
||||
event = {
|
||||
"type": "response.output_audio.delta",
|
||||
"response_id": "response-1",
|
||||
"sample_rate_hz": e2e_driver.SAMPLE_RATE_HZ,
|
||||
"metadata": {
|
||||
"vllm_omni": {
|
||||
"runtime_impl": "scheduler_data_plane",
|
||||
"uses_model_runner_scheduler": True,
|
||||
"runner_kv_backed": True,
|
||||
}
|
||||
},
|
||||
}
|
||||
events = SimpleNamespace(
|
||||
events=[event],
|
||||
response_audio={"response-1": [raw]},
|
||||
audio_bytes=lambda: raw,
|
||||
)
|
||||
return SimpleNamespace(events=events)
|
||||
|
||||
|
||||
def test_realtime_audio_frame_stats_separate_voiced_and_silent_frames() -> None:
|
||||
silent = np.zeros(e2e_driver.FRAME_SAMPLES, dtype="<i2")
|
||||
voiced = np.full(e2e_driver.FRAME_SAMPLES, 1000, dtype="<i2")
|
||||
|
||||
stats = e2e_driver._audio_frame_stats(
|
||||
silent.tobytes() + voiced.tobytes(),
|
||||
input_frames=3,
|
||||
voiced_frame_rms_threshold=1e-4,
|
||||
)
|
||||
|
||||
assert stats == {
|
||||
"output_frames": 2,
|
||||
"voiced_frames": 1,
|
||||
"silent_frames": 1,
|
||||
"frame_deficit": 1,
|
||||
"frame_coverage_ratio": pytest.approx(2 / 3),
|
||||
}
|
||||
|
||||
|
||||
def test_realtime_audio_frame_stats_reject_partial_codec_frame() -> None:
|
||||
partial_frame = np.zeros(e2e_driver.FRAME_SAMPLES - 1, dtype="<i2")
|
||||
|
||||
with pytest.raises(AssertionError, match="whole 80 ms codec frames"):
|
||||
e2e_driver._audio_frame_stats(
|
||||
partial_frame.tobytes(),
|
||||
input_frames=1,
|
||||
voiced_frame_rms_threshold=1e-4,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("frame_count", [10, 40])
|
||||
def test_e2e_driver_rejects_inaudible_sessions(frame_count: int) -> None:
|
||||
args = SimpleNamespace(
|
||||
max_frame_deficit=4,
|
||||
voiced_frame_rms_threshold=1e-3,
|
||||
min_voiced_frames=5,
|
||||
)
|
||||
|
||||
with pytest.raises(AssertionError, match="audible speech"):
|
||||
e2e_driver._session_result(
|
||||
_audio_client(frame_count),
|
||||
input_frames=frame_count,
|
||||
args=args,
|
||||
minimum_chunks=1,
|
||||
)
|
||||
|
||||
|
||||
def test_e2e_driver_uses_absolute_audible_floor() -> None:
|
||||
args = SimpleNamespace(
|
||||
max_frame_deficit=4,
|
||||
voiced_frame_rms_threshold=1e-3,
|
||||
min_voiced_frames=5,
|
||||
)
|
||||
|
||||
e2e_driver._session_result(
|
||||
_audio_client(200, voiced_frames=5),
|
||||
input_frames=200,
|
||||
args=args,
|
||||
minimum_chunks=1,
|
||||
)
|
||||
|
||||
|
||||
def test_e2e_driver_defaults_replacement_to_full_input() -> None:
|
||||
args = e2e_driver.parse_args(
|
||||
[
|
||||
"--model",
|
||||
"/model",
|
||||
"--input-wav",
|
||||
"/input.wav",
|
||||
]
|
||||
)
|
||||
|
||||
assert args.replacement_frames == 0
|
||||
assert args.min_voiced_frames == 5
|
||||
|
||||
|
||||
def test_e2e_driver_records_and_validates_input_sha256(tmp_path: Path) -> None:
|
||||
input_wav = tmp_path / "input.wav"
|
||||
input_wav.write_bytes(b"fixture-wav")
|
||||
|
||||
identity = e2e_driver._input_identity(
|
||||
input_wav,
|
||||
expected_sha256="bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
|
||||
)
|
||||
|
||||
assert identity == {
|
||||
"path": str(input_wav.resolve()),
|
||||
"sha256": "bade4b3c163edde390ff391207d34a887257d8a9cc3b621cc8c618b6e6761304",
|
||||
}
|
||||
|
||||
|
||||
def test_e2e_driver_rejects_unexpected_input_sha256(tmp_path: Path) -> None:
|
||||
input_wav = tmp_path / "input.wav"
|
||||
input_wav.write_bytes(b"fixture-wav")
|
||||
|
||||
with pytest.raises(ValueError, match="input WAV SHA-256 mismatch"):
|
||||
e2e_driver._input_identity(input_wav, expected_sha256="0" * 64)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("attribute", "expected"),
|
||||
[
|
||||
(
|
||||
"duplex_runtime_extension",
|
||||
"vllm_omni.model_executor.models.personaplex.duplex.runtime_extension.PersonaPlexDuplexRuntimeExtension",
|
||||
),
|
||||
(
|
||||
"duplex_serving_adapter",
|
||||
"vllm_omni.model_executor.models.personaplex.duplex.serving_adapter.PersonaPlexServingRuntimeAdapter",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_personaplex_pipeline_registers_unified_duplex_components(
|
||||
attribute: str,
|
||||
expected: str,
|
||||
) -> None:
|
||||
assert PERSONAPLEX_PIPELINE.duplex_control_enabled is True
|
||||
assert getattr(PERSONAPLEX_PIPELINE, attribute) == expected
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("engine_arg", "expected"),
|
||||
[
|
||||
("skip_tokenizer_init", True),
|
||||
("enable_prefix_caching", False),
|
||||
],
|
||||
)
|
||||
def test_personaplex_stage_engine_args(
|
||||
engine_arg: str,
|
||||
expected: bool,
|
||||
) -> None:
|
||||
deploy_path = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml"
|
||||
stages = merge_pipeline_deploy(
|
||||
PERSONAPLEX_PIPELINE,
|
||||
load_deploy_config(deploy_path),
|
||||
)
|
||||
|
||||
assert [stage.yaml_engine_args.get(engine_arg) for stage in stages] == [
|
||||
expected,
|
||||
expected,
|
||||
]
|
||||
|
||||
|
||||
def test_personaplex_duplex_capacity_is_propagated_to_all_model_stages() -> None:
|
||||
deploy_path = Path(__file__).parents[5] / "vllm_omni" / "deploy" / "personaplex.yaml"
|
||||
deploy = load_deploy_config(deploy_path)
|
||||
stages = merge_pipeline_deploy(PERSONAPLEX_PIPELINE, deploy)
|
||||
|
||||
assert deploy.duplex_session.max_sessions == 2
|
||||
assert [stage.yaml_engine_args.get("duplex_max_sessions") for stage in stages] == [2, 2]
|
||||
assert "personaplex_codec_max_sessions" not in deploy.connectors["connector_of_shared_memory"]["extra"]
|
||||
|
||||
|
||||
def test_personaplex_capabilities_are_honest() -> None:
|
||||
single = PersonaPlexServingRuntimeAdapter(lambda *_: None).capabilities(max_sessions=1)
|
||||
multi = PersonaPlexServingRuntimeAdapter(lambda *_: None).capabilities(max_sessions=2)
|
||||
|
||||
assert multi.implementation_level == "model_native_duplex"
|
||||
assert multi.input_modes == ["append_audio_chunk"]
|
||||
assert multi.chunk_period_ms == 80
|
||||
assert single.supports_multi_session is False
|
||||
assert multi.supports_multi_session is True
|
||||
assert multi.supports_multi_session_same_replica is True
|
||||
assert multi.supports_barge_in is False
|
||||
|
||||
|
||||
def test_personaplex_runtime_config_update_rejects_changed_persona() -> None:
|
||||
adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
|
||||
short_persona = "You are a helpful assistant."
|
||||
longer_persona = (
|
||||
"You are now a swashbuckling pirate captain who speaks entirely in "
|
||||
"nautical slang, references buried treasure constantly, and never "
|
||||
"breaks character no matter what the user asks."
|
||||
)
|
||||
current = {"personaplex_persona": short_persona, "personaplex_voice_prompt": "NATF2.pt"}
|
||||
|
||||
with pytest.raises(ServingRuntimeConfigError, match="persona"):
|
||||
adapter.runtime_config_for_update(
|
||||
SimpleNamespace(instructions=longer_persona, extra_body={}),
|
||||
current,
|
||||
)
|
||||
|
||||
unchanged = adapter.runtime_config_for_update(
|
||||
SimpleNamespace(instructions=short_persona, extra_body={}),
|
||||
current,
|
||||
)
|
||||
assert unchanged["personaplex_persona"] == short_persona
|
||||
|
||||
|
||||
def test_personaplex_runtime_config_update_rejects_changed_voice() -> None:
|
||||
adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
|
||||
current = {"personaplex_persona": DEFAULT_PERSONA, "personaplex_voice_prompt": "NATF2.pt"}
|
||||
|
||||
with pytest.raises(ServingRuntimeConfigError, match="voice"):
|
||||
adapter.runtime_config_for_update(
|
||||
SimpleNamespace(instructions=None, voice="OtherVoice.pt", extra_body={}),
|
||||
current,
|
||||
)
|
||||
|
||||
unchanged = adapter.runtime_config_for_update(
|
||||
SimpleNamespace(instructions=None, voice="NATF2.pt", extra_body={}),
|
||||
current,
|
||||
)
|
||||
assert unchanged["personaplex_voice_prompt"] == "NATF2.pt"
|
||||
|
||||
|
||||
def test_personaplex_rejects_native_duplex_opt_out_flag() -> None:
|
||||
# PersonaPlex is always model-native; the per-session opt-in knob is
|
||||
# server-owned there, under both the canonical name and its deprecated
|
||||
# model-prefixed alias.
|
||||
adapter = PersonaPlexServingRuntimeAdapter(lambda *_: None)
|
||||
|
||||
with pytest.raises(ServingRuntimeConfigError, match="native_duplex"):
|
||||
adapter.validate_client_extra_body({"native_duplex": False})
|
||||
with pytest.raises(ServingRuntimeConfigError, match="minicpmo45_native_duplex"):
|
||||
adapter.validate_client_extra_body({"minicpmo45_native_duplex": False})
|
||||
|
||||
|
||||
def test_pcm_buffer_emits_one_80ms_frame_transactionally() -> None:
|
||||
buffer = PersonaPlexPcmAppendBuffer()
|
||||
reservation = buffer.prepare_append(
|
||||
_pcm_payload(np.arange(1920, dtype=np.float32)),
|
||||
operation_id="op-1",
|
||||
chunk_period_ms=80,
|
||||
allow_emit=True,
|
||||
)
|
||||
|
||||
assert reservation is not None
|
||||
assert reservation.byte_count == 1920 * 4
|
||||
assert buffer.pending_byte_count == 0
|
||||
reservation.rollback()
|
||||
assert buffer.pending_byte_count == 1920 * 4
|
||||
|
||||
|
||||
def test_pcm_buffer_rejects_changed_sample_rate() -> None:
|
||||
buffer = PersonaPlexPcmAppendBuffer()
|
||||
assert (
|
||||
buffer.prepare_append(
|
||||
_pcm_payload(np.zeros(100, np.float32)),
|
||||
operation_id="op-1",
|
||||
chunk_period_ms=80,
|
||||
allow_emit=False,
|
||||
)
|
||||
is None
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="sample_rate"):
|
||||
buffer.prepare_append(
|
||||
_pcm_payload(np.zeros(100, np.float32), sample_rate_hz=16000),
|
||||
operation_id="op-2",
|
||||
chunk_period_ms=80,
|
||||
allow_emit=False,
|
||||
)
|
||||
|
||||
|
||||
def test_pcm_buffer_rejects_non_finite_samples() -> None:
|
||||
buffer = PersonaPlexPcmAppendBuffer()
|
||||
|
||||
with pytest.raises(ValueError, match="finite"):
|
||||
buffer.prepare_append(
|
||||
_pcm_payload(np.array([0.0, np.nan], np.float32)),
|
||||
operation_id="op-1",
|
||||
chunk_period_ms=80,
|
||||
allow_emit=False,
|
||||
)
|
||||
|
||||
|
||||
def test_runtime_extension_builds_one_frame_scheduler_append() -> None:
|
||||
extension = PersonaPlexDuplexRuntimeExtension()
|
||||
defaults = (
|
||||
SamplingParams(temperature=0.8, top_k=10, max_tokens=10),
|
||||
SamplingParams(max_tokens=1024),
|
||||
)
|
||||
|
||||
configured = extension.configure_sampling_params(runtime_config={}, defaults=defaults)
|
||||
assert configured[0].temperature == 0.0
|
||||
assert configured[0].top_k == 1
|
||||
assert configured[0].max_tokens == 1
|
||||
assert configured[1] is defaults[1]
|
||||
|
||||
plan = extension.plan_append(
|
||||
request_id="req",
|
||||
fence=DuplexFence("session", incarnation=2, epoch=3),
|
||||
session_config={"instructions": "Be concise."},
|
||||
runtime_config={"personaplex_prefill_slots": 4},
|
||||
seq=1,
|
||||
turn_seq=1,
|
||||
mode=DuplexInputMode.APPEND_AUDIO_CHUNK,
|
||||
payload=_pcm_payload(np.zeros(1920, np.float32)),
|
||||
final=False,
|
||||
sampling_params=configured[0],
|
||||
)
|
||||
assert len(plan.prompt["prompt_token_ids"]) == 5
|
||||
duplex = plan.prompt["model_intermediate_buffer"]["duplex"]
|
||||
assert duplex["session_id"] == "session"
|
||||
assert duplex["incarnation"] == 2
|
||||
assert duplex["epoch"] == 3
|
||||
assert duplex["seq"] == 1
|
||||
assert duplex["data_plane"] is True
|
||||
|
||||
|
||||
def test_projector_emits_only_cumulative_audio_and_text_suffixes() -> None:
|
||||
encoded_sizes: list[int] = []
|
||||
|
||||
def encode_audio(audio, _sample_rate, _response_format, _speed):
|
||||
size = int(np.asarray(audio, dtype=np.float32).size)
|
||||
encoded_sizes.append(size)
|
||||
return f"audio-{size}"
|
||||
|
||||
projector = PersonaPlexDataPlaneSession(encode_audio)
|
||||
projector.begin_request("req")
|
||||
context = PersonaPlexDataPlaneContext(response_format="wav")
|
||||
|
||||
first = SimpleNamespace(
|
||||
request_id="req",
|
||||
outputs=[SimpleNamespace(text="he", multimodal_output={})],
|
||||
multimodal_output={"model_outputs": np.arange(4, dtype=np.float32), "sr": 24000},
|
||||
finished=False,
|
||||
)
|
||||
second = SimpleNamespace(
|
||||
request_id="req",
|
||||
outputs=[SimpleNamespace(text="hello", multimodal_output={})],
|
||||
multimodal_output={"model_outputs": np.arange(6, dtype=np.float32), "sr": 24000},
|
||||
finished=False,
|
||||
)
|
||||
|
||||
projected = list(
|
||||
projector.project(
|
||||
{"data_plane_outputs": [first, second]},
|
||||
context=context,
|
||||
)
|
||||
)
|
||||
|
||||
assert encoded_sizes == [4, 2]
|
||||
assert [item["text"] for item in projected] == ["he", "llo"]
|
||||
assert [item["audio_data"] for item in projected] == ["audio-4", "audio-2"]
|
||||
assert all(item["sample_rate_hz"] == 24000 for item in projected)
|
||||
assert all(item["end_of_turn"] is False for item in projected)
|
||||
@@ -324,10 +324,10 @@ class PipelineConfig:
|
||||
duplex_plugin: str | None = None
|
||||
# Preserve legacy turn deployments when adding an optional duplex plugin.
|
||||
default_session_mode: str | None = None
|
||||
# Legacy duplex wiring of the models that are not ported to the plugin
|
||||
# framework yet (PersonaPlex, Nemotron VoiceChat). Nothing reads them: a
|
||||
# pipeline that only declares these is served turn-based. Each field goes
|
||||
# away with the follow-up PR that ports its model to ``duplex_plugin``.
|
||||
# Legacy duplex wiring of the model that is not ported to the plugin
|
||||
# framework yet (Nemotron VoiceChat, RFC vllm-omni#7181 PR 4). Nothing
|
||||
# reads them: a pipeline that only declares these is served turn-based.
|
||||
# The fields go away with the PR that ports it to ``duplex_plugin``.
|
||||
duplex_runtime_extension: str | None = None
|
||||
duplex_serving_adapter: str | None = None
|
||||
duplex_control_enabled: bool = False
|
||||
|
||||
@@ -37,6 +37,7 @@ from vllm_omni.diffusion.models.qwen_image.qwen_image_transformer import (
|
||||
QwenImageTransformer2DModel,
|
||||
)
|
||||
from vllm_omni.diffusion.models.qwen_image.rope_utils import txt_seq_lens_from_embeds
|
||||
from vllm_omni.diffusion.offloader.config import OffloadStrategy, resolve_offload_strategy
|
||||
from vllm_omni.diffusion.profiler.diffusion_pipeline_profiler import DiffusionPipelineProfilerMixin
|
||||
from vllm_omni.diffusion.utils.prompt_utils import (
|
||||
validate_prompt_sequence_lengths,
|
||||
@@ -341,7 +342,7 @@ class QwenImagePipeline(
|
||||
# do not share VRAM with DiT construction (#7555). The DiT follows the
|
||||
# loader's default-device context (CUDA for online / AutoRound INT under
|
||||
# offload, CPU for layerwise / unquantized HSDP defer).
|
||||
cpu_offload = bool(getattr(self.od_config, "enable_cpu_offload", False))
|
||||
cpu_offload = resolve_offload_strategy(self.od_config) is OffloadStrategy.MODEL_LEVEL
|
||||
enc_vae_device = torch.device("cpu") if cpu_offload else self.device
|
||||
self.text_encoder = self.text_encoder.to(enc_vae_device)
|
||||
self.vae = from_pretrained_with_prefetch(
|
||||
|
||||
@@ -16,9 +16,12 @@ import asyncio
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Callable, Iterable, Mapping
|
||||
from dataclasses import dataclass
|
||||
from functools import lru_cache
|
||||
from importlib import import_module
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pybase64 as base64
|
||||
|
||||
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
|
||||
from vllm_omni.engine.duplex.contracts import (
|
||||
DuplexAppendPlan,
|
||||
@@ -219,6 +222,20 @@ class DefaultDuplexModelSessionState(DuplexModelSessionState):
|
||||
self.silence_deadline_monotonic = None
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class DuplexDataPlaneContext:
|
||||
"""Session state the runner hands a data plane to project one stage output."""
|
||||
|
||||
epoch: int = 0
|
||||
turn_id: int = 0
|
||||
active_response_turn_id: int | None = None
|
||||
active_response_id: str | None = None
|
||||
auto_responds: bool = False
|
||||
response_format: str = "wav"
|
||||
speed: float | None = None
|
||||
modalities: tuple[str, ...] = ()
|
||||
|
||||
|
||||
class DuplexDataPlane(ABC):
|
||||
"""Projects raw stage outputs of one model into internal duplex events."""
|
||||
|
||||
@@ -277,12 +294,29 @@ class DuplexModelPlugin(ABC):
|
||||
private_runtime_config_keys: frozenset[str] = frozenset()
|
||||
#: Samples per silence unit the runner appends to keep a model turn going.
|
||||
silence_continuation_samples: int = 16000
|
||||
#: Sample rate of that unit: the runner submits it through ``plan_append``
|
||||
#: exactly like client audio, so it must be a unit the model accepts.
|
||||
silence_continuation_sample_rate_hz: int = 16000
|
||||
data_plane: DuplexDataPlane
|
||||
|
||||
def __init__(self, encode_audio: EncodeAudio) -> None:
|
||||
# Constructor-only: concrete plugins hand the encoder to their data plane.
|
||||
del encode_audio
|
||||
|
||||
def silence_unit_payload(self) -> dict[str, object]:
|
||||
"""One silence unit as an append payload (``pcm_f32le`` zeros).
|
||||
|
||||
Used by the runner's turn continuation and by the startup warmup; a
|
||||
model whose unit is not plain zero PCM overrides it.
|
||||
"""
|
||||
samples = int(self.silence_continuation_samples)
|
||||
return {
|
||||
"type": "audio",
|
||||
"audio": _silence_pcm_f32le_base64(samples),
|
||||
"format": "pcm_f32le",
|
||||
"sample_rate_hz": int(self.silence_continuation_sample_rate_hz),
|
||||
}
|
||||
|
||||
# ---- engine policy (was DuplexRuntimeExtension) ----
|
||||
|
||||
@abstractmethod
|
||||
@@ -430,8 +464,15 @@ class DuplexModelPlugin(ABC):
|
||||
@abstractmethod
|
||||
def capabilities(self, *, max_sessions: int) -> DuplexCapabilities: ...
|
||||
|
||||
@abstractmethod
|
||||
def validate_client_extra_body(self, extra_body: object) -> None: ...
|
||||
def validate_client_extra_body(self, extra_body: object) -> None:
|
||||
"""Refuse client ``extra_body`` keys the server owns (``private_runtime_config_keys``)."""
|
||||
if not isinstance(extra_body, Mapping):
|
||||
return
|
||||
private_keys = sorted(self.private_runtime_config_keys.intersection(extra_body))
|
||||
if private_keys:
|
||||
raise DuplexRuntimeConfigError(
|
||||
f"{self.plugin_id} runtime configuration is server-owned: " + ", ".join(private_keys)
|
||||
)
|
||||
|
||||
@abstractmethod
|
||||
async def prepare_runtime_config(
|
||||
@@ -445,7 +486,6 @@ class DuplexModelPlugin(ABC):
|
||||
current: Mapping[str, object],
|
||||
) -> dict[str, object]: ...
|
||||
|
||||
@abstractmethod
|
||||
def data_plane_context(
|
||||
self,
|
||||
*,
|
||||
@@ -457,7 +497,18 @@ class DuplexModelPlugin(ABC):
|
||||
response_format: str,
|
||||
speed: float | None,
|
||||
modalities: tuple[str, ...],
|
||||
) -> object: ...
|
||||
) -> object:
|
||||
"""The context handed to ``data_plane.project``; the default is the generic dataclass."""
|
||||
return DuplexDataPlaneContext(
|
||||
epoch=epoch,
|
||||
turn_id=turn_id,
|
||||
active_response_turn_id=active_response_turn_id,
|
||||
active_response_id=active_response_id,
|
||||
auto_responds=auto_responds,
|
||||
response_format=response_format,
|
||||
speed=speed,
|
||||
modalities=modalities,
|
||||
)
|
||||
|
||||
# Optional hook: build the runtime config patch for a function-call output
|
||||
# item. Plugins without tools keep the default (no change).
|
||||
@@ -515,6 +566,11 @@ def validate_duplex_plugin_sampling(plugin: DuplexModelPlugin, *, sampling_defau
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=8)
|
||||
def _silence_pcm_f32le_base64(samples: int) -> str:
|
||||
return base64.b64encode(bytes(max(0, samples) * 4)).decode("ascii")
|
||||
|
||||
|
||||
def payload_turn_id(payload: object) -> int | None:
|
||||
if not isinstance(payload, Mapping):
|
||||
return None
|
||||
@@ -537,6 +593,7 @@ def coerce_int(value: object) -> int | None:
|
||||
__all__ = [
|
||||
"DefaultDuplexModelSessionState",
|
||||
"DuplexDataPlane",
|
||||
"DuplexDataPlaneContext",
|
||||
"DuplexModelPlugin",
|
||||
"DuplexModelSessionState",
|
||||
"DuplexRuntimeConfigError",
|
||||
|
||||
@@ -121,7 +121,14 @@ class SessionEmitter:
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def auto_responds(self) -> bool:
|
||||
"""Whether the session answers committed input without a ``response.create``."""
|
||||
"""Whether the session answers committed input without a ``response.create``.
|
||||
|
||||
A model that takes no client commits (``supports_client_commit`` off, a
|
||||
lockstep model) can only auto-respond; for every other model the
|
||||
client opts in through ``extra_body.auto_response``.
|
||||
"""
|
||||
if not self._ctx.session.capabilities.supports_client_commit:
|
||||
return True
|
||||
extra = getattr(self._ctx.session.config, "extra_body", None)
|
||||
if not isinstance(extra, dict):
|
||||
return False
|
||||
|
||||
@@ -29,7 +29,6 @@ from collections.abc import Awaitable, Callable, Mapping
|
||||
from dataclasses import replace
|
||||
from typing import TYPE_CHECKING, Protocol
|
||||
|
||||
import pybase64 as base64
|
||||
from vllm.logger import init_logger
|
||||
|
||||
from vllm_omni.engine.duplex.config import DuplexSessionState
|
||||
@@ -83,7 +82,6 @@ class ModelChannel:
|
||||
"""Appends out to the model, events back from it, for one session."""
|
||||
|
||||
# One MiniCPM model unit (1 s at 16 kHz) is the compatibility default.
|
||||
_SILENCE_UNIT_PAYLOAD_AUDIO = base64.b64encode(bytes(16000 * 4)).decode("ascii")
|
||||
_RESPONSE_MAX_CONTINUATION_UNITS = 8
|
||||
_AUTO_RESPONSE_MAX_CONTINUATION_UNITS = 64
|
||||
|
||||
@@ -1210,13 +1208,8 @@ class ModelChannel:
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def silence_unit_payload(self) -> dict[str, object]:
|
||||
samples = int(self._ctx.plugin.silence_continuation_samples)
|
||||
audio = (
|
||||
self._SILENCE_UNIT_PAYLOAD_AUDIO
|
||||
if samples == 16000
|
||||
else base64.b64encode(bytes(samples * 4)).decode("ascii")
|
||||
)
|
||||
return {"type": "audio", "audio": audio, "format": "pcm_f32le", "sample_rate_hz": 16000}
|
||||
"""One silence unit in the model's own format: the plugin defines it (rate and length)."""
|
||||
return dict(self._ctx.plugin.silence_unit_payload())
|
||||
|
||||
def response_continuations_remaining(self, response_id: str) -> bool:
|
||||
model_state = self._ctx.model_state
|
||||
|
||||
@@ -94,6 +94,26 @@ def _warmup_jpeg_b64() -> str:
|
||||
return base64.b64encode(buf.getvalue()).decode("ascii")
|
||||
|
||||
|
||||
_DEFAULT_SILENCE_SAMPLES = 16000
|
||||
_DEFAULT_SILENCE_SAMPLE_RATE_HZ = 16000
|
||||
|
||||
|
||||
def warmup_silence_unit(plugin: object | None) -> dict[str, object]:
|
||||
"""The append the silent-frames warmup sends: the plugin's silence unit, or the 16 kHz default."""
|
||||
factory = getattr(plugin, "silence_unit_payload", None)
|
||||
if callable(factory):
|
||||
unit = dict(factory())
|
||||
if isinstance(unit.get("audio"), str) and isinstance(unit.get("format"), str):
|
||||
unit.setdefault("sample_rate_hz", _DEFAULT_SILENCE_SAMPLE_RATE_HZ)
|
||||
return unit
|
||||
samples = int(getattr(plugin, "silence_continuation_samples", _DEFAULT_SILENCE_SAMPLES))
|
||||
return {
|
||||
"audio": base64.b64encode(bytes(samples * 4)).decode("ascii"),
|
||||
"format": "pcm_f32le",
|
||||
"sample_rate_hz": _DEFAULT_SILENCE_SAMPLE_RATE_HZ,
|
||||
}
|
||||
|
||||
|
||||
async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
|
||||
"""Run one throwaway realtime session before real clients are admitted.
|
||||
|
||||
@@ -121,9 +141,14 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
|
||||
model_name = served
|
||||
else:
|
||||
model_name = args.model
|
||||
# One silence unit as the engine-side plugin defines it (DuplexOmniEngine.plugin).
|
||||
frame_samples = int(getattr(plugin, "silence_continuation_samples", 16000))
|
||||
native_append = plugin.capabilities(max_sessions=1).supports_core_resumable_request if plugin else True
|
||||
# One silence unit as the engine-side plugin defines it (DuplexOmniEngine.plugin):
|
||||
# the unit's format, rate and length are the model's, not a fixed 16 kHz.
|
||||
unit = warmup_silence_unit(plugin)
|
||||
frame_samples = len(base64.b64decode(str(unit["audio"]))) // 4
|
||||
capabilities = getattr(plugin, "capabilities", None)
|
||||
native_append = (
|
||||
bool(capabilities(max_sessions=1).supports_core_resumable_request) if callable(capabilities) else True
|
||||
)
|
||||
from vllm_omni.clients.duplex import build_realtime_url
|
||||
|
||||
url = (
|
||||
@@ -163,6 +188,10 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
|
||||
"turn_detection": None,
|
||||
"extra_body": {"auto_response": native_append},
|
||||
}
|
||||
if kind != "video_turn":
|
||||
session["input_audio_format"] = unit["format"]
|
||||
session["sample_rate_hz"] = unit["sample_rate_hz"]
|
||||
session["audio"] = {"input": {"sample_rate_hz": unit["sample_rate_hz"]}}
|
||||
if kind == "video_turn":
|
||||
# The default AURA prompt answers with <|silent|> unless the user
|
||||
# asked for something. This session is thrown away; force one short
|
||||
@@ -205,9 +234,17 @@ async def _warmup_duplex_realtime(app, args, warmup_frames: int) -> None:
|
||||
)
|
||||
sent = 1
|
||||
else:
|
||||
silence = base64.b64encode(bytes(frame_samples * 4)).decode("ascii")
|
||||
while sent < warmup_frames:
|
||||
await ws.send(json.dumps({"type": "input_audio_buffer.append", "audio": silence}))
|
||||
await ws.send(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "input_audio_buffer.append",
|
||||
"audio": unit["audio"],
|
||||
"format": unit["format"],
|
||||
"sample_rate_hz": unit["sample_rate_hz"],
|
||||
}
|
||||
)
|
||||
)
|
||||
sent += 1
|
||||
await asyncio.sleep(0.08)
|
||||
if not native_append:
|
||||
@@ -251,4 +288,5 @@ __all__ = [
|
||||
"_warmup_duplex_realtime",
|
||||
"lookup_duplex_plugin",
|
||||
"startup_warmup_kind",
|
||||
"warmup_silence_unit",
|
||||
]
|
||||
@@ -0,0 +1,8 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Model-neutral helpers shared between model packages.
|
||||
|
||||
``audio`` and ``request_outputs`` carry no duplex vocabulary and may be used
|
||||
from any model. ``duplex`` builds on the duplex framework contracts and is
|
||||
imported only from a model's ``duplex`` package.
|
||||
"""
|
||||
@@ -0,0 +1,3 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Generic audio helpers shared between model packages."""
|
||||
@@ -0,0 +1,54 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Little-endian float32 PCM (``pcm_f32le``) helpers.
|
||||
|
||||
Plain audio plumbing every model that takes raw PCM needs: decode a base64
|
||||
chunk, check it is whole finite samples, count and materialise them. No
|
||||
duplex vocabulary lives here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import binascii
|
||||
|
||||
import numpy as np
|
||||
import pybase64 as base64
|
||||
|
||||
PCM_F32LE_BYTES_PER_SAMPLE = 4
|
||||
|
||||
|
||||
def decode_pcm_f32le_base64(encoded: object, *, model: str = "audio") -> bytes:
|
||||
"""Decode base64 ``pcm_f32le`` into raw bytes, refusing partial or non-finite samples.
|
||||
|
||||
``model`` names the caller in error messages so a client sees which model
|
||||
rejected its audio.
|
||||
"""
|
||||
if not isinstance(encoded, str):
|
||||
raise ValueError(f"{model} audio must be base64 pcm_f32le")
|
||||
try:
|
||||
raw = base64.b64decode(encoded, validate=True)
|
||||
except (binascii.Error, ValueError) as exc:
|
||||
raise ValueError(f"{model} audio is not valid base64") from exc
|
||||
if len(raw) % PCM_F32LE_BYTES_PER_SAMPLE:
|
||||
raise ValueError(f"{model} pcm_f32le byte length must be divisible by four")
|
||||
samples = np.frombuffer(raw, dtype="<f4")
|
||||
if samples.size and not bool(np.isfinite(samples).all()):
|
||||
raise ValueError(f"{model} pcm_f32le samples must be finite")
|
||||
return raw
|
||||
|
||||
|
||||
def pcm_f32le_sample_count(raw: bytes) -> int:
|
||||
return len(raw) // PCM_F32LE_BYTES_PER_SAMPLE
|
||||
|
||||
|
||||
def pcm_f32le_samples(raw: bytes) -> np.ndarray:
|
||||
"""The samples as a writable, contiguous float32 array (safe for zero-copy tensor views)."""
|
||||
return np.ascontiguousarray(np.frombuffer(raw, dtype="<f4"), dtype=np.float32).copy()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PCM_F32LE_BYTES_PER_SAMPLE",
|
||||
"decode_pcm_f32le_base64",
|
||||
"pcm_f32le_sample_count",
|
||||
"pcm_f32le_samples",
|
||||
]
|
||||
@@ -0,0 +1,8 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Shared building blocks for ``DuplexModelPlugin`` implementations.
|
||||
|
||||
Everything here depends on the duplex framework contracts
|
||||
(``vllm_omni.engine.duplex.plugin``), so it is imported only from a model's
|
||||
``duplex`` package, never from the turn-based stack.
|
||||
"""
|
||||
@@ -0,0 +1,135 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Data plane for models whose final stage streams cumulative audio and text.
|
||||
|
||||
The stage runner re-sends the whole waveform (and transcript) it has produced
|
||||
so far for one request; the session needs only what is new. This data plane
|
||||
keeps one cursor per request, slices the delta, encodes it for the wire and
|
||||
reports it as one internal ``response.output_audio.delta``-shaped result.
|
||||
Model-specific constants (stage role, runtime metadata, default rate) are
|
||||
class attributes for a subclass to set.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Iterator
|
||||
from dataclasses import dataclass
|
||||
|
||||
from vllm_omni.engine.duplex.contracts import duplex_resource_request_belongs_to_session
|
||||
from vllm_omni.engine.duplex.plugin import DuplexDataPlane, DuplexDataPlaneContext
|
||||
from vllm_omni.model_executor.common.request_outputs import (
|
||||
audio_sample_count,
|
||||
audio_value,
|
||||
multimodal_output,
|
||||
sample_rate_hz,
|
||||
slice_audio_delta,
|
||||
text_delta,
|
||||
text_value,
|
||||
unwrap_request_output,
|
||||
)
|
||||
|
||||
EncodeAudio = Callable[[object, int, str, float | None], str | None]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _RequestCursor:
|
||||
audio_samples: int = 0
|
||||
text: str = ""
|
||||
terminal: bool = False
|
||||
|
||||
|
||||
class CumulativeAudioTextDataPlane(DuplexDataPlane):
|
||||
"""Project cumulative staged audio/text output into per-request deltas."""
|
||||
|
||||
stage_role: str = "tts"
|
||||
runtime_impl: str = "scheduler_data_plane"
|
||||
uses_model_runner_scheduler: bool = True
|
||||
runner_kv_backed: bool = True
|
||||
owned_runtime: bool = False
|
||||
default_sample_rate_hz: int = 24000
|
||||
|
||||
def __init__(self, encode_audio: EncodeAudio) -> None:
|
||||
self._encode_audio = encode_audio
|
||||
self._requests: dict[str, _RequestCursor] = {}
|
||||
|
||||
# ---- request / session bookkeeping ----
|
||||
|
||||
def begin_request(self, request_id: str) -> None:
|
||||
self._requests.setdefault(request_id, _RequestCursor()).terminal = False
|
||||
|
||||
def is_terminal(self, request_id: str | None) -> bool:
|
||||
if request_id is None:
|
||||
return False
|
||||
state = self._requests.get(request_id)
|
||||
return state is not None and state.terminal
|
||||
|
||||
def mark_terminal(self, request_id: str) -> None:
|
||||
self._requests.setdefault(request_id, _RequestCursor()).terminal = True
|
||||
|
||||
def close_stream(self, request_id: str) -> None:
|
||||
self._requests.pop(request_id, None)
|
||||
|
||||
def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None:
|
||||
if active_request_id is not None:
|
||||
self._requests.pop(active_request_id, None)
|
||||
for request_id in list(self._requests):
|
||||
if duplex_resource_request_belongs_to_session(request_id, session_id):
|
||||
self._requests.pop(request_id, None)
|
||||
|
||||
# ---- projection ----
|
||||
|
||||
def project(self, result: object, *, context: object | None = None) -> Iterator[dict[str, object]]:
|
||||
if not isinstance(result, dict):
|
||||
return
|
||||
outputs = result.get("data_plane_outputs")
|
||||
if not isinstance(outputs, list):
|
||||
return
|
||||
if not isinstance(context, DuplexDataPlaneContext):
|
||||
context = DuplexDataPlaneContext(response_format="wav", modalities=("audio", "text"))
|
||||
for output in outputs:
|
||||
projected = self._project_output(output, context=context)
|
||||
if projected is not None:
|
||||
yield projected
|
||||
|
||||
def _project_output(self, output: object, *, context: DuplexDataPlaneContext) -> dict[str, object] | None:
|
||||
output, completion = unwrap_request_output(output)
|
||||
request_id = getattr(output, "request_id", None)
|
||||
if not isinstance(request_id, str) or not request_id:
|
||||
request_id = None
|
||||
state = self._requests.setdefault(request_id, _RequestCursor()) if request_id is not None else _RequestCursor()
|
||||
multimodal = multimodal_output(output, completion)
|
||||
audio = audio_value(multimodal)
|
||||
audio_delta = slice_audio_delta(audio, state.audio_samples)
|
||||
total_samples = audio_sample_count(audio)
|
||||
if total_samples is not None:
|
||||
state.audio_samples = total_samples
|
||||
|
||||
text = text_value(multimodal, completion)
|
||||
delta_text = text_delta(text, state.text)
|
||||
if text:
|
||||
state.text = text
|
||||
|
||||
rate = sample_rate_hz(multimodal, default=self.default_sample_rate_hz)
|
||||
encoded = self._encode_audio(audio_delta, rate, context.response_format, context.speed)
|
||||
if not encoded and not delta_text:
|
||||
return None
|
||||
delta_samples = audio_sample_count(audio_delta) or 0
|
||||
return {
|
||||
"supported": True,
|
||||
"stage_role": self.stage_role,
|
||||
"is_listen": False,
|
||||
"data_plane_request_id": request_id,
|
||||
"text": delta_text,
|
||||
"audio_data": encoded or "",
|
||||
"audio_format": context.response_format,
|
||||
"sample_rate_hz": rate,
|
||||
"audio_duration_ms": round(delta_samples * 1000 / max(1, rate)),
|
||||
"end_of_turn": False,
|
||||
"uses_model_runner_scheduler": self.uses_model_runner_scheduler,
|
||||
"runner_kv_backed": self.runner_kv_backed,
|
||||
"runtime_impl": self.runtime_impl,
|
||||
"owned_runtime": self.owned_runtime,
|
||||
}
|
||||
|
||||
|
||||
__all__ = ["CumulativeAudioTextDataPlane"]
|
||||
@@ -0,0 +1,63 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Validation of one duplex audio append payload.
|
||||
|
||||
The runner hands a plugin the append as a mapping ``{format, sample_rate_hz,
|
||||
audio}`` (``audio`` base64 ``pcm_f32le``). Models with a fixed input rate and
|
||||
frame size validate it here; the PCM itself is decoded by
|
||||
``vllm_omni.model_executor.common.audio.pcm``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
|
||||
from vllm_omni.model_executor.common.audio.pcm import (
|
||||
decode_pcm_f32le_base64,
|
||||
pcm_f32le_sample_count,
|
||||
)
|
||||
|
||||
|
||||
def payload_audio(payload: object) -> object | None:
|
||||
"""The encoded audio field of an append payload (``audio``, or the older ``data``)."""
|
||||
if not isinstance(payload, Mapping):
|
||||
return None
|
||||
return payload.get("audio") or payload.get("data")
|
||||
|
||||
|
||||
def decode_pcm_f32le_payload(
|
||||
payload: object,
|
||||
*,
|
||||
sample_rate_hz: int,
|
||||
exact_samples: int | None = None,
|
||||
model: str = "duplex",
|
||||
) -> bytes:
|
||||
"""Raw PCM of one append that must be ``pcm_f32le`` at ``sample_rate_hz``.
|
||||
|
||||
``exact_samples`` pins the unit length (a lockstep model takes exactly one
|
||||
frame per append); ``None`` accepts any whole number of samples.
|
||||
"""
|
||||
if not isinstance(payload, Mapping):
|
||||
raise ValueError(f"{model} duplex append payload must be a mapping")
|
||||
if payload.get("format") != "pcm_f32le":
|
||||
raise ValueError(f"{model} duplex append format must be pcm_f32le")
|
||||
if payload.get("sample_rate_hz") != sample_rate_hz:
|
||||
raise ValueError(f"{model} duplex append sample_rate_hz must be {sample_rate_hz}")
|
||||
raw = decode_pcm_f32le_base64(payload_audio(payload), model=model)
|
||||
if exact_samples is not None and pcm_f32le_sample_count(raw) != exact_samples:
|
||||
raise ValueError(f"{model} duplex append must contain exactly {exact_samples} samples")
|
||||
return raw
|
||||
|
||||
|
||||
def payload_sample_count(payload: object) -> int | None:
|
||||
"""Sample count of a ``pcm_f32le`` append, or ``None`` when it is not one (lenient, for budgeting)."""
|
||||
if not isinstance(payload, Mapping) or payload.get("format") != "pcm_f32le":
|
||||
return None
|
||||
try:
|
||||
raw = decode_pcm_f32le_base64(payload_audio(payload))
|
||||
except ValueError:
|
||||
return None
|
||||
return pcm_f32le_sample_count(raw)
|
||||
|
||||
|
||||
__all__ = ["decode_pcm_f32le_payload", "payload_audio", "payload_sample_count"]
|
||||
@@ -0,0 +1,194 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Transactional framing of client PCM into fixed model units.
|
||||
|
||||
A frame-locked model (PersonaPlex: 1920 samples every 80 ms at 24 kHz) takes
|
||||
exactly one frame per append. Clients send arbitrary chunks, so the session
|
||||
runner buffers them here and takes one whole frame out at a time as a
|
||||
*reservation*: committed once the stage accepted the append, rolled back
|
||||
(the bytes go back to the front of the buffer) when it did not.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pybase64 as base64
|
||||
|
||||
from vllm_omni.engine.duplex.plugin import PcmAppendBuffer, PcmAppendReservation
|
||||
from vllm_omni.model_executor.common.audio.pcm import (
|
||||
PCM_F32LE_BYTES_PER_SAMPLE,
|
||||
decode_pcm_f32le_base64,
|
||||
)
|
||||
|
||||
|
||||
class FixedFramePcmAppendReservation(PcmAppendReservation):
|
||||
__slots__ = ("_active", "_owner", "_raw", "operation_id", "payload")
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
owner: FixedFramePcmAppendBuffer,
|
||||
operation_id: str,
|
||||
payload: dict[str, object] | None,
|
||||
raw: bytes,
|
||||
) -> None:
|
||||
self._owner = owner
|
||||
self.operation_id = operation_id
|
||||
self.payload = payload
|
||||
self._raw = raw
|
||||
self._active = True
|
||||
|
||||
@property
|
||||
def active(self) -> bool:
|
||||
return self._active
|
||||
|
||||
@property
|
||||
def byte_count(self) -> int:
|
||||
return len(self._raw)
|
||||
|
||||
def commit(self) -> None:
|
||||
self._owner._commit_reservation(self)
|
||||
|
||||
def rollback(self) -> None:
|
||||
self._owner._rollback_reservation(self)
|
||||
|
||||
|
||||
class FixedFramePcmAppendBuffer(PcmAppendBuffer):
|
||||
"""Frame ``pcm_f32le`` at one sample rate into ``frame_samples`` units, transactionally."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
sample_rate_hz: int,
|
||||
frame_samples: int,
|
||||
chunk_period_ms: int,
|
||||
model: str = "duplex",
|
||||
) -> None:
|
||||
if sample_rate_hz <= 0 or frame_samples <= 0 or chunk_period_ms <= 0:
|
||||
raise ValueError("FixedFramePcmAppendBuffer needs positive sample_rate_hz, frame_samples, chunk_period_ms")
|
||||
self.sample_rate_hz = int(sample_rate_hz)
|
||||
self.frame_samples = int(frame_samples)
|
||||
self.chunk_period_ms = int(chunk_period_ms)
|
||||
self.model = model
|
||||
self._buffer = bytearray()
|
||||
self._reservations: list[FixedFramePcmAppendReservation] = []
|
||||
|
||||
@property
|
||||
def frame_bytes(self) -> int:
|
||||
return self.frame_samples * PCM_F32LE_BYTES_PER_SAMPLE
|
||||
|
||||
@property
|
||||
def pending_byte_count(self) -> int:
|
||||
return len(self._buffer)
|
||||
|
||||
def has_pending(self) -> bool:
|
||||
return bool(self._buffer)
|
||||
|
||||
def has_reserved(self) -> bool:
|
||||
return any(reservation.active for reservation in self._reservations)
|
||||
|
||||
def clear_force_listen(self) -> None:
|
||||
return
|
||||
|
||||
def clear(self) -> None:
|
||||
for reservation in self._reservations:
|
||||
reservation._active = False
|
||||
self._reservations.clear()
|
||||
self._buffer.clear()
|
||||
|
||||
def prepare_append(
|
||||
self,
|
||||
payload: dict[str, object],
|
||||
*,
|
||||
operation_id: str,
|
||||
chunk_period_ms: int,
|
||||
allow_emit: bool,
|
||||
) -> FixedFramePcmAppendReservation | None:
|
||||
self._require_chunk_period(chunk_period_ms)
|
||||
if any(reservation.active and reservation.operation_id == operation_id for reservation in self._reservations):
|
||||
raise ValueError(f"{self.model} duplicate active operation_id: {operation_id}")
|
||||
self._buffer.extend(self._decode_payload(payload))
|
||||
if not allow_emit or len(self._buffer) < self.frame_bytes:
|
||||
return None
|
||||
return self._reserve_frame(payload, operation_id=operation_id, flush=False)
|
||||
|
||||
def prepare_commit(self, *, operation_id: str, chunk_period_ms: int) -> FixedFramePcmAppendReservation:
|
||||
self._require_chunk_period(chunk_period_ms)
|
||||
if not self._buffer:
|
||||
reservation = FixedFramePcmAppendReservation(owner=self, operation_id=operation_id, payload=None, raw=b"")
|
||||
self._reservations.append(reservation)
|
||||
return reservation
|
||||
payload: dict[str, object] = {
|
||||
"type": "audio",
|
||||
"format": "pcm_f32le",
|
||||
"sample_rate_hz": self.sample_rate_hz,
|
||||
"audio": "",
|
||||
"final": True,
|
||||
}
|
||||
return self._reserve_frame(payload, operation_id=operation_id, flush=True)
|
||||
|
||||
def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
|
||||
reservation = self.prepare_commit(operation_id=f"{self.model}-flush", chunk_period_ms=chunk_period_ms)
|
||||
payload = reservation.payload
|
||||
reservation.commit()
|
||||
return payload
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Internals #
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def _require_chunk_period(self, chunk_period_ms: int) -> None:
|
||||
if chunk_period_ms != self.chunk_period_ms:
|
||||
raise ValueError(f"{self.model} chunk_period_ms must be {self.chunk_period_ms}")
|
||||
|
||||
def _decode_payload(self, payload: dict[str, object]) -> bytes:
|
||||
if payload.get("format") != "pcm_f32le":
|
||||
raise ValueError(f"{self.model} input format must be pcm_f32le")
|
||||
if payload.get("sample_rate_hz") != self.sample_rate_hz:
|
||||
raise ValueError(f"{self.model} sample_rate_hz must be {self.sample_rate_hz}")
|
||||
return decode_pcm_f32le_base64(payload.get("audio"), model=self.model)
|
||||
|
||||
def _reserve_frame(
|
||||
self,
|
||||
source_payload: dict[str, object],
|
||||
*,
|
||||
operation_id: str,
|
||||
flush: bool,
|
||||
) -> FixedFramePcmAppendReservation:
|
||||
frame_bytes = self.frame_bytes
|
||||
consumed_bytes = min(len(self._buffer), frame_bytes) if flush else frame_bytes
|
||||
raw = bytes(self._buffer[:consumed_bytes])
|
||||
del self._buffer[:consumed_bytes]
|
||||
encoded_raw = raw + b"\x00" * (frame_bytes - consumed_bytes)
|
||||
payload = dict(source_payload)
|
||||
payload["type"] = "audio"
|
||||
payload["format"] = "pcm_f32le"
|
||||
payload["sample_rate_hz"] = self.sample_rate_hz
|
||||
payload["audio"] = base64.b64encode(encoded_raw).decode("ascii")
|
||||
reservation = FixedFramePcmAppendReservation(owner=self, operation_id=operation_id, payload=payload, raw=raw)
|
||||
self._reservations.append(reservation)
|
||||
return reservation
|
||||
|
||||
def _commit_reservation(self, reservation: FixedFramePcmAppendReservation) -> None:
|
||||
if not reservation.active:
|
||||
return
|
||||
reservation._active = False
|
||||
self._reservations.remove(reservation)
|
||||
|
||||
def _rollback_reservation(self, reservation: FixedFramePcmAppendReservation) -> None:
|
||||
if not reservation.active:
|
||||
return
|
||||
try:
|
||||
index = self._reservations.index(reservation)
|
||||
except ValueError:
|
||||
reservation._active = False
|
||||
return
|
||||
# Later reservations were taken after this one: they go back too, in order.
|
||||
rolled_back = self._reservations[index:]
|
||||
restored = b"".join(item._raw for item in rolled_back if item.active)
|
||||
self._buffer[:0] = restored
|
||||
for item in rolled_back:
|
||||
item._active = False
|
||||
del self._reservations[index:]
|
||||
|
||||
|
||||
__all__ = ["FixedFramePcmAppendBuffer", "FixedFramePcmAppendReservation"]
|
||||
@@ -0,0 +1,172 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Reading a vLLM ``RequestOutput`` the way model-side projection code needs to.
|
||||
|
||||
Stage outputs arrive as duck-typed objects (a ``RequestOutput``, an
|
||||
``OmniRequestOutput`` wrapping one, or a test double): these helpers unwrap
|
||||
them, read the ``multimodal_output`` mapping, coerce tensor scalars, and turn
|
||||
cumulative audio/text into the delta since the last read. No duplex
|
||||
vocabulary lives here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
from typing import Any, cast
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
def unwrap_request_output(output: object) -> tuple[object, object | None]:
|
||||
"""Return ``(request_output, first_completion)`` for a stage output or a wrapper around one."""
|
||||
inner = getattr(output, "request_output", None)
|
||||
if inner is not None and inner is not output:
|
||||
output = inner
|
||||
return output, first_completion(output)
|
||||
|
||||
|
||||
def first_completion(output: object) -> object | None:
|
||||
outputs = getattr(output, "outputs", None)
|
||||
return outputs[0] if isinstance(outputs, list) and outputs else None
|
||||
|
||||
|
||||
def multimodal_output(output: object, completion: object | None = None) -> dict[str, object]:
|
||||
"""The first non-empty ``multimodal_output`` mapping of the output or its completion, copied."""
|
||||
for candidate in (
|
||||
getattr(output, "multimodal_output", None),
|
||||
getattr(completion, "multimodal_output", None) if completion is not None else None,
|
||||
):
|
||||
if isinstance(candidate, Mapping) and candidate:
|
||||
return dict(candidate)
|
||||
return {}
|
||||
|
||||
|
||||
def coerce_int(value: object) -> int | None:
|
||||
"""``int(value)`` for scalars, one-element tensors/arrays and numeric strings; ``None`` otherwise."""
|
||||
detach = getattr(value, "detach", None)
|
||||
if callable(detach):
|
||||
try:
|
||||
flat = detach().cpu().reshape(-1)
|
||||
if flat.numel() == 0:
|
||||
return None
|
||||
value = flat[0].item()
|
||||
except (RuntimeError, TypeError, ValueError, IndexError):
|
||||
return None
|
||||
elif isinstance(value, np.ndarray):
|
||||
if value.size == 0:
|
||||
return None
|
||||
value = value.reshape(-1)[0].item()
|
||||
try:
|
||||
return int(cast(Any, value)) # Any: duck-typed scalar (int/float/str/tensor item)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def coerce_int_list(value: object) -> list[int]:
|
||||
if value is None:
|
||||
return []
|
||||
if hasattr(value, "detach"):
|
||||
try:
|
||||
value = value.detach().cpu().reshape(-1).tolist()
|
||||
except (RuntimeError, TypeError, ValueError):
|
||||
return []
|
||||
elif isinstance(value, np.ndarray):
|
||||
value = value.reshape(-1).tolist()
|
||||
if not isinstance(value, (list, tuple)):
|
||||
return []
|
||||
return [token_id for item in value if (token_id := coerce_int(item)) is not None]
|
||||
|
||||
|
||||
def audio_value(multimodal: Mapping[str, object]) -> object | None:
|
||||
"""The audio carried by a ``multimodal_output`` (``audio`` / ``model_outputs`` / ``latent``)."""
|
||||
value = next(
|
||||
(multimodal[key] for key in ("audio", "model_outputs", "latent") if key in multimodal),
|
||||
None,
|
||||
)
|
||||
if isinstance(value, list) and len(value) == 1:
|
||||
return value[0]
|
||||
return value
|
||||
|
||||
|
||||
def text_value(multimodal: Mapping[str, object], completion: object | None) -> str:
|
||||
for candidate in (
|
||||
multimodal.get("text"),
|
||||
multimodal.get("llm_output_text"),
|
||||
getattr(completion, "text", None) if completion is not None else None,
|
||||
):
|
||||
if isinstance(candidate, str) and candidate:
|
||||
return candidate
|
||||
return ""
|
||||
|
||||
|
||||
def text_delta(text: str, previous: str) -> str:
|
||||
"""What ``text`` adds over ``previous`` when outputs are cumulative; the whole text on a restart."""
|
||||
if not text:
|
||||
return ""
|
||||
if text == previous:
|
||||
return ""
|
||||
if previous and text.startswith(previous):
|
||||
return text[len(previous) :]
|
||||
return text
|
||||
|
||||
|
||||
def audio_sample_count(audio: object | None) -> int | None:
|
||||
if audio is None:
|
||||
return None
|
||||
try:
|
||||
import torch
|
||||
|
||||
if isinstance(audio, torch.Tensor):
|
||||
return int(audio.numel())
|
||||
except ImportError:
|
||||
pass
|
||||
try:
|
||||
return int(np.asarray(audio, dtype=np.float32).size)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def slice_audio_delta(audio: object | None, offset: int) -> object | None:
|
||||
"""The samples of a cumulative ``audio`` past ``offset``; the whole audio when it restarted."""
|
||||
samples = audio_sample_count(audio)
|
||||
if samples is None or samples <= 0:
|
||||
return None
|
||||
if offset <= 0 or samples < offset:
|
||||
return audio
|
||||
if samples == offset:
|
||||
return None
|
||||
try:
|
||||
import torch
|
||||
|
||||
if isinstance(audio, torch.Tensor):
|
||||
return audio.reshape(-1)[offset:].contiguous()
|
||||
except ImportError:
|
||||
pass
|
||||
return np.asarray(audio, dtype=np.float32).reshape(-1)[offset:]
|
||||
|
||||
|
||||
def sample_rate_hz(multimodal: Mapping[str, object], *, default: int) -> int:
|
||||
value = multimodal.get("sr", multimodal.get("sample_rate_hz", default))
|
||||
if isinstance(value, list) and value:
|
||||
value = value[0]
|
||||
if hasattr(value, "item"):
|
||||
try:
|
||||
value = value.item()
|
||||
except (RuntimeError, TypeError, ValueError):
|
||||
value = default
|
||||
return int(value) if isinstance(value, int | float) and value > 0 else default
|
||||
|
||||
|
||||
__all__ = [
|
||||
"audio_sample_count",
|
||||
"audio_value",
|
||||
"coerce_int",
|
||||
"coerce_int_list",
|
||||
"first_completion",
|
||||
"multimodal_output",
|
||||
"sample_rate_hz",
|
||||
"slice_audio_delta",
|
||||
"text_delta",
|
||||
"text_value",
|
||||
"unwrap_request_output",
|
||||
]
|
||||
@@ -12,7 +12,7 @@ from vllm.logger import init_logger
|
||||
from vllm_omni.engine.duplex.contracts import (
|
||||
duplex_resource_request_belongs_to_session,
|
||||
)
|
||||
from vllm_omni.engine.duplex.plugin import DuplexDataPlane
|
||||
from vllm_omni.engine.duplex.plugin import DuplexDataPlane, DuplexDataPlaneContext
|
||||
from vllm_omni.outputs.duplex import get_duplex_output_decision
|
||||
|
||||
logger = init_logger(__name__)
|
||||
@@ -21,17 +21,8 @@ EncodeAudio = Callable[[object, int, str, float | None], str | None]
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class MiniCPMO45DataPlaneContext:
|
||||
"""Serving state needed to project one MiniCPM data-plane output."""
|
||||
|
||||
epoch: int = 0
|
||||
turn_id: int = 0
|
||||
active_response_turn_id: int | None = None
|
||||
active_response_id: str | None = None
|
||||
auto_responds: bool = False
|
||||
response_format: str = "wav"
|
||||
speed: float | None = None
|
||||
modalities: tuple[str, ...] = ()
|
||||
class MiniCPMO45DataPlaneContext(DuplexDataPlaneContext):
|
||||
"""Serving state needed to project one MiniCPM data-plane output (the framework fields)."""
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
|
||||
@@ -18,7 +18,7 @@ from base64 import b64decode
|
||||
from binascii import Error as BinasciiError
|
||||
from collections.abc import Mapping
|
||||
from copy import deepcopy
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import numpy as np
|
||||
from numpy.typing import NDArray
|
||||
@@ -40,6 +40,11 @@ from vllm_omni.engine.duplex.plugin import (
|
||||
reject_changed_runtime_value,
|
||||
reject_private_runtime_keys,
|
||||
)
|
||||
from vllm_omni.model_executor.common.duplex.payload import payload_sample_count as _duplex_pcm_sample_count
|
||||
from vllm_omni.model_executor.common.request_outputs import coerce_int as _coerce_int
|
||||
from vllm_omni.model_executor.common.request_outputs import coerce_int_list as _coerce_int_list
|
||||
from vllm_omni.model_executor.common.request_outputs import first_completion as _first_completion
|
||||
from vllm_omni.model_executor.common.request_outputs import multimodal_output as _multimodal_output
|
||||
from vllm_omni.model_executor.models.minicpmo_4_5.duplex.capabilities import (
|
||||
minicpmo45_native_capabilities,
|
||||
)
|
||||
@@ -56,7 +61,6 @@ from vllm_omni.model_executor.models.minicpmo_4_5.duplex.session import (
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import torch
|
||||
from transformers import PreTrainedTokenizerBase
|
||||
from vllm.config import ModelConfig
|
||||
|
||||
@@ -170,19 +174,6 @@ def _duplex_vision_tokens(payload: object, *, tile_pixels: int | None = None) ->
|
||||
return blocks * _DUPLEX_VISION_TOKENS_PER_FRAME
|
||||
|
||||
|
||||
def _duplex_pcm_sample_count(payload: object) -> int | None:
|
||||
if not isinstance(payload, dict):
|
||||
return None
|
||||
audio = payload.get("audio") or payload.get("data")
|
||||
if payload.get("format") != "pcm_f32le" or not isinstance(audio, str):
|
||||
return None
|
||||
try:
|
||||
raw = b64decode(audio, validate=True)
|
||||
except (BinasciiError, ValueError):
|
||||
return None
|
||||
return len(raw) // 4
|
||||
|
||||
|
||||
def duplex_payload_is_exact_chunks(payload: object) -> bool:
|
||||
sample_count = _duplex_pcm_sample_count(payload)
|
||||
return sample_count is not None and sample_count != 0 and sample_count % _DUPLEX_CHUNK_SAMPLES == 0
|
||||
@@ -288,48 +279,6 @@ def build_duplex_data_plane_prompt(
|
||||
# ---- engine policy helpers: listen decision ----
|
||||
|
||||
|
||||
def _coerce_int(value: object) -> int | None:
|
||||
detach = getattr(value, "detach", None)
|
||||
if callable(detach):
|
||||
try:
|
||||
flat: torch.Tensor = detach().cpu().reshape(-1)
|
||||
if flat.numel() == 0:
|
||||
return None
|
||||
value = flat[0].item()
|
||||
except Exception:
|
||||
return None
|
||||
try:
|
||||
return int(cast(Any, value)) # Any: duck-typed scalar (int/float/str/tensor item)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _coerce_int_list(value: object) -> list[int]:
|
||||
if value is None:
|
||||
return []
|
||||
if hasattr(value, "detach"):
|
||||
try:
|
||||
value = value.detach().cpu().reshape(-1).tolist()
|
||||
except Exception:
|
||||
return []
|
||||
if not isinstance(value, (list, tuple)):
|
||||
return []
|
||||
return [token_id for item in value if (token_id := _coerce_int(item)) is not None]
|
||||
|
||||
|
||||
def _first_completion(output: object) -> object | None:
|
||||
outputs = getattr(output, "outputs", None)
|
||||
return outputs[0] if isinstance(outputs, list) and outputs else None
|
||||
|
||||
|
||||
def _multimodal_output(output: object, completion: object | None) -> dict[str, object]:
|
||||
metadata = getattr(output, "multimodal_output", None)
|
||||
if isinstance(metadata, dict):
|
||||
return metadata
|
||||
metadata = getattr(completion, "multimodal_output", None) if completion is not None else None
|
||||
return metadata if isinstance(metadata, dict) else {}
|
||||
|
||||
|
||||
def _special_token_ids(metadata: dict[str, object]) -> dict[str, int]:
|
||||
sources: list[object] = [metadata.get("special_token_ids"), metadata.get("meta")]
|
||||
sources.append(
|
||||
|
||||
@@ -3,39 +3,22 @@
|
||||
"""PersonaPlex full-duplex integration.
|
||||
|
||||
PersonaPlex (``nvidia/personaplex-7b-v1``) is a Moshi finetune: a pure-lockstep
|
||||
speech-to-speech model. This package plugs it into the generic duplex serving
|
||||
stack through the standard plugin seams (``duplex_serving_adapter`` /
|
||||
``duplex_runtime_extension`` dotted strings in the model's ``pipeline.py``):
|
||||
speech-to-speech model. This package plugs it into the unified full-duplex
|
||||
framework through the one seam the framework has, ``PipelineConfig.duplex_plugin``:
|
||||
|
||||
- :class:`PersonaPlexConfig` immutable session config (voice / persona / sampling)
|
||||
- :class:`PersonaPlexServingRuntimeAdapter` the ``ServingRuntimeAdapter`` impl
|
||||
- :class:`PersonaPlexDuplexRuntimeExtension` the engine ``DuplexRuntimeExtension``
|
||||
- :class:`PersonaPlexStage0DuplexRuntime` Stage 0 session state and prefill
|
||||
- :class:`PersonaPlexPcmAppendBuffer` PCM input framing
|
||||
- :class:`PersonaPlexDuplexPlugin` the ``DuplexModelPlugin`` (engine + session policy)
|
||||
- :class:`PersonaPlexStage0DuplexRuntime` worker-side lockstep state and first-append prefill
|
||||
- :class:`PersonaPlexPcmAppendBuffer` 80 ms PCM input framing
|
||||
"""
|
||||
|
||||
from .config import (
|
||||
PersonaPlexConfig,
|
||||
)
|
||||
from .input import (
|
||||
PersonaPlexPcmAppendBuffer,
|
||||
)
|
||||
from .input import PersonaPlexPcmAppendBuffer
|
||||
from .plugin import PersonaPlexDuplexPlugin
|
||||
from .policy import PrefillStep
|
||||
from .runtime_extension import (
|
||||
PersonaPlexDuplexRuntimeExtension,
|
||||
)
|
||||
from .serving_adapter import (
|
||||
PersonaPlexServingRuntimeAdapter,
|
||||
)
|
||||
from .stage0 import (
|
||||
PersonaPlexStage0DuplexRuntime,
|
||||
)
|
||||
from .stage0 import PersonaPlexStage0DuplexRuntime
|
||||
|
||||
__all__ = [
|
||||
"PersonaPlexConfig",
|
||||
"PersonaPlexDuplexRuntimeExtension",
|
||||
"PersonaPlexDuplexPlugin",
|
||||
"PersonaPlexPcmAppendBuffer",
|
||||
"PersonaPlexServingRuntimeAdapter",
|
||||
"PersonaPlexStage0DuplexRuntime",
|
||||
"PrefillStep",
|
||||
]
|
||||
@@ -0,0 +1,55 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from vllm_omni.engine.duplex.config import DuplexCapabilities
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import CHUNK_PERIOD_MS
|
||||
|
||||
|
||||
def personaplex_capabilities(*, max_sessions: int = 1) -> DuplexCapabilities:
|
||||
"""What a PersonaPlex session advertises in ``session.created``.
|
||||
|
||||
PersonaPlex is a pure-lockstep model: audio flows both ways at the codec
|
||||
frame rate, the model itself decides when to speak, and there are no
|
||||
client commits or external turn signals. Barge-in is native model
|
||||
behaviour but destructive output interruption and model-state rewind are
|
||||
not validated, so it is not advertised. No text seeding exists, so the
|
||||
model cannot serve ``/v1/chat/completions``.
|
||||
"""
|
||||
supports_multi_session = max_sessions > 1
|
||||
return DuplexCapabilities(
|
||||
supports_model_native_turn_policy=True,
|
||||
supports_external_turn_signal=False,
|
||||
supports_client_commit=False,
|
||||
supports_barge_in=False,
|
||||
supports_playback_ack=True,
|
||||
supports_input_append=True,
|
||||
supports_replace_latest_chunk=False,
|
||||
supports_reencode_context=False,
|
||||
supports_rollback_to_checkpoint=False,
|
||||
supports_turn_commit_only=False,
|
||||
supports_model_internal_state=True,
|
||||
supports_stage_resumption=True,
|
||||
supports_core_resumable_request=True,
|
||||
supports_stage_connector_handoff=True,
|
||||
supports_independent_io_streams=True,
|
||||
supports_realtime_endpoint=True,
|
||||
supports_multi_session=supports_multi_session,
|
||||
supports_multi_session_same_replica=supports_multi_session,
|
||||
supports_session_lease=True,
|
||||
supports_session_resume=False,
|
||||
session_admission_mode="engine_managed",
|
||||
supports_audio_truncate=False,
|
||||
supports_chat_completions=False,
|
||||
requires_model_runner_kv=True,
|
||||
requires_native_stage_role=True,
|
||||
adapter_patterns=["scheduler_data_plane"],
|
||||
signal_sources=["model_native", "client_event"],
|
||||
stage_handoff_transport="scheduler_data_plane",
|
||||
chunk_period_ms=CHUNK_PERIOD_MS,
|
||||
target_barge_in_latency_ms=None,
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["personaplex_capabilities"]
|
||||
@@ -1,64 +1,24 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Configuration for the PersonaPlex full-duplex backend.
|
||||
"""Constants of the PersonaPlex full-duplex integration.
|
||||
|
||||
PersonaPlex (``nvidia/personaplex-7b-v1``) is a Moshi finetune: a pure-lockstep
|
||||
full-duplex speech-to-speech model running at the Mimi codec frame rate
|
||||
(12.5 Hz / 80 ms). One config drives both the offline driver and the duplex
|
||||
adapter. Defaults mirror the PersonaPlex reference loop.
|
||||
(12.5 Hz / 80 ms). The values are fixed by the pretrained checkpoint; they
|
||||
live here so the rest of the package never hard-codes them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
# Mimi codec constants (loaders.py: SAMPLE_RATE / FRAME_RATE). Fixed by the
|
||||
# pretrained checkpoint; exposed here only so the rest of the package never
|
||||
# hard-codes them.
|
||||
# Mimi codec constants (loaders.py: SAMPLE_RATE / FRAME_RATE).
|
||||
SAMPLE_RATE = 24000
|
||||
FRAME_RATE = 12.5
|
||||
FRAME_SIZE = int(SAMPLE_RATE / FRAME_RATE) # 1920 samples per 80 ms frame
|
||||
CHUNK_PERIOD_MS = int(round(1000 / FRAME_RATE)) # 80
|
||||
|
||||
# Default assistant persona shipped with PersonaPlex (offline.py:335).
|
||||
DEFAULT_PERSONA = "You are a wise and friendly teacher. Answer questions or provide advice in a clear and engaging way."
|
||||
# Bundled voice prompt used when a session names none.
|
||||
DEFAULT_VOICE = "NATF2.pt"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PersonaPlexConfig:
|
||||
"""Immutable session configuration for a PersonaPlex conversation.
|
||||
|
||||
Attributes:
|
||||
hf_repo: HuggingFace repo holding the weights, Mimi codec and tokenizer.
|
||||
voice_prompt: Voice-clone reference. Either a bundled basename
|
||||
(``"NATF2.pt"`` / ``"NATM1.pt"`` from ``voices.tgz``) or a path to a
|
||||
``.pt`` embedding bundle or a reference ``.wav``.
|
||||
persona: System role text; injected as ``<system> ... <system>`` into the
|
||||
inner-monologue stream at session start.
|
||||
device: Torch device for the backend (``"cuda"`` / ``"cpu"``).
|
||||
cpu_offload: Offload LM layers to CPU when GPU memory is tight (needs
|
||||
``accelerate``).
|
||||
batch_size: Concurrent conversation slots sharing one engine. ``1`` is
|
||||
the single-session path; ``> 1`` enables elastic batching with
|
||||
per-slot recycle for new callers.
|
||||
|
||||
Note: the native stepper decodes greedily (argmax) for both the text head
|
||||
and the depformer, so there are no sampling knobs here yet. Temperature /
|
||||
top-k / seed fields will be added if and when a sampling path is wired in.
|
||||
"""
|
||||
|
||||
hf_repo: str = "nvidia/personaplex-7b-v1"
|
||||
voice_prompt: str = "NATF2.pt"
|
||||
persona: str = DEFAULT_PERSONA
|
||||
|
||||
device: str = "cuda"
|
||||
cpu_offload: bool = False
|
||||
|
||||
batch_size: int = 1
|
||||
|
||||
@property
|
||||
def sample_rate(self) -> int:
|
||||
return SAMPLE_RATE
|
||||
|
||||
@property
|
||||
def frame_size(self) -> int:
|
||||
return FRAME_SIZE
|
||||
__all__ = ["CHUNK_PERIOD_MS", "DEFAULT_PERSONA", "DEFAULT_VOICE", "FRAME_RATE", "FRAME_SIZE", "SAMPLE_RATE"]
|
||||
@@ -1,228 +1,17 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""PersonaPlex output projection: cumulative Code2Wav audio and inner-monologue text into deltas."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Callable, Iterator, Mapping
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
from vllm_omni.engine.duplex.contracts import (
|
||||
duplex_resource_request_belongs_to_session,
|
||||
)
|
||||
|
||||
EncodeAudio = Callable[[object, int, str, float | None], str | None]
|
||||
from vllm_omni.model_executor.common.duplex.data_plane import CumulativeAudioTextDataPlane
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import SAMPLE_RATE
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class PersonaPlexDataPlaneContext:
|
||||
epoch: int = 0
|
||||
turn_id: int = 0
|
||||
active_response_turn_id: int | None = None
|
||||
active_response_id: str | None = None
|
||||
auto_responds: bool = True
|
||||
response_format: str = "wav"
|
||||
speed: float | None = None
|
||||
modalities: tuple[str, ...] = ("audio", "text")
|
||||
class PersonaPlexDataPlaneSession(CumulativeAudioTextDataPlane):
|
||||
"""Project cumulative staged PersonaPlex output into Realtime deltas (24 kHz by default)."""
|
||||
|
||||
default_sample_rate_hz = SAMPLE_RATE
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _RequestCursor:
|
||||
audio_samples: int = 0
|
||||
text: str = ""
|
||||
terminal: bool = False
|
||||
|
||||
|
||||
class PersonaPlexDataPlaneSession:
|
||||
"""Project cumulative staged PersonaPlex output into Realtime deltas."""
|
||||
|
||||
def __init__(self, encode_audio: EncodeAudio) -> None:
|
||||
self._encode_audio = encode_audio
|
||||
self._requests: dict[str, _RequestCursor] = {}
|
||||
|
||||
def begin_request(self, request_id: str) -> None:
|
||||
self._requests.setdefault(request_id, _RequestCursor()).terminal = False
|
||||
|
||||
def is_terminal(self, request_id: str | None) -> bool:
|
||||
if request_id is None:
|
||||
return False
|
||||
state = self._requests.get(request_id)
|
||||
return state is not None and state.terminal
|
||||
|
||||
def mark_terminal(self, request_id: str) -> None:
|
||||
self._requests.setdefault(request_id, _RequestCursor()).terminal = True
|
||||
|
||||
def close_stream(self, request_id: str) -> None:
|
||||
self._requests.pop(request_id, None)
|
||||
|
||||
def close_session(self, session_id: str, *, active_request_id: str | None = None) -> None:
|
||||
if active_request_id is not None:
|
||||
self._requests.pop(active_request_id, None)
|
||||
for request_id in list(self._requests):
|
||||
if duplex_resource_request_belongs_to_session(request_id, session_id):
|
||||
self._requests.pop(request_id, None)
|
||||
|
||||
def project(
|
||||
self,
|
||||
result: object,
|
||||
*,
|
||||
context: PersonaPlexDataPlaneContext | None = None,
|
||||
) -> Iterator[dict[str, object]]:
|
||||
if not isinstance(result, dict):
|
||||
return
|
||||
outputs = result.get("data_plane_outputs")
|
||||
if not isinstance(outputs, list):
|
||||
return
|
||||
context = context or PersonaPlexDataPlaneContext()
|
||||
for output in outputs:
|
||||
projected = self._project_output(output, context=context)
|
||||
if projected is not None:
|
||||
yield projected
|
||||
|
||||
def _project_output(
|
||||
self,
|
||||
output: object,
|
||||
*,
|
||||
context: PersonaPlexDataPlaneContext,
|
||||
) -> dict[str, object] | None:
|
||||
output, completion = _unwrap_output(output)
|
||||
request_id = getattr(output, "request_id", None)
|
||||
if not isinstance(request_id, str) or not request_id:
|
||||
request_id = None
|
||||
state = self._requests.setdefault(request_id, _RequestCursor()) if request_id is not None else _RequestCursor()
|
||||
multimodal = _multimodal_output(output, completion)
|
||||
audio = _audio_value(multimodal)
|
||||
audio_delta = _slice_audio(audio, state.audio_samples)
|
||||
audio_samples = _num_samples(audio)
|
||||
if audio_samples is not None:
|
||||
state.audio_samples = audio_samples
|
||||
|
||||
text = _text_value(multimodal, completion)
|
||||
text_delta = _text_delta(text, state.text)
|
||||
if text:
|
||||
state.text = text
|
||||
|
||||
sample_rate_hz = _sample_rate(multimodal)
|
||||
encoded = self._encode_audio(audio_delta, sample_rate_hz, context.response_format, context.speed)
|
||||
if not encoded and not text_delta:
|
||||
return None
|
||||
delta_samples = _num_samples(audio_delta) or 0
|
||||
return {
|
||||
"supported": True,
|
||||
"stage_role": "tts",
|
||||
"is_listen": False,
|
||||
"data_plane_request_id": request_id,
|
||||
"text": text_delta,
|
||||
"audio_data": encoded or "",
|
||||
"audio_format": context.response_format,
|
||||
"sample_rate_hz": sample_rate_hz,
|
||||
"audio_duration_ms": round(delta_samples * 1000 / max(1, sample_rate_hz)),
|
||||
"end_of_turn": False,
|
||||
"uses_model_runner_scheduler": True,
|
||||
"runner_kv_backed": True,
|
||||
"runtime_impl": "scheduler_data_plane",
|
||||
"owned_runtime": False,
|
||||
}
|
||||
|
||||
|
||||
def _unwrap_output(output: object) -> tuple[object, object | None]:
|
||||
inner = getattr(output, "request_output", None)
|
||||
if inner is not None and inner is not output:
|
||||
output = inner
|
||||
outputs = getattr(output, "outputs", None)
|
||||
completion = outputs[0] if isinstance(outputs, list) and outputs else None
|
||||
return output, completion
|
||||
|
||||
|
||||
def _multimodal_output(output: object, completion: object | None) -> dict[str, object]:
|
||||
for candidate in (
|
||||
getattr(output, "multimodal_output", None),
|
||||
getattr(completion, "multimodal_output", None) if completion is not None else None,
|
||||
):
|
||||
if isinstance(candidate, Mapping) and candidate:
|
||||
return dict(candidate)
|
||||
return {}
|
||||
|
||||
|
||||
def _audio_value(multimodal: dict[str, object]) -> object | None:
|
||||
value = next(
|
||||
(multimodal[key] for key in ("audio", "model_outputs", "latent") if key in multimodal),
|
||||
None,
|
||||
)
|
||||
if isinstance(value, list) and len(value) == 1:
|
||||
return value[0]
|
||||
return value
|
||||
|
||||
|
||||
def _text_value(multimodal: dict[str, object], completion: object | None) -> str:
|
||||
for candidate in (
|
||||
multimodal.get("text"),
|
||||
multimodal.get("llm_output_text"),
|
||||
getattr(completion, "text", None) if completion is not None else None,
|
||||
):
|
||||
if isinstance(candidate, str) and candidate:
|
||||
return candidate
|
||||
return ""
|
||||
|
||||
|
||||
def _text_delta(text: str, previous: str) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
if text == previous:
|
||||
return ""
|
||||
if previous and text.startswith(previous):
|
||||
return text[len(previous) :]
|
||||
return text
|
||||
|
||||
|
||||
def _slice_audio(audio: object | None, offset: int) -> object | None:
|
||||
samples = _num_samples(audio)
|
||||
if samples is None or samples <= 0:
|
||||
return None
|
||||
if offset <= 0 or samples < offset:
|
||||
return audio
|
||||
if samples == offset:
|
||||
return None
|
||||
try:
|
||||
import torch
|
||||
|
||||
if isinstance(audio, torch.Tensor):
|
||||
return audio.reshape(-1)[offset:].contiguous()
|
||||
except Exception:
|
||||
pass
|
||||
return np.asarray(audio, dtype=np.float32).reshape(-1)[offset:]
|
||||
|
||||
|
||||
def _num_samples(audio: object | None) -> int | None:
|
||||
if audio is None:
|
||||
return None
|
||||
try:
|
||||
import torch
|
||||
|
||||
if isinstance(audio, torch.Tensor):
|
||||
return int(audio.numel())
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
return int(np.asarray(audio, dtype=np.float32).size)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _sample_rate(multimodal: dict[str, object]) -> int:
|
||||
value = multimodal.get("sr", multimodal.get("sample_rate_hz", 24000))
|
||||
if isinstance(value, list) and value:
|
||||
value = value[0]
|
||||
if hasattr(value, "item"):
|
||||
try:
|
||||
value = value.item()
|
||||
except Exception:
|
||||
value = 24000
|
||||
return int(value) if isinstance(value, int | float) else 24000
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PersonaPlexDataPlaneContext",
|
||||
"PersonaPlexDataPlaneSession",
|
||||
]
|
||||
__all__ = ["PersonaPlexDataPlaneSession"]
|
||||
@@ -1,213 +1,27 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""PCM input framing for PersonaPlex: 24 kHz ``pcm_f32le`` into 1920-sample (80 ms) units."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
|
||||
import numpy as np
|
||||
|
||||
_SAMPLE_RATE_HZ = 24000
|
||||
_FRAME_SAMPLES = 1920
|
||||
_BYTES_PER_SAMPLE = 4
|
||||
from vllm_omni.model_executor.common.duplex.pcm_buffer import FixedFramePcmAppendBuffer
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import (
|
||||
CHUNK_PERIOD_MS,
|
||||
FRAME_SIZE,
|
||||
SAMPLE_RATE,
|
||||
)
|
||||
|
||||
|
||||
class PersonaPlexPcmAppendReservation:
|
||||
__slots__ = (
|
||||
"_active",
|
||||
"_owner",
|
||||
"_raw",
|
||||
"operation_id",
|
||||
"payload",
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
owner: PersonaPlexPcmAppendBuffer,
|
||||
operation_id: str,
|
||||
payload: dict[str, object] | None,
|
||||
raw: bytes,
|
||||
) -> None:
|
||||
self._owner = owner
|
||||
self.operation_id = operation_id
|
||||
self.payload = payload
|
||||
self._raw = raw
|
||||
self._active = True
|
||||
|
||||
@property
|
||||
def active(self) -> bool:
|
||||
return self._active
|
||||
|
||||
@property
|
||||
def byte_count(self) -> int:
|
||||
return len(self._raw)
|
||||
|
||||
def commit(self) -> None:
|
||||
self._owner._commit_reservation(self)
|
||||
|
||||
def rollback(self) -> None:
|
||||
self._owner._rollback_reservation(self)
|
||||
|
||||
|
||||
class PersonaPlexPcmAppendBuffer:
|
||||
class PersonaPlexPcmAppendBuffer(FixedFramePcmAppendBuffer):
|
||||
"""Transactionally frame 24 kHz float PCM into PersonaPlex 80 ms units."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._buffer = bytearray()
|
||||
self._sample_rate_hz: int | None = None
|
||||
self._reservations: list[PersonaPlexPcmAppendReservation] = []
|
||||
|
||||
@property
|
||||
def pending_byte_count(self) -> int:
|
||||
return len(self._buffer)
|
||||
|
||||
def has_pending(self) -> bool:
|
||||
return bool(self._buffer)
|
||||
|
||||
def has_reserved(self) -> bool:
|
||||
return any(reservation.active for reservation in self._reservations)
|
||||
|
||||
def clear_force_listen(self) -> None:
|
||||
return
|
||||
|
||||
def clear(self) -> None:
|
||||
for reservation in self._reservations:
|
||||
reservation._active = False
|
||||
self._reservations.clear()
|
||||
self._buffer.clear()
|
||||
self._sample_rate_hz = None
|
||||
|
||||
def prepare_append(
|
||||
self,
|
||||
payload: dict[str, object],
|
||||
*,
|
||||
operation_id: str,
|
||||
chunk_period_ms: int,
|
||||
allow_emit: bool,
|
||||
) -> PersonaPlexPcmAppendReservation | None:
|
||||
if chunk_period_ms != 80:
|
||||
raise ValueError("PersonaPlex chunk_period_ms must be 80")
|
||||
if any(reservation.active and reservation.operation_id == operation_id for reservation in self._reservations):
|
||||
raise ValueError(f"PersonaPlex duplicate active operation_id: {operation_id}")
|
||||
|
||||
raw = self._decode_payload(payload)
|
||||
self._buffer.extend(raw)
|
||||
if not allow_emit or len(self._buffer) < _FRAME_SAMPLES * _BYTES_PER_SAMPLE:
|
||||
return None
|
||||
return self._reserve_frame(payload, operation_id=operation_id, flush=False)
|
||||
|
||||
def prepare_commit(
|
||||
self,
|
||||
*,
|
||||
operation_id: str,
|
||||
chunk_period_ms: int,
|
||||
) -> PersonaPlexPcmAppendReservation:
|
||||
if chunk_period_ms != 80:
|
||||
raise ValueError("PersonaPlex chunk_period_ms must be 80")
|
||||
if not self._buffer:
|
||||
reservation = PersonaPlexPcmAppendReservation(
|
||||
owner=self,
|
||||
operation_id=operation_id,
|
||||
payload=None,
|
||||
raw=b"",
|
||||
)
|
||||
self._reservations.append(reservation)
|
||||
return reservation
|
||||
payload: dict[str, object] = {
|
||||
"type": "audio",
|
||||
"format": "pcm_f32le",
|
||||
"sample_rate_hz": self._sample_rate_hz or _SAMPLE_RATE_HZ,
|
||||
"audio": "",
|
||||
"final": True,
|
||||
}
|
||||
return self._reserve_frame(payload, operation_id=operation_id, flush=True)
|
||||
|
||||
def flush(self, *, chunk_period_ms: int) -> dict[str, object] | None:
|
||||
reservation = self.prepare_commit(
|
||||
operation_id="personaplex-flush",
|
||||
chunk_period_ms=chunk_period_ms,
|
||||
super().__init__(
|
||||
sample_rate_hz=SAMPLE_RATE,
|
||||
frame_samples=FRAME_SIZE,
|
||||
chunk_period_ms=CHUNK_PERIOD_MS,
|
||||
model="PersonaPlex",
|
||||
)
|
||||
payload = reservation.payload
|
||||
reservation.commit()
|
||||
return payload
|
||||
|
||||
def _decode_payload(self, payload: dict[str, object]) -> bytes:
|
||||
if payload.get("format") != "pcm_f32le":
|
||||
raise ValueError("PersonaPlex input format must be pcm_f32le")
|
||||
sample_rate_hz = payload.get("sample_rate_hz")
|
||||
if sample_rate_hz != _SAMPLE_RATE_HZ:
|
||||
raise ValueError("PersonaPlex sample_rate_hz must be 24000")
|
||||
if self._sample_rate_hz is not None and self._sample_rate_hz != sample_rate_hz:
|
||||
raise ValueError("PersonaPlex sample_rate_hz changed within a session")
|
||||
audio = payload.get("audio")
|
||||
if not isinstance(audio, str):
|
||||
raise ValueError("PersonaPlex audio must be base64 pcm_f32le")
|
||||
try:
|
||||
raw = base64.b64decode(audio, validate=True)
|
||||
except (binascii.Error, ValueError) as exc:
|
||||
raise ValueError("PersonaPlex audio is not valid base64") from exc
|
||||
if len(raw) % _BYTES_PER_SAMPLE:
|
||||
raise ValueError("PersonaPlex pcm_f32le byte length must be divisible by four")
|
||||
samples = np.frombuffer(raw, dtype="<f4")
|
||||
if not np.isfinite(samples).all():
|
||||
raise ValueError("PersonaPlex pcm_f32le samples must be finite")
|
||||
self._sample_rate_hz = int(sample_rate_hz)
|
||||
return raw
|
||||
|
||||
def _reserve_frame(
|
||||
self,
|
||||
source_payload: dict[str, object],
|
||||
*,
|
||||
operation_id: str,
|
||||
flush: bool,
|
||||
) -> PersonaPlexPcmAppendReservation:
|
||||
frame_bytes = _FRAME_SAMPLES * _BYTES_PER_SAMPLE
|
||||
if flush:
|
||||
consumed_bytes = min(len(self._buffer), frame_bytes)
|
||||
else:
|
||||
consumed_bytes = frame_bytes
|
||||
raw = bytes(self._buffer[:consumed_bytes])
|
||||
del self._buffer[:consumed_bytes]
|
||||
encoded_raw = raw + b"\x00" * (frame_bytes - consumed_bytes)
|
||||
payload = dict(source_payload)
|
||||
payload["type"] = "audio"
|
||||
payload["format"] = "pcm_f32le"
|
||||
payload["sample_rate_hz"] = _SAMPLE_RATE_HZ
|
||||
payload["audio"] = base64.b64encode(encoded_raw).decode("ascii")
|
||||
reservation = PersonaPlexPcmAppendReservation(
|
||||
owner=self,
|
||||
operation_id=operation_id,
|
||||
payload=payload,
|
||||
raw=raw,
|
||||
)
|
||||
self._reservations.append(reservation)
|
||||
return reservation
|
||||
|
||||
def _commit_reservation(self, reservation: PersonaPlexPcmAppendReservation) -> None:
|
||||
if not reservation.active:
|
||||
return
|
||||
reservation._active = False
|
||||
self._reservations.remove(reservation)
|
||||
|
||||
def _rollback_reservation(self, reservation: PersonaPlexPcmAppendReservation) -> None:
|
||||
if not reservation.active:
|
||||
return
|
||||
try:
|
||||
index = self._reservations.index(reservation)
|
||||
except ValueError:
|
||||
reservation._active = False
|
||||
return
|
||||
rolled_back = self._reservations[index:]
|
||||
restored = b"".join(item._raw for item in rolled_back if item.active)
|
||||
self._buffer[:0] = restored
|
||||
for item in rolled_back:
|
||||
item._active = False
|
||||
del self._reservations[index:]
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PersonaPlexPcmAppendBuffer",
|
||||
"PersonaPlexPcmAppendReservation",
|
||||
]
|
||||
__all__ = ["PersonaPlexPcmAppendBuffer"]
|
||||
@@ -0,0 +1,269 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""PersonaPlex full-duplex model plugin: engine policy and session policy in one class.
|
||||
|
||||
PersonaPlex is a pure-lockstep model. Engine policy is one 80 ms frame per
|
||||
resumable Stage 0 append with greedy one-token decoding; session policy is
|
||||
the bundled voice prompt and the persona text, resolved once at open into the
|
||||
server-owned runtime configuration the worker replays as its prefill.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Mapping
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePath
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from vllm.sampling_params import SamplingParams
|
||||
|
||||
from vllm_omni.engine.duplex.config import DuplexCapabilities, DuplexSessionConfig
|
||||
from vllm_omni.engine.duplex.contracts import (
|
||||
DuplexAppendPlan,
|
||||
DuplexFence,
|
||||
DuplexOutputDecision,
|
||||
)
|
||||
from vllm_omni.engine.duplex.plugin import (
|
||||
DefaultDuplexModelSessionState,
|
||||
DuplexModelPlugin,
|
||||
DuplexRuntimeConfigError,
|
||||
EncodeAudio,
|
||||
reject_changed_runtime_value,
|
||||
)
|
||||
from vllm_omni.model_executor.common.duplex.payload import decode_pcm_f32le_payload
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.capabilities import personaplex_capabilities
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import (
|
||||
DEFAULT_PERSONA,
|
||||
DEFAULT_VOICE,
|
||||
FRAME_SIZE,
|
||||
SAMPLE_RATE,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import PersonaPlexDataPlaneSession
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.input import PersonaPlexPcmAppendBuffer
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from vllm.config import ModelConfig
|
||||
|
||||
#: Runtime-config keys the server derives from the session fields; a client
|
||||
#: cannot set them through ``extra_body``.
|
||||
PRIVATE_RUNTIME_CONFIG_KEYS = frozenset(
|
||||
{
|
||||
"personaplex_prefill_slots",
|
||||
"personaplex_model_path",
|
||||
"personaplex_voice_prompt",
|
||||
"personaplex_persona",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PersonaPlexSessionState(DefaultDuplexModelSessionState):
|
||||
"""Per-session model state: the framework flags plus the 80 ms framing buffer."""
|
||||
|
||||
audio_buffer: PersonaPlexPcmAppendBuffer = field(default_factory=PersonaPlexPcmAppendBuffer)
|
||||
|
||||
|
||||
def voice_name(value: object) -> str:
|
||||
"""A bundled ``.pt`` basename (``NATF2.pt``); paths are refused so the worker only reads the checkpoint."""
|
||||
voice = value if isinstance(value, str) and value else DEFAULT_VOICE
|
||||
path = PurePath(voice)
|
||||
if path.name != voice or path.suffix != ".pt" or any(part == ".." for part in path.parts):
|
||||
raise DuplexRuntimeConfigError("PersonaPlex voice must be a bundled .pt basename", code="invalid_voice")
|
||||
return voice
|
||||
|
||||
|
||||
def persona_text(value: object) -> str:
|
||||
return str(value) if isinstance(value, str) and value else DEFAULT_PERSONA
|
||||
|
||||
|
||||
def prefill_slot_count(runtime_config: Mapping[str, object]) -> int:
|
||||
raw = runtime_config.get("personaplex_prefill_slots", 0)
|
||||
try:
|
||||
return max(0, int(raw)) # type: ignore[arg-type]
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("personaplex_prefill_slots must be a non-negative integer") from exc
|
||||
|
||||
|
||||
class PersonaPlexDuplexPlugin(DuplexModelPlugin):
|
||||
"""PersonaPlex-owned sampling policy, append planning, session state and output projection."""
|
||||
|
||||
plugin_id = "personaplex"
|
||||
private_runtime_config_keys = PRIVATE_RUNTIME_CONFIG_KEYS
|
||||
#: One codec frame: the runner keeps a model turn clocked with these when the client pauses.
|
||||
silence_continuation_samples = FRAME_SIZE
|
||||
silence_continuation_sample_rate_hz = SAMPLE_RATE
|
||||
|
||||
def __init__(self, encode_audio: EncodeAudio) -> None:
|
||||
super().__init__(encode_audio)
|
||||
self.data_plane = PersonaPlexDataPlaneSession(encode_audio)
|
||||
# The prefill length depends only on (model, voice, persona); the
|
||||
# voice bundle and tokenizer are read once per distinct triple.
|
||||
self._prefill_slots: dict[tuple[str, str, str], int] = {}
|
||||
self._prefill_lock = asyncio.Lock()
|
||||
|
||||
# ---- engine policy (the resumable Stage 0 request) ----
|
||||
|
||||
def configure_sampling_params(
|
||||
self,
|
||||
*,
|
||||
runtime_config: dict[str, object],
|
||||
defaults: tuple[object, ...],
|
||||
) -> tuple[object, ...]:
|
||||
del runtime_config
|
||||
if not defaults:
|
||||
return defaults
|
||||
configured = list(defaults)
|
||||
stage0 = defaults[0]
|
||||
if isinstance(stage0, SamplingParams):
|
||||
# Greedy, one temporal token per frame: the native stepper takes
|
||||
# argmax for the text head and the depformer.
|
||||
stage0 = stage0.clone()
|
||||
stage0.temperature = 0.0
|
||||
stage0.top_k = 1
|
||||
stage0.max_tokens = 1
|
||||
configured[0] = stage0
|
||||
return tuple(configured)
|
||||
|
||||
def plan_append(
|
||||
self,
|
||||
*,
|
||||
request_id: str,
|
||||
fence: DuplexFence,
|
||||
session_config: dict[str, object],
|
||||
runtime_config: dict[str, object],
|
||||
seq: int,
|
||||
turn_seq: int,
|
||||
payload: object,
|
||||
final: bool,
|
||||
sampling_params: object,
|
||||
) -> DuplexAppendPlan:
|
||||
del sampling_params
|
||||
decode_pcm_f32le_payload(payload, sample_rate_hz=SAMPLE_RATE, exact_samples=FRAME_SIZE, model="PersonaPlex")
|
||||
normalized_payload = dict(payload) # type: ignore[call-overload]
|
||||
# One scheduler slot per frame; the first append of an epoch also
|
||||
# carries the voice/persona prefill (a new epoch is a new Stage 0
|
||||
# request with fresh KV, so the worker replays it).
|
||||
prompt_slots = 1 + (prefill_slot_count(runtime_config) if seq <= 1 else 0)
|
||||
return DuplexAppendPlan(
|
||||
prompt={
|
||||
"prompt_token_ids": [0] * prompt_slots,
|
||||
"model_intermediate_buffer": {
|
||||
"request_id": request_id,
|
||||
"global_request_id": [fence.session_id],
|
||||
"duplex": {
|
||||
"data_plane": True,
|
||||
"fence": fence,
|
||||
"session_id": fence.session_id,
|
||||
"epoch": fence.epoch,
|
||||
"turn_id": fence.turn_id,
|
||||
"seq": seq,
|
||||
"turn_seq": turn_seq,
|
||||
"mode": "append_audio_chunk",
|
||||
"payload": normalized_payload,
|
||||
"final": final,
|
||||
"session_config": dict(session_config),
|
||||
"runtime_config": dict(runtime_config),
|
||||
"scheduler_token_budget": prompt_slots,
|
||||
},
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
def decide_output(
|
||||
self,
|
||||
*,
|
||||
stage_id: int,
|
||||
final_stage_id: int,
|
||||
segment_finished: bool,
|
||||
segment_token_ids: tuple[int, ...],
|
||||
segment_output_metadata: dict[str, object],
|
||||
output: object,
|
||||
) -> DuplexOutputDecision | None:
|
||||
# Always-clocked model: what the client hears comes from the final
|
||||
# stage data plane, never from a Stage 0 listen/speak decision.
|
||||
del stage_id, final_stage_id, segment_finished, segment_token_ids, segment_output_metadata, output
|
||||
return None
|
||||
|
||||
# ---- session policy ----
|
||||
|
||||
def create_session_state(self) -> PersonaPlexSessionState:
|
||||
return PersonaPlexSessionState()
|
||||
|
||||
def capabilities(self, *, max_sessions: int) -> DuplexCapabilities:
|
||||
return personaplex_capabilities(max_sessions=max_sessions)
|
||||
|
||||
async def prepare_runtime_config(
|
||||
self, config: DuplexSessionConfig, *, model_config: ModelConfig | None
|
||||
) -> dict[str, object]:
|
||||
self.validate_client_extra_body(config.extra_body)
|
||||
model_path = getattr(model_config, "model", None)
|
||||
if not isinstance(model_path, str) or not model_path:
|
||||
raise DuplexRuntimeConfigError("PersonaPlex model path is unavailable", code="model_path_unavailable")
|
||||
voice = voice_name(config.voice)
|
||||
persona = persona_text(config.instructions)
|
||||
prefill_slots = await self._prefill_slots_for(model_path, voice, persona)
|
||||
return {
|
||||
"personaplex_model_path": model_path,
|
||||
"personaplex_voice_prompt": voice,
|
||||
"personaplex_persona": persona,
|
||||
"personaplex_prefill_slots": prefill_slots,
|
||||
}
|
||||
|
||||
async def _prefill_slots_for(self, model_path: str, voice: str, persona: str) -> int:
|
||||
key = (model_path, voice, persona)
|
||||
cached = self._prefill_slots.get(key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.stage0 import personaplex_prefill_slots
|
||||
|
||||
async with self._prefill_lock:
|
||||
cached = self._prefill_slots.get(key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
try:
|
||||
# Reads the voice bundle and the tokenizer: off the orchestrator loop.
|
||||
slots = await asyncio.to_thread(personaplex_prefill_slots, model_path, voice, persona)
|
||||
except Exception as exc:
|
||||
raise DuplexRuntimeConfigError(
|
||||
f"PersonaPlex voice/persona prefill could not be prepared: {exc}",
|
||||
code="prefill_unavailable",
|
||||
) from exc
|
||||
self._prefill_slots[key] = slots
|
||||
return slots
|
||||
|
||||
def runtime_config_for_update(
|
||||
self,
|
||||
config: DuplexSessionConfig,
|
||||
current: Mapping[str, object],
|
||||
) -> dict[str, object]:
|
||||
self.validate_client_extra_body(config.extra_body)
|
||||
new_persona = persona_text(config.instructions)
|
||||
reject_changed_runtime_value(
|
||||
new_persona,
|
||||
current.get("personaplex_persona"),
|
||||
message="PersonaPlex persona (instructions) cannot be changed after the session is created",
|
||||
code="persona_update_unsupported",
|
||||
)
|
||||
new_voice = voice_name(config.voice)
|
||||
reject_changed_runtime_value(
|
||||
new_voice,
|
||||
current.get("personaplex_voice_prompt"),
|
||||
message="PersonaPlex voice cannot be changed after the session is created",
|
||||
code="voice_update_unsupported",
|
||||
)
|
||||
runtime_config = deepcopy(dict(current))
|
||||
runtime_config["personaplex_voice_prompt"] = new_voice
|
||||
runtime_config["personaplex_persona"] = new_persona
|
||||
return runtime_config
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PRIVATE_RUNTIME_CONFIG_KEYS",
|
||||
"PersonaPlexDuplexPlugin",
|
||||
"PersonaPlexSessionState",
|
||||
"persona_text",
|
||||
"prefill_slot_count",
|
||||
"voice_name",
|
||||
]
|
||||
@@ -1,129 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
from typing import Any
|
||||
|
||||
from vllm.sampling_params import SamplingParams
|
||||
|
||||
from vllm_omni.engine.duplex.messages import DuplexFence
|
||||
from vllm_omni.engine.duplex.runtime import (
|
||||
DuplexAppendPlan,
|
||||
DuplexInputMode,
|
||||
DuplexOutputDecision,
|
||||
)
|
||||
|
||||
_FRAME_BYTES = 1920 * 4
|
||||
|
||||
|
||||
def _validated_frame_payload(payload: object) -> dict[str, object]:
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("PersonaPlex duplex append payload must be a mapping")
|
||||
if payload.get("format") != "pcm_f32le":
|
||||
raise ValueError("PersonaPlex duplex append format must be pcm_f32le")
|
||||
if payload.get("sample_rate_hz") != 24000:
|
||||
raise ValueError("PersonaPlex duplex append sample_rate_hz must be 24000")
|
||||
audio = payload.get("audio")
|
||||
if not isinstance(audio, str):
|
||||
raise ValueError("PersonaPlex duplex append audio must be base64 pcm_f32le")
|
||||
try:
|
||||
raw = base64.b64decode(audio, validate=True)
|
||||
except (binascii.Error, ValueError) as exc:
|
||||
raise ValueError("PersonaPlex duplex append audio is not valid base64") from exc
|
||||
if len(raw) != _FRAME_BYTES:
|
||||
raise ValueError("PersonaPlex duplex append must contain exactly 1920 samples")
|
||||
return dict(payload)
|
||||
|
||||
|
||||
class PersonaPlexDuplexRuntimeExtension:
|
||||
"""PersonaPlex policy for the engine-owned resumable duplex request."""
|
||||
|
||||
def configure_sampling_params(
|
||||
self,
|
||||
*,
|
||||
runtime_config: dict[str, Any],
|
||||
defaults: tuple[object, ...],
|
||||
) -> tuple[object, ...]:
|
||||
del runtime_config
|
||||
if not defaults:
|
||||
return defaults
|
||||
configured = list(defaults)
|
||||
stage0 = defaults[0]
|
||||
if isinstance(stage0, SamplingParams):
|
||||
stage0 = stage0.clone()
|
||||
stage0.temperature = 0.0
|
||||
stage0.top_k = 1
|
||||
stage0.max_tokens = 1
|
||||
configured[0] = stage0
|
||||
return tuple(configured)
|
||||
|
||||
def plan_append(
|
||||
self,
|
||||
*,
|
||||
request_id: str,
|
||||
fence: DuplexFence,
|
||||
session_config: dict[str, Any],
|
||||
runtime_config: dict[str, Any],
|
||||
seq: int,
|
||||
turn_seq: int,
|
||||
mode: DuplexInputMode,
|
||||
payload: object,
|
||||
final: bool,
|
||||
sampling_params: object,
|
||||
) -> DuplexAppendPlan:
|
||||
del sampling_params
|
||||
if mode is not DuplexInputMode.APPEND_AUDIO_CHUNK:
|
||||
raise ValueError(f"PersonaPlex does not support duplex input mode {mode.value!r}")
|
||||
normalized_payload = _validated_frame_payload(payload)
|
||||
prefill_slots = runtime_config.get("personaplex_prefill_slots", 0)
|
||||
try:
|
||||
prefill_slots = max(0, int(prefill_slots))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ValueError("personaplex_prefill_slots must be a non-negative integer") from exc
|
||||
prompt_slots = 1 + (prefill_slots if seq <= 1 else 0)
|
||||
return DuplexAppendPlan(
|
||||
prompt={
|
||||
"prompt_token_ids": [0] * prompt_slots,
|
||||
"model_intermediate_buffer": {
|
||||
"request_id": request_id,
|
||||
"global_request_id": [fence.session_id],
|
||||
"duplex": {
|
||||
"data_plane": True,
|
||||
"fence": fence,
|
||||
"session_id": fence.session_id,
|
||||
"incarnation": fence.incarnation,
|
||||
"epoch": fence.epoch,
|
||||
"turn_id": fence.turn_id,
|
||||
"response_seq": fence.response_seq,
|
||||
"seq": seq,
|
||||
"turn_seq": turn_seq,
|
||||
"mode": mode.value,
|
||||
"payload": normalized_payload,
|
||||
"final": final,
|
||||
"session_config": dict(session_config),
|
||||
"runtime_config": dict(runtime_config),
|
||||
"scheduler_token_budget": prompt_slots,
|
||||
},
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
def decide_output(
|
||||
self,
|
||||
*,
|
||||
stage_id: int,
|
||||
final_stage_id: int,
|
||||
segment_finished: bool,
|
||||
segment_token_ids: tuple[int, ...],
|
||||
segment_output_metadata: dict[str, Any],
|
||||
output: object,
|
||||
) -> DuplexOutputDecision | None:
|
||||
del stage_id, final_stage_id, segment_finished
|
||||
del segment_token_ids, segment_output_metadata, output
|
||||
return None
|
||||
|
||||
|
||||
__all__ = ["PersonaPlexDuplexRuntimeExtension"]
|
||||
@@ -1,253 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
from copy import deepcopy
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import PurePath
|
||||
from typing import Any
|
||||
|
||||
from vllm_omni.entrypoints.duplex.protocol import (
|
||||
DuplexCapabilities,
|
||||
)
|
||||
from vllm_omni.entrypoints.duplex.runtime_adapter import (
|
||||
ServingRuntimeConfigError,
|
||||
reject_changed_runtime_value,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import DEFAULT_PERSONA
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.data_plane import (
|
||||
PersonaPlexDataPlaneContext,
|
||||
PersonaPlexDataPlaneSession,
|
||||
)
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.input import (
|
||||
PersonaPlexPcmAppendBuffer,
|
||||
)
|
||||
|
||||
EncodeAudio = Callable[[object, int, str, float | None], str | None]
|
||||
|
||||
_PRIVATE_RUNTIME_CONFIG_KEYS = frozenset(
|
||||
{
|
||||
"personaplex_prefill_slots",
|
||||
"personaplex_model_path",
|
||||
# PersonaPlex is always model-native; the per-session opt-in knob
|
||||
# (canonical name and its deprecated alias) is server-owned here.
|
||||
"native_duplex",
|
||||
"minicpmo45_native_duplex",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class PersonaPlexServingSessionState:
|
||||
audio_buffer: PersonaPlexPcmAppendBuffer = field(default_factory=PersonaPlexPcmAppendBuffer)
|
||||
input_since_commit: bool = False
|
||||
speech_since_commit: bool = False
|
||||
native_context_locked: bool = False
|
||||
committed_audio_payload: dict[str, object] | None = None
|
||||
committed_audio_operation_id: str | None = None
|
||||
committed_audio_reserved_bytes: int = 0
|
||||
deferred_response_create: bool = False
|
||||
deferred_precreate_response: bool = False
|
||||
data_plane_task: asyncio.Task[None] | None = None
|
||||
data_plane_restart_requested: bool = False
|
||||
continuation_owner_id: str | None = None
|
||||
continuation_units: int = 0
|
||||
pending_silence_task: asyncio.Task[bool] | None = None
|
||||
pending_silence_owner_id: str | None = None
|
||||
silence_continuation_scheduler: Callable[..., Awaitable[bool]] | None = None
|
||||
|
||||
def retain_committed_audio(
|
||||
self,
|
||||
payload: dict[str, object],
|
||||
*,
|
||||
operation_id: str | None,
|
||||
reserved_bytes: int = 0,
|
||||
) -> None:
|
||||
self.committed_audio_payload = payload
|
||||
self.committed_audio_operation_id = operation_id
|
||||
self.committed_audio_reserved_bytes += max(0, int(reserved_bytes))
|
||||
|
||||
def clear_committed_audio(self) -> int:
|
||||
reserved_bytes = self.committed_audio_reserved_bytes
|
||||
self.committed_audio_payload = None
|
||||
self.committed_audio_operation_id = None
|
||||
self.committed_audio_reserved_bytes = 0
|
||||
self.deferred_response_create = False
|
||||
self.deferred_precreate_response = False
|
||||
return reserved_bytes
|
||||
|
||||
def clear_continuation(self) -> None:
|
||||
self.continuation_owner_id = None
|
||||
self.continuation_units = 0
|
||||
self.pending_silence_task = None
|
||||
self.pending_silence_owner_id = None
|
||||
|
||||
|
||||
class PersonaPlexServingRuntimeAdapter:
|
||||
adapter_id = "personaplex"
|
||||
clean_response_done_prefix = ""
|
||||
interrupted_tts_prefix = ""
|
||||
private_runtime_config_keys = _PRIVATE_RUNTIME_CONFIG_KEYS
|
||||
|
||||
def __init__(self, encode_audio: EncodeAudio) -> None:
|
||||
self.session_states: dict[str, PersonaPlexServingSessionState] = {}
|
||||
self.data_plane = PersonaPlexDataPlaneSession(encode_audio)
|
||||
|
||||
def create_session_state(self) -> PersonaPlexServingSessionState:
|
||||
return PersonaPlexServingSessionState()
|
||||
|
||||
def session_state(self, session_id: str) -> PersonaPlexServingSessionState:
|
||||
return self.session_states.setdefault(session_id, self.create_session_state())
|
||||
|
||||
def remove_session_state(self, session_id: str) -> None:
|
||||
self.session_states.pop(session_id, None)
|
||||
|
||||
@staticmethod
|
||||
def is_enabled(config: object) -> bool:
|
||||
del config
|
||||
return True
|
||||
|
||||
@staticmethod
|
||||
def capabilities(*, max_sessions: int) -> DuplexCapabilities:
|
||||
supports_multi_session = max_sessions > 1
|
||||
return DuplexCapabilities(
|
||||
supports_model_native_turn_policy=True,
|
||||
supports_external_turn_signal=False,
|
||||
supports_client_commit=False,
|
||||
supports_barge_in=False,
|
||||
supports_playback_ack=True,
|
||||
supports_input_append=True,
|
||||
supports_replace_latest_chunk=False,
|
||||
supports_reencode_context=False,
|
||||
supports_rollback_to_checkpoint=False,
|
||||
supports_turn_commit_only=False,
|
||||
supports_model_internal_state=True,
|
||||
supports_stage_resumption=True,
|
||||
supports_core_resumable_request=True,
|
||||
supports_stage_connector_handoff=True,
|
||||
supports_independent_io_streams=True,
|
||||
supports_realtime_endpoint=True,
|
||||
supports_multi_session=supports_multi_session,
|
||||
supports_multi_session_same_replica=supports_multi_session,
|
||||
supports_session_lease=True,
|
||||
supports_session_resume=False,
|
||||
session_admission_mode="engine_managed",
|
||||
supports_audio_truncate=False,
|
||||
requires_model_runner_kv=True,
|
||||
requires_native_stage_role=True,
|
||||
implementation_level="model_native_duplex",
|
||||
adapter_patterns=["scheduler_data_plane"],
|
||||
input_modes=["append_audio_chunk"],
|
||||
signal_sources=["model_native", "client_event"],
|
||||
stage_handoff_transport="scheduler_data_plane",
|
||||
chunk_period_ms=80,
|
||||
target_barge_in_latency_ms=None,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def validate_client_extra_body(extra_body: object) -> None:
|
||||
if not isinstance(extra_body, dict):
|
||||
return
|
||||
private = sorted(_PRIVATE_RUNTIME_CONFIG_KEYS.intersection(extra_body))
|
||||
if private:
|
||||
raise ServingRuntimeConfigError("PersonaPlex runtime configuration is server-owned: " + ", ".join(private))
|
||||
|
||||
@classmethod
|
||||
async def prepare_runtime_config(
|
||||
cls,
|
||||
config: object,
|
||||
*,
|
||||
model_config: Any,
|
||||
) -> dict[str, object]:
|
||||
extra_body = getattr(config, "extra_body", None)
|
||||
cls.validate_client_extra_body(extra_body)
|
||||
model_path = getattr(model_config, "model", None)
|
||||
if not isinstance(model_path, str) or not model_path:
|
||||
raise ServingRuntimeConfigError("PersonaPlex model path is unavailable")
|
||||
voice = cls._voice_name(getattr(config, "voice", None))
|
||||
instructions = getattr(config, "instructions", None) or DEFAULT_PERSONA
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.stage0 import (
|
||||
personaplex_prefill_slots,
|
||||
)
|
||||
|
||||
try:
|
||||
prefill_slots = await asyncio.to_thread(
|
||||
personaplex_prefill_slots,
|
||||
model_path,
|
||||
voice,
|
||||
str(instructions),
|
||||
)
|
||||
except Exception as exc:
|
||||
raise ServingRuntimeConfigError(f"PersonaPlex voice/persona prefill could not be prepared: {exc}") from exc
|
||||
return {
|
||||
"personaplex_model_path": model_path,
|
||||
"personaplex_voice_prompt": voice,
|
||||
"personaplex_persona": str(instructions),
|
||||
"personaplex_prefill_slots": prefill_slots,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def runtime_config_for_update(
|
||||
cls,
|
||||
config: object,
|
||||
current: Mapping[str, object],
|
||||
) -> dict[str, object]:
|
||||
cls.validate_client_extra_body(getattr(config, "extra_body", None))
|
||||
new_persona = str(getattr(config, "instructions", None) or DEFAULT_PERSONA)
|
||||
reject_changed_runtime_value(
|
||||
new_persona,
|
||||
current.get("personaplex_persona"),
|
||||
message="PersonaPlex persona (instructions) cannot be changed after the session is created",
|
||||
code="persona_update_unsupported",
|
||||
)
|
||||
new_voice = cls._voice_name(getattr(config, "voice", None))
|
||||
reject_changed_runtime_value(
|
||||
new_voice,
|
||||
current.get("personaplex_voice_prompt"),
|
||||
message="PersonaPlex voice cannot be changed after the session is created",
|
||||
code="voice_update_unsupported",
|
||||
)
|
||||
runtime_config = deepcopy(dict(current))
|
||||
runtime_config["personaplex_voice_prompt"] = new_voice
|
||||
runtime_config["personaplex_persona"] = new_persona
|
||||
return runtime_config
|
||||
|
||||
@staticmethod
|
||||
def data_plane_context(
|
||||
*,
|
||||
epoch: int,
|
||||
turn_id: int,
|
||||
active_response_turn_id: int | None,
|
||||
active_response_id: str | None,
|
||||
auto_responds: bool,
|
||||
response_format: str,
|
||||
speed: float | None,
|
||||
modalities: tuple[str, ...],
|
||||
) -> PersonaPlexDataPlaneContext:
|
||||
return PersonaPlexDataPlaneContext(
|
||||
epoch=epoch,
|
||||
turn_id=turn_id,
|
||||
active_response_turn_id=active_response_turn_id,
|
||||
active_response_id=active_response_id,
|
||||
auto_responds=auto_responds,
|
||||
response_format=response_format,
|
||||
speed=speed,
|
||||
modalities=modalities,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _voice_name(value: object) -> str:
|
||||
voice = value if isinstance(value, str) and value else "NATF2.pt"
|
||||
path = PurePath(voice)
|
||||
if path.name != voice or path.suffix != ".pt" or any(part == ".." for part in path.parts):
|
||||
raise ServingRuntimeConfigError("PersonaPlex voice must be a bundled .pt basename")
|
||||
return voice
|
||||
|
||||
|
||||
__all__ = [
|
||||
"PersonaPlexServingRuntimeAdapter",
|
||||
"PersonaPlexServingSessionState",
|
||||
]
|
||||
@@ -3,17 +3,19 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import binascii
|
||||
import io
|
||||
import tarfile
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass, field
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from vllm_omni.model_executor.common.audio.pcm import pcm_f32le_samples
|
||||
from vllm_omni.model_executor.common.duplex.payload import decode_pcm_f32le_payload
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.config import FRAME_SIZE, SAMPLE_RATE
|
||||
from vllm_omni.model_executor.models.personaplex.duplex.policy import (
|
||||
AUDIO_SILENCE_FRAME_CNT,
|
||||
SILENCE_TOKENS,
|
||||
@@ -22,7 +24,7 @@ from vllm_omni.model_executor.models.personaplex.duplex.policy import (
|
||||
wrap_with_system_tags,
|
||||
)
|
||||
|
||||
_FRAME_SAMPLES = 1920
|
||||
_FRAME_SAMPLES = FRAME_SIZE
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
@@ -37,8 +39,10 @@ class PersonaPlexStage0PreparedAppend:
|
||||
|
||||
@dataclass(slots=True)
|
||||
class PersonaPlexStage0SessionState:
|
||||
"""Lockstep state of one (session, epoch): a new epoch is a new Stage 0 request with fresh KV."""
|
||||
|
||||
session_id: str
|
||||
incarnation: int
|
||||
epoch: int
|
||||
user_codes: Any | None = None
|
||||
last_text_token: Any | None = None
|
||||
last_agent_codes: Any | None = None
|
||||
@@ -94,14 +98,29 @@ def load_personaplex_voice_state(model_path: str, voice: str) -> dict[str, Any]:
|
||||
raise FileNotFoundError(f"PersonaPlex bundled voice prompt {voice!r} was not found under {model_path!r}")
|
||||
|
||||
|
||||
def personaplex_prefill_slots(model_path: str, voice: str, persona: str) -> int:
|
||||
@lru_cache(maxsize=4)
|
||||
def _cached_tokenizer(model_path: str):
|
||||
return load_personaplex_tokenizer(model_path)
|
||||
|
||||
|
||||
@lru_cache(maxsize=16)
|
||||
def _cached_voice_embedding_rows(model_path: str, voice: str) -> int:
|
||||
state = load_personaplex_voice_state(model_path, voice)
|
||||
embeddings = state.get("embeddings")
|
||||
if not hasattr(embeddings, "shape") or len(embeddings.shape) < 1:
|
||||
raise ValueError(f"PersonaPlex voice prompt {voice!r} has no embeddings")
|
||||
tokenizer = load_personaplex_tokenizer(model_path)
|
||||
persona_tokens = tokenizer(wrap_with_system_tags(persona)) if persona else []
|
||||
return int(embeddings.shape[0]) + 2 * AUDIO_SILENCE_FRAME_CNT + len(persona_tokens)
|
||||
return int(embeddings.shape[0])
|
||||
|
||||
|
||||
def personaplex_prefill_slots(model_path: str, voice: str, persona: str) -> int:
|
||||
"""Scheduler slots the first append of a session needs for the voice + persona prefill.
|
||||
|
||||
The voice bundle row count and the tokenizer are cached per model path:
|
||||
they are constant, and this runs on every session open.
|
||||
"""
|
||||
voice_rows = _cached_voice_embedding_rows(model_path, voice)
|
||||
persona_tokens = _cached_tokenizer(model_path)(wrap_with_system_tags(persona)) if persona else []
|
||||
return voice_rows + 2 * AUDIO_SILENCE_FRAME_CNT + len(persona_tokens)
|
||||
|
||||
|
||||
class PersonaPlexStage0DuplexRuntime:
|
||||
@@ -147,19 +166,24 @@ class PersonaPlexStage0DuplexRuntime:
|
||||
session_id = duplex.get("session_id")
|
||||
if not isinstance(session_id, str) or not session_id:
|
||||
raise ValueError("PersonaPlex duplex append requires session_id")
|
||||
incarnation = _coerce_non_negative_int(duplex.get("incarnation"), "incarnation")
|
||||
epoch = _coerce_non_negative_int(duplex.get("epoch"), "epoch")
|
||||
seq = _coerce_positive_int(duplex.get("seq"), "seq")
|
||||
identity = (epoch, seq)
|
||||
key = (session_id, incarnation)
|
||||
key = (session_id, epoch)
|
||||
|
||||
state = self.sessions.get(key)
|
||||
if state is None:
|
||||
# A newer epoch supersedes the session's earlier lockstep state:
|
||||
# the engine aborted that request, but its finish notification
|
||||
# may still be in flight, so release it here rather than let the
|
||||
# two epochs share the codec budget.
|
||||
for stale_key in [k for k in self.sessions if k[0] == session_id and k[1] < epoch]:
|
||||
self.close_session(*stale_key)
|
||||
if len(self.sessions) >= self.max_sessions:
|
||||
raise RuntimeError(f"PersonaPlex Stage 0 session capacity {self.max_sessions} is exhausted")
|
||||
state = PersonaPlexStage0SessionState(
|
||||
session_id=session_id,
|
||||
incarnation=incarnation,
|
||||
epoch=epoch,
|
||||
codec=self._acquire_codec(),
|
||||
)
|
||||
self.sessions[key] = state
|
||||
@@ -291,7 +315,6 @@ class PersonaPlexStage0DuplexRuntime:
|
||||
"stage0_prepared": True,
|
||||
"prefill_applied": first_append,
|
||||
"session_id": session_id,
|
||||
"incarnation": incarnation,
|
||||
"epoch": epoch,
|
||||
"seq": seq,
|
||||
},
|
||||
@@ -363,8 +386,8 @@ class PersonaPlexStage0DuplexRuntime:
|
||||
if not state.request_ids:
|
||||
self.close_session(*key)
|
||||
|
||||
def close_session(self, session_id: str, incarnation: int) -> None:
|
||||
key = (session_id, incarnation)
|
||||
def close_session(self, session_id: str, epoch: int) -> None:
|
||||
key = (session_id, epoch)
|
||||
state = self.sessions.pop(key, None)
|
||||
if state is None:
|
||||
return
|
||||
@@ -421,23 +444,13 @@ class PersonaPlexStage0DuplexRuntime:
|
||||
|
||||
@staticmethod
|
||||
def _decode_pcm(payload: object) -> np.ndarray:
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("PersonaPlex duplex payload must be a mapping")
|
||||
if payload.get("format") != "pcm_f32le" or payload.get("sample_rate_hz") != 24000:
|
||||
raise ValueError("PersonaPlex Stage 0 requires 24 kHz pcm_f32le")
|
||||
audio = payload.get("audio")
|
||||
if not isinstance(audio, str):
|
||||
raise ValueError("PersonaPlex Stage 0 requires base64 audio")
|
||||
try:
|
||||
raw = base64.b64decode(audio, validate=True)
|
||||
except (binascii.Error, ValueError) as exc:
|
||||
raise ValueError("PersonaPlex Stage 0 audio is not valid base64") from exc
|
||||
samples = np.frombuffer(raw, dtype="<f4")
|
||||
if samples.size != _FRAME_SAMPLES:
|
||||
raise ValueError(f"PersonaPlex Stage 0 requires {_FRAME_SAMPLES} samples per append")
|
||||
if not np.isfinite(samples).all():
|
||||
raise ValueError("PersonaPlex Stage 0 samples must be finite")
|
||||
return np.ascontiguousarray(samples, dtype=np.float32).copy()
|
||||
raw = decode_pcm_f32le_payload(
|
||||
payload,
|
||||
sample_rate_hz=SAMPLE_RATE,
|
||||
exact_samples=_FRAME_SAMPLES,
|
||||
model="PersonaPlex Stage 0",
|
||||
)
|
||||
return pcm_f32le_samples(raw)
|
||||
|
||||
|
||||
def _coerce_non_negative_int(value: object, name: str) -> int:
|
||||
|
||||
@@ -2,9 +2,10 @@
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""PersonaPlex pipeline: Talker (AR decode -> Mimi codebooks) -> Code2Wav (codebooks -> 24 kHz PCM).
|
||||
|
||||
PersonaPlex is a Moshi finetune (full-duplex speech-to-speech). For offline/batch
|
||||
runs it is served as a 2-stage vllm-omni audio->audio pipeline, reusing the
|
||||
Qwen3-TTS staged topology:
|
||||
PersonaPlex is a Moshi finetune (full-duplex speech-to-speech), served as a
|
||||
2-stage vllm-omni audio->audio pipeline that reuses the Qwen3-TTS staged
|
||||
topology (online: one resumable Stage 0 request per duplex session, see
|
||||
``duplex/plugin.py``):
|
||||
|
||||
* Stage 0 (``personaplex``) is the AR talker: the Helium temporal transformer
|
||||
plus the depformer (both built by the lead). It emits the per-frame audio
|
||||
@@ -31,13 +32,8 @@ PERSONAPLEX_PIPELINE = PipelineConfig(
|
||||
# Pipeline-level default; the code2wav stage overrides per-stage below.
|
||||
model_arch="PersonaPlexTalkerForConditionalGeneration",
|
||||
default_deploy_config_name="personaplex.yaml",
|
||||
duplex_runtime_extension=(
|
||||
"vllm_omni.model_executor.models.personaplex.duplex.runtime_extension.PersonaPlexDuplexRuntimeExtension"
|
||||
),
|
||||
duplex_serving_adapter=(
|
||||
"vllm_omni.model_executor.models.personaplex.duplex.serving_adapter.PersonaPlexServingRuntimeAdapter"
|
||||
),
|
||||
duplex_control_enabled=True,
|
||||
# Served full duplex over /v1/realtime?duplex=1 by the unified framework.
|
||||
duplex_plugin="vllm_omni.model_executor.models.personaplex.duplex.plugin.PersonaPlexDuplexPlugin",
|
||||
stages=(
|
||||
StagePipelineConfig(
|
||||
stage_id=0,
|
||||
|
||||
Reference in new issue
Block a user