[Core][Model] Fit PersonaPlex into the Unified Full-duplex Framework (RFC #7181 PR 2) (#7695)

Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
authored and GitHub committed 2026-09-27 03:02:24 +00:00
1 parent 7ab582ae25
commit 798d52ce55
49 files changed
+3056 -1715

No files matched your search

+179 -190
View File
@@ -1,142 +1,137 @@
# PersonaPlex Unified Full-Duplex Design
# PersonaPlex on the Unified Full-Duplex Framework
## Status and target
This design adapts PersonaPlex PR #4771 to the unified full-duplex runtime on
top of:
- vLLM-Omni `origin/main`: `67c54777bb22e9e7e08fdf7c47a64f06b566fc47`
- PersonaPlex PR head: `477fb7c225f0c06991bc8aa55eadbd908ba282e4`
The target is the engine-native path:
PersonaPlex (`nvidia/personaplex-7b-v1`, a Moshi finetune) is served by the
[Unified Full-Duplex Framework](fullduplex.md) through one model plugin,
`PersonaPlexDuplexPlugin` (`vllm_omni/model_executor/models/personaplex/duplex/plugin.py`),
selected by `PipelineConfig.duplex_plugin`. The serving path is:
```text
/v1/realtime?duplex=1
-> OpenAI Realtime session actor
-> DuplexRequestClient
-> AsyncOmni correlated RPC
-> DuplexControlPlane
-> resumable Stage 0 request
-> PersonaPlex Talker
-> streaming PersonaPlex Code2Wav
-> PersonaPlex data-plane projector
-> OmniDuplexSessionHandler (thin: websocket I/O, Realtime <-> DuplexCommand/DuplexEvent)
-> DuplexOmni / DuplexOmniEngine
-> DuplexSessionManager -> DuplexSessionRunner (one per session)
plugin.plan_append: one 80 ms frame -> one resumable Stage 0 append
-> PersonaPlex Talker (Stage 0, lockstep temporal transformer + depformer)
-> streaming PersonaPlex Code2Wav (Stage 1, Mimi)
-> plugin.data_plane: cumulative audio/text -> deltas
-> response.output_audio.delta + response.output_audio_transcript.delta
```
The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
original PR was demo-only and has been removed from the tree; the unified
engine path above is the only serving surface. It was never evidence that the
unified engine path works.
## Why configuration-only enablement was invalid
The staged pipeline as shipped by the original PR was explicitly turn based.
Its Talker read `pplex_user_codes`, `pplex_prefill_text`, and
`pplex_silence_codes`, but no production staged input path wrote those fields.
The voice prompt, persona prefill, and streaming Mimi state lived only in the
standalone `PersonaPlexEngine` (demo-only, since removed).
Setting only the following fields would therefore advertise an endpoint whose
model never receives the live microphone stream:
```python
duplex_control_enabled = True
duplex_runtime_extension = "..."
duplex_serving_adapter = "..."
```
The adapter must supply a real scheduler data plane, not just endpoint
capabilities.
The pre-framework pair (engine runtime extension plus serving adapter) and
the standalone `/api/chat` / `/v1/audio/duplex` server are gone; the plugin is
the only integration surface, and this document describes it.
## Supported scope
The unified implementation supports:
The integration supports:
- up to two engine-owned sessions on one replica;
- 24 kHz mono float PCM input;
- one 1920-sample, 80 ms model frame per physical append unit;
- continuous user input while assistant audio is generated or played;
- bundled `.pt` voice prompts and a session persona;
- greedy text and depformer sampling, matching the current PersonaPlex port;
- `/v1/realtime?duplex=1` (the wire vocabulary is catalogued in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide);
- up to `duplex_session.max_sessions` engine-owned sessions on one replica
(the shipped deploy sets two);
- 24 kHz mono float PCM input, one 1920-sample (80 ms) model frame per
physical append unit; clients may send any chunking, the session buffers
whole frames;
- continuous user input while assistant audio is generated or played
(pure lockstep: the model listens while it speaks);
- bundled `.pt` voice prompts (`voice`) and a session persona (`instructions`);
- greedy text and depformer sampling (one temporal token per frame);
- the public client preset
`vllm_omni.clients.personaplex.create_duplex_session_config()` (24 kHz
`pcm_f32le` input format, voice prompt, persona) as the canonical
session-config source for `DuplexClient` consumers;
- engine lease close, disconnect cleanup, reconnect after cleanup, and explicit
response cancellation without cross-session state reuse.
`pcm_f32le` input, voice, persona) as the canonical session-config source
for `DuplexClient` / `InlineDuplexClient` consumers;
- engine lease close, disconnect cleanup and explicit response cancellation
without cross-session state reuse.
The implementation does not claim:
It does not claim:
- more than two simultaneous PersonaPlex sessions on one replica;
- arbitrary WAV voice cloning;
- turn-based `response.create` semantics for an otherwise continuous model;
- scheduler migration of live codec state between replicas;
- exact output equality with the standalone engine after different scheduling
boundaries.
- arbitrary WAV voice cloning (voices are bundled basenames, resolved by the
worker under the checkpoint);
- turn-based `response.create` semantics, client commits or external turn
signals (`supports_client_commit=false`, `supports_external_turn_signal=false`);
- text seeding, hence no `/v1/chat/completions` route
(`supports_chat_completions=false`);
- session resume across a transport drop (`supports_session_resume=false`);
- destructive output interruption or model-state rewind at a playback cursor
(`supports_barge_in=false`, `supports_audio_truncate=false`): overlapping
speech is native model behaviour, not a barge-in contract.
The capability payload derives multi-session support from the configured
session limit. The shipped two-session deployment reports
`supports_multi_session=true` and `supports_multi_session_same_replica=true`.
`duplex_session.max_sessions` is the only capacity source: config resolution
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools read
that model-config value. Connector extras do not carry a second model-specific
capacity knob that could drift from engine admission.
It still reports `supports_barge_in=false`: the generic epoch fence can suppress
stale transport output, but neither PersonaPlex nor the current MiniCPM-o 4.5
adapter proves that model-owned streaming state can be destructively rewound or
restarted at a playback cursor. Continuous overlapping speech is model-native
duplex behavior, not by itself a barge-in contract.
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools
(Stage 0 encoders, Stage 1 decoders) read that model-config value. The
capability payload derives `supports_multi_session` /
`supports_multi_session_same_replica` from it.
## Components and ownership
### Serving adapter
### The plugin (engine side)
`PersonaPlexServingRuntimeAdapter` owns only serving-side state:
`PersonaPlexDuplexPlugin` owns both halves of the contract.
- a transactional PCM append buffer;
- validation of `voice_prompt` and `instructions`;
- public capabilities;
- PersonaPlex data-plane output projection.
Engine policy:
It accepts 24 kHz `pcm_f32le`, groups client packets into whole 1920-sample
frames, zero-pads only the final residual, and rolls a reservation back when an
engine append fails. It never loads CUDA weights and never encodes user audio.
- `configure_sampling_params`: Stage 0 greedy (`temperature=0`, `top_k=1`,
`max_tokens=1`), other stages untouched.
- `plan_append`: validates exactly one 1920-sample 24 kHz `pcm_f32le` frame
(through `model_executor/common/duplex/payload.py`) and reserves one
scheduler slot per frame plus, on the first append of an epoch (`seq == 1`),
the `personaplex_prefill_slots` of the voice/persona prefill. The prompt
carries the fence, `seq`, the payload and the runtime config under
`model_intermediate_buffer["duplex"]`.
- `decide_output`: never decides. PersonaPlex is always-clocked; what the
client hears comes from the final-stage data plane.
- `silence_unit_payload`: one frame of zeros at 24 kHz
(`silence_continuation_samples=1920`,
`silence_continuation_sample_rate_hz=24000`). The runner uses it to keep a
model turn clocked when the client pauses; the startup warmup sends it.
Client input cannot provide local filesystem paths. A voice is a bundled
basename such as `NATF2.pt`; the worker resolves it under the local model
checkpoint. `instructions` is the persona string.
Session policy:
### Runtime extension
- `capabilities`: `personaplex_capabilities(max_sessions)` (80 ms units,
append-only, no commits, no barge-in, no resume, no chat route).
- `prepare_runtime_config`: `voice` must be a bundled `.pt` basename,
`instructions` defaults to the shipped persona; the prefill slot count is
computed once per `(model, voice, persona)` off the orchestrator loop
(`personaplex_prefill_slots`, itself caching the tokenizer and the voice
bundle row count). The result is server-owned runtime config
(`personaplex_model_path`, `personaplex_voice_prompt`,
`personaplex_persona`, `personaplex_prefill_slots`); the same keys are
refused in a client's `extra_body`.
- `runtime_config_for_update`: persona and voice are immutable for the
session (`persona_update_unsupported`, `voice_update_unsupported`).
- session state, extra-body validation and data-plane context are the
framework defaults (`DefaultDuplexModelSessionState` with the 80 ms
`PersonaPlexPcmAppendBuffer`, `DuplexDataPlaneContext`).
`PersonaPlexDuplexRuntimeExtension` is pure model policy. It:
Because `supports_client_commit` is off, the session auto-responds without
`extra_body.auto_response`: a stock Realtime client streams audio and hears
the model without any vendor flag.
- configures greedy Stage 0 sampling and bounded segment lengths;
- maps each accepted PCM append to a scheduler prompt;
- places immutable session identity, append sequence, PCM payload, voice, and
persona under `model_intermediate_buffer["duplex"]`;
- reserves exactly one scheduler prompt slot per encoded Mimi frame, plus the
first-append voice/persona prefill length;
- never performs model inference or owns session state.
### Input framing
The extension returns no turn/listen decision. PersonaPlex is an always-clocked
model, so visible audio/text comes from the final stage data plane.
`PersonaPlexPcmAppendBuffer` is the `FixedFramePcmAppendBuffer` of
`model_executor/common/duplex/pcm_buffer.py` at 24 kHz / 1920 samples /
80 ms. It accepts 24 kHz `pcm_f32le` only, groups client packets into whole
frames, takes one frame out per append as a reservation (committed when the
stage accepted the append, rolled back to the front of the buffer when it did
not), zero-pads only a final residual on commit, and never encodes audio.
### Stage 0 streaming runtime
### Stage 0 streaming runtime (worker side)
The Talker owns a `PersonaPlexStage0DuplexRuntime` helper, analogous to
MiniCPM-o's Stage 0 helper but with PersonaPlex lockstep semantics.
The Talker owns a `PersonaPlexStage0DuplexRuntime`
(`model_executor/models/personaplex/duplex/stage0.py`), created lazily in
`_duplex_stage0_runtime()` and released through `on_requests_finished`.
For each admitted session it owns:
For each live `(session_id, epoch)` it owns:
- streaming Mimi encoder convolution and transformer state;
- the selected voice embedding bundle;
- persona tokenization and prefill embeddings;
- the prior user code frame needed by the one-frame acoustic delay;
- append identity used to make a retried scheduler update idempotent.
- the streaming Mimi encoder convolution and transformer state;
- the selected voice embedding bundle and the persona prefill embeddings;
- the prior user code frames needed by the one- and two-frame acoustic
delays;
- the append identity `(epoch, seq)` that makes a retried scheduler update
idempotent.
The first append builds this ordered prefill:
The first append of an epoch builds this ordered prefill:
```text
voice embeddings
@@ -146,45 +141,47 @@ voice embeddings
-> first live user frame
```
Later appends encode only new 1920-sample frames. The helper returns the
Later appends encode only new 1920-sample frames. The runtime returns the
per-frame user codes and prompt embeddings through the request's
`model_intermediate_buffer`. The normal vLLM runner remains authoritative for
attention metadata, block tables, KV allocation, scheduling, and sampling.
`model_intermediate_buffer`; the normal vLLM runner remains authoritative for
attention metadata, block tables, KV allocation, scheduling and sampling.
The Talker must distinguish resumable prompt prefill from decode positions.
Prompt rows consume the exact prepared embeddings; sampled decode rows continue
to use the existing delayed agent/user frame construction. No code path may
fall back to an all-initial user stream for a duplex request.
**Epochs.** The framework identifies a Stage 0 request by
`(session_id, epoch)`; `response.cancel` and `output_audio_buffer.clear`
advance the epoch, abort the current request and start the next append at
`seq == 1` on a fresh request with fresh KV. The Stage 0 runtime therefore
keys its state by `(session_id, epoch)`: when an append of a newer epoch
arrives, any older-epoch state of the same session is closed first (its Mimi
encoder returns to the pool, so the codec budget never counts a superseded
epoch), and the voice/persona prefill is replayed because the plan reserved
the slots again. The user-visible consequence is that a cancel restarts the
model's conversation context. A late `on_requests_finished` for the aborted
request finds nothing to close.
Cleanup is keyed by the full `(session_id, incarnation)` identity. Every live
session has an independent Mimi encoder instance; encoder convolution/KV state
is never shared between asynchronously scheduled sessions. A finished or
Every live session has an independent Mimi encoder instance; encoder state is
never shared between asynchronously scheduled sessions. A finished or
aborted scheduler request resets and returns only that session's encoder.
### Stage 1 streaming decoder
The current `PersonaPlexCode2Wav` calls one-shot `MimiModel.decode` and is not
CUDA-graph safe. Unified duplex uses eager Stage 1 and maintains an independent
streaming Mimi decoder for every active request. Decoder ownership is keyed by
the stable Stage 1 request id and released by `on_requests_finished`; a mixed
batch must never advance another request's convolution or transformer state.
Each Stage 0 segment emits de-delayed agent codebooks. Stage 1 decodes only the
new code frames, emits only the new PCM suffix, and resets state when the
request is closed. Connector chunk boundaries retain the final raw code frame
needed to de-delay the next chunk.
The deploy default sets Stage 1 `enforce_eager: true`; a default configuration
that fails during CUDA graph capture is not an acceptable deployment profile.
`PersonaPlexCode2Wav` runs eager (`enforce_eager: true` in the deploy) and
maintains an independent streaming Mimi decoder for every active request,
keyed by the Stage 1 request id and released by `on_requests_finished`. Each
Stage 0 segment emits de-delayed agent codebooks; Stage 1 decodes only the new
code frames and emits only the new PCM suffix. Connector chunk boundaries
retain the final raw code frame needed to de-delay the next chunk.
### Data-plane projector
`PersonaPlexDataPlaneSession` converts cumulative or delta Stage 1 output into
model-neutral native results:
`PersonaPlexDataPlaneSession` is the `CumulativeAudioTextDataPlane` of
`model_executor/common/duplex/data_plane.py` with a 24 kHz default rate. It
keeps one audio/text cursor per request so cumulative Stage 1 output cannot
replay old audio, and yields one internal result per new suffix:
```python
{
"stage_role": "tts",
"is_listen": False,
"data_plane_request_id": request_id,
"text": text_delta,
"audio_data": encoded_audio_delta,
@@ -192,82 +189,74 @@ model-neutral native results:
"sample_rate_hz": 24000,
"audio_duration_ms": delta_duration,
"end_of_turn": False,
"runtime_impl": "scheduler_data_plane",
...
}
```
It owns per-request audio and text cursors so a cumulative output cannot replay
old audio. The generic projector turns each native result into the Realtime
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair (and
`response.output_text.delta` for text) under one `response_id`; the full
mapping is the name map in the
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
PersonaPlex keeps one visible response open while continuous output arrives.
Session close or cancellation terminates that response through the generic
Realtime lifecycle; a codec segment finishing is not a conversational turn
boundary. PersonaPlex advertises `supports_barge_in=false` and
`supports_session_resume=false`, so `barge_in`, `turn_detection.server_vad`,
and `session.resume` are rejected on this model.
The runner turns each result into the Realtime
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair
under one `response_id`. PersonaPlex keeps one visible response open while
continuous output arrives; session close or cancellation terminates it through
the generic Realtime lifecycle. A codec segment finishing is not a
conversational turn boundary.
## Error and lifecycle contracts
- Unsupported sample rate, malformed base64, non-finite PCM, invalid voice
basename, or changed format fails before scheduler submission.
- Unsupported sample rate, malformed base64, non-finite PCM, a partial frame
at the plan, an invalid voice basename or a changed format fails before
scheduler submission, as a typed `error` event.
- Append is prepare/submit/commit. Failure rolls back the exact reserved PCM
bytes and does not advance the model frame cursor.
- A repeated `operation_id` must not encode or submit the same frame twice.
- Input iterator exceptions execute the same cleanup as explicit close.
- Cancellation does not release the engine lease until the stage request and
any in-flight codec operation are actually finished.
- A bounded cleanup timeout returns a cleanup error and keeps the session in
the closing admission set; it must not make the slot available while work
still mutates shared state.
- New sessions cannot observe the previous voice, persona, PCM tail, Mimi
convolution state, or Talker delayed code frame.
(The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path,
which followed the same exception-safe rule, have been removed from the tree.)
- A repeated `(epoch, seq)` identity does not encode or submit the same frame
twice.
- Cancellation advances the epoch; the aborted request's Stage 0 state is
released by the next-epoch append or by `on_requests_finished`, whichever
comes first.
- Close releases the admission slot only once stage cleanup succeeded; the
reaper retries a failed cleanup while the slot stays held.
- New sessions cannot observe a previous voice, persona, PCM tail, Mimi
convolution state or Talker delayed code frame.
## Testing and acceptance
### Contract tests
### CPU contract tests
Tests first cover:
- `tests/model_executor/models/personaplex/duplex/test_plugin.py`: pipeline
binding and plugin load, honest capabilities, private keys, voice/persona
resolution and caching, immutable updates, greedy sampling, one-slot-plus-
prefill planning, frame validation, the 24 kHz silence unit, and the
CPU-checked helpers of the E2E driver.
- `tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py`:
first-append prefill, depformer teacher forcing, causal user-frame delays,
idempotent retries, independent encoders per session, capacity, epoch
restart and late finish of a superseded request.
- `tests/engine/duplex/test_session_runner_personaplex.py`: the session
runner with the real plugin -- one submission per frame, prefill on the
first append only, half frames buffered, wrong rate refused, cumulative
Code2Wav output projected as 24 kHz deltas, cancel restarting the epoch,
close aborting the request.
- `tests/model_executor/executor_common/`: the shared toolbox (PCM helpers, payload
validation, fixed-frame buffer, cumulative data plane, request-output
readers).
- pipeline registration enables the control plane and selects both PersonaPlex
adapters;
- the capability payload is two-session, 80 ms, append-only, and honest;
- PCM reservation commit/rollback, partial-frame flush, invalid input, and
operation idempotency;
- runtime prompt fields and exact token budgeting;
- first-append voice/persona prefill followed by live user codes;
- later appends retain per-session Mimi state and do not replay prefill;
- interleaved Stage 0 and Stage 1 work preserves independent codec histories;
- output projection emits only audio/text deltas;
- close, exception, timeout, and reconnect cleanup;
- ordinary non-duplex imports do not load PersonaPlex modules.
### GPU validation
### Remote H20 validation
`tests/e2e/online_serving/personaplex_realtime_duplex.py` (wrapped by
`tests/e2e/online_serving/test_personaplex_duplex.py` when
`PERSONAPLEX_MODEL_PATH` is set) drives the default `personaplex.yaml`
deployment and requires:
Validation runs in an isolated remote worktree using the ModelScope
`nv-community/personaplex-7b-v1` checkpoint and its Mimi dependency.
The required evidence is:
1. default `personaplex.yaml` reaches ready without a local eager override;
2. `/health` returns 200;
3. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`,
and two-session admission;
4. paced 24 kHz PCM appends produce finite, non-silent 24 kHz audio deltas and
text deltas;
5. microphone input continues during assistant output without cancelling the
scheduler request;
6. two paced sessions simultaneously produce independent non-empty audio, and
a third session is rejected with `resource_exhausted`;
7. closing either session frees only its scheduler request and GPU codec state,
after which a replacement session can use a different persona without state
leakage;
8. malformed input returns one typed error and does not poison the next append;
9. all focused unit tests and `git diff --check` pass.
1. the server reaches ready without a local eager override; `/health` is 200;
2. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`
and two-session admission, with server-allocated session ids;
3. paced 24 kHz PCM appends produce finite, non-silent, whole-frame 24 kHz
audio deltas and text deltas;
4. two paced sessions simultaneously produce independent non-empty audio, and
a third session is refused with `resource_exhausted`;
5. closing one session frees only its scheduler request and codec state, after
which a replacement session with a different persona is admitted with a
fresh id and no state leakage.
Audio that is empty, all zero, non-finite, or only a protocol `listen` event is
not a successful end-to-end result.
+1
View File
@@ -107,6 +107,7 @@ th {
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
| `PersonaPlexTalkerForConditionalGeneration` | PersonaPlex (full duplex, 24 kHz speech in/out over `/v1/realtime?duplex=1`) | `nvidia/personaplex-7b-v1` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/PersonaPlex.md) |
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
+8 -4
View File
@@ -384,9 +384,9 @@ with no OpenAI counterpart.
The event vocabulary is uniform, but several surfaces are gated by the
`capabilities` object the server returns in `session.created`; a client must
branch on those flags rather than on the model name. MiniCPM-o 4.5 is the
only model on the plugin contract today; the other two columns record what
their integrations advertise once the follow-up PRs port them:
branch on those flags rather than on the model name. MiniCPM-o 4.5 and
PersonaPlex are on the plugin contract today; the Nemotron VoiceChat column
records what its integration advertises once the follow-up PR ports it:
| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
| --- | --- | --- | --- | --- |
@@ -401,7 +401,11 @@ their integrations advertise once the follow-up PRs port them:
Everything else in the catalogue — session lifecycle, heartbeat and event
acknowledgement, append/commit/clear, the response envelope, playback
acknowledgement, and the error envelope — behaves identically for every
model.
model. Two PersonaPlex specifics follow from its capabilities rather than from
special-casing: a model with `supports_client_commit=false` auto-responds
without `extra_body.auto_response`, and `response.cancel` /
`output_audio_buffer.clear` restart its conversation context (a new Stage 0
request replays the voice/persona prefill).
### Compatibility with the OpenAI Realtime protocol