Signed-off-by: chickeyton <ngton2014@gmail.com> Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
49 files changed
+3056
-1715
No files matched your search
@@ -1,142 +1,137 @@
|
||||
# PersonaPlex Unified Full-Duplex Design
|
||||
# PersonaPlex on the Unified Full-Duplex Framework
|
||||
|
||||
## Status and target
|
||||
|
||||
This design adapts PersonaPlex PR #4771 to the unified full-duplex runtime on
|
||||
top of:
|
||||
|
||||
- vLLM-Omni `origin/main`: `67c54777bb22e9e7e08fdf7c47a64f06b566fc47`
|
||||
- PersonaPlex PR head: `477fb7c225f0c06991bc8aa55eadbd908ba282e4`
|
||||
|
||||
The target is the engine-native path:
|
||||
PersonaPlex (`nvidia/personaplex-7b-v1`, a Moshi finetune) is served by the
|
||||
[Unified Full-Duplex Framework](fullduplex.md) through one model plugin,
|
||||
`PersonaPlexDuplexPlugin` (`vllm_omni/model_executor/models/personaplex/duplex/plugin.py`),
|
||||
selected by `PipelineConfig.duplex_plugin`. The serving path is:
|
||||
|
||||
```text
|
||||
/v1/realtime?duplex=1
|
||||
-> OpenAI Realtime session actor
|
||||
-> DuplexRequestClient
|
||||
-> AsyncOmni correlated RPC
|
||||
-> DuplexControlPlane
|
||||
-> resumable Stage 0 request
|
||||
-> PersonaPlex Talker
|
||||
-> streaming PersonaPlex Code2Wav
|
||||
-> PersonaPlex data-plane projector
|
||||
-> OmniDuplexSessionHandler (thin: websocket I/O, Realtime <-> DuplexCommand/DuplexEvent)
|
||||
-> DuplexOmni / DuplexOmniEngine
|
||||
-> DuplexSessionManager -> DuplexSessionRunner (one per session)
|
||||
plugin.plan_append: one 80 ms frame -> one resumable Stage 0 append
|
||||
-> PersonaPlex Talker (Stage 0, lockstep temporal transformer + depformer)
|
||||
-> streaming PersonaPlex Code2Wav (Stage 1, Mimi)
|
||||
-> plugin.data_plane: cumulative audio/text -> deltas
|
||||
-> response.output_audio.delta + response.output_audio_transcript.delta
|
||||
```
|
||||
|
||||
The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
|
||||
original PR was demo-only and has been removed from the tree; the unified
|
||||
engine path above is the only serving surface. It was never evidence that the
|
||||
unified engine path works.
|
||||
|
||||
## Why configuration-only enablement was invalid
|
||||
|
||||
The staged pipeline as shipped by the original PR was explicitly turn based.
|
||||
Its Talker read `pplex_user_codes`, `pplex_prefill_text`, and
|
||||
`pplex_silence_codes`, but no production staged input path wrote those fields.
|
||||
The voice prompt, persona prefill, and streaming Mimi state lived only in the
|
||||
standalone `PersonaPlexEngine` (demo-only, since removed).
|
||||
|
||||
Setting only the following fields would therefore advertise an endpoint whose
|
||||
model never receives the live microphone stream:
|
||||
|
||||
```python
|
||||
duplex_control_enabled = True
|
||||
duplex_runtime_extension = "..."
|
||||
duplex_serving_adapter = "..."
|
||||
```
|
||||
|
||||
The adapter must supply a real scheduler data plane, not just endpoint
|
||||
capabilities.
|
||||
The pre-framework pair (engine runtime extension plus serving adapter) and
|
||||
the standalone `/api/chat` / `/v1/audio/duplex` server are gone; the plugin is
|
||||
the only integration surface, and this document describes it.
|
||||
|
||||
## Supported scope
|
||||
|
||||
The unified implementation supports:
|
||||
The integration supports:
|
||||
|
||||
- up to two engine-owned sessions on one replica;
|
||||
- 24 kHz mono float PCM input;
|
||||
- one 1920-sample, 80 ms model frame per physical append unit;
|
||||
- continuous user input while assistant audio is generated or played;
|
||||
- bundled `.pt` voice prompts and a session persona;
|
||||
- greedy text and depformer sampling, matching the current PersonaPlex port;
|
||||
- `/v1/realtime?duplex=1` (the wire vocabulary is catalogued in the
|
||||
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide);
|
||||
- up to `duplex_session.max_sessions` engine-owned sessions on one replica
|
||||
(the shipped deploy sets two);
|
||||
- 24 kHz mono float PCM input, one 1920-sample (80 ms) model frame per
|
||||
physical append unit; clients may send any chunking, the session buffers
|
||||
whole frames;
|
||||
- continuous user input while assistant audio is generated or played
|
||||
(pure lockstep: the model listens while it speaks);
|
||||
- bundled `.pt` voice prompts (`voice`) and a session persona (`instructions`);
|
||||
- greedy text and depformer sampling (one temporal token per frame);
|
||||
- the public client preset
|
||||
`vllm_omni.clients.personaplex.create_duplex_session_config()` (24 kHz
|
||||
`pcm_f32le` input format, voice prompt, persona) as the canonical
|
||||
session-config source for `DuplexClient` consumers;
|
||||
- engine lease close, disconnect cleanup, reconnect after cleanup, and explicit
|
||||
response cancellation without cross-session state reuse.
|
||||
`pcm_f32le` input, voice, persona) as the canonical session-config source
|
||||
for `DuplexClient` / `InlineDuplexClient` consumers;
|
||||
- engine lease close, disconnect cleanup and explicit response cancellation
|
||||
without cross-session state reuse.
|
||||
|
||||
The implementation does not claim:
|
||||
It does not claim:
|
||||
|
||||
- more than two simultaneous PersonaPlex sessions on one replica;
|
||||
- arbitrary WAV voice cloning;
|
||||
- turn-based `response.create` semantics for an otherwise continuous model;
|
||||
- scheduler migration of live codec state between replicas;
|
||||
- exact output equality with the standalone engine after different scheduling
|
||||
boundaries.
|
||||
- arbitrary WAV voice cloning (voices are bundled basenames, resolved by the
|
||||
worker under the checkpoint);
|
||||
- turn-based `response.create` semantics, client commits or external turn
|
||||
signals (`supports_client_commit=false`, `supports_external_turn_signal=false`);
|
||||
- text seeding, hence no `/v1/chat/completions` route
|
||||
(`supports_chat_completions=false`);
|
||||
- session resume across a transport drop (`supports_session_resume=false`);
|
||||
- destructive output interruption or model-state rewind at a playback cursor
|
||||
(`supports_barge_in=false`, `supports_audio_truncate=false`): overlapping
|
||||
speech is native model behaviour, not a barge-in contract.
|
||||
|
||||
The capability payload derives multi-session support from the configured
|
||||
session limit. The shipped two-session deployment reports
|
||||
`supports_multi_session=true` and `supports_multi_session_same_replica=true`.
|
||||
`duplex_session.max_sessions` is the only capacity source: config resolution
|
||||
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools read
|
||||
that model-config value. Connector extras do not carry a second model-specific
|
||||
capacity knob that could drift from engine admission.
|
||||
It still reports `supports_barge_in=false`: the generic epoch fence can suppress
|
||||
stale transport output, but neither PersonaPlex nor the current MiniCPM-o 4.5
|
||||
adapter proves that model-owned streaming state can be destructively rewound or
|
||||
restarted at a playback cursor. Continuous overlapping speech is model-native
|
||||
duplex behavior, not by itself a barge-in contract.
|
||||
propagates it to every stage as `duplex_max_sessions`, and both Mimi pools
|
||||
(Stage 0 encoders, Stage 1 decoders) read that model-config value. The
|
||||
capability payload derives `supports_multi_session` /
|
||||
`supports_multi_session_same_replica` from it.
|
||||
|
||||
## Components and ownership
|
||||
|
||||
### Serving adapter
|
||||
### The plugin (engine side)
|
||||
|
||||
`PersonaPlexServingRuntimeAdapter` owns only serving-side state:
|
||||
`PersonaPlexDuplexPlugin` owns both halves of the contract.
|
||||
|
||||
- a transactional PCM append buffer;
|
||||
- validation of `voice_prompt` and `instructions`;
|
||||
- public capabilities;
|
||||
- PersonaPlex data-plane output projection.
|
||||
Engine policy:
|
||||
|
||||
It accepts 24 kHz `pcm_f32le`, groups client packets into whole 1920-sample
|
||||
frames, zero-pads only the final residual, and rolls a reservation back when an
|
||||
engine append fails. It never loads CUDA weights and never encodes user audio.
|
||||
- `configure_sampling_params`: Stage 0 greedy (`temperature=0`, `top_k=1`,
|
||||
`max_tokens=1`), other stages untouched.
|
||||
- `plan_append`: validates exactly one 1920-sample 24 kHz `pcm_f32le` frame
|
||||
(through `model_executor/common/duplex/payload.py`) and reserves one
|
||||
scheduler slot per frame plus, on the first append of an epoch (`seq == 1`),
|
||||
the `personaplex_prefill_slots` of the voice/persona prefill. The prompt
|
||||
carries the fence, `seq`, the payload and the runtime config under
|
||||
`model_intermediate_buffer["duplex"]`.
|
||||
- `decide_output`: never decides. PersonaPlex is always-clocked; what the
|
||||
client hears comes from the final-stage data plane.
|
||||
- `silence_unit_payload`: one frame of zeros at 24 kHz
|
||||
(`silence_continuation_samples=1920`,
|
||||
`silence_continuation_sample_rate_hz=24000`). The runner uses it to keep a
|
||||
model turn clocked when the client pauses; the startup warmup sends it.
|
||||
|
||||
Client input cannot provide local filesystem paths. A voice is a bundled
|
||||
basename such as `NATF2.pt`; the worker resolves it under the local model
|
||||
checkpoint. `instructions` is the persona string.
|
||||
Session policy:
|
||||
|
||||
### Runtime extension
|
||||
- `capabilities`: `personaplex_capabilities(max_sessions)` (80 ms units,
|
||||
append-only, no commits, no barge-in, no resume, no chat route).
|
||||
- `prepare_runtime_config`: `voice` must be a bundled `.pt` basename,
|
||||
`instructions` defaults to the shipped persona; the prefill slot count is
|
||||
computed once per `(model, voice, persona)` off the orchestrator loop
|
||||
(`personaplex_prefill_slots`, itself caching the tokenizer and the voice
|
||||
bundle row count). The result is server-owned runtime config
|
||||
(`personaplex_model_path`, `personaplex_voice_prompt`,
|
||||
`personaplex_persona`, `personaplex_prefill_slots`); the same keys are
|
||||
refused in a client's `extra_body`.
|
||||
- `runtime_config_for_update`: persona and voice are immutable for the
|
||||
session (`persona_update_unsupported`, `voice_update_unsupported`).
|
||||
- session state, extra-body validation and data-plane context are the
|
||||
framework defaults (`DefaultDuplexModelSessionState` with the 80 ms
|
||||
`PersonaPlexPcmAppendBuffer`, `DuplexDataPlaneContext`).
|
||||
|
||||
`PersonaPlexDuplexRuntimeExtension` is pure model policy. It:
|
||||
Because `supports_client_commit` is off, the session auto-responds without
|
||||
`extra_body.auto_response`: a stock Realtime client streams audio and hears
|
||||
the model without any vendor flag.
|
||||
|
||||
- configures greedy Stage 0 sampling and bounded segment lengths;
|
||||
- maps each accepted PCM append to a scheduler prompt;
|
||||
- places immutable session identity, append sequence, PCM payload, voice, and
|
||||
persona under `model_intermediate_buffer["duplex"]`;
|
||||
- reserves exactly one scheduler prompt slot per encoded Mimi frame, plus the
|
||||
first-append voice/persona prefill length;
|
||||
- never performs model inference or owns session state.
|
||||
### Input framing
|
||||
|
||||
The extension returns no turn/listen decision. PersonaPlex is an always-clocked
|
||||
model, so visible audio/text comes from the final stage data plane.
|
||||
`PersonaPlexPcmAppendBuffer` is the `FixedFramePcmAppendBuffer` of
|
||||
`model_executor/common/duplex/pcm_buffer.py` at 24 kHz / 1920 samples /
|
||||
80 ms. It accepts 24 kHz `pcm_f32le` only, groups client packets into whole
|
||||
frames, takes one frame out per append as a reservation (committed when the
|
||||
stage accepted the append, rolled back to the front of the buffer when it did
|
||||
not), zero-pads only a final residual on commit, and never encodes audio.
|
||||
|
||||
### Stage 0 streaming runtime
|
||||
### Stage 0 streaming runtime (worker side)
|
||||
|
||||
The Talker owns a `PersonaPlexStage0DuplexRuntime` helper, analogous to
|
||||
MiniCPM-o's Stage 0 helper but with PersonaPlex lockstep semantics.
|
||||
The Talker owns a `PersonaPlexStage0DuplexRuntime`
|
||||
(`model_executor/models/personaplex/duplex/stage0.py`), created lazily in
|
||||
`_duplex_stage0_runtime()` and released through `on_requests_finished`.
|
||||
|
||||
For each admitted session it owns:
|
||||
For each live `(session_id, epoch)` it owns:
|
||||
|
||||
- streaming Mimi encoder convolution and transformer state;
|
||||
- the selected voice embedding bundle;
|
||||
- persona tokenization and prefill embeddings;
|
||||
- the prior user code frame needed by the one-frame acoustic delay;
|
||||
- append identity used to make a retried scheduler update idempotent.
|
||||
- the streaming Mimi encoder convolution and transformer state;
|
||||
- the selected voice embedding bundle and the persona prefill embeddings;
|
||||
- the prior user code frames needed by the one- and two-frame acoustic
|
||||
delays;
|
||||
- the append identity `(epoch, seq)` that makes a retried scheduler update
|
||||
idempotent.
|
||||
|
||||
The first append builds this ordered prefill:
|
||||
The first append of an epoch builds this ordered prefill:
|
||||
|
||||
```text
|
||||
voice embeddings
|
||||
@@ -146,45 +141,47 @@ voice embeddings
|
||||
-> first live user frame
|
||||
```
|
||||
|
||||
Later appends encode only new 1920-sample frames. The helper returns the
|
||||
Later appends encode only new 1920-sample frames. The runtime returns the
|
||||
per-frame user codes and prompt embeddings through the request's
|
||||
`model_intermediate_buffer`. The normal vLLM runner remains authoritative for
|
||||
attention metadata, block tables, KV allocation, scheduling, and sampling.
|
||||
`model_intermediate_buffer`; the normal vLLM runner remains authoritative for
|
||||
attention metadata, block tables, KV allocation, scheduling and sampling.
|
||||
|
||||
The Talker must distinguish resumable prompt prefill from decode positions.
|
||||
Prompt rows consume the exact prepared embeddings; sampled decode rows continue
|
||||
to use the existing delayed agent/user frame construction. No code path may
|
||||
fall back to an all-initial user stream for a duplex request.
|
||||
**Epochs.** The framework identifies a Stage 0 request by
|
||||
`(session_id, epoch)`; `response.cancel` and `output_audio_buffer.clear`
|
||||
advance the epoch, abort the current request and start the next append at
|
||||
`seq == 1` on a fresh request with fresh KV. The Stage 0 runtime therefore
|
||||
keys its state by `(session_id, epoch)`: when an append of a newer epoch
|
||||
arrives, any older-epoch state of the same session is closed first (its Mimi
|
||||
encoder returns to the pool, so the codec budget never counts a superseded
|
||||
epoch), and the voice/persona prefill is replayed because the plan reserved
|
||||
the slots again. The user-visible consequence is that a cancel restarts the
|
||||
model's conversation context. A late `on_requests_finished` for the aborted
|
||||
request finds nothing to close.
|
||||
|
||||
Cleanup is keyed by the full `(session_id, incarnation)` identity. Every live
|
||||
session has an independent Mimi encoder instance; encoder convolution/KV state
|
||||
is never shared between asynchronously scheduled sessions. A finished or
|
||||
Every live session has an independent Mimi encoder instance; encoder state is
|
||||
never shared between asynchronously scheduled sessions. A finished or
|
||||
aborted scheduler request resets and returns only that session's encoder.
|
||||
|
||||
### Stage 1 streaming decoder
|
||||
|
||||
The current `PersonaPlexCode2Wav` calls one-shot `MimiModel.decode` and is not
|
||||
CUDA-graph safe. Unified duplex uses eager Stage 1 and maintains an independent
|
||||
streaming Mimi decoder for every active request. Decoder ownership is keyed by
|
||||
the stable Stage 1 request id and released by `on_requests_finished`; a mixed
|
||||
batch must never advance another request's convolution or transformer state.
|
||||
|
||||
Each Stage 0 segment emits de-delayed agent codebooks. Stage 1 decodes only the
|
||||
new code frames, emits only the new PCM suffix, and resets state when the
|
||||
request is closed. Connector chunk boundaries retain the final raw code frame
|
||||
needed to de-delay the next chunk.
|
||||
|
||||
The deploy default sets Stage 1 `enforce_eager: true`; a default configuration
|
||||
that fails during CUDA graph capture is not an acceptable deployment profile.
|
||||
`PersonaPlexCode2Wav` runs eager (`enforce_eager: true` in the deploy) and
|
||||
maintains an independent streaming Mimi decoder for every active request,
|
||||
keyed by the Stage 1 request id and released by `on_requests_finished`. Each
|
||||
Stage 0 segment emits de-delayed agent codebooks; Stage 1 decodes only the new
|
||||
code frames and emits only the new PCM suffix. Connector chunk boundaries
|
||||
retain the final raw code frame needed to de-delay the next chunk.
|
||||
|
||||
### Data-plane projector
|
||||
|
||||
`PersonaPlexDataPlaneSession` converts cumulative or delta Stage 1 output into
|
||||
model-neutral native results:
|
||||
`PersonaPlexDataPlaneSession` is the `CumulativeAudioTextDataPlane` of
|
||||
`model_executor/common/duplex/data_plane.py` with a 24 kHz default rate. It
|
||||
keeps one audio/text cursor per request so cumulative Stage 1 output cannot
|
||||
replay old audio, and yields one internal result per new suffix:
|
||||
|
||||
```python
|
||||
{
|
||||
"stage_role": "tts",
|
||||
"is_listen": False,
|
||||
"data_plane_request_id": request_id,
|
||||
"text": text_delta,
|
||||
"audio_data": encoded_audio_delta,
|
||||
@@ -192,82 +189,74 @@ model-neutral native results:
|
||||
"sample_rate_hz": 24000,
|
||||
"audio_duration_ms": delta_duration,
|
||||
"end_of_turn": False,
|
||||
"runtime_impl": "scheduler_data_plane",
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
It owns per-request audio and text cursors so a cumulative output cannot replay
|
||||
old audio. The generic projector turns each native result into the Realtime
|
||||
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair (and
|
||||
`response.output_text.delta` for text) under one `response_id`; the full
|
||||
mapping is the name map in the
|
||||
[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
|
||||
PersonaPlex keeps one visible response open while continuous output arrives.
|
||||
Session close or cancellation terminates that response through the generic
|
||||
Realtime lifecycle; a codec segment finishing is not a conversational turn
|
||||
boundary. PersonaPlex advertises `supports_barge_in=false` and
|
||||
`supports_session_resume=false`, so `barge_in`, `turn_detection.server_vad`,
|
||||
and `session.resume` are rejected on this model.
|
||||
The runner turns each result into the Realtime
|
||||
`response.output_audio.delta` / `response.output_audio_transcript.delta` pair
|
||||
under one `response_id`. PersonaPlex keeps one visible response open while
|
||||
continuous output arrives; session close or cancellation terminates it through
|
||||
the generic Realtime lifecycle. A codec segment finishing is not a
|
||||
conversational turn boundary.
|
||||
|
||||
## Error and lifecycle contracts
|
||||
|
||||
- Unsupported sample rate, malformed base64, non-finite PCM, invalid voice
|
||||
basename, or changed format fails before scheduler submission.
|
||||
- Unsupported sample rate, malformed base64, non-finite PCM, a partial frame
|
||||
at the plan, an invalid voice basename or a changed format fails before
|
||||
scheduler submission, as a typed `error` event.
|
||||
- Append is prepare/submit/commit. Failure rolls back the exact reserved PCM
|
||||
bytes and does not advance the model frame cursor.
|
||||
- A repeated `operation_id` must not encode or submit the same frame twice.
|
||||
- Input iterator exceptions execute the same cleanup as explicit close.
|
||||
- Cancellation does not release the engine lease until the stage request and
|
||||
any in-flight codec operation are actually finished.
|
||||
- A bounded cleanup timeout returns a cleanup error and keeps the session in
|
||||
the closing admission set; it must not make the slot available while work
|
||||
still mutates shared state.
|
||||
- New sessions cannot observe the previous voice, persona, PCM tail, Mimi
|
||||
convolution state, or Talker delayed code frame.
|
||||
|
||||
(The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path,
|
||||
which followed the same exception-safe rule, have been removed from the tree.)
|
||||
- A repeated `(epoch, seq)` identity does not encode or submit the same frame
|
||||
twice.
|
||||
- Cancellation advances the epoch; the aborted request's Stage 0 state is
|
||||
released by the next-epoch append or by `on_requests_finished`, whichever
|
||||
comes first.
|
||||
- Close releases the admission slot only once stage cleanup succeeded; the
|
||||
reaper retries a failed cleanup while the slot stays held.
|
||||
- New sessions cannot observe a previous voice, persona, PCM tail, Mimi
|
||||
convolution state or Talker delayed code frame.
|
||||
|
||||
## Testing and acceptance
|
||||
|
||||
### Contract tests
|
||||
### CPU contract tests
|
||||
|
||||
Tests first cover:
|
||||
- `tests/model_executor/models/personaplex/duplex/test_plugin.py`: pipeline
|
||||
binding and plugin load, honest capabilities, private keys, voice/persona
|
||||
resolution and caching, immutable updates, greedy sampling, one-slot-plus-
|
||||
prefill planning, frame validation, the 24 kHz silence unit, and the
|
||||
CPU-checked helpers of the E2E driver.
|
||||
- `tests/model_executor/models/personaplex/duplex/test_stage0_runtime.py`:
|
||||
first-append prefill, depformer teacher forcing, causal user-frame delays,
|
||||
idempotent retries, independent encoders per session, capacity, epoch
|
||||
restart and late finish of a superseded request.
|
||||
- `tests/engine/duplex/test_session_runner_personaplex.py`: the session
|
||||
runner with the real plugin -- one submission per frame, prefill on the
|
||||
first append only, half frames buffered, wrong rate refused, cumulative
|
||||
Code2Wav output projected as 24 kHz deltas, cancel restarting the epoch,
|
||||
close aborting the request.
|
||||
- `tests/model_executor/executor_common/`: the shared toolbox (PCM helpers, payload
|
||||
validation, fixed-frame buffer, cumulative data plane, request-output
|
||||
readers).
|
||||
|
||||
- pipeline registration enables the control plane and selects both PersonaPlex
|
||||
adapters;
|
||||
- the capability payload is two-session, 80 ms, append-only, and honest;
|
||||
- PCM reservation commit/rollback, partial-frame flush, invalid input, and
|
||||
operation idempotency;
|
||||
- runtime prompt fields and exact token budgeting;
|
||||
- first-append voice/persona prefill followed by live user codes;
|
||||
- later appends retain per-session Mimi state and do not replay prefill;
|
||||
- interleaved Stage 0 and Stage 1 work preserves independent codec histories;
|
||||
- output projection emits only audio/text deltas;
|
||||
- close, exception, timeout, and reconnect cleanup;
|
||||
- ordinary non-duplex imports do not load PersonaPlex modules.
|
||||
### GPU validation
|
||||
|
||||
### Remote H20 validation
|
||||
`tests/e2e/online_serving/personaplex_realtime_duplex.py` (wrapped by
|
||||
`tests/e2e/online_serving/test_personaplex_duplex.py` when
|
||||
`PERSONAPLEX_MODEL_PATH` is set) drives the default `personaplex.yaml`
|
||||
deployment and requires:
|
||||
|
||||
Validation runs in an isolated remote worktree using the ModelScope
|
||||
`nv-community/personaplex-7b-v1` checkpoint and its Mimi dependency.
|
||||
|
||||
The required evidence is:
|
||||
|
||||
1. default `personaplex.yaml` reaches ready without a local eager override;
|
||||
2. `/health` returns 200;
|
||||
3. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`,
|
||||
and two-session admission;
|
||||
4. paced 24 kHz PCM appends produce finite, non-silent 24 kHz audio deltas and
|
||||
text deltas;
|
||||
5. microphone input continues during assistant output without cancelling the
|
||||
scheduler request;
|
||||
6. two paced sessions simultaneously produce independent non-empty audio, and
|
||||
a third session is rejected with `resource_exhausted`;
|
||||
7. closing either session frees only its scheduler request and GPU codec state,
|
||||
after which a replacement session can use a different persona without state
|
||||
leakage;
|
||||
8. malformed input returns one typed error and does not poison the next append;
|
||||
9. all focused unit tests and `git diff --check` pass.
|
||||
1. the server reaches ready without a local eager override; `/health` is 200;
|
||||
2. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`
|
||||
and two-session admission, with server-allocated session ids;
|
||||
3. paced 24 kHz PCM appends produce finite, non-silent, whole-frame 24 kHz
|
||||
audio deltas and text deltas;
|
||||
4. two paced sessions simultaneously produce independent non-empty audio, and
|
||||
a third session is refused with `resource_exhausted`;
|
||||
5. closing one session frees only its scheduler request and codec state, after
|
||||
which a replacement session with a different persona is admitted with a
|
||||
fresh id and no state leakage.
|
||||
|
||||
Audio that is empty, all zero, non-finite, or only a protocol `listen` event is
|
||||
not a successful end-to-end result.
|
||||
|
||||
@@ -107,6 +107,7 @@ th {
|
||||
| `VoxtralTTSForConditionalGeneration` | Voxtral TTS | `mistralai/Voxtral-4B-TTS-2603` | ✅︎ | ✅︎ | | | — |
|
||||
| `CovoAudioForConditionalGeneration` | Covo-Audio-Chat | `tencent/Covo-Audio-Chat` | ✅︎ | | | | — |
|
||||
| `MiniCPMO45OmniForConditionalGeneration` | MiniCPM-o 4.5 | `openbmb/MiniCPM-o-4_5` | ✅︎ | | ✅︎ | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/OpenBMB/MiniCPM-o-4_5.md) |
|
||||
| `PersonaPlexTalkerForConditionalGeneration` | PersonaPlex (full duplex, 24 kHz speech in/out over `/v1/realtime?duplex=1`) | `nvidia/personaplex-7b-v1` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/NVIDIA/PersonaPlex.md) |
|
||||
| `ErnieImagePipeline` | ERNIE-Image | `baidu/ERNIE-Image`, `baidu/ERNIE-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
| `GepardTalkerForConditionalGeneration` | Gepard-1.0 | `nineninesix/gepard-1.0` | ✅︎ | | | ✅︎ | — |
|
||||
| `HiDreamImagePipeline` | HiDream-I1-Full | `HiDream-ai/HiDream-I1-Full` | ✅︎ | ✅︎ | | | — |
|
||||
|
||||
@@ -384,9 +384,9 @@ with no OpenAI counterpart.
|
||||
|
||||
The event vocabulary is uniform, but several surfaces are gated by the
|
||||
`capabilities` object the server returns in `session.created`; a client must
|
||||
branch on those flags rather than on the model name. MiniCPM-o 4.5 is the
|
||||
only model on the plugin contract today; the other two columns record what
|
||||
their integrations advertise once the follow-up PRs port them:
|
||||
branch on those flags rather than on the model name. MiniCPM-o 4.5 and
|
||||
PersonaPlex are on the plugin contract today; the Nemotron VoiceChat column
|
||||
records what its integration advertises once the follow-up PR ports it:
|
||||
|
||||
| Capability | MiniCPM-o 4.5 | PersonaPlex | Nemotron VoiceChat | Gated surface |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -401,7 +401,11 @@ their integrations advertise once the follow-up PRs port them:
|
||||
Everything else in the catalogue — session lifecycle, heartbeat and event
|
||||
acknowledgement, append/commit/clear, the response envelope, playback
|
||||
acknowledgement, and the error envelope — behaves identically for every
|
||||
model.
|
||||
model. Two PersonaPlex specifics follow from its capabilities rather than from
|
||||
special-casing: a model with `supports_client_commit=false` auto-responds
|
||||
without `extra_body.auto_response`, and `response.cancel` /
|
||||
`output_audio_buffer.clear` restart its conversation context (a new Stage 0
|
||||
request replays the voice/persona prefill).
|
||||
|
||||
### Compatibility with the OpenAI Realtime protocol
|
||||
|
||||
|
||||
Reference in new issue
Block a user