[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196)

Signed-off-by: chickeyton <ngton2014@gmail.com>
Signed-off-by: Gao Han <hgaoaf@connect.ust.hk>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>

docs/.nav.yml

@@ -17,6 +17,7 @@ nav:
       - Streaming and Realtime:
         - Realtime Audio: serving/realtime_api.md
         - Full Duplex: serving/full_duplex_api.md
+        - Realtime Duplex Reference: serving/realtime_duplex_api.md
         - Streaming Video Input: serving/video_stream_api.md
         - Streaming Video Output: serving/streaming_video_output_api.md
         - OpenPI Robot Policy: serving/openpi_api.md

docs/README.md

@@ -23,7 +23,6 @@ Easy, fast, and cheap omni-modality model serving for everyone
 <a class="github-button" href="https://github.com/vllm-project/vllm-omni/fork" data-show-count="true" data-icon="octicon-repo-forked" data-size="large" aria-label="Fork">Fork</a>
 </p>
 
-
 ## About
 
 [vLLM](https://github.com/vllm-project/vllm) was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
@@ -52,7 +51,7 @@ vLLM-Omni is flexible and easy to use with:
 - Tensor, pipeline, data and expert parallelism support for distributed inference
 - Streaming outputs
 - OpenAI-compatible API server
-- Full-duplex realtime serving with streaming audio input and output (experimental)
+- [Full-duplex realtime serving](serving/realtime_duplex_api.md) with streaming audio input and output
 
 vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
 

docs/api/README.md

@@ -15,6 +15,36 @@ Main entry points for vLLM-Omni inference and serving.
 - [vllm_omni.entrypoints.omni_base.OmniBase][]
 - [vllm_omni.entrypoints.pd_utils.PDDisaggregationMixin][]
 
+## Clients
+
+Client libraries that talk to a running vLLM-Omni server (the full-duplex Realtime client and its per-model session presets).
+
+- [vllm_omni.clients.duplex.AudioDelta][]
+- [vllm_omni.clients.duplex.AudioFormat][]
+- [vllm_omni.clients.duplex.ConnectionResumed][]
+- [vllm_omni.clients.duplex.DuplexClient][]
+- [vllm_omni.clients.duplex.DuplexClientError][]
+- [vllm_omni.clients.duplex.DuplexConnectionError][]
+- [vllm_omni.clients.duplex.DuplexEvent][]
+- [vllm_omni.clients.duplex.DuplexProtocolError][]
+- [vllm_omni.clients.duplex.DuplexSessionClosedError][]
+- [vllm_omni.clients.duplex.ErrorEvent][]
+- [vllm_omni.clients.duplex.EventCollector][]
+- [vllm_omni.clients.duplex.ListenDecision][]
+- [vllm_omni.clients.duplex.ReconnectPolicy][]
+- [vllm_omni.clients.duplex.ResponseCreated][]
+- [vllm_omni.clients.duplex.ResponseDone][]
+- [vllm_omni.clients.duplex.ResponseHandle][]
+- [vllm_omni.clients.duplex.SessionClosed][]
+- [vllm_omni.clients.duplex.SessionConfig][]
+- [vllm_omni.clients.duplex.SessionCreated][]
+- [vllm_omni.clients.duplex.SessionExpired][]
+- [vllm_omni.clients.duplex.SessionResumed][]
+- [vllm_omni.clients.duplex.SpeakDecision][]
+- [vllm_omni.clients.duplex.TextDelta][]
+- [vllm_omni.clients.duplex.TranscriptDelta][]
+- [vllm_omni.clients.duplex.WebSocketTransport][]
+
 ## Inputs
 
 Input data structures for multi-modal inputs.

docs/cli/bench/serve.md

@@ -1,11 +1,15 @@
 # vLLM-Omni Benchmark CLI Guide
+
 The vllm bench command launches the vLLM-Omni benchmark to evaluate the performance of multimodal models.
 
 ## Notes
+
 vLLM-Omni registers the `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, `daily-omni`, and `openai-realtime-duplex` serving benchmark backends. It also adds the `omniinteract` dataset.
 
 ## Basic Parameter Description
+
 You can use `vllm bench serve --omni --help=all` to get descriptions of all parameters. The commonly used parameters are described below:
+
 - `--omni`
   Enable Omni (multimodal) mode, supporting multimodal inputs and outputs such as images, videos, and audio.
 
@@ -77,8 +81,8 @@ Specify to save benchmark results to a json file
 - `--random-prefix-len`
   Number of fixed prefix tokens before the random context in a request.
   The total input length is the sum of random-prefix-len and a random
-  context length sampled from [input_len * (1 - range_ratio),
-  input_len * (1 + range_ratio)].Only the random and random-mm modes
+  context length sampled from [input_len *(1 - range_ratio),
+  input_len* (1 + range_ratio)].Only the random and random-mm modes
   support this parameter.
 
 - `--random-input-len`
@@ -91,7 +95,7 @@ Specify to save benchmark results to a json file
   Range ratio for sampling input/output length,
   used only for random sampling. Must be in the range [0, 1) to define
   a symmetric sampling range
-  [length * (1 - range_ratio), length * (1 + range_ratio)].
+  [length *(1 - range_ratio), length* (1 + range_ratio)].
   Only the random and random-mm modes support this parameter.
 
 - `--random-mm-base-items-per-request`
@@ -137,6 +141,7 @@ Specify to save benchmark results to a json file
 ## Usage Examples
 
 ### Online Benchmark
+
 <details class="admonition abstract" markdown="1">
 <summary>Show more</summary>
 
@@ -162,7 +167,9 @@ vllm bench serve \
   --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
   --percentile-metrics ttft,tpot,itl,e2el
 ```
+
 If successful, you will see the following output:
+
 ```text
 ============ Serving Benchmark Result ============
 Successful requests:                     2
@@ -264,6 +271,7 @@ Median AUDIO_RTF:                        0.47
 P99 AUDIO_RTF:                           0.48
 ==================================================
 ```
+
 Notes:
 We use audio generation time / audio duration to calculate RTF.
 
@@ -301,8 +309,12 @@ options apply. Use `--omniinteract-require-response` only for functional E2E cas
 Each completed case writes `output.wav`, `wav_transcript.json`, `events.json`, `result.json`, and a final `.done` marker under
 `--omniinteract-output-dir`. The root also contains `batch_summary.json` and `official_eval_manifest.jsonl`; failed cases write
 `.failed.json`. Runs sharing one output root are serialized. Completion validates transport, response lifecycle, and artifacts,
-not answer accuracy. Transcript timestamps are serialized playback-queue times. Clipped or cancelled outputs are ineligible and
-omitted from the official manifest; `audio_clipped_bytes` records output beyond the rounded video horizon.
+not answer accuracy. Transcript timestamps are serialized playback-queue times. Playback ACKs report cumulative progress
+incrementally along that serialized clock, like a live listener; the first ack for a response goes out as soon as its audio
+arrives, checkpointing the response's history position so a later committed user input updates it in place. A residual
+`playback_ack_too_late` rejection is recorded as an artifact warning rather than failing the case. Clipped or cancelled
+outputs are ineligible and omitted from the official manifest; `audio_clipped_bytes` records output beyond the rounded video
+horizon.
 
 TTFT, TTFP, and RTF start at client receipt of `response.created`. TPOT/ITL use engine stage-0 timing; ITL is emitted only when
 every token interval is present.
@@ -339,8 +351,8 @@ Generate synthetic image、video、audio inputs alongside random text prompts to
 Notes:
 
 - Works only with online benchmark via:
-  - the OpenAI chat backend (`--backend openai-chat-omni`) and endpoint `/v1/chat/completions`.
-  - the OpenAI edit image backend (`--backend openai-image-edits-omni`) and endpoint `/v1/images/edits`.
+    - the OpenAI chat backend (`--backend openai-chat-omni`) and endpoint `/v1/chat/completions`.
+    - the OpenAI edit image backend (`--backend openai-image-edits-omni`) and endpoint `/v1/images/edits`.
 
 Start the server (example):
 
@@ -351,6 +363,7 @@ vllm serve Qwen/Qwen2.5-Omni-7B --omni
 It is recommended to use the flag `--ignore-eos` to simulate real responses. You can set the size of the output via the arg `random-output-len`.
 
 Then run the benchmarking script:
+
 ```bash
 vllm bench serve \
     --omni \
@@ -420,6 +433,7 @@ How sampling works:
 - If a modality (e.g., image) reaches its limit from `--random-mm-limit-mm-per-prompt`, all buckets of that modality are excluded and the remaining bucket probabilities are renormalized before continuing.
 This should be seen as an edge case, and if this behavior can be avoided by setting `--random-mm-limit-mm-per-prompt` to a large number. Note that this might result in errors due to engine config `--limit-mm-per-prompt`.
 - The resulting request contains synthetic image data in `multi_modal_data` (OpenAI Chat format). When `random-mm` is used with the OpenAI Chat backend, prompts remain text and MM content is attached via `multi_modal_data`.
+
 </details>
 
 ### Multi-Stage Benchmark
@@ -434,6 +448,7 @@ vllm serve --omni --port 29999 --model ~/Qwen3-Omni-30B-A3B-Instruct
 ```
 
 Then run the benchmarking script (remember to add the flag `--print-stage`):
+
 ```bash
 vllm bench serve --omni \
   --dataset-name random \
@@ -453,6 +468,7 @@ vllm bench serve --omni \
 ```
 
 Besides the "Serving Benchmark Result", there will be "Stage Benchmark Result" below:
+
 ```text
 ============= Stage Benchmark Result =============
 =============== Stage 0 (thinker) ================
@@ -509,7 +525,9 @@ Mean AUDIO_RTF:                          0.24
 Median AUDIO_RTF:                        0.26
 P99 AUDIO_RTF:                           0.27
 ```
+
 Explanation:
+
 - stage_gen_time: Time from submitting a request to a specific stage to that stage finishing generation.
 
 - Serving TTFC (Time to First Chunk): Time from the HTTP request being accepted by the serving frontend to

docs/design/architecture_overview.md

@@ -418,7 +418,7 @@ The public interfaces map onto the same engine and stage boundaries:
 flowchart LR
     offline["Offline Python<br/>Omni.generate()"] --> engine["AsyncOmniEngine"]
     online["OpenAI-compatible serving<br/>vllm serve ... --omni"] --> engine
-    duplex["Experimental duplex WebSocket<br/>/v1/duplex or realtime duplex"] --> engine
+    duplex["Duplex WebSocket<br/>/v1/realtime?duplex=1 or /v1/duplex"] --> engine
     engine --> stages["Configured AR and diffusion stages"]
     stages --> result["Streaming or final multimodal output"]
 ```

docs/design/fullduplex-personaplex.md

@@ -0,0 +1,273 @@
+# PersonaPlex Unified Full-Duplex Design
+
+## Status and target
+
+This design adapts PersonaPlex PR #4771 to the unified full-duplex runtime on
+top of:
+
+- vLLM-Omni `origin/main`: `67c54777bb22e9e7e08fdf7c47a64f06b566fc47`
+- PersonaPlex PR head: `477fb7c225f0c06991bc8aa55eadbd908ba282e4`
+
+The target is the engine-native path:
+
+```text
+/v1/realtime?duplex=1
+  -> OpenAI Realtime session actor
+  -> DuplexRequestClient
+  -> AsyncOmni correlated RPC
+  -> DuplexControlPlane
+  -> resumable Stage 0 request
+  -> PersonaPlex Talker
+  -> streaming PersonaPlex Code2Wav
+  -> PersonaPlex data-plane projector
+  -> response.audio.delta + response.audio_transcript.delta
+```
+
+The standalone `/api/chat` and `/v1/audio/duplex` server that accompanied the
+original PR was demo-only and has been removed from the tree; the unified
+engine path above is the only serving surface. It was never evidence that the
+unified engine path works.
+
+## Why configuration-only enablement was invalid
+
+The staged pipeline as shipped by the original PR was explicitly turn based.
+Its Talker read `pplex_user_codes`, `pplex_prefill_text`, and
+`pplex_silence_codes`, but no production staged input path wrote those fields.
+The voice prompt, persona prefill, and streaming Mimi state lived only in the
+standalone `PersonaPlexEngine` (demo-only, since removed).
+
+Setting only the following fields would therefore advertise an endpoint whose
+model never receives the live microphone stream:
+
+```python
+duplex_control_enabled = True
+duplex_runtime_extension = "..."
+duplex_serving_adapter = "..."
+```
+
+The adapter must supply a real scheduler data plane, not just endpoint
+capabilities.
+
+## Supported scope
+
+The unified implementation supports:
+
+- up to two engine-owned sessions on one replica;
+- 24 kHz mono float PCM input;
+- one 1920-sample, 80 ms model frame per physical append unit;
+- continuous user input while assistant audio is generated or played;
+- bundled `.pt` voice prompts and a session persona;
+- greedy text and depformer sampling, matching the current PersonaPlex port;
+- `/v1/realtime?duplex=1` (the wire vocabulary is catalogued in the
+  [Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide);
+- the public client preset
+  `vllm_omni.clients.personaplex.create_duplex_session_config()` (24 kHz
+  `pcm_f32le` input format, voice prompt, persona) as the canonical
+  session-config source for `DuplexClient` consumers;
+- engine lease close, disconnect cleanup, reconnect after cleanup, and explicit
+  response cancellation without cross-session state reuse.
+
+The implementation does not claim:
+
+- more than two simultaneous PersonaPlex sessions on one replica;
+- arbitrary WAV voice cloning;
+- turn-based `response.create` semantics for an otherwise continuous model;
+- scheduler migration of live codec state between replicas;
+- exact output equality with the standalone engine after different scheduling
+  boundaries.
+
+The capability payload derives multi-session support from the configured
+session limit. The shipped two-session deployment reports
+`supports_multi_session=true` and `supports_multi_session_same_replica=true`.
+`duplex_session.max_sessions` is the only capacity source: config resolution
+propagates it to every stage as `duplex_max_sessions`, and both Mimi pools read
+that model-config value. Connector extras do not carry a second model-specific
+capacity knob that could drift from engine admission.
+It still reports `supports_barge_in=false`: the generic epoch fence can suppress
+stale transport output, but neither PersonaPlex nor the current MiniCPM-o 4.5
+adapter proves that model-owned streaming state can be destructively rewound or
+restarted at a playback cursor. Continuous overlapping speech is model-native
+duplex behavior, not by itself a barge-in contract.
+
+## Components and ownership
+
+### Serving adapter
+
+`PersonaPlexServingRuntimeAdapter` owns only serving-side state:
+
+- a transactional PCM append buffer;
+- validation of `voice_prompt` and `instructions`;
+- public capabilities;
+- PersonaPlex data-plane output projection.
+
+It accepts 24 kHz `pcm_f32le`, groups client packets into whole 1920-sample
+frames, zero-pads only the final residual, and rolls a reservation back when an
+engine append fails. It never loads CUDA weights and never encodes user audio.
+
+Client input cannot provide local filesystem paths. A voice is a bundled
+basename such as `NATF2.pt`; the worker resolves it under the local model
+checkpoint. `instructions` is the persona string.
+
+### Runtime extension
+
+`PersonaPlexDuplexRuntimeExtension` is pure model policy. It:
+
+- configures greedy Stage 0 sampling and bounded segment lengths;
+- maps each accepted PCM append to a scheduler prompt;
+- places immutable session identity, append sequence, PCM payload, voice, and
+  persona under `model_intermediate_buffer["duplex"]`;
+- reserves exactly one scheduler prompt slot per encoded Mimi frame, plus the
+  first-append voice/persona prefill length;
+- never performs model inference or owns session state.
+
+The extension returns no turn/listen decision. PersonaPlex is an always-clocked
+model, so visible audio/text comes from the final stage data plane.
+
+### Stage 0 streaming runtime
+
+The Talker owns a `PersonaPlexStage0DuplexRuntime` helper, analogous to
+MiniCPM-o's Stage 0 helper but with PersonaPlex lockstep semantics.
+
+For each admitted session it owns:
+
+- streaming Mimi encoder convolution and transformer state;
+- the selected voice embedding bundle;
+- persona tokenization and prefill embeddings;
+- the prior user code frame needed by the one-frame acoustic delay;
+- append identity used to make a retried scheduler update idempotent.
+
+The first append builds this ordered prefill:
+
+```text
+voice embeddings
+  -> encoded silence
+  -> <system> persona <system> tokens with encoded silence
+  -> encoded silence
+  -> first live user frame
+```
+
+Later appends encode only new 1920-sample frames. The helper returns the
+per-frame user codes and prompt embeddings through the request's
+`model_intermediate_buffer`. The normal vLLM runner remains authoritative for
+attention metadata, block tables, KV allocation, scheduling, and sampling.
+
+The Talker must distinguish resumable prompt prefill from decode positions.
+Prompt rows consume the exact prepared embeddings; sampled decode rows continue
+to use the existing delayed agent/user frame construction. No code path may
+fall back to an all-initial user stream for a duplex request.
+
+Cleanup is keyed by the full `(session_id, incarnation)` identity. Every live
+session has an independent Mimi encoder instance; encoder convolution/KV state
+is never shared between asynchronously scheduled sessions. A finished or
+aborted scheduler request resets and returns only that session's encoder.
+
+### Stage 1 streaming decoder
+
+The current `PersonaPlexCode2Wav` calls one-shot `MimiModel.decode` and is not
+CUDA-graph safe. Unified duplex uses eager Stage 1 and maintains an independent
+streaming Mimi decoder for every active request. Decoder ownership is keyed by
+the stable Stage 1 request id and released by `on_requests_finished`; a mixed
+batch must never advance another request's convolution or transformer state.
+
+Each Stage 0 segment emits de-delayed agent codebooks. Stage 1 decodes only the
+new code frames, emits only the new PCM suffix, and resets state when the
+request is closed. Connector chunk boundaries retain the final raw code frame
+needed to de-delay the next chunk.
+
+The deploy default sets Stage 1 `enforce_eager: true`; a default configuration
+that fails during CUDA graph capture is not an acceptable deployment profile.
+
+### Data-plane projector
+
+`PersonaPlexDataPlaneSession` converts cumulative or delta Stage 1 output into
+model-neutral native results:
+
+```python
+{
+    "stage_role": "tts",
+    "data_plane_request_id": request_id,
+    "text": text_delta,
+    "audio_data": encoded_audio_delta,
+    "audio_format": response_format,
+    "sample_rate_hz": 24000,
+    "audio_duration_ms": delta_duration,
+    "end_of_turn": False,
+}
+```
+
+It owns per-request audio and text cursors so a cumulative output cannot replay
+old audio. The generic projector turns each native result into the Realtime
+`response.audio.delta` / `response.audio_transcript.delta` pair (and
+`response.output_text.delta` for text) under one `response_id`; the full
+mapping is the name map in the
+[Realtime Duplex API](../serving/realtime_duplex_api.md) serving guide.
+PersonaPlex keeps one visible response open while continuous output arrives.
+Session close or cancellation terminates that response through the generic
+Realtime lifecycle; a codec segment finishing is not a conversational turn
+boundary. PersonaPlex advertises `supports_barge_in=false` and
+`supports_session_resume=false`, so `barge_in`, `turn_detection.server_vad`,
+and `session.resume` are rejected on this model.
+
+## Error and lifecycle contracts
+
+- Unsupported sample rate, malformed base64, non-finite PCM, invalid voice
+  basename, or changed format fails before scheduler submission.
+- Append is prepare/submit/commit. Failure rolls back the exact reserved PCM
+  bytes and does not advance the model frame cursor.
+- A repeated `operation_id` must not encode or submit the same frame twice.
+- Input iterator exceptions execute the same cleanup as explicit close.
+- Cancellation does not release the engine lease until the stage request and
+  any in-flight codec operation are actually finished.
+- A bounded cleanup timeout returns a cleanup error and keeps the session in
+  the closing admission set; it must not make the slot available while work
+  still mutates shared state.
+- New sessions cannot observe the previous voice, persona, PCM tail, Mimi
+  convolution state, or Talker delayed code frame.
+
+(The legacy `PersonaPlexDuplexRuntime.run()` and standalone server drain path,
+which followed the same exception-safe rule, have been removed from the tree.)
+
+## Testing and acceptance
+
+### Contract tests
+
+Tests first cover:
+
+- pipeline registration enables the control plane and selects both PersonaPlex
+  adapters;
+- the capability payload is two-session, 80 ms, append-only, and honest;
+- PCM reservation commit/rollback, partial-frame flush, invalid input, and
+  operation idempotency;
+- runtime prompt fields and exact token budgeting;
+- first-append voice/persona prefill followed by live user codes;
+- later appends retain per-session Mimi state and do not replay prefill;
+- interleaved Stage 0 and Stage 1 work preserves independent codec histories;
+- output projection emits only audio/text deltas;
+- close, exception, timeout, and reconnect cleanup;
+- ordinary non-duplex imports do not load PersonaPlex modules.
+
+### Remote H20 validation
+
+Validation runs in an isolated remote worktree using the ModelScope
+`nv-community/personaplex-7b-v1` checkpoint and its Mimi dependency.
+
+The required evidence is:
+
+1. default `personaplex.yaml` reaches ready without a local eager override;
+2. `/health` returns 200;
+3. `/v1/realtime?duplex=1` reports `model_native_duplex`, `chunk_period_ms=80`,
+   and two-session admission;
+4. paced 24 kHz PCM appends produce finite, non-silent 24 kHz audio deltas and
+   text deltas;
+5. microphone input continues during assistant output without cancelling the
+   scheduler request;
+6. two paced sessions simultaneously produce independent non-empty audio, and
+   a third session is rejected with `resource_exhausted`;
+7. closing either session frees only its scheduler request and GPU codec state,
+   after which a replacement session can use a different persona without state
+   leakage;
+8. malformed input returns one typed error and does not poison the next append;
+9. all focused unit tests and `git diff --check` pass.
+
+Audio that is empty, all zero, non-finite, or only a protocol `listen` event is
+not a successful end-to-end result.

docs/design/index.md

@@ -17,6 +17,8 @@ implementation contract; it is not, by itself, a general support claim.
 
 ### Runtime and stage execution
 
+- [Full-Duplex Runtime (MiniCPM-o 4.5)](fullduplex.md)
+- [Full-Duplex Runtime (PersonaPlex)](fullduplex-personaplex.md)
 - [Disaggregated Inference](feature/disaggregated_inference.md)
 - [Host Weight Runtime](feature/host_weight_runtime.md)
 - [Async Chunk](feature/async_chunk.md)

docs/design/module/engine_orchestration.md

@@ -28,6 +28,9 @@ related_code_paths:
   - vllm_omni/engine/stage_pool.py
   - vllm_omni/outputs/output_processor.py
   - vllm_omni/distributed/omni_coordinator/**
+  # Duplex kernel design is owned by ../fullduplex.md; listed here because the
+  # orchestrator hosts the duplex control plane.
+  - vllm_omni/engine/duplex/**
 depends_on:
   - input_output_modality_contracts.md
   - error_contracts.md
@@ -44,7 +47,8 @@ validation_paths:
   - tests/engine/test_orchestrator_stage_input_bridge.py
   - tests/engine/test_cfg_companion_lifecycle.py
   - tests/engine/test_rpc_result_router.py
-  - tests/e2e/features/fullduplex/engine/**
+  - tests/engine/duplex/**
+  - tests/engine/test_duplex_import_boundary.py
 upstream_refs:
   - vllm.v1.engine.EngineCoreRequest
   - vllm.v1.engine.EngineCoreOutputs

docs/design/module/entrypoints.md

@@ -26,9 +26,18 @@ primary_code_paths:
   - vllm_omni/entrypoints/client_request_state.py
   - vllm_omni/entrypoints/stage_utils.py
   - vllm_omni/entrypoints/utils.py
+  - vllm_omni/entrypoints/duplex/**
+  - vllm_omni/entrypoints/duplex_request_client.py
+  - vllm_omni/clients/**
 primary_path_exceptions:
   - path: vllm_omni/entrypoints/openai/errors.py
     owner: error_contracts.md
+  - path: vllm_omni/entrypoints/duplex/**
+    owner: ../fullduplex.md
+  - path: vllm_omni/entrypoints/duplex_request_client.py
+    owner: ../fullduplex.md
+  - path: vllm_omni/clients/**
+    owner: ../fullduplex.md
 related_code_paths:
   - vllm_omni/errors.py
   - vllm_omni/inputs/**

docs/mkdocs/hooks/generate_api_readme.py

@@ -22,6 +22,13 @@ CATEGORIES = {
         "name": "Entry Points",
         "description": "Main entry points for vLLM-Omni inference and serving.",
     },
+    "clients": {
+        "name": "Clients",
+        "description": (
+            "Client libraries that talk to a running vLLM-Omni server "
+            "(the full-duplex Realtime client and its per-model session presets)."
+        ),
+    },
     "inputs": {
         "name": "Inputs",
         "description": "Input data structures for multi-modal inputs.",

docs/serving/README.md

@@ -43,7 +43,7 @@ Completions for conversational or heterogeneous omni pipelines, rather than as
 the default wrapper for every generation task.
 
 | Task | Endpoint | Request | Response | Details |
-|------|----------|---------|----------|---------|
+| ------ | ---------- | --------- | ---------- | --------- |
 | Conversation or multimodal understanding/generation | `POST /v1/chat/completions` | JSON | OpenAI-style JSON or SSE | [Chat Completions](chat_completions_api.md) |
 | Text-to-speech | `POST /v1/audio/speech` | JSON | Audio bytes or SSE | [Speech](speech_api.md) |
 | Sound, music, or ambient audio generation | `POST /v1/audio/generate` | JSON | Audio bytes | [Audio Generation](audio_generate_api.md) |
@@ -141,7 +141,7 @@ These WebSocket APIs have different event schemas and cannot be used
 interchangeably.
 
 | Workload | Endpoint | Interaction model | Details |
-|----------|----------|-------------------|---------|
+| ---------- | ---------- | ------------------- | --------- |
 | Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
 | Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
 | Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
@@ -158,7 +158,7 @@ ordinary realtime handler when duplex is unavailable.
 ## Related Endpoints
 
 | Purpose | Endpoints | Reference |
-|---------|-----------|-----------|
+| --------- | ----------- | ----------- |
 | Discovery and readiness | `GET /health`, `GET /v1/models` | This page |
 | Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) |
 | Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) |
@@ -168,11 +168,10 @@ ordinary realtime handler when duplex is unavailable.
 
 ## Standalone Experimental Servers
 
-PersonaPlex also provides a Moshi-compatible WebSocket server, while JoyVL
-provides a stateful interaction orchestrator in front of another model server.
-These are separate processes with their own routes; they are not additional
+JoyVL provides a stateful interaction orchestrator in front of another model
+server. It is a separate process with its own routes; they are not additional
 paths on every `vllm serve --omni` instance. See [Standalone Experimental
-Servers](standalone_servers.md) before deploying either one.
+Servers](standalone_servers.md) before deploying it.
 
 ## Compatibility Notes
 

docs/serving/full_duplex_api.md

@@ -1,16 +1,20 @@
 # Full-Duplex WebSocket API
 
-vLLM-Omni provides an experimental full-duplex runtime for models that can
+vLLM-Omni provides a full-duplex runtime for models that can
 continue receiving speech while producing speech. It adds persistent session
 state, model-specific turn policy, overlap handling, playback acknowledgement,
 and optional session resume.
 
 Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md).
 
+This page is the endpoint overview. For the complete wire contract, the
+`vllm_omni.clients.duplex.DuplexClient` Python library, and the per-model
+capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
+
 ## Choose an Endpoint
 
 | Endpoint | Protocol | Recommended use |
-|----------|----------|-----------------|
+| --- | --- | --- |
 | `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
 | `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
 
@@ -41,8 +45,10 @@ and control plane as part of their registered pipeline.
 
 The current unified-runtime integrations are:
 
-- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`;
-- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex.
+- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5.yaml`
+  (`session_mode: duplex`);
+- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex;
+- Nemotron VoiceChat, via its registered duplex plugin package.
 
 JoyVL is a separate HTTP interaction orchestrator and does not use these
 WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
@@ -53,7 +59,7 @@ Start the duplex deployment:
 
 ```bash
 vllm serve openbmb/MiniCPM-o-4_5 --omni \
-  --deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \
+  --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
   --trust-remote-code \
   --port 8091
 ```
@@ -112,5 +118,5 @@ applications should use the provided Realtime client where possible.
 
 See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
 [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
-and [full-duplex runtime design](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/experimental/fullduplex/DESIGN.md)
+and the [full-duplex runtime design](../design/fullduplex.md)
 for model-specific validation and architecture details.

docs/serving/realtime_api.md

@@ -5,7 +5,7 @@ receive incremental transcription and synthesized response audio. This is a
 turn-based streaming API: it starts one generation over an audio stream and
 finishes after the client closes that input stream.
 
-For a model that listens while it is already speaking, use the experimental
+For a model that listens while it is already speaking, use the
 [Full-Duplex API](full_duplex_api.md) instead.
 
 ## Quick Start
@@ -35,7 +35,7 @@ Messages in both directions are JSON text frames. Audio carried inside an
 event is base64-encoded raw PCM16.
 
 | Direction | Event | Purpose |
-|-----------|-------|---------|
+| ----------- | ------- | --------- |
 | Server to client | `session.created` | Confirms that the WebSocket connection is ready |
 | Client to server | `session.update` | Selects and validates the served model |
 | Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream |

docs/serving/standalone_servers.md

@@ -1,61 +1,19 @@
 # Standalone Experimental Servers
 
 Most vLLM-Omni endpoints run on the unified server started with
-`vllm serve <model> --omni`. The experimental full-duplex package also contains
-two separate user-facing server processes. Their routes are not added to the
-unified API server.
+`vllm serve <model> --omni`. The experimental package also contains one
+separate user-facing server process; its routes are not added to the unified
+API server.
 
 | Server | Transport | Role |
-|--------|-----------|------|
-| PersonaPlex compatibility server | WebSocket | Hosts the official PersonaPlex browser protocol and a raw-PCM alternative |
+| --- | --- | --- |
 | JoyVL interaction server | HTTP | Adds state, memory, proactive decisions, and delegation in front of a separate OpenAI-compatible model backend |
 
-## PersonaPlex Compatibility Server
-
-Use this server when a client requires the official Moshi/PersonaPlex wire
-protocol or the bundled browser UI. For the engine-owned unified duplex path,
-use [`WS /v1/realtime?duplex=1`](full_duplex_api.md) instead.
-
-Start it with:
-
-```bash
-HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \
-  vllm_omni.experimental.fullduplex.personaplex.serving.server \
-  --port 8091
-```
-
-| Route | Purpose |
-|-------|---------|
-| `GET /` | Official PersonaPlex browser client, when its assets are available |
-| `GET /health` | Readiness check |
-| `WS /api/chat` | Moshi-compatible tagged binary protocol with Opus audio |
-| `WS /v1/audio/duplex` | Simpler float32 raw-PCM protocol |
-
-### `/api/chat` Protocol
-
-The first byte of every binary frame is a message tag:
-
-| Direction | Tag | Payload |
-|-----------|-----|---------|
-| Server to client | `0x00` | Ready handshake |
-| Client to server | `0x01` | Opus microphone audio at 24 kHz |
-| Server to client | `0x01` | Opus response audio |
-| Server to client | `0x02` | UTF-8 inner-monologue text |
-
-The optional `text_prompt` and `voice_prompt` query parameters select a persona
-and voice. The default server admits one conversation; use `--batch-size N` to
-configure multiple lockstep slots. Connections beyond capacity close with
-WebSocket code 1013.
-
-### `/v1/audio/duplex` Protocol
-
-Send `{"type":"open","persona":"...","voice":"..."}` as a JSON text frame,
-then exchange binary float32 PCM frames. The server returns generated float32
-audio as binary frames and text as `{"type":"text","text":"..."}`. Send
-`{"type":"close"}` and wait for `{"type":"done"}` to flush and finish.
-
-See the [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
-for the browser and headless clients.
+PersonaPlex no longer ships a standalone compatibility server: it is served
+through the unified duplex path (`vllm_omni/deploy/personaplex.yaml`) over
+[`WS /v1/realtime?duplex=1`](full_duplex_api.md); see the
+[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
+for its clients.
 
 ## JoyVL Interaction Server
 
@@ -80,7 +38,7 @@ python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \
 ```
 
 | Method and route | Purpose |
-|------------------|---------|
+| ------------------ | --------- |
 | `GET /health` | Readiness check |
 | `GET /v1/models` | Reports the configured interaction model |
 | `POST /v1/chat/completions` | Processes one frame or interaction tick |

docs/usage/faq.md

@@ -18,4 +18,4 @@ A: AWQ is available through the inherited vLLM quantization registry, and vLLM-O
 
 > Q: Does vLLM-Omni support multimodal streaming input and output?
 
-A: Yes, for specific models and serving paths. Current capabilities include [streaming video input](../serving/video_stream_api.md), [streaming diffusion output](../user_guide/diffusion/execution_modes.md#streaming-output), and experimental full-duplex real-time audio described in the [project overview](../README.md). Check the model and API documentation because input and output support is model-specific.
+A: Yes, for specific models and serving paths. Current capabilities include [streaming video input](../serving/video_stream_api.md), [streaming diffusion output](../user_guide/diffusion/execution_modes.md#streaming-output), and [full-duplex realtime audio](../serving/realtime_duplex_api.md). Check the model and API documentation because input and output support is model-specific.