[Core] Graduate MiniCPM-o 4.5 and PersonaPlex full-duplex serving out of experimental (#6196)
Signed-off-by: chickeyton <ngton2014@gmail.com> Signed-off-by: Gao Han <hgaoaf@connect.ust.hk> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Gao Han <hgaoaf@connect.ust.hk>
This commit is contained in:
179 files changed
+6462
-3832
No files matched your search
@@ -43,7 +43,7 @@ Completions for conversational or heterogeneous omni pipelines, rather than as
|
||||
the default wrapper for every generation task.
|
||||
|
||||
| Task | Endpoint | Request | Response | Details |
|
||||
|------|----------|---------|----------|---------|
|
||||
| ------ | ---------- | --------- | ---------- | --------- |
|
||||
| Conversation or multimodal understanding/generation | `POST /v1/chat/completions` | JSON | OpenAI-style JSON or SSE | [Chat Completions](chat_completions_api.md) |
|
||||
| Text-to-speech | `POST /v1/audio/speech` | JSON | Audio bytes or SSE | [Speech](speech_api.md) |
|
||||
| Sound, music, or ambient audio generation | `POST /v1/audio/generate` | JSON | Audio bytes | [Audio Generation](audio_generate_api.md) |
|
||||
@@ -141,7 +141,7 @@ These WebSocket APIs have different event schemas and cannot be used
|
||||
interchangeably.
|
||||
|
||||
| Workload | Endpoint | Interaction model | Details |
|
||||
|----------|----------|-------------------|---------|
|
||||
| ---------- | ---------- | ------------------- | --------- |
|
||||
| Incremental text input for speech synthesis | `WS /v1/audio/speech/stream` | Send text events and receive audio | [Streaming Text to Speech](speech_api.md#streaming-text-input-websocket) |
|
||||
| Live video understanding | `WS /v1/video/chat/stream` | Send video frames and receive text/audio | [Streaming Video Input](video_stream_api.md) |
|
||||
| Turn-based realtime audio | `WS /v1/realtime` | Stream one audio input and receive transcript/audio events | [Realtime Audio](realtime_api.md) |
|
||||
@@ -158,7 +158,7 @@ ordinary realtime handler when duplex is unavailable.
|
||||
## Related Endpoints
|
||||
|
||||
| Purpose | Endpoints | Reference |
|
||||
|---------|-----------|-----------|
|
||||
| --------- | ----------- | ----------- |
|
||||
| Discovery and readiness | `GET /health`, `GET /v1/models` | This page |
|
||||
| Batched conversations | `POST /v1/chat/completions/batch` | [Batch requests](chat_completions_api.md#batch-requests) |
|
||||
| Batched speech | `POST /v1/audio/speech/batch` | [Batch speech generation](speech_api.md#batch-speech-generation) |
|
||||
@@ -168,11 +168,10 @@ ordinary realtime handler when duplex is unavailable.
|
||||
|
||||
## Standalone Experimental Servers
|
||||
|
||||
PersonaPlex also provides a Moshi-compatible WebSocket server, while JoyVL
|
||||
provides a stateful interaction orchestrator in front of another model server.
|
||||
These are separate processes with their own routes; they are not additional
|
||||
JoyVL provides a stateful interaction orchestrator in front of another model
|
||||
server. It is a separate process with its own routes; they are not additional
|
||||
paths on every `vllm serve --omni` instance. See [Standalone Experimental
|
||||
Servers](standalone_servers.md) before deploying either one.
|
||||
Servers](standalone_servers.md) before deploying it.
|
||||
|
||||
## Compatibility Notes
|
||||
|
||||
|
||||
@@ -1,16 +1,20 @@
|
||||
# Full-Duplex WebSocket API
|
||||
|
||||
vLLM-Omni provides an experimental full-duplex runtime for models that can
|
||||
vLLM-Omni provides a full-duplex runtime for models that can
|
||||
continue receiving speech while producing speech. It adds persistent session
|
||||
state, model-specific turn policy, overlap handling, playback acknowledgement,
|
||||
and optional session resume.
|
||||
|
||||
Full duplex is distinct from the turn-based [Realtime Audio API](realtime_api.md).
|
||||
|
||||
This page is the endpoint overview. For the complete wire contract, the
|
||||
`vllm_omni.clients.duplex.DuplexClient` Python library, and the per-model
|
||||
capability gates, see the [Realtime Duplex API guide](realtime_duplex_api.md).
|
||||
|
||||
## Choose an Endpoint
|
||||
|
||||
| Endpoint | Protocol | Recommended use |
|
||||
|----------|----------|-----------------|
|
||||
| --- | --- | --- |
|
||||
| `WS /v1/realtime?duplex=1` | OpenAI Realtime-style event projection | Applications and browser clients |
|
||||
| `WS /v1/duplex` | Native vLLM-Omni duplex events | Runtime integration and low-level testing |
|
||||
|
||||
@@ -41,8 +45,10 @@ and control plane as part of their registered pipeline.
|
||||
|
||||
The current unified-runtime integrations are:
|
||||
|
||||
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5_duplex.yaml`;
|
||||
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex.
|
||||
- MiniCPM-o 4.5, using `vllm_omni/deploy/minicpmo_4_5.yaml`
|
||||
(`session_mode: duplex`);
|
||||
- PersonaPlex, whose default `vllm_omni/deploy/personaplex.yaml` enables duplex;
|
||||
- Nemotron VoiceChat, via its registered duplex plugin package.
|
||||
|
||||
JoyVL is a separate HTTP interaction orchestrator and does not use these
|
||||
WebSocket endpoints. See [Standalone Experimental Servers](standalone_servers.md).
|
||||
@@ -53,7 +59,7 @@ Start the duplex deployment:
|
||||
|
||||
```bash
|
||||
vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
||||
--deploy-config vllm_omni/deploy/minicpmo_4_5_duplex.yaml \
|
||||
--deploy-config vllm_omni/deploy/minicpmo_4_5.yaml \
|
||||
--trust-remote-code \
|
||||
--port 8091
|
||||
```
|
||||
@@ -112,5 +118,5 @@ applications should use the provided Realtime client where possible.
|
||||
|
||||
See the [MiniCPM-o example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/minicpmo),
|
||||
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex),
|
||||
and [full-duplex runtime design](https://github.com/vllm-project/vllm-omni/blob/main/vllm_omni/experimental/fullduplex/DESIGN.md)
|
||||
and the [full-duplex runtime design](../design/fullduplex.md)
|
||||
for model-specific validation and architecture details.
|
||||
@@ -5,7 +5,7 @@ receive incremental transcription and synthesized response audio. This is a
|
||||
turn-based streaming API: it starts one generation over an audio stream and
|
||||
finishes after the client closes that input stream.
|
||||
|
||||
For a model that listens while it is already speaking, use the experimental
|
||||
For a model that listens while it is already speaking, use the
|
||||
[Full-Duplex API](full_duplex_api.md) instead.
|
||||
|
||||
## Quick Start
|
||||
@@ -35,7 +35,7 @@ Messages in both directions are JSON text frames. Audio carried inside an
|
||||
event is base64-encoded raw PCM16.
|
||||
|
||||
| Direction | Event | Purpose |
|
||||
|-----------|-------|---------|
|
||||
| ----------- | ------- | --------- |
|
||||
| Server to client | `session.created` | Confirms that the WebSocket connection is ready |
|
||||
| Client to server | `session.update` | Selects and validates the served model |
|
||||
| Client to server | `input_audio_buffer.commit` with `final: false` | Starts generation over the incoming stream |
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,61 +1,19 @@
|
||||
# Standalone Experimental Servers
|
||||
|
||||
Most vLLM-Omni endpoints run on the unified server started with
|
||||
`vllm serve <model> --omni`. The experimental full-duplex package also contains
|
||||
two separate user-facing server processes. Their routes are not added to the
|
||||
unified API server.
|
||||
`vllm serve <model> --omni`. The experimental package also contains one
|
||||
separate user-facing server process; its routes are not added to the unified
|
||||
API server.
|
||||
|
||||
| Server | Transport | Role |
|
||||
|--------|-----------|------|
|
||||
| PersonaPlex compatibility server | WebSocket | Hosts the official PersonaPlex browser protocol and a raw-PCM alternative |
|
||||
| --- | --- | --- |
|
||||
| JoyVL interaction server | HTTP | Adds state, memory, proactive decisions, and delegation in front of a separate OpenAI-compatible model backend |
|
||||
|
||||
## PersonaPlex Compatibility Server
|
||||
|
||||
Use this server when a client requires the official Moshi/PersonaPlex wire
|
||||
protocol or the bundled browser UI. For the engine-owned unified duplex path,
|
||||
use [`WS /v1/realtime?duplex=1`](full_duplex_api.md) instead.
|
||||
|
||||
Start it with:
|
||||
|
||||
```bash
|
||||
HF_TOKEN=... CUDA_VISIBLE_DEVICES=0 python -m \
|
||||
vllm_omni.experimental.fullduplex.personaplex.serving.server \
|
||||
--port 8091
|
||||
```
|
||||
|
||||
| Route | Purpose |
|
||||
|-------|---------|
|
||||
| `GET /` | Official PersonaPlex browser client, when its assets are available |
|
||||
| `GET /health` | Readiness check |
|
||||
| `WS /api/chat` | Moshi-compatible tagged binary protocol with Opus audio |
|
||||
| `WS /v1/audio/duplex` | Simpler float32 raw-PCM protocol |
|
||||
|
||||
### `/api/chat` Protocol
|
||||
|
||||
The first byte of every binary frame is a message tag:
|
||||
|
||||
| Direction | Tag | Payload |
|
||||
|-----------|-----|---------|
|
||||
| Server to client | `0x00` | Ready handshake |
|
||||
| Client to server | `0x01` | Opus microphone audio at 24 kHz |
|
||||
| Server to client | `0x01` | Opus response audio |
|
||||
| Server to client | `0x02` | UTF-8 inner-monologue text |
|
||||
|
||||
The optional `text_prompt` and `voice_prompt` query parameters select a persona
|
||||
and voice. The default server admits one conversation; use `--batch-size N` to
|
||||
configure multiple lockstep slots. Connections beyond capacity close with
|
||||
WebSocket code 1013.
|
||||
|
||||
### `/v1/audio/duplex` Protocol
|
||||
|
||||
Send `{"type":"open","persona":"...","voice":"..."}` as a JSON text frame,
|
||||
then exchange binary float32 PCM frames. The server returns generated float32
|
||||
audio as binary frames and text as `{"type":"text","text":"..."}`. Send
|
||||
`{"type":"close"}` and wait for `{"type":"done"}` to flush and finish.
|
||||
|
||||
See the [PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
|
||||
for the browser and headless clients.
|
||||
PersonaPlex no longer ships a standalone compatibility server: it is served
|
||||
through the unified duplex path (`vllm_omni/deploy/personaplex.yaml`) over
|
||||
[`WS /v1/realtime?duplex=1`](full_duplex_api.md); see the
|
||||
[PersonaPlex example](https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/personaplex)
|
||||
for its clients.
|
||||
|
||||
## JoyVL Interaction Server
|
||||
|
||||
@@ -80,7 +38,7 @@ python -m vllm_omni.experimental.fullduplex.joyvl.serving.server \
|
||||
```
|
||||
|
||||
| Method and route | Purpose |
|
||||
|------------------|---------|
|
||||
| ------------------ | --------- |
|
||||
| `GET /health` | Readiness check |
|
||||
| `GET /v1/models` | Reports the configured interaction model |
|
||||
| `POST /v1/chat/completions` | Processes one frame or interaction tick |
|
||||
|
||||
Reference in new issue
Block a user