Merge vLLM-Omni v0.29.0rc1 into edge-infer/omni-0.29.0
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while our 0.28-based tree needs seven modules v0.29.0 no longer has (vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args, openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant, models.hunyuan_v1). Ten files conflicted. Upstream's structure is the base everywhere; the fork's features are re-attached to it: - deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config pops deploy_config (async_omni_engine, cli/serve); the deploy-profile selector and the resolve_model_config_path chain it needs stay in entrypoints/utils.py; stage_overrides takes rc1's dict type. - config_factory keeps the lazy diffusion output-type import; the diffusion registry import in rc1's new config/resolver.py is made lazy for the same WP6 reason (TTS path must not load diffusion). - requirements: our core/media split kept; rc1's transformers>=5.13 floor and kernels==0.16.1 applied in core.txt / media.txt. - TTS: rc1's split_granularity and bounded seed plus our output_mode= codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation; our private _conditioning_cache_salt is superseded by rc1's public tts_adapters/base.py::conditioning_cache_salt. - stage_runtime: rc1's single ExitStack G2 path (lock released before READY) replaces our two-path block, with the four worker_phase names the init profiler consumes still emitted; _cpu_placed / host admission kept. Not yet run against vLLM 0.29.0: the tests listed in the W6 record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
commit
33de5d8629
513 files changed
+35146
-10927
No files matched your search
@@ -64,7 +64,7 @@ connect
|
||||
<- msgpack(policy_server_config)
|
||||
|
||||
infer
|
||||
-> msgpack({"endpoint": "infer", "session_id": "...", ...observation})
|
||||
-> msgpack({"endpoint": "infer", "session_id": "...", "seed": <optional int>, ...observation})
|
||||
<- msgpack(ndarray | dict[str, ndarray])
|
||||
|
||||
reset
|
||||
@@ -92,6 +92,12 @@ frames are not observation messages.
|
||||
- The policy pipeline owns observation transforms and persistent model state,
|
||||
normally keyed by `session_id`; the API layer forwards the raw observation
|
||||
dictionary.
|
||||
- An optional integer `seed` in an inference message becomes the engine
|
||||
request's `sampling_params.seed`. A policy whose sampling consumes the
|
||||
request generator (GR00T-N1.7) returns the same action chunk for the same
|
||||
observation and seed. If omitted, the engine assigns a random per-request
|
||||
seed, so unseeded inferences are independent of each other. The API layer
|
||||
consumes `seed`; it is not forwarded inside the observation.
|
||||
|
||||
## Limits and Errors
|
||||
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
# Realtime Duplex API
|
||||
|
||||
vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
|
||||
`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
|
||||
`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a
|
||||
set of duplex extensions: the model decides when to listen and when to speak,
|
||||
the user can talk over the assistant, playback progress is reported back so
|
||||
history stays honest, and a dropped connection can resume the same session.
|
||||
This page covers how to run a duplex deployment, how to drive it from Python
|
||||
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
|
||||
|
||||
The endpoint is served for models that ship a duplex plugin — currently
|
||||
MiniCPM-o 4.5, PersonaPlex, and Nemotron VoiceChat — and only for deploy
|
||||
configurations that declare `session_mode: duplex`. The runtime architecture
|
||||
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
The handler is enabled by either a model-native `session_mode: duplex`
|
||||
deployment or an explicit `duplex_session` configuration for turn-based
|
||||
Server VAD. The runtime architecture is described in
|
||||
[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -27,7 +27,7 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
|
||||
|
||||
`vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
|
||||
`duplex_session.max_sessions: 4`, which mounts the WebSocket routes
|
||||
`ws://<host>:8099/v1/realtime?duplex=1` (this page) and `ws://<host>:8099/v1/duplex`
|
||||
`ws://<host>:8099/v1/realtime` (this page) and `ws://<host>:8099/v1/duplex`
|
||||
(the native dialect for raw-protocol clients) next to the usual HTTP API.
|
||||
PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and
|
||||
`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way.
|
||||
@@ -240,7 +240,7 @@ on the same helpers.
|
||||
|
||||
## API Reference
|
||||
|
||||
This section is the complete wire contract of `/v1/realtime?duplex=1`: how
|
||||
This section is the complete wire contract of `/v1/realtime`: how
|
||||
the dialect relates to the OpenAI Realtime protocol, which surfaces each
|
||||
model's capability flags gate, a catalogue of every client-to-server and
|
||||
server-to-client event, and a JSON example for each. `DuplexClient` speaks
|
||||
@@ -253,10 +253,11 @@ its protocol tests.
|
||||
|
||||
### Endpoint and transport
|
||||
|
||||
The endpoint is `ws(s)://<host>/v1/realtime?duplex=1`; the `duplex` query
|
||||
parameter (`1`, `true`, or `on`) selects the duplex session handler, and the
|
||||
route is mounted only for deployments whose deploy configuration declares
|
||||
`session_mode: duplex`. Optional query parameters are `model`, `session_id`,
|
||||
The endpoint is `ws(s)://<host>/v1/realtime`. Bare connections select the
|
||||
session handler when the deployment declares either `session_mode: duplex`
|
||||
or `duplex_session`. `?duplex=1` remains a compatibility alias, while
|
||||
`?duplex=0` selects the legacy Realtime handler. Optional query parameters
|
||||
are `model`, `session_id`,
|
||||
`autostart` (`0` means resume-only), `resume`, and `native_duplex`
|
||||
(`minicpmo45_native_duplex` is accepted as a deprecated alias and folded
|
||||
into the canonical name). Every message is one JSON object per WebSocket
|
||||
@@ -342,9 +343,14 @@ Semantic divergences hidden behind shared names:
|
||||
before any commit; a commit may end in `response.listen` and no
|
||||
response, and the model may open a response with no commit at all.
|
||||
OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
|
||||
- `turn_detection.interrupt_response=false` is rejected;
|
||||
`create_response` is ignored; `semantic_vad` is unsupported; the VAD
|
||||
runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
|
||||
- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response`
|
||||
defaults to `false` on turn-based models such as Qwen3-Omni, enabling
|
||||
endpointing and automatic responses without interruption. Explicit `true`
|
||||
is rejected there. Model-native duplex keeps the `true` default and rejects
|
||||
`false`; its VAD implies `overlap_policy=barge_in_on_speech`.
|
||||
This capability-dependent default differs from OpenAI's `true` default;
|
||||
the effective configuration is returned in `session.created` / `session.updated`.
|
||||
`semantic_vad` is unsupported.
|
||||
- `rate_limits.updated` is always an empty list (compatibility only).
|
||||
- Cancellation never reuses a `response_id`; truncation is driven by
|
||||
`playback.ack` as well as `conversation.item.truncate`.
|
||||
@@ -1201,9 +1207,10 @@ Realtime projector renames and fans out these events before they reach a
|
||||
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
|
||||
or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
|
||||
calls are produced only by Nemotron VoiceChat.
|
||||
- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
|
||||
`semantic_vad`, `interrupt_response=false`, and `create_response` are not
|
||||
supported. Server VAD requires a model-native session.
|
||||
- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`.
|
||||
Turn-based models support endpointing without interruption, including
|
||||
`create_response=false` for explicit response creation. Model-native duplex
|
||||
requires `interrupt_response=true` and `create_response=true`.
|
||||
- `input_audio_transcription` and `input_audio_noise_reduction` are accepted
|
||||
and echoed but no separate transcription or noise-reduction stage runs;
|
||||
transcripts come from the model.
|
||||
|
||||
+31
-12
@@ -279,10 +279,15 @@ curl -X POST http://localhost:8091/v1/audio/voices \
|
||||
|
||||
## Streaming Text Input (WebSocket)
|
||||
|
||||
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally and generates audio per sentence as boundaries are detected.
|
||||
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally.
|
||||
By default (`split_granularity=none`) it buffers until `input.done` and
|
||||
synthesizes that flush as **one** TTS request, which keeps long-form timbre
|
||||
stable. Set `split_granularity` to `sentence` or `clause` to emit a request
|
||||
at each detected boundary (lower time-to-first-audio for STT/LLM pipelines).
|
||||
|
||||
> Note: text input is always streamed incrementally. Audio output remains sentence-scoped:
|
||||
> use `stream_audio=false` for one binary frame per sentence, or `stream_audio=true` for one or more PCM chunks per sentence.
|
||||
> Note: `stream_audio` only changes how **audio bytes** are framed (one WAV/PCM
|
||||
> payload vs chunked PCM). Text segmentation is controlled separately by
|
||||
> `split_granularity`.
|
||||
|
||||
### WebSocket Protocol
|
||||
|
||||
@@ -314,14 +319,16 @@ upstream LLM) pays the WebSocket handshake once instead of once per utterance.
|
||||
|
||||
- The session config is sticky. Send `input.text` again straight after
|
||||
`session.done` to reuse it, or send another `session.config` first to change
|
||||
voice, format, or reference audio. A `session.config` sent while text is
|
||||
still buffered is rejected so no pending input is silently dropped.
|
||||
- An utterance is the flush unit, not a linguistic one: it is whatever text was
|
||||
buffered when `input.done` arrived, of any length, synthesized as one request.
|
||||
`utterance_index` counts those flushes across the connection, so it tells you
|
||||
which `input.done` a frame belongs to. `sentence_index` counts within one
|
||||
flush and so pairs with `total_sentences`, which means every utterance reports
|
||||
`sentence_index: 0` of `total_sentences: 1` (or `0` for an empty buffer).
|
||||
voice, format, or reference audio. A `session.config` sent in the middle of
|
||||
an utterance is rejected so no pending input is silently dropped and so a
|
||||
split utterance cannot end up half in one voice and half in another.
|
||||
- An utterance is the flush unit, not a linguistic one: it is one `input.done`
|
||||
cycle. `utterance_index` counts those flushes across the connection, so it
|
||||
tells you which `input.done` a frame belongs to. `sentence_index` counts the
|
||||
TTS requests inside one flush and so pairs with `total_sentences`: with the
|
||||
default `split_granularity=none` that is always `sentence_index: 0` of
|
||||
`total_sentences: 1` (or `0` for an empty buffer), while `sentence` or
|
||||
`clause` counts the linguistic units actually synthesized.
|
||||
- End the connection with `session.close`, or by closing the socket. An idle
|
||||
connection is still closed after the server's idle timeout, which now also
|
||||
applies to the gap between utterances.
|
||||
@@ -332,7 +339,15 @@ All REST API parameters are supported, plus:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
| ----------- | ------ | --------- | ------------- |
|
||||
| `stream_audio` | bool | false | Stream one or more PCM chunks for the buffered input over WebSocket |
|
||||
| `stream_audio` | bool | false | Stream one or more PCM chunks for each TTS request over WebSocket |
|
||||
| `split_granularity` | string | `"none"` | `"none"`: one request per `input.done`. `"sentence"`: split on `.!?` plus CJK `。!?…`, Indic danda `।॥`, and Arabic `؟`. `"clause"`: also split on `,;,;،؛`. |
|
||||
| `seed` | integer | null | Forwarded to the speech engine for this session |
|
||||
|
||||
ASCII punctuation only ends a unit when whitespace or `input.done` follows it,
|
||||
and decimals (`3.14`), thousands separators (`1,000`), abbreviations (`Dr.`,
|
||||
`e.g.`) and initials (`J. R.`) are not treated as boundaries. A punctuation run
|
||||
and any closing quote or bracket stay with the unit they close, so `Wait...`
|
||||
and `He said "Hello."` are one request each.
|
||||
|
||||
```bash
|
||||
DELETE /v1/audio/voices/{name}
|
||||
@@ -752,6 +767,10 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
|
||||
| ------- | ------------- |
|
||||
| `k2-fsa/OmniVoice` | Pure-diffusion TTS. Supports voice cloning via `ref_audio` (with optional `ref_text`); no built-in voice presets. |
|
||||
|
||||
OmniVoice uses packed variable-length attention for batched generator execution. The attention operator accepts FP16 and BF16 inputs, so its query,
|
||||
key, and value tensors are evaluated in BF16 even when the stage is configured with `dtype: float32`; the attention output is converted back to the model's
|
||||
hidden-state dtype before the output projection. Consequently, a float32 stage configuration does not imply FP32 attention arithmetic.
|
||||
|
||||
### VoxCPM2
|
||||
|
||||
| Model | Description |
|
||||
|
||||
Reference in new issue
Block a user