Merge vLLM-Omni v0.29.0rc1 into edge-infer/omni-0.29.0

The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).

Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
  pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
  selector and the resolve_model_config_path chain it needs stay in
  entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
  registry import in rc1's new config/resolver.py is made lazy for the same
  WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
  and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
  codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
  our private _conditioning_cache_salt is superseded by rc1's public
  tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
  replaces our two-path block, with the four worker_phase names the init
  profiler consumes still emitted; _cpu_placed / host admission kept.

Not yet run against vLLM 0.29.0: the tests listed in the W6 record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
tzhouamandClaude Fable 5.1 committed 2026-09-18 09:57:20 +08:00
commit 33de5d8629
513 files changed
+35146 -10927

No files matched your search

+7 -1
View File
@@ -64,7 +64,7 @@ connect
<- msgpack(policy_server_config)
infer
-> msgpack({"endpoint": "infer", "session_id": "...", ...observation})
-> msgpack({"endpoint": "infer", "session_id": "...", "seed": <optional int>, ...observation})
<- msgpack(ndarray | dict[str, ndarray])
reset
@@ -92,6 +92,12 @@ frames are not observation messages.
- The policy pipeline owns observation transforms and persistent model state,
normally keyed by `session_id`; the API layer forwards the raw observation
dictionary.
- An optional integer `seed` in an inference message becomes the engine
request's `sampling_params.seed`. A policy whose sampling consumes the
request generator (GR00T-N1.7) returns the same action chunk for the same
observation and seed. If omitted, the engine assigns a random per-request
seed, so unseeded inferences are independent of each other. The API layer
consumes `seed`; it is not forwarded inside the observation.
## Limits and Errors
+24 -17
View File
@@ -1,17 +1,17 @@
# Realtime Duplex API
vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a
set of duplex extensions: the model decides when to listen and when to speak,
the user can talk over the assistant, playback progress is reported back so
history stays honest, and a dropped connection can resume the same session.
This page covers how to run a duplex deployment, how to drive it from Python
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
The endpoint is served for models that ship a duplex plugin — currently
MiniCPM-o 4.5, PersonaPlex, and Nemotron VoiceChat — and only for deploy
configurations that declare `session_mode: duplex`. The runtime architecture
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
The handler is enabled by either a model-native `session_mode: duplex`
deployment or an explicit `duplex_session` configuration for turn-based
Server VAD. The runtime architecture is described in
[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
## Quick Start
@@ -27,7 +27,7 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
`vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
`duplex_session.max_sessions: 4`, which mounts the WebSocket routes
`ws://<host>:8099/v1/realtime?duplex=1` (this page) and `ws://<host>:8099/v1/duplex`
`ws://<host>:8099/v1/realtime` (this page) and `ws://<host>:8099/v1/duplex`
(the native dialect for raw-protocol clients) next to the usual HTTP API.
PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and
`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way.
@@ -240,7 +240,7 @@ on the same helpers.
## API Reference
This section is the complete wire contract of `/v1/realtime?duplex=1`: how
This section is the complete wire contract of `/v1/realtime`: how
the dialect relates to the OpenAI Realtime protocol, which surfaces each
model's capability flags gate, a catalogue of every client-to-server and
server-to-client event, and a JSON example for each. `DuplexClient` speaks
@@ -253,10 +253,11 @@ its protocol tests.
### Endpoint and transport
The endpoint is `ws(s)://<host>/v1/realtime?duplex=1`; the `duplex` query
parameter (`1`, `true`, or `on`) selects the duplex session handler, and the
route is mounted only for deployments whose deploy configuration declares
`session_mode: duplex`. Optional query parameters are `model`, `session_id`,
The endpoint is `ws(s)://<host>/v1/realtime`. Bare connections select the
session handler when the deployment declares either `session_mode: duplex`
or `duplex_session`. `?duplex=1` remains a compatibility alias, while
`?duplex=0` selects the legacy Realtime handler. Optional query parameters
are `model`, `session_id`,
`autostart` (`0` means resume-only), `resume`, and `native_duplex`
(`minicpmo45_native_duplex` is accepted as a deprecated alias and folded
into the canonical name). Every message is one JSON object per WebSocket
@@ -342,9 +343,14 @@ Semantic divergences hidden behind shared names:
before any commit; a commit may end in `response.listen` and no
response, and the model may open a response with no commit at all.
OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
- `turn_detection.interrupt_response=false` is rejected;
`create_response` is ignored; `semantic_vad` is unsupported; the VAD
runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response`
defaults to `false` on turn-based models such as Qwen3-Omni, enabling
endpointing and automatic responses without interruption. Explicit `true`
is rejected there. Model-native duplex keeps the `true` default and rejects
`false`; its VAD implies `overlap_policy=barge_in_on_speech`.
This capability-dependent default differs from OpenAI's `true` default;
the effective configuration is returned in `session.created` / `session.updated`.
`semantic_vad` is unsupported.
- `rate_limits.updated` is always an empty list (compatibility only).
- Cancellation never reuses a `response_id`; truncation is driven by
`playback.ack` as well as `conversation.item.truncate`.
@@ -1201,9 +1207,10 @@ Realtime projector renames and fans out these events before they reach a
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
calls are produced only by Nemotron VoiceChat.
- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
`semantic_vad`, `interrupt_response=false`, and `create_response` are not
supported. Server VAD requires a model-native session.
- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`.
Turn-based models support endpointing without interruption, including
`create_response=false` for explicit response creation. Model-native duplex
requires `interrupt_response=true` and `create_response=true`.
- `input_audio_transcription` and `input_audio_noise_reduction` are accepted
and echoed but no separate transcription or noise-reduction stage runs;
transcripts come from the model.
+31 -12
View File
@@ -279,10 +279,15 @@ curl -X POST http://localhost:8091/v1/audio/voices \
## Streaming Text Input (WebSocket)
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally and generates audio per sentence as boundaries are detected.
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally.
By default (`split_granularity=none`) it buffers until `input.done` and
synthesizes that flush as **one** TTS request, which keeps long-form timbre
stable. Set `split_granularity` to `sentence` or `clause` to emit a request
at each detected boundary (lower time-to-first-audio for STT/LLM pipelines).
> Note: text input is always streamed incrementally. Audio output remains sentence-scoped:
> use `stream_audio=false` for one binary frame per sentence, or `stream_audio=true` for one or more PCM chunks per sentence.
> Note: `stream_audio` only changes how **audio bytes** are framed (one WAV/PCM
> payload vs chunked PCM). Text segmentation is controlled separately by
> `split_granularity`.
### WebSocket Protocol
@@ -314,14 +319,16 @@ upstream LLM) pays the WebSocket handshake once instead of once per utterance.
- The session config is sticky. Send `input.text` again straight after
`session.done` to reuse it, or send another `session.config` first to change
voice, format, or reference audio. A `session.config` sent while text is
still buffered is rejected so no pending input is silently dropped.
- An utterance is the flush unit, not a linguistic one: it is whatever text was
buffered when `input.done` arrived, of any length, synthesized as one request.
`utterance_index` counts those flushes across the connection, so it tells you
which `input.done` a frame belongs to. `sentence_index` counts within one
flush and so pairs with `total_sentences`, which means every utterance reports
`sentence_index: 0` of `total_sentences: 1` (or `0` for an empty buffer).
voice, format, or reference audio. A `session.config` sent in the middle of
an utterance is rejected so no pending input is silently dropped and so a
split utterance cannot end up half in one voice and half in another.
- An utterance is the flush unit, not a linguistic one: it is one `input.done`
cycle. `utterance_index` counts those flushes across the connection, so it
tells you which `input.done` a frame belongs to. `sentence_index` counts the
TTS requests inside one flush and so pairs with `total_sentences`: with the
default `split_granularity=none` that is always `sentence_index: 0` of
`total_sentences: 1` (or `0` for an empty buffer), while `sentence` or
`clause` counts the linguistic units actually synthesized.
- End the connection with `session.close`, or by closing the socket. An idle
connection is still closed after the server's idle timeout, which now also
applies to the gap between utterances.
@@ -332,7 +339,15 @@ All REST API parameters are supported, plus:
| Parameter | Type | Default | Description |
| ----------- | ------ | --------- | ------------- |
| `stream_audio` | bool | false | Stream one or more PCM chunks for the buffered input over WebSocket |
| `stream_audio` | bool | false | Stream one or more PCM chunks for each TTS request over WebSocket |
| `split_granularity` | string | `"none"` | `"none"`: one request per `input.done`. `"sentence"`: split on `.!?` plus CJK `。!?…`, Indic danda `।॥`, and Arabic `؟`. `"clause"`: also split on `,;,;،؛`. |
| `seed` | integer | null | Forwarded to the speech engine for this session |
ASCII punctuation only ends a unit when whitespace or `input.done` follows it,
and decimals (`3.14`), thousands separators (`1,000`), abbreviations (`Dr.`,
`e.g.`) and initials (`J. R.`) are not treated as boundaries. A punctuation run
and any closing quote or bracket stay with the unit they close, so `Wait...`
and `He said "Hello."` are one request each.
```bash
DELETE /v1/audio/voices/{name}
@@ -752,6 +767,10 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
| ------- | ------------- |
| `k2-fsa/OmniVoice` | Pure-diffusion TTS. Supports voice cloning via `ref_audio` (with optional `ref_text`); no built-in voice presets. |
OmniVoice uses packed variable-length attention for batched generator execution. The attention operator accepts FP16 and BF16 inputs, so its query,
key, and value tensors are evaluated in BF16 even when the stage is configured with `dtype: float32`; the attention output is converted back to the model's
hidden-state dtype before the output projection. Consequently, a float32 stage configuration does not imply FP32 attention arithmetic.
### VoxCPM2
| Model | Description |