[Bugfix] Fix Qwen3-Omni realtime playback interruption and conversation history (#7791)

Signed-off-by: psv666 <2693925048@qq.com>
This commit is contained in:
psv666 authored and GitHub committed 2026-09-20 14:42:17 +00:00
1 parent d3396fc5fb
commit 2d0b03f717
13 files changed
+1292 -68

No files matched your search

+31 -2
View File
@@ -8,12 +8,41 @@ history stays honest, and a dropped connection can resume the same session.
This page covers how to run a duplex deployment, how to drive it from Python
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
The endpoint is served for models that ship a duplex plugin — currently
MiniCPM-o 4.5 only — and just for deploy configurations that declare
The endpoint is served for models that ship a duplex plugin — including
MiniCPM-o 4.5 and Qwen3-Omni — and just for deploy configurations that declare
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
## Qwen3-Omni conversation history
Qwen3-Omni keeps incoming conversation items independently addressable by item
ID. Before applying the model's chat template, it merges consecutive user
items into one multimodal user message. An assistant message, including an
empty message reserved for an interrupted or unheard answer, separates turns.
The final user message places media before text instructions, following the
[official Qwen3-Omni demo's `format_history()`](https://github.com/QwenLM/Qwen3-Omni/blob/e4235853125589c789f06a2dd83e9f4126df5e9d/web_demo.py#L84-L179).
A camera image or text item may arrive after an audio commit and before
`response.create`. Those items belong to the same pending user turn: the
committed audio remains associated with its history item even when that item
is no longer last. Prompt preparation snapshots the selected inputs; later
items do not modify an already prepared request.
The duplex adapter retains its existing budgets: at most four audio payloads
(including the current input), an 8 MiB encoded-audio history budget (keeping
at least the newest payload), and at most eight prompt images. The recent
16-message window is rounded outwards to a complete turn. If any audio in an
older turn is no longer retained, the entire turn's model context, including
its images/text and assistant replies, is omitted. Images from that turn are
therefore **not** permanent session-wide model context. These limits differ
from the demo's one image-bearing turn and five audio-bearing turns; the
shared rule is grouping before pruning, rather than copying its UI budgets.
Grouping and pruning affect only the model-input view. They do not merge or
delete the source conversation items exposed to clients, and do not change
playback acknowledgement or interruption handling.
## Quick Start
### Start the Server