[Bugfix] Fix Qwen3-Omni realtime playback interruption and conversation history (#7791)
Signed-off-by: psv666 <2693925048@qq.com>
This commit is contained in:
1 parent
d3396fc5fb
commit
2d0b03f717
13 files changed
+1292
-68
No files matched your search
@@ -8,12 +8,41 @@ history stays honest, and a dropped connection can resume the same session.
|
||||
This page covers how to run a duplex deployment, how to drive it from Python
|
||||
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
|
||||
|
||||
The endpoint is served for models that ship a duplex plugin — currently
|
||||
MiniCPM-o 4.5 only — and just for deploy configurations that declare
|
||||
The endpoint is served for models that ship a duplex plugin — including
|
||||
MiniCPM-o 4.5 and Qwen3-Omni — and just for deploy configurations that declare
|
||||
`session_mode: duplex`. PersonaPlex and Nemotron VoiceChat are ported to the
|
||||
plugin contract in follow-up PRs and are not served over this endpoint yet. The runtime architecture
|
||||
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
|
||||
|
||||
## Qwen3-Omni conversation history
|
||||
|
||||
Qwen3-Omni keeps incoming conversation items independently addressable by item
|
||||
ID. Before applying the model's chat template, it merges consecutive user
|
||||
items into one multimodal user message. An assistant message, including an
|
||||
empty message reserved for an interrupted or unheard answer, separates turns.
|
||||
The final user message places media before text instructions, following the
|
||||
[official Qwen3-Omni demo's `format_history()`](https://github.com/QwenLM/Qwen3-Omni/blob/e4235853125589c789f06a2dd83e9f4126df5e9d/web_demo.py#L84-L179).
|
||||
|
||||
A camera image or text item may arrive after an audio commit and before
|
||||
`response.create`. Those items belong to the same pending user turn: the
|
||||
committed audio remains associated with its history item even when that item
|
||||
is no longer last. Prompt preparation snapshots the selected inputs; later
|
||||
items do not modify an already prepared request.
|
||||
|
||||
The duplex adapter retains its existing budgets: at most four audio payloads
|
||||
(including the current input), an 8 MiB encoded-audio history budget (keeping
|
||||
at least the newest payload), and at most eight prompt images. The recent
|
||||
16-message window is rounded outwards to a complete turn. If any audio in an
|
||||
older turn is no longer retained, the entire turn's model context, including
|
||||
its images/text and assistant replies, is omitted. Images from that turn are
|
||||
therefore **not** permanent session-wide model context. These limits differ
|
||||
from the demo's one image-bearing turn and five audio-bearing turns; the
|
||||
shared rule is grouping before pruning, rather than copying its UI budgets.
|
||||
|
||||
Grouping and pruning affect only the model-input view. They do not merge or
|
||||
delete the source conversation items exposed to clients, and do not change
|
||||
playback acknowledgement or interruption handling.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Start the Server
|
||||
|
||||
Reference in new issue
Block a user