[Bugfix][Frontend] Preserve duplex chat audio format and duration (#7316)

Signed-off-by: Nolen Liang <nliang@nvidia.com>
This commit is contained in:
bcsdhjew authored and GitHub committed 2026-09-14 09:05:55 -07:00
1 parent 1b6cd282f0
commit e0c4f97f32
13 files changed
+684 -40

No files matched your search

+43
View File
@@ -387,6 +387,49 @@ Semantic divergences hidden behind shared names:
the Tier 3 surface on top of the same Tier 1 events, so both kinds of
client can share one deployment and one event log.
### Turn-based Chat audio metadata
For turn-based models served through Chat fallback, non-empty audio choices
produced by the Omni Chat audio encoder include `audio_metadata`, in both
streaming and complete Chat responses. Text-only choices do not require it.
| Field | Meaning |
| --- | --- |
| `format` | Encoding of the returned audio, such as `pcm` or `wav`; `pcm16` requests normalize to `pcm`. |
| `sample_rate_hz` | Sample rate used by the encoder, in samples per second. |
| `frame_count` | Samples per channel in this chunk, or in the complete waveform for a non-streaming choice. Not a byte count. |
| `channels` | Number of audio channels. |
These values describe the waveform after audio transformations. Compressed
formats may add codec padding that is not included in `frame_count`.
The encoder uses a model-reported sample rate when available; Qwen3-Omni
currently omits it and uses the existing 24 kHz default. This metadata does
not independently determine a model's sample rate.
Chat fallback requests the session's output format explicitly. Its internal
`response.output_audio.delta` carries `format`, `sample_rate_hz`, `channels`,
and `audio_duration_ms`. The Realtime endpoint exposes output `format` and
`sample_rate_hz` on `response.audio.delta`, with duration in
`metadata.audio_duration_ms`; it does not currently forward `channels`.
The duration is cumulative for the response:
integer frames are summed before converting to milliseconds and rounding down.
Non-empty audio without valid metadata, or a sample-rate change within one
response, fails that fallback response with `response_error`.
In this Chat fallback path, `response.done.response.metadata.playback` reports:
- `generated_ms`: cumulative source-waveform duration.
- `sent_ms`: cumulative audio accepted into the server output path, including
queued or journaled audio. A journaled event remains accepted if the
subsequent send fails; this is not proof of socket delivery.
- `played_ms`: cumulative playback reported by the client.
- `committed_ms`: playback position used for history reconciliation.
The native model path still records audio before sending; it does not yet
share Chat fallback's output-acceptance timing. Only client playback reports
establish what was played. The existing duration-proportional text truncation
remains an approximation, not word-level audio alignment.
### Event catalogue
Every event type the Realtime route accepts or emits, with a worked example in the next section: 21