0 Fork 0

Files

T

tzhouamandClaude Fable 5.1 49a839ecf9 [Edge][WP5] Add codec-token streaming mode with on-device decoding

* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage,
  final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml;
  a terminal talker publishes its per-step codes.audio rows to the client
  (omni_client_multimodal_output_keys).
* /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams
  codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows,
  vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel
  message aborts an in-flight generation for both codec and PCM sessions
  (single reader task + session inbox replaces direct socket reads); optional
  session.config.seed; audio.done gains "cancelled" only when it happened.
* serving_speech._generate_codec_chunks aligns the cumulative code tensor to
  output token ids (drops prefill placeholder/EOS rows) and emits deltas.
* TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16
  codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz).
* examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py
  decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame
  context, cancel/reset), codec_parity_check.py and
  benchmarks/tts/codec_stream_e2e.py compare against the two-stage server.
* docs/serving/codec_token_streaming.md; CPU unit tests for the wire format,
  alignment, session config and handler (frames, unsupported adapter, cancel);
  L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-09 11:10:13 +08:00

2.8 KiB

Raw Blame History

Codec-token streaming (on-device decoding)

For edge deployments the expensive part of a TTS pipeline that must run near the user is the audio decoder, while the autoregressive talker can stay on a server. Codec-token streaming serves the talker only and streams its codec frames to a client that decodes them locally (Qwen3-TTS: 16 codebooks at 12.5 Hz, about 400 B/s of payload before framing).

Server

Deploy the talker-only pipeline:

vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --omni --port 8001 \
  --deploy-config qwen3_tts_talker_only.yaml --trust-remote-code

qwen3_tts_talker_only.yaml selects the registered qwen3_tts_talker_only pipeline (stage 0 only, final_output_type: latent). The talker publishes its per-step codes.audio rows to the client (omni_client_multimodal_output_keys), and the WebSocket handler emits them as binary frames.

Protocol (`/v1/audio/speech/stream`)

Client → server: the usual session.config with

{"type": "session.config", "output_mode": "codec_tokens", "stream_audio": true,
 "response_format": "pcm", "voice": "vivian", "task_type": "CustomVoice"}

then input.text / input.done. input.cancel aborts the in-flight generation (works for PCM sessions too).

Server → client per utterance:

message	content
`codec.start` (JSON)	`codebooks`, `codebook_size`, `frame_rate_hz`, `sample_rate`, `decoder_id`, `header_format`
binary frames	header `<IHB` = `seq:uint32, n_frames:uint16, flags:uint8`, then `int16[n_frames × codebooks]`; flags `EOS=1`, `SEGMENT_END=2`, `RESET=4`
`codec.done` (JSON)	`total_frames`, `seq_last`, `error`, `cancelled`
`session.done`	as for PCM sessions

Frames are emitted as they are produced (one talker step = one frame); the chunking policy for the decoder (first small chunk, ramp to 25 frames, 72-frame left context) is the client's choice. The wire format lives in vllm_omni/entrypoints/openai/codec_stream.py.

Client

examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py loads Qwen3TTSTokenizerV2Decoder from the checkpoint's speech_tokenizer/ folder (CPU or CUDA), applies the same chunk schedule and state carry-over as the server's Code2Wav stage, writes a WAV and reports TTFA, playback-start latency and RTF. codec_parity_check.py compares the client-decoded audio with the full two-stage server's output (same tts_local_seed) and reports SNR.

Limits

Only adapters with supports_codec_stream = True (Qwen3-TTS today) accept output_mode: codec_tokens; others return an error message.
The talker's per-step output carries the cumulative code tensor, so a long utterance costs O(N²) bytes between engine and API server (about 9 MB for 30 s of audio); the wire stream itself is incremental.
word_timestamps and speed != 1.0 are not available in this mode.

2.8 KiB Raw Blame History Unescape Escape