Files
vllm-omni/vllm_omni/deploy/qwen3_tts_talker_only.yaml
T
tzhouamandClaude Fable 5.1 49a839ecf9 [Edge][WP5] Add codec-token streaming mode with on-device decoding
* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage,
  final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml;
  a terminal talker publishes its per-step codes.audio rows to the client
  (omni_client_multimodal_output_keys).
* /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams
  codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows,
  vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel
  message aborts an in-flight generation for both codec and PCM sessions
  (single reader task + session inbox replaces direct socket reads); optional
  session.config.seed; audio.done gains "cancelled" only when it happened.
* serving_speech._generate_codec_chunks aligns the cumulative code tensor to
  output token ids (drops prefill placeholder/EOS rows) and emits deltas.
* TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16
  codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz).
* examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py
  decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame
  context, cancel/reset), codec_parity_check.py and
  benchmarks/tts/codec_stream_e2e.py compare against the two-stage server.
* docs/serving/codec_token_streaming.md; CPU unit tests for the wire format,
  alignment, session config and handler (frames, unsupported adapter, cancel);
  L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-09 11:10:13 +08:00

31 lines
961 B
YAML

# Qwen3-TTS talker-only deployment: stage 0 is terminal and publishes codec
# frames (codes.audio, 16 codebooks @ 12.5 Hz) to the client instead of PCM.
# Used by the codec-token streaming mode of /v1/audio/speech/stream
# (session.config.output_mode: "codec_tokens") with decoding on the client
# device (see examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py).
# Stage settings mirror stage 0 of qwen3_tts.yaml.
pipeline: qwen3_tts_talker_only
async_chunk: false
stages:
- stage_id: 0
max_num_seqs: 64
gpu_memory_utilization: 0.3
trust_remote_code: true
enable_prefix_caching: false
async_scheduling: true
max_num_batched_tokens: 32768
max_model_len: 4096
devices: '0'
default_sampling_params:
temperature: 0.9
top_k: 50
max_tokens: 4096
min_tokens: 2
repetition_penalty: 1.05
silence_ban_frames: 0
subtalker_sampling_params:
do_sample: true
temperature: 0.9
top_k: 50
top_p: 1.0