diff --git a/docs/design/feature/async_chunk.md b/docs/design/feature/async_chunk.md index 068a5ac1b..df5a70e58 100644 --- a/docs/design/feature/async_chunk.md +++ b/docs/design/feature/async_chunk.md @@ -246,6 +246,33 @@ stages: from_stage_0: connector_of_shared_memory ``` +### Chunk Ramp and the Edge Profile + +The default schedule emits one 80 ms frame first (`initial_codec_chunk_frames: 1`) +and the next 25-frame chunk about 160 ms later, so a player that starts at the +first audio stalls for roughly 80 ms (measured on L20X: playback-start latency +116 ms vs TTFA 32 ms for the 1.7B model). `codec_chunk_ramp` replaces the +initial/steady pair with a growing schedule so the audio buffered ahead always +covers the wait for the next chunk: + +```yaml +connectors: + connector_of_shared_memory: + name: SharedMemoryConnector + extra: + codec_chunk_frames: 25 + codec_chunk_ramp: [2, 4, 8, 16, 25] # last entry == codec_chunk_frames + decode_batch_max_size: 1 # edge: 1-4 concurrent utterances +``` + +`vllm_omni/deploy/edge/qwen3_tts.yaml` bundles this ramp with single-device, +small-batch stage settings and concurrent stage init; select it with +`Omni(model=..., deploy_profile="edge")` or `vllm serve --omni --deploy-profile edge` +(`deploy_profile="auto"` additionally probes the hardware, see +`vllm_omni/edge/`). With the ramp the measured playback-start latency equals +TTFA (stall 0 ms, TTFA 41-44 ms) at unchanged RTF; results live under +`benchmarks/tts/edge_results/`. + ### Code2Wav Batch Configuration For optimal performance with async_chunk, the code2wav stage should be configured with batching: diff --git a/recipes/Qwen/Qwen3-TTS-Edge.md b/recipes/Qwen/Qwen3-TTS-Edge.md new file mode 100644 index 000000000..ef51bfafd --- /dev/null +++ b/recipes/Qwen/Qwen3-TTS-Edge.md @@ -0,0 +1,85 @@ +# Qwen3-TTS on edge-class hardware (edge/omni-edge branch) + +This recipe covers the edge deploy profile, the CPU platform, hardware +auto-adaptation, codec-token streaming with on-device decoding, and the +profiling tools added on the `edge/omni-edge` branch. Numbers quoted here were +measured on one L20X (48 GB) and a 2x56-core Xeon 8480C host on 2026-09-09; the +JSON files are committed under `benchmarks/tts/edge_results/` +(`python benchmarks/tts/edge_results/summarize.py` renders them). + +## 1. Edge profile (one GPU, small batch, ramped chunks) + +```bash +vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --omni --deploy-profile edge --trust-remote-code +# offline +python - <<'PY' +from vllm_omni import Omni +omni = Omni(model="Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice", deploy_profile="edge") +PY +``` + +`vllm_omni/deploy/edge/qwen3_tts.yaml` inherits the default deploy YAML and sets +`codec_chunk_ramp: [2, 4, 8, 16, 25]`, `decode_batch_max_size: 1`, +`max_num_seqs: 4`, both stages on one device with 0.25/0.15 memory fractions, +and `orchestrator: parallel_stage_init: true` (both stages initialise +concurrently; 94.6 s vs 152 s warm init for 1.7B on L20X). + +| model | deploy | TTFA p50 | playback start p50 | stall at TTFA | RTF | +|---|---|---|---|---|---| +| 1.7B | default | 32 ms | 116 ms | 85 ms | 0.10 | +| 1.7B | edge | 44 ms | 44 ms | 0 ms | 0.10 | +| 0.6B | default | 28 ms | 95 ms | 67 ms | 0.10 | +| 0.6B | edge | 41 ms | 41 ms | 0 ms | 0.10 | + +Playback-start latency = max over chunks of (arrival time − audio already +buffered); the ramp removes the initial 80 ms gap at the cost of ~12 ms TTFA +(first chunk is 2 frames instead of 1). + +## 2. Hardware auto-adaptation (`deploy_profile="auto"`) + +`vllm_omni/edge/hardware_probe.py` classifies the host (jetson / arm64_cpu / +x86_cpu / cuda_discrete / npu_phone), `adapt.derive_overrides` derives dtype, +absolute KV budgets (never allowing recompute preemption: +`max_num_seqs x max_model_len` talker tokens must fit), thread binding, eager +mode and adaptive-chunk parameters, and `deploy/edge/hardware/.yaml` +overlays pin per-class values. `VLLM_OMNI_HW_PROFILE=` overrides the probe +(used by `benchmarks/tts/hw_emulation.py` to emulate other classes on one host). + +## 3. CPU platform + +Install the `+cpu` vLLM wheel (see `docs/getting_started/installation/cpu.md`), +then: + +```bash +VLLM_TARGET_DEVICE=cpu python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py \ + --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --query-type CustomVoice --streaming +``` + +The `platforms: cpu:` overlay in `vllm_omni/deploy/qwen3_tts.yaml` sizes the two +stages and binds disjoint OpenMP thread ranges; `benchmarks/tts/cpu_thread_sweep.py` +measures 8/16/32 threads. On the 8480C host: 8 threads TTFA 426 ms, RTF 2.8; +12+4 threads RTF 1.7 (bf16/AMX). The CPU path is usable for offline synthesis, +not for real-time streaming on this class of CPU. + +## 4. Codec-token streaming + on-device decoder + +Run the talker-only deployment and decode on the client (CPU or an exported +decoder): + +```bash +vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --omni --trust-remote-code \ + --deploy-config vllm_omni/deploy/qwen3_tts_talker_only.yaml +python examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py \ + --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice --text "..." --out out.wav +``` + +See `docs/serving/codec_token_streaming.md` for the wire format (binary frames +with a `/manifest.json`. + +## 5. Profiling tools + +- `benchmarks/tts/stream_latency_bench.py` — TTFA / playback-start / RTF / memory JSON; `--step-stats` adds per-step overhead attribution (scheduler, engine step, chunk hop, orchestrator dispatch) as a share of the 80 ms frame budget. +- `benchmarks/tts/init_profile.py` — init-phase timeline per configuration (`VLLM_OMNI_INIT_TIMELINE`). +- `benchmarks/tts/hw_emulation.py`, `benchmarks/tts/cpu_thread_sweep.py`, `benchmarks/tts/codec_stream_e2e.py`.