Files
vllm-omni/vllm_omni/deploy/qwen3_tts.yaml
T
tzhouamandClaude Fable 5.1 35e43b9983 [Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform)
  selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed
  (builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes
  like vLLM does). NUMA-based memory queries, one logical device, no graph
  wrapper (sentinel type for isinstance checks), inductor opt-in via
  VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported.
* vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose
  vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner /
  CPUGenerationModelRunner compose the Omni runners with CPUModelRunner
  (no graphs, no async omni output; sleep/wake return error ACKs).
* patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder
  and prompt-embeds dtype follow the model dtype (float32/float16 on CPU).
* vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async
  scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges).
* requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md.

Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads):
Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms,
RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline
committed under benchmarks/tts/edge_results/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-09 11:02:14 +08:00

177 lines
7.1 KiB
YAML

# Qwen3-TTS deploy: talker → code2wav via shared-memory chunk streaming.
# Verified on 1x H100.
#
# Fields omitted from a stage fall back to StageDeployConfig defaults (see
# vllm_omni/config/stage_config.py). Notable choices for this model:
# * Stage 0 (talker) sets max_num_batched_tokens=512 for async-chunk
# latency tuning (not correctness) — small per-step batches keep
# first-chunk latency low.
# * Stage 1 (code2wav) sets max_num_batched_tokens=65536 for correctness:
# codec prefill length (Q * num_frames) exceeds the 32k default.
# * Stage 0 does not set enforce_eager — talker runs cudagraph by default.
# Stage 1 defaults to false so Code2Wav can use its inner cudagraph;
# set true to disable that path in eager mode.
async_chunk: true
connectors:
connector_of_shared_memory:
name: SharedMemoryConnector
extra:
codec_streaming: true
connector_get_sleep_s: 0.01
connector_get_max_wait_first_chunk: 3000
connector_get_max_wait: 300
# Must match the decoder sliding attention window.
codec_chunk_frames: 25
codec_left_context_frames: 72
# Emit only the first audio chunk early, then return to codec_chunk_frames.
initial_codec_chunk_frames: 1
# Chunk ramp (opt-in): gradual size increase for first N chunks to
# eliminate the IC→steady cliff (1→25). Uncomment to enable.
# Each entry is the size of that chunk index; indices past the table
# fall back to codec_chunk_frames. When enabled, ramp overrides
# initial_codec_chunk_frames (including the per-request API field).
# Example:
# codec_chunk_ramp: [4, 4, 8, 16, 25]
# Adaptive chunk ramp (opt-in, takes precedence over fixed codec_chunk_ramp):
# Chunk 0 uses dynamic IC; chunk 1+ sizes from live buffer-depth feedback.
# Self-tunes across hardware (NPU/NVIDIA) and load conditions.
# codec_chunk_adaptive: true
# codec_chunk_min_frames: 2
# Code2Wav decode + transfer margin
# codec_chunk_safety_margin_ms: 50
# EWMA->delta divisor (delta=max(delta_min, int(ewma/divisor))); higher=gentler ramp
# codec_chunk_ramp_divisor: 15
# min chunk-size growth per emit
# codec_chunk_ramp_delta_min: 2
# Decoder CUDA Graph sizing:
# * async_chunk=true derives all stateful ICL/xvec shapes from
# codec_chunk_ramp when enabled, otherwise from
# initial_codec_chunk_frames, codec_chunk_frames, and sliding_window;
# decode_cudagraph_capture_sizes is ignored in this mode.
# * async_chunk=false captures stateless shapes 150 and
# decode_chunk_frames + decode_left_context_frames (325 by default).
# Extra stateless shapes may be added (union, not intersection) with:
# decode_cudagraph_capture_sizes: [97, 400]
# Graph batch size defaults to 1. Add larger buckets only when needed:
# decode_cudagraph_batch_sizes: [1, 2, 4]
# Limit eager batches to reduce temporary stateful cache allocations; CUDA Graph batching uses the buckets above.
decode_batch_max_size: 4
stages:
- stage_id: 0
max_num_seqs: 64
gpu_memory_utilization: 0.3
trust_remote_code: true
# Prefix caching on by default. The HS opt-out (talker only consumes the
# last token's hidden state) and deferred mm-output writes added in this
# PR make per-step bookkeeping negligible, and any text repetition across
# requests (same voice or shared ref_audio) is a net throughput win.
# Disable explicitly if memory headroom is tight on a single-GPU deploy.
enable_prefix_caching: false
async_scheduling: true
max_num_batched_tokens: 32768
max_model_len: 4096
devices: "0"
output_connectors:
to_stage_1: connector_of_shared_memory
default_sampling_params:
temperature: 0.9
top_k: 50
max_tokens: 4096
# Official qwen-tts generates the talker with min_new_tokens=2; without
# it, checkpoints that sample codec EOS on the first frame return empty
# audio (HTTP 400). See #4962.
min_tokens: 2
repetition_penalty: 1.05
# Suppress codebook-0 silence tokens for the first N decode frames of a
# request (#4966). Applies to every request on this stage, so intended
# for x-vector-only (Base voice-clone) deploys. 0 = off.
silence_ban_frames: 0
subtalker_sampling_params:
do_sample: true
temperature: 0.9
top_k: 50
top_p: 1.0
- stage_id: 1
# Tuned for Base voice clone; CustomVoice / VoiceDesign are TTFA-optimal at 1.
max_num_seqs: 64
gpu_memory_utilization: 0.3
enforce_eager: false
trust_remote_code: true
enable_prefix_caching: false
async_scheduling: true
# Must be divisible by num_code_groups and cover (left_context + chunk).
# Prefill length is Q * num_frames (e.g. 16 * 2148 = 34368); keep
# headroom past 32k.
max_num_batched_tokens: 65536
# async_chunk appends windows per step; max_model_len must cover the
# accumulated flat codec stream.
max_model_len: 65536
devices: "0"
input_connectors:
from_stage_0: connector_of_shared_memory
default_sampling_params:
temperature: 0.0
top_p: 1.0
top_k: -1
max_tokens: 65536
repetition_penalty: 1.0
platforms:
rocm:
stages:
- stage_id: 1
# MIOpen Conv1d cannot be invoked while vLLM captures the outer
# Code2Wav model graph. Keep CUDA's graph-enabled default, but run
# Stage 1 eager on ROCm so decoder convolutions execute normally.
enforce_eager: true
npu:
stages:
- stage_id: 0
enforce_eager: false
compilation_config:
cudagraph_mode: PIECEWISE
- stage_id: 1
compilation_config:
cudagraph_mode: PIECEWISE
# CPU OmniPlatform (VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel). vLLM's
# CpuPlatform already forces async_scheduling=False, no CUDA graphs and
# CPU_ATTN; the values below size the two stages for a CPU box. Nested
# layout: engine args under ``engine_args``, placement under ``runtime``.
# gpu_memory_utilization is read by the CPU worker as a NUMA-node memory
# fraction, so an absolute kv_cache_memory_bytes is set instead.
# Thread ranges are per-stage (disjoint) so the two workers do not pin the
# same cores; deploy_profile="auto" derives them from the probed topology.
cpu:
stages:
- stage_id: 0
engine_args:
enforce_eager: true
async_scheduling: false
max_num_seqs: 2
max_num_batched_tokens: 2048
max_model_len: 2048
gpu_memory_utilization: 0.05
dtype: bfloat16
kv_cache_memory_bytes: 4294967296
runtime:
devices: "cpu"
env:
VLLM_CPU_OMP_THREADS_BIND: "0-11"
- stage_id: 1
engine_args:
enforce_eager: true
async_scheduling: false
max_num_seqs: 2
max_num_batched_tokens: 8192
max_model_len: 8192
gpu_memory_utilization: 0.05
dtype: bfloat16
kv_cache_memory_bytes: 2147483648
runtime:
devices: "cpu"
env:
VLLM_CPU_OMP_THREADS_BIND: "12-15"