tzhouamandClaude Opus 5
ca4dd105e4
`CpuPlatform.check_and_update_config` ends with a block whose own comment
reads "Lagecy setting":
env_key = "VLLM_CPU_KVCACHE_SPACE"
if env_key in os.environ and os.environ[env_key] != "":
cache_config.kv_cache_memory_bytes = int(os.environ[env_key]) * GiB
It runs unconditionally, so `--kv-cache-memory-bytes` is silently discarded
whenever the env var is set -- which it has to be on CPU, or the engine tries
to claim most of the machine. Two consequences for edge serving: the budget
can only be expressed in whole GiB (parsed with `int()`, so the smallest
non-zero cache is 1 GiB), and vllm-omni's per-stage `kv_cache_memory_bytes`,
which is how the deploy YAMLs express per-stage budgets, has no effect at all
on CPU.
Verified by allocation rather than by memory, because allocation cannot drift:
with the env at 1 GiB, asking for 128 MiB produced the same 17 881 cache
tokens as asking for nothing, and with the env at 4 GiB it produced 71 527 --
the explicit value never applied. After restoring the obvious precedence
(explicit wins, env stays the default for callers that set nothing) the same
request yields 2 231 tokens, the 128 MiB it asked for.
The Qwen3-TTS CPU overlay now sizes stage 0 from the model -- 112 KiB/token,
so 2 seqs at 2048 tokens needs 560 MiB, not 4 GiB. Stage 1 is left alone on
purpose: Code2Wav is a generation stage whose cache is not a plain attention
KV, and under-sizing makes vLLM preempt by recompute, which on a single-stream
edge deployment is a stall.
What this saves in resident memory is deliberately *not* claimed. VmRSS on
this host moves 1.2-2.0 GB between identical runs, because it counts the
mmap'd checkpoint's file pages; an earlier 3.1 GB figure came from a
comparison that changed the model file as well as the budget and does not
stand. RssAnon is the metric to use and that measurement is still in flight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
195 lines
8.4 KiB
YAML
195 lines
8.4 KiB
YAML
# Qwen3-TTS deploy: talker → code2wav via shared-memory chunk streaming.
|
|
# Verified on 1x H100.
|
|
#
|
|
# Fields omitted from a stage fall back to StageDeployConfig defaults (see
|
|
# vllm_omni/config/stage_config.py). Notable choices for this model:
|
|
# * Stage 0 (talker) sets max_num_batched_tokens=512 for async-chunk
|
|
# latency tuning (not correctness) — small per-step batches keep
|
|
# first-chunk latency low.
|
|
# * Stage 1 (code2wav) sets max_num_batched_tokens=65536 for correctness:
|
|
# codec prefill length (Q * num_frames) exceeds the 32k default.
|
|
# * Stage 0 does not set enforce_eager — talker runs cudagraph by default.
|
|
# Stage 1 defaults to false so Code2Wav can use its inner cudagraph;
|
|
# set true to disable that path in eager mode.
|
|
async_chunk: true
|
|
|
|
connectors:
|
|
connector_of_shared_memory:
|
|
name: SharedMemoryConnector
|
|
extra:
|
|
codec_streaming: true
|
|
connector_get_sleep_s: 0.01
|
|
connector_get_max_wait_first_chunk: 3000
|
|
connector_get_max_wait: 300
|
|
# Must match the decoder sliding attention window.
|
|
codec_chunk_frames: 25
|
|
codec_left_context_frames: 72
|
|
# Emit only the first audio chunk early, then return to codec_chunk_frames.
|
|
initial_codec_chunk_frames: 1
|
|
# Chunk ramp (opt-in): gradual size increase for first N chunks to
|
|
# eliminate the IC→steady cliff (1→25). Uncomment to enable.
|
|
# Each entry is the size of that chunk index; indices past the table
|
|
# fall back to codec_chunk_frames. When enabled, ramp overrides
|
|
# initial_codec_chunk_frames (including the per-request API field).
|
|
# Example:
|
|
# codec_chunk_ramp: [4, 4, 8, 16, 25]
|
|
# Adaptive chunk ramp (opt-in, takes precedence over fixed codec_chunk_ramp):
|
|
# Chunk 0 uses dynamic IC; chunk 1+ sizes from live buffer-depth feedback.
|
|
# Self-tunes across hardware (NPU/NVIDIA) and load conditions.
|
|
# codec_chunk_adaptive: true
|
|
# codec_chunk_min_frames: 2
|
|
# Code2Wav decode + transfer margin
|
|
# codec_chunk_safety_margin_ms: 50
|
|
# EWMA->delta divisor (delta=max(delta_min, int(ewma/divisor))); higher=gentler ramp
|
|
# codec_chunk_ramp_divisor: 15
|
|
# min chunk-size growth per emit
|
|
# codec_chunk_ramp_delta_min: 2
|
|
# Decoder CUDA Graph sizing:
|
|
# * async_chunk=true derives all stateful ICL/xvec shapes from
|
|
# codec_chunk_ramp when enabled, otherwise from
|
|
# initial_codec_chunk_frames, codec_chunk_frames, and sliding_window;
|
|
# decode_cudagraph_capture_sizes is ignored in this mode.
|
|
# * async_chunk=false captures stateless shapes 150 and
|
|
# decode_chunk_frames + decode_left_context_frames (325 by default).
|
|
# Extra stateless shapes may be added (union, not intersection) with:
|
|
# decode_cudagraph_capture_sizes: [97, 400]
|
|
# Graph batch size defaults to 1. Add larger buckets only when needed:
|
|
# decode_cudagraph_batch_sizes: [1, 2, 4]
|
|
# Limit eager batches to reduce temporary stateful cache allocations; CUDA Graph batching uses the buckets above.
|
|
decode_batch_max_size: 4
|
|
|
|
stages:
|
|
- stage_id: 0
|
|
max_num_seqs: 64
|
|
gpu_memory_utilization: 0.3
|
|
trust_remote_code: true
|
|
# Prefix caching on by default. The HS opt-out (talker only consumes the
|
|
# last token's hidden state) and deferred mm-output writes added in this
|
|
# PR make per-step bookkeeping negligible, and any text repetition across
|
|
# requests (same voice or shared ref_audio) is a net throughput win.
|
|
# Disable explicitly if memory headroom is tight on a single-GPU deploy.
|
|
enable_prefix_caching: false
|
|
async_scheduling: true
|
|
max_num_batched_tokens: 32768
|
|
max_model_len: 4096
|
|
devices: "0"
|
|
output_connectors:
|
|
to_stage_1: connector_of_shared_memory
|
|
default_sampling_params:
|
|
temperature: 0.9
|
|
top_k: 50
|
|
max_tokens: 4096
|
|
# Official qwen-tts generates the talker with min_new_tokens=2; without
|
|
# it, checkpoints that sample codec EOS on the first frame return empty
|
|
# audio (HTTP 400). See #4962.
|
|
min_tokens: 2
|
|
repetition_penalty: 1.05
|
|
# Suppress codebook-0 silence tokens for the first N decode frames of a
|
|
# request (#4966). Applies to every request on this stage, so intended
|
|
# for x-vector-only (Base voice-clone) deploys. 0 = off.
|
|
silence_ban_frames: 0
|
|
subtalker_sampling_params:
|
|
do_sample: true
|
|
temperature: 0.9
|
|
top_k: 50
|
|
top_p: 1.0
|
|
|
|
- stage_id: 1
|
|
# Tuned for Base voice clone; CustomVoice / VoiceDesign are TTFA-optimal at 1.
|
|
max_num_seqs: 64
|
|
gpu_memory_utilization: 0.3
|
|
enforce_eager: false
|
|
trust_remote_code: true
|
|
enable_prefix_caching: false
|
|
async_scheduling: true
|
|
# Must be divisible by num_code_groups and cover (left_context + chunk).
|
|
# Prefill length is Q * num_frames (e.g. 16 * 2148 = 34368); keep
|
|
# headroom past 32k.
|
|
max_num_batched_tokens: 65536
|
|
# async_chunk appends windows per step; max_model_len must cover the
|
|
# accumulated flat codec stream.
|
|
max_model_len: 65536
|
|
devices: "0"
|
|
input_connectors:
|
|
from_stage_0: connector_of_shared_memory
|
|
default_sampling_params:
|
|
temperature: 0.0
|
|
top_p: 1.0
|
|
top_k: -1
|
|
max_tokens: 65536
|
|
repetition_penalty: 1.0
|
|
|
|
platforms:
|
|
rocm:
|
|
stages:
|
|
- stage_id: 1
|
|
# MIOpen Conv1d cannot be invoked while vLLM captures the outer
|
|
# Code2Wav model graph. Keep CUDA's graph-enabled default, but run
|
|
# Stage 1 eager on ROCm so decoder convolutions execute normally.
|
|
enforce_eager: true
|
|
npu:
|
|
stages:
|
|
- stage_id: 0
|
|
enforce_eager: false
|
|
compilation_config:
|
|
cudagraph_mode: PIECEWISE
|
|
- stage_id: 1
|
|
compilation_config:
|
|
cudagraph_mode: PIECEWISE
|
|
# CPU OmniPlatform (VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel). vLLM's
|
|
# CpuPlatform already forces async_scheduling=False, no CUDA graphs and
|
|
# CPU_ATTN; the values below size the two stages for a CPU box. Nested
|
|
# layout: engine args under ``engine_args``, placement under ``runtime``.
|
|
# gpu_memory_utilization is read by the CPU worker as a NUMA-node memory
|
|
# fraction, so an absolute kv_cache_memory_bytes is set instead.
|
|
# Thread ranges are per-stage (disjoint) so the two workers do not pin the
|
|
# same cores; deploy_profile="auto" derives them from the probed topology.
|
|
cpu:
|
|
stages:
|
|
- stage_id: 0
|
|
engine_args:
|
|
enforce_eager: true
|
|
async_scheduling: false
|
|
max_num_seqs: 2
|
|
max_num_batched_tokens: 2048
|
|
max_model_len: 2048
|
|
gpu_memory_utilization: 0.05
|
|
dtype: bfloat16
|
|
# Sized from the model rather than from a server default: the
|
|
# talker is 28 layers x 8 KV heads x 128 head_dim x 2 (K,V) x 2 B
|
|
# = 112 KiB/token, so max_num_seqs 2 at max_model_len 2048 needs
|
|
# 560 MiB against the 4 GiB this used to ask for -- 7.3x the need.
|
|
# (vllm_omni/edge/kv_budget.py computes it from the HF config.)
|
|
#
|
|
# This only takes effect with the patch in vllm_omni/patch.py:
|
|
# CpuPlatform.check_and_update_config overwrites
|
|
# kv_cache_memory_bytes from VLLM_CPU_KVCACHE_SPACE unconditionally,
|
|
# so on CPU this key was dead. Verified by allocation: the same
|
|
# request went from 71 527 cache tokens to 2 231 once the explicit
|
|
# value won. What that saves in resident memory is *not* established
|
|
# -- RssAnon is the metric to use, since VmRSS here moves 1-2 GB
|
|
# between identical runs as the mmap'd checkpoint's pages come and go.
|
|
kv_cache_memory_bytes: 671088640
|
|
runtime:
|
|
devices: "cpu"
|
|
env:
|
|
VLLM_CPU_OMP_THREADS_BIND: "0-11"
|
|
- stage_id: 1
|
|
engine_args:
|
|
enforce_eager: true
|
|
async_scheduling: false
|
|
max_num_seqs: 2
|
|
max_num_batched_tokens: 8192
|
|
max_model_len: 8192
|
|
gpu_memory_utilization: 0.05
|
|
dtype: bfloat16
|
|
# Left at 2 GiB deliberately: Code2Wav is a generation stage and its
|
|
# cache geometry is not a plain attention KV, so the figure cannot be
|
|
# derived the way stage 0's can. Measure before shrinking it --
|
|
# under-sizing makes vLLM preempt by recompute, which is a stall.
|
|
kv_cache_memory_bytes: 2147483648
|
|
runtime:
|
|
devices: "cpu"
|
|
env:
|
|
VLLM_CPU_OMP_THREADS_BIND: "12-15"
|