tzhouamandClaude Fable 5.1
35e43b9983
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform) selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed (builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes like vLLM does). NUMA-based memory queries, one logical device, no graph wrapper (sentinel type for isinstance checks), inductor opt-in via VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported. * vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner / CPUGenerationModelRunner compose the Omni runners with CPUModelRunner (no graphs, no async omni output; sleep/wake return error ACKs). * patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder and prompt-embeds dtype follow the model dtype (float32/float16 on CPU). * vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges). * requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md. Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads): Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms, RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline committed under benchmarks/tts/edge_results/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2.5 KiB
2.5 KiB
CPU
vLLM-Omni can run autoregressive and generation stages (for example the Qwen3-TTS talker → code2wav pipeline) on vLLM's CPU backend. Diffusion stages are not supported on CPU.
Requirements
- x86-64 with AVX-512 (bf16 needs
avx512_bf16or AMX) or aarch64 with NEON (fp16/bf16extensions recommended). See the vLLM CPU installation page for the supported ISA matrix. - A vLLM CPU build (
+cpuwheel). The CUDA wheel does not contain the CPU kernels. - Python 3.10–3.13.
Install
uv venv --python 3.12 && source .venv/bin/activate
# vLLM CPU wheel (x86-64 example; see vLLM docs for aarch64 and per-commit wheels)
uv pip install --torch-backend cpu \
https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0+cpu-cp38-abi3-manylinux_2_34_x86_64.whl
# vLLM-Omni without re-resolving torch, then its CPU requirements
VLLM_TARGET_DEVICE=cpu uv pip install --torch-backend cpu --no-deps -e .
VLLM_TARGET_DEVICE=cpu uv pip install --torch-backend cpu -r requirements/cpu.txt
Run
export VLLM_TARGET_DEVICE=cpu # selects the CPU OmniPlatform
export LD_PRELOAD=$(python -c "import glob,sys; print(glob.glob(sys.prefix+'/lib/libiomp5.so')[0])")
# one utterance, Qwen3-TTS 0.6B, CPU overlay from vllm_omni/deploy/qwen3_tts.yaml (platforms: cpu:)
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
# or let the hardware probe size everything (threads, KV budget, dtype, chunking):
python benchmarks/tts/stream_latency_bench.py --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice \
--query-type CustomVoice --deploy-profile auto
What the CPU platform changes (vllm_omni/platforms/cpu/platform.py):
| Item | CPU behaviour |
|---|---|
| Workers | CPUARWorker / CPUGenerationWorker compose vLLM's CPUWorker with the Omni runners |
| Scheduling | async_scheduling=False (forced by vLLM's CpuPlatform), no CUDA graphs |
| Memory | gpu_memory_utilization is a NUMA-node memory fraction; prefer kv_cache_memory_bytes |
| Threads | per-stage VLLM_CPU_OMP_THREADS_BIND ranges (platforms: cpu: overlay or deploy_profile="auto") |
| Compile | code-predictor torch.compile off by default; VLLM_OMNI_CPU_INDUCTOR=1 enables it |
| dtype | bf16 with avx512_bf16/AMX or ARM bf16, otherwise float32 (deploy_profile="auto") |
| Sleep mode | not supported (returns an error ACK) |
Tests
VLLM_TARGET_DEVICE=cpu pytest tests/platforms/test_cpu_platform.py tests/worker/test_cpu_ar_model_runner.py -m "core_model and cpu"