tzhouamandClaude Fable 5.1
35e43b9983
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform) selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed (builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes like vLLM does). NUMA-based memory queries, one logical device, no graph wrapper (sentinel type for isinstance checks), inductor opt-in via VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported. * vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner / CPUGenerationModelRunner compose the Omni runners with CPUModelRunner (no graphs, no async omni output; sleep/wake return error ACKs). * patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder and prompt-embeds dtype follow the model dtype (float32/float16 on CPU). * vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges). * requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md. Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads): Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms, RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline committed under benchmarks/tts/edge_results/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
57 lines
2.5 KiB
Markdown
57 lines
2.5 KiB
Markdown
# CPU
|
||
|
||
vLLM-Omni can run **autoregressive and generation stages** (for example the
|
||
Qwen3-TTS talker → code2wav pipeline) on vLLM's CPU backend. Diffusion stages
|
||
are not supported on CPU.
|
||
|
||
## Requirements
|
||
|
||
- x86-64 with AVX-512 (bf16 needs `avx512_bf16` or AMX) or aarch64 with NEON
|
||
(`fp16`/`bf16` extensions recommended). See the vLLM CPU installation page
|
||
for the supported ISA matrix.
|
||
- A vLLM **CPU** build (`+cpu` wheel). The CUDA wheel does not contain the
|
||
CPU kernels.
|
||
- Python 3.10–3.13.
|
||
|
||
## Install
|
||
|
||
```bash
|
||
uv venv --python 3.12 && source .venv/bin/activate
|
||
# vLLM CPU wheel (x86-64 example; see vLLM docs for aarch64 and per-commit wheels)
|
||
uv pip install --torch-backend cpu \
|
||
https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0+cpu-cp38-abi3-manylinux_2_34_x86_64.whl
|
||
# vLLM-Omni without re-resolving torch, then its CPU requirements
|
||
VLLM_TARGET_DEVICE=cpu uv pip install --torch-backend cpu --no-deps -e .
|
||
VLLM_TARGET_DEVICE=cpu uv pip install --torch-backend cpu -r requirements/cpu.txt
|
||
```
|
||
|
||
## Run
|
||
|
||
```bash
|
||
export VLLM_TARGET_DEVICE=cpu # selects the CPU OmniPlatform
|
||
export LD_PRELOAD=$(python -c "import glob,sys; print(glob.glob(sys.prefix+'/lib/libiomp5.so')[0])")
|
||
# one utterance, Qwen3-TTS 0.6B, CPU overlay from vllm_omni/deploy/qwen3_tts.yaml (platforms: cpu:)
|
||
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
|
||
# or let the hardware probe size everything (threads, KV budget, dtype, chunking):
|
||
python benchmarks/tts/stream_latency_bench.py --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice \
|
||
--query-type CustomVoice --deploy-profile auto
|
||
```
|
||
|
||
What the CPU platform changes (`vllm_omni/platforms/cpu/platform.py`):
|
||
|
||
| Item | CPU behaviour |
|
||
|---|---|
|
||
| Workers | `CPUARWorker` / `CPUGenerationWorker` compose vLLM's `CPUWorker` with the Omni runners |
|
||
| Scheduling | `async_scheduling=False` (forced by vLLM's `CpuPlatform`), no CUDA graphs |
|
||
| Memory | `gpu_memory_utilization` is a NUMA-node memory fraction; prefer `kv_cache_memory_bytes` |
|
||
| Threads | per-stage `VLLM_CPU_OMP_THREADS_BIND` ranges (`platforms: cpu:` overlay or `deploy_profile="auto"`) |
|
||
| Compile | code-predictor `torch.compile` off by default; `VLLM_OMNI_CPU_INDUCTOR=1` enables it |
|
||
| dtype | bf16 with `avx512_bf16`/AMX or ARM `bf16`, otherwise `float32` (`deploy_profile="auto"`) |
|
||
| Sleep mode | not supported (returns an error ACK) |
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
VLLM_TARGET_DEVICE=cpu pytest tests/platforms/test_cpu_platform.py tests/worker/test_cpu_ar_model_runner.py -m "core_model and cpu"
|
||
```
|