[Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform) selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed (builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes like vLLM does). NUMA-based memory queries, one logical device, no graph wrapper (sentinel type for isinstance checks), inductor opt-in via VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported. * vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner / CPUGenerationModelRunner compose the Omni runners with CPUModelRunner (no graphs, no async omni output; sleep/wake return error ACKs). * patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder and prompt-embeds dtype follow the model dtype (float32/float16 on CPU). * vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges). * requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md. Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads): Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms, RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline committed under benchmarks/tts/edge_results/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
55192de984
commit
35e43b9983
18 files changed
+1740
-7
No files matched your search
@@ -0,0 +1,56 @@
|
||||
# CPU
|
||||
|
||||
vLLM-Omni can run **autoregressive and generation stages** (for example the
|
||||
Qwen3-TTS talker → code2wav pipeline) on vLLM's CPU backend. Diffusion stages
|
||||
are not supported on CPU.
|
||||
|
||||
## Requirements
|
||||
|
||||
- x86-64 with AVX-512 (bf16 needs `avx512_bf16` or AMX) or aarch64 with NEON
|
||||
(`fp16`/`bf16` extensions recommended). See the vLLM CPU installation page
|
||||
for the supported ISA matrix.
|
||||
- A vLLM **CPU** build (`+cpu` wheel). The CUDA wheel does not contain the
|
||||
CPU kernels.
|
||||
- Python 3.10–3.13.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
# vLLM CPU wheel (x86-64 example; see vLLM docs for aarch64 and per-commit wheels)
|
||||
uv pip install --torch-backend cpu \
|
||||
https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0+cpu-cp38-abi3-manylinux_2_34_x86_64.whl
|
||||
# vLLM-Omni without re-resolving torch, then its CPU requirements
|
||||
VLLM_TARGET_DEVICE=cpu uv pip install --torch-backend cpu --no-deps -e .
|
||||
VLLM_TARGET_DEVICE=cpu uv pip install --torch-backend cpu -r requirements/cpu.txt
|
||||
```
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
export VLLM_TARGET_DEVICE=cpu # selects the CPU OmniPlatform
|
||||
export LD_PRELOAD=$(python -c "import glob,sys; print(glob.glob(sys.prefix+'/lib/libiomp5.so')[0])")
|
||||
# one utterance, Qwen3-TTS 0.6B, CPU overlay from vllm_omni/deploy/qwen3_tts.yaml (platforms: cpu:)
|
||||
python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type CustomVoice
|
||||
# or let the hardware probe size everything (threads, KV budget, dtype, chunking):
|
||||
python benchmarks/tts/stream_latency_bench.py --model Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice \
|
||||
--query-type CustomVoice --deploy-profile auto
|
||||
```
|
||||
|
||||
What the CPU platform changes (`vllm_omni/platforms/cpu/platform.py`):
|
||||
|
||||
| Item | CPU behaviour |
|
||||
|---|---|
|
||||
| Workers | `CPUARWorker` / `CPUGenerationWorker` compose vLLM's `CPUWorker` with the Omni runners |
|
||||
| Scheduling | `async_scheduling=False` (forced by vLLM's `CpuPlatform`), no CUDA graphs |
|
||||
| Memory | `gpu_memory_utilization` is a NUMA-node memory fraction; prefer `kv_cache_memory_bytes` |
|
||||
| Threads | per-stage `VLLM_CPU_OMP_THREADS_BIND` ranges (`platforms: cpu:` overlay or `deploy_profile="auto"`) |
|
||||
| Compile | code-predictor `torch.compile` off by default; `VLLM_OMNI_CPU_INDUCTOR=1` enables it |
|
||||
| dtype | bf16 with `avx512_bf16`/AMX or ARM `bf16`, otherwise `float32` (`deploy_profile="auto"`) |
|
||||
| Sleep mode | not supported (returns an error ACK) |
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
VLLM_TARGET_DEVICE=cpu pytest tests/platforms/test_cpu_platform.py tests/worker/test_cpu_ar_model_runner.py -m "core_model and cpu"
|
||||
```
|
||||
Reference in new issue
Block a user