[Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)
* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform) selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed (builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes like vLLM does). NUMA-based memory queries, one logical device, no graph wrapper (sentinel type for isinstance checks), inductor opt-in via VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported. * vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner / CPUGenerationModelRunner compose the Omni runners with CPUModelRunner (no graphs, no async omni output; sleep/wake return error ACKs). * patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder and prompt-embeds dtype follow the model dtype (float32/float16 on CPU). * vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges). * requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md. Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads): Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms, RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline committed under benchmarks/tts/edge_results/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
55192de984
commit
35e43b9983
18 files changed
+1740
-7
No files matched your search
+10
-2
@@ -328,8 +328,8 @@ def _patch_chat_template_registry():
|
||||
)
|
||||
|
||||
if "qwen3_omni_moe" not in _MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK:
|
||||
_MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK["qwen3_omni_moe"] = (
|
||||
lambda _: CHAT_TEMPLATES_DIR / "template_chatml.jinja"
|
||||
_MODEL_TYPE_TO_CHAT_TEMPLATE_FALLBACK["qwen3_omni_moe"] = lambda _: (
|
||||
CHAT_TEMPLATES_DIR / "template_chatml.jinja"
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
@@ -529,6 +529,14 @@ _patch_inductor_factorable_divisibility()
|
||||
# (not a slot or C extension). If CuMemAllocator is rewritten in C/Cython,
|
||||
# this monkey-patch will silently become a no-op.
|
||||
def _patch_cumem_free_callback_cuda() -> None:
|
||||
try:
|
||||
import torch
|
||||
|
||||
if torch.version.cuda is None and getattr(torch.version, "hip", None) is None:
|
||||
_PATCH_LOGGER.debug("[cumem-cuda] CPU-only torch; skipping patch")
|
||||
return
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
from vllm.device_allocator.cumem import CuMemAllocator
|
||||
except ImportError:
|
||||
|
||||
Reference in new issue
Block a user