0 Fork 0

Files

T

tzhouamandClaude Fable 5.1 35e43b9983 [Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)

* vllm_omni/platforms/cpu/platform.py: CPUOmniPlatform(OmniPlatform, CpuPlatform)
  selected when VLLM_TARGET_DEVICE=cpu or a +cpu vLLM wheel is installed
  (builtin plugin "cpu"; explicit CPU target short-circuits accelerator probes
  like vLLM does). NUMA-based memory queries, one logical device, no graph
  wrapper (sentinel type for isinstance checks), inductor opt-in via
  VLLM_OMNI_CPU_INDUCTOR=1, diffusion stages unsupported.
* vllm_omni/platforms/cpu/worker: CPUARWorker / CPUGenerationWorker compose
  vLLM's CPUWorker with the Omni mixin (XPU pattern); CPUARModelRunner /
  CPUGenerationModelRunner compose the Omni runners with CPUModelRunner
  (no graphs, no async omni output; sleep/wake return error ACKs).
* patch.py skips the CuMem patch on CPU-only torch; Qwen3-TTS speaker encoder
  and prompt-embeds dtype follow the model dtype (float32/float16 on CPU).
* vllm_omni/deploy/qwen3_tts.yaml: `platforms: cpu:` overlay (eager, no async
  scheduling, absolute kv_cache_memory_bytes, disjoint OMP thread ranges).
* requirements/cpu.txt (+psutil, intel-openmp), docs/getting_started/installation/cpu.md.

Measured (this x86 Xeon 8480C host, vllm 0.28.0+cpu, bf16, 12+4 threads):
Qwen3-TTS 0.6B CustomVoice end-to-end on CPU, 12 prompts: TTFA p50 292 ms,
RTF ~1.7 (25-frame chunk ~4.6 s), init 185 s. Result JSON + init timeline
committed under benchmarks/tts/edge_results/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

2026-09-09 11:02:14 +08:00

stream_latency

[Edge][WP3] Add CPU OmniPlatform (AR + generation stages on vLLM's CPU backend)

2026-09-09 11:02:14 +08:00

README.md

[Edge][WP0] Add offline streaming-latency benchmark for chunked TTS

2026-09-09 10:24:53 +08:00

README.md

Edge benchmark results

Committed JSON produced by the edge branch benchmarks (kept small; WAVs are never committed):

stream_latency/<model>/<deploy>/<timestamp>[_tag].json — stream_latency_bench.py output (schema 1): meta (model, deploy config, git SHA, GPU), init (init_s, optional init timeline), resources (peak RSS, peak GPU MiB, sampling mode), requests[] (per-chunk arrivals, TTFA, playback-start, stall at TTFA, RTF streamed / total) and summary (p50/p90/max).
init_profile/<timestamp>/summary.json — init_profile.py comparison of initialization phases per configuration.
hw_emulation/, scheduling/ — per-work-package comparison tables.

Metric definitions live in benchmarks/tts/stream_latency_metrics.py.