tzhouamandClaude Opus 5
b36a3431bd
Turns a branch of accumulated edge/CPU work into something someone else can
read, reproduce and continue.
README.md states what the fork is for (multimodal models on edge CPUs rather
than servers), the headline memory and speed results with their costs, and why
the numbers are worth reading -- five retracted conclusions, all measurement
errors rather than coding errors. Upstream's README is preserved at
docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the
files this fork changes.
docs/edge/:
methodology.md the measurement discipline, which is the most transferable
thing here: what each of the five retractions was, and the
rule it left behind. Includes the two traps specific to this
stack -- vLLM's AOT compile cache keying on config so two
arms sharing an artifact load the wrong graph, and allocated
not being resident until mincore(2) says so.
results.md every measured figure with its spread and baseline, and an
explicit note that the four memory levers overlap and must
not be added.
cpu-4bit.md the two 4-bit paths, the packed layout as measured rather
than assumed, why the path choice is a config field, and the
GPTQ output-head trap that invalidated a whole round of W4A8
numbers.
harness.md how to run a comparison, and the two arguments that matter
more than they look (--context-len, --prompt-len), each of
which has produced a wrong conclusion by being left at its
default.
ROADMAP.md orders by whether the verification method is correct rather than by
expected size, lists what is done with results, what is next with expected
gains and how each would be verified, and -- at equal length -- what was
considered and rejected, including the CPU megakernel with the measurements
that rule it out for now.
benchmarks/edge_harness/ brings the tooling into the repo so the numbers are
reproducible: the grid runner, the single-run benchmarks, the memory ledger,
the fidelity metric, the Android agent suite, the quantizers, and the tests
that pin the harness's judgement. 82 tests pass from the new location.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Benchmarks
This directory contains benchmark suites for evaluating different model families and infrastructure components in vLLM-Omni. Each subfolder targets a different benchmark family with its own scripts, configs, and metrics. See the per-subfolder READMEs for detailed usage.
Benchmark families
TTS — Text-to-Speech
Model-agnostic serving benchmarks for TTS models, including Qwen3-TTS and VoxCPM2.
- Layout:
tts/bench_tts.py(serving benchmark driver),tts/model_configs.yaml(model registry),tts/plot_results.py(result plotting) - Dataset: Seed-TTS full or text-only datasets, plus bundled smoke/design prompts under
build_dataset/ - Key metrics: TTFP (time to first audio packet), E2E latency, RTF (real-time factor), throughput (audio seconds / wall-clock second)
Diffusion — Image and Video Generation
Online-serving benchmark for diffusion image/video models, sending requests to the configured vLLM serving endpoint (/v1/chat/completions, /v1/images/generations, or /v1/videos, depending on backend/task).
- Tasks: text-to-image, text-to-video, image-to-image, image-to-video, text+image-to-image, text+image-to-video
- Datasets:
vbench,trace,random - Key metrics: request throughput, latency percentiles, optional SLO attainment
GLM-Image — Text-to-Image and Image-to-Image
Benchmarks for GLM-Image performance across HuggingFace baseline, vLLM-Omni offline inference, and vLLM-Omni online serving.
- Layout:
glm_image/huggingface/(HF baseline),glm_image/vllm-omni/(offline inference),glm_image/benchmark_glm_image.py(online serving) - Tasks: text-to-image and image-to-image
- Key metrics: request/image throughput, latency percentiles, optional per-stage pipeline timings
LingBot-Video — Dense and MoE Parity
Manual cross-runtime validation for the dense LingBot-Video pipeline and the LingBot-Video MoE transformer.
- Dense pipeline: decoded-video MAE, MSE, PSNR, latency, and optional steady-state timings
- MoE transformer: bitwise router, sparse-block, shared-expert, and full-transformer parity
Distributed — RDMA Connector Testing
RDMA environment setup and transfer tests for MooncakeTransferEngineConnector, including pytest-based single-node checks and manual cross-node benchmarks.
- Transfer modes:
copy,zerocopy,gpu(GPUDirect) - Supports: single-node pytest suites and manual multi-node/cross-node transfer testing
Accuracy — Image Generation and Editing Quality
Accuracy benchmarks for image generation/editing models, adapting external suites to vLLM-Omni serving and local judge-evaluation flows.
- Layout:
accuracy/text_to_image/(GEBench),accuracy/image_to_image/(GEdit-Bench) - Method: generation and judge scoring both run through local
vllm-omni serveendpoints
Common serving metrics framework
vllm_omni/benchmarks/ extends vllm bench serve --omni with Omni-specific datasets, backends, and multimodal metrics. Key metrics include:
- Text output: TTFT (time to first token), TPOT (time per output token), ITL (inter-token latency)
- Audio output: TTFP (time to first audio packet), E2E latency, RTF (real-time factor)
- Throughput: request throughput, output token throughput, total token throughput, audio throughput
See vllm_omni/benchmarks/serve.py for the vllm bench serve --omni runner wrapper and vllm_omni/benchmarks/metrics/ for Omni metric definitions.
Adding a new benchmark
- Create a subfolder under
benchmarks/<name>/with scripts, configs if needed, and aREADME.md. - If comparing against another runtime, use clear backend subfolders where applicable, such as
huggingface/andvllm-omni/, or follow the shared TTS serving benchmark pattern intts/. - Add reusable dataset or prompt-building utilities to
build_dataset/if applicable. - Update this README with a link to the new benchmark family.