tzhouamandClaude Opus 5
9264ff81ab
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
Benchmarks
This directory contains benchmark suites for evaluating different model families and infrastructure components in vLLM-Omni. Each subfolder targets a different benchmark family with its own scripts, configs, and metrics. See the per-subfolder READMEs for detailed usage.
Benchmark families
TTS — Text-to-Speech
Model-agnostic serving benchmarks for TTS models, including Qwen3-TTS and VoxCPM2.
- Layout:
tts/bench_tts.py(serving benchmark driver),tts/model_configs.yaml(model registry),tts/plot_results.py(result plotting) - Dataset: Seed-TTS full or text-only datasets, plus bundled smoke/design prompts under
build_dataset/ - Key metrics: TTFP (time to first audio packet), E2E latency, RTF (real-time factor), throughput (audio seconds / wall-clock second)
Diffusion — Image and Video Generation
Online-serving benchmark for diffusion image/video models, sending requests to the configured vLLM serving endpoint (/v1/chat/completions, /v1/images/generations, or /v1/videos, depending on backend/task).
- Tasks: text-to-image, text-to-video, image-to-image, image-to-video, text+image-to-image, text+image-to-video
- Datasets:
vbench,trace,random - Key metrics: request throughput, latency percentiles, optional SLO attainment
GLM-Image — Text-to-Image and Image-to-Image
Benchmarks for GLM-Image performance across HuggingFace baseline, vLLM-Omni offline inference, and vLLM-Omni online serving.
- Layout:
glm_image/huggingface/(HF baseline),glm_image/vllm-omni/(offline inference),glm_image/benchmark_glm_image.py(online serving) - Tasks: text-to-image and image-to-image
- Key metrics: request/image throughput, latency percentiles, optional per-stage pipeline timings
LingBot-Video — Dense and MoE Parity
Manual cross-runtime validation for the dense LingBot-Video pipeline and the LingBot-Video MoE transformer.
- Dense pipeline: decoded-video MAE, MSE, PSNR, latency, and optional steady-state timings
- MoE transformer: bitwise router, sparse-block, shared-expert, and full-transformer parity
Distributed — RDMA Connector Testing
RDMA environment setup and transfer tests for MooncakeTransferEngineConnector, including pytest-based single-node checks and manual cross-node benchmarks.
- Transfer modes:
copy,zerocopy,gpu(GPUDirect) - Supports: single-node pytest suites and manual multi-node/cross-node transfer testing
Accuracy — Image Generation and Editing Quality
Accuracy benchmarks for image generation/editing models, adapting external suites to vLLM-Omni serving and local judge-evaluation flows.
- Layout:
accuracy/text_to_image/(GEBench),accuracy/image_to_image/(GEdit-Bench) - Method: generation and judge scoring both run through local
vllm-omni serveendpoints
Common serving metrics framework
vllm_omni/benchmarks/ extends vllm bench serve --omni with Omni-specific datasets, backends, and multimodal metrics. Key metrics include:
- Text output: TTFT (time to first token), TPOT (time per output token), ITL (inter-token latency)
- Audio output: TTFP (time to first audio packet), E2E latency, RTF (real-time factor)
- Throughput: request throughput, output token throughput, total token throughput, audio throughput
See vllm_omni/benchmarks/serve.py for the vllm bench serve --omni runner wrapper and vllm_omni/benchmarks/metrics/ for Omni metric definitions.
Adding a new benchmark
- Create a subfolder under
benchmarks/<name>/with scripts, configs if needed, and aREADME.md. - If comparing against another runtime, use clear backend subfolders where applicable, such as
huggingface/andvllm-omni/, or follow the shared TTS serving benchmark pattern intts/. - Add reusable dataset or prompt-building utilities to
build_dataset/if applicable. - Update this README with a link to the new benchmark family.