tzhouamandClaude Fable 5.1
33de5d8629
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while our 0.28-based tree needs seven modules v0.29.0 no longer has (vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args, openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant, models.hunyuan_v1). Ten files conflicted. Upstream's structure is the base everywhere; the fork's features are re-attached to it: - deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config pops deploy_config (async_omni_engine, cli/serve); the deploy-profile selector and the resolve_model_config_path chain it needs stay in entrypoints/utils.py; stage_overrides takes rc1's dict type. - config_factory keeps the lazy diffusion output-type import; the diffusion registry import in rc1's new config/resolver.py is made lazy for the same WP6 reason (TTS path must not load diffusion). - requirements: our core/media split kept; rc1's transformers>=5.13 floor and kernels==0.16.1 applied in core.txt / media.txt. - TTS: rc1's split_granularity and bounded seed plus our output_mode= codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation; our private _conditioning_cache_salt is superseded by rc1's public tts_adapters/base.py::conditioning_cache_salt. - stage_runtime: rc1's single ExitStack G2 path (lock released before READY) replaces our two-path block, with the four worker_phase names the init profiler consumes still emitted; _cpu_placed / host admission kept. Not yet run against vLLM 0.29.0: the tests listed in the W6 record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Benchmarks
This directory contains benchmark suites for evaluating different model families and infrastructure components in vLLM-Omni. Each subfolder targets a different benchmark family with its own scripts, configs, and metrics. See the per-subfolder READMEs for detailed usage.
Benchmark families
TTS — Text-to-Speech
Model-agnostic serving benchmarks for TTS models, including Qwen3-TTS and VoxCPM2.
- Layout:
tts/bench_tts.py(serving benchmark driver),tts/model_configs.yaml(model registry),tts/plot_results.py(result plotting) - Dataset: Seed-TTS full or text-only datasets, plus bundled smoke/design prompts under
build_dataset/ - Key metrics: TTFP (time to first audio packet), E2E latency, RTF (real-time factor), throughput (audio seconds / wall-clock second)
Diffusion — Image and Video Generation
Online-serving benchmark for diffusion image/video models, sending requests to the configured vLLM serving endpoint (/v1/chat/completions, /v1/images/generations, or /v1/videos, depending on backend/task).
- Tasks: text-to-image, text-to-video, image-to-image, image-to-video, text+image-to-image, text+image-to-video
- Datasets:
vbench,trace,random - Key metrics: request throughput, latency percentiles, optional SLO attainment
GLM-Image — Text-to-Image and Image-to-Image
Benchmarks for GLM-Image performance across HuggingFace baseline, vLLM-Omni offline inference, and vLLM-Omni online serving.
- Layout:
glm_image/huggingface/(HF baseline),glm_image/vllm-omni/(offline inference),glm_image/benchmark_glm_image.py(online serving) - Tasks: text-to-image and image-to-image
- Key metrics: request/image throughput, latency percentiles, optional per-stage pipeline timings
LingBot-Video — Dense and MoE Parity
Manual cross-runtime validation for the dense LingBot-Video pipeline and the LingBot-Video MoE transformer.
- Dense pipeline: decoded-video MAE, MSE, PSNR, latency, and optional steady-state timings
- MoE transformer: bitwise router, sparse-block, shared-expert, and full-transformer parity
Distributed — RDMA Connector Testing
RDMA environment setup and transfer tests for MooncakeTransferEngineConnector, including pytest-based single-node checks and manual cross-node benchmarks.
- Transfer modes:
copy,zerocopy,gpu(GPUDirect) - Supports: single-node pytest suites and manual multi-node/cross-node transfer testing
Accuracy — Image Generation and Editing Quality
Accuracy benchmarks for image generation/editing models, adapting external suites to vLLM-Omni serving and local judge-evaluation flows.
- Layout:
accuracy/text_to_image/(GEBench),accuracy/image_to_image/(GEdit-Bench) - Method: generation and judge scoring both run through local
vllm-omni serveendpoints
Common serving metrics framework
vllm_omni/benchmarks/ extends vllm bench serve --omni with Omni-specific datasets, backends, and multimodal metrics. Key metrics include:
- Text output: TTFT (time to first token), TPOT (time per output token), ITL (inter-token latency)
- Audio output: TTFP (time to first audio packet), E2E latency, RTF (real-time factor)
- Throughput: request throughput, output token throughput, total token throughput, audio throughput
See vllm_omni/benchmarks/serve.py for the vllm bench serve --omni runner wrapper and vllm_omni/benchmarks/metrics/ for Omni metric definitions.
Adding a new benchmark
- Create a subfolder under
benchmarks/<name>/with scripts, configs if needed, and aREADME.md. - If comparing against another runtime, use clear backend subfolders where applicable, such as
huggingface/andvllm-omni/, or follow the shared TTS serving benchmark pattern intts/. - Add reusable dataset or prompt-building utilities to
build_dataset/if applicable. - Update this README with a link to the new benchmark family.