tzhouamandClaude Opus 5
f3629320e7
The W4A16 path kept two packed weight layouts because the tinygemm layout was treated as opaque: the prefill path needed something row-major to dequantize from, so `_finalize_int4` built `weight_halves` plus cloned scale/zero beside the kernel's own buffer. That duplicate is the 740 MB the report attributes to "the dual weight layout". The layout is not opaque. Measured on this host by packing an index pattern and reading back where each nibble landed: it is [N/64][K][32 bytes] -- K is the middle axis, so a K range is contiguous inside each n-block -- and within every 32-byte run, byte d carries output channel d in its low nibble and d+32 in its high one, the same permutation for every k and every block. Decoding it is a deinterleave. _dequantize_from_int4pack does that, bit-exact against the old dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048, and the duplicate is no longer allocated. Measured with the two paths toggled in otherwise identical code, 3 arms x 2 interleaved passes, RssAnon summed over the process tree, 1536-token context: 615 MB of resident memory and ~915 MB of peak, landing statistically indistinguishable from prefill_dequant: false (the harness refused to rank them, 0.4% apart inside the spread). Config arithmetic predicts 764 MB; ~150 MB is allocator retention malloc_trim does not return. It is not free, and the docstring says so. The duplicate was not only a copy: to_split_halves de-interleaves once at load, so each prefill call read two contiguous planes, while this redoes the de-interleave every call. pp512 drops 2272 -> 1210 tok/s. Dequantizing to [K, N] and using matmul to dodge the transpose was tried and is worse (20.3 ms vs 14.4 per gate_up at 8 threads); recovering the prefill wants a C++ de-interleave, not an ATen expression chain. So it is a dial. Against prefill_dequant: false it is strictly better -- same memory, 2.7x the prefill, and dequant_threshold stays in force so multi-row batches never reach the tinygemm kernel and the documented SIGILL hazard. That setting is now dominated. Against keeping both layouts it trades 615 MB for 1.9x prefill; default on, because this path exists for devices chosen for their memory ceiling. supports_direct_dequant gates the reshape on N tiling the 64-channel block, which every Spark linear does (2048, 3072, 13312, 131072). An N like 80 packs as 64 + a 16 tail the reshape would mis-read, so those layers keep the copy rather than being quietly wrong. The choice is a `direct_dequant` field of the checkpoint's quantization_config, not only an env var, and that matters: the two paths build different graphs (one passes weight_halves into the custom op, the other passes None), so they must not share a compiled artifact. ModelConfig.compute_hash hashes config fields, so a config field reaches vLLM's compile-cache key; an env var does not. Toggling the env var against a warm cache loaded the wrong graph and killed three benchmark passes with `KeyError: 'weight_halves'` inside the AOT-loaded forward -- found because the harness records failures instead of dropping them. VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT remains as an experiment override, documented as needing its own VLLM_CACHE_ROOT. Tests: 18 passed, including bit-exactness at three shapes, a pin on the packed layout itself (so a torch release that repacks differently fails loudly rather than returning wrong weights), the N=80 fallback, and both paths producing identical output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Easy, fast, and cheap omni-modality model serving for everyone
| Documentation | DeepWiki | User Forum | Developer Slack | WeChat | Paper | Slides |
Latest News 🔥
- [2026/08] We released 0.28.0, featuring production-ready MiniMax H3 serving on GPU and NPU, a unified AR/DiT paged KV cache runtime, and enhanced realtime full-duplex serving for the MiniCPM-o series.
- [2026/08] VeRL-Omni
v0.2.0is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the release notes. - [2026/08] We released 0.26.0 - aligned with the vLLM 0.26 release line, featuring MiniMax H3 joint video/audio generation, an experimental full-duplex realtime runtime for MiniCPM-o 4.5, distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
- [2026/07] We released 0.24.0 - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
- [2026/06] Starting with 0.14.0, vLLM-Omni publishes a stable release aligned with every even-numbered upstream vLLM minor version. 0.16.0, 0.18.0, 0.20.0, and 0.22.0 continued this cadence, expanding omni and world-model support with NVIDIA Cosmos3 and DreamZero, adding models such as MiniCPM-o 4.5, MOSS-TTS, and Lance, and advancing TTS, diffusion, distributed execution, quantization, RL integration through VeRL-Omni, and CUDA/ROCm/MUSA/NPU/XPU coverage.
- [2026/03] Check out our first public project deepdive at the vLLM Hong Kong Meetup!
- [2025/11] vLLM community officially released vllm-project/vllm-omni in order to support omni-modality models serving.
About
vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
- Omni-modality: Text, image, audio, video, and action data processing
- Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
- Heterogeneous outputs: from traditional text generation to multimodal and action outputs
vLLM-Omni is fast with:
- State-of-the-art AR support by leveraging efficient KV cache management from vLLM
- Pipelined stage execution overlapping for high throughput performance
- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages
vLLM-Omni is flexible and easy to use with:
- Heterogeneous pipeline abstraction to manage complex model workflows
- Seamless integration with popular Hugging Face models
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
- Robot-policy and action models (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
Getting Started
Visit our documentation to learn more.
- Installation
- Quickstart
- Nightly CUDA Docker images built from the latest
main - List of Supported Models
- Deployment Recipes for vLLM-Omni model serving
Contributing
We welcome and value any contributions and collaborations. Please check out Contributing to vLLM-Omni for how to get involved.
Citation
If you use vLLM-Omni for your research, please cite our paper:
@article{yin2026vllmomni,
title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
journal={arXiv preprint arXiv:2602.02204},
year={2026}
}
Join the Community
Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in #sig-omni slack channel at slack.vllm.ai or vLLM user forum at discuss.vllm.ai.
Star History
License
Apache License 2.0, as found in the LICENSE file.