vllm-omni/docs/README.vllm-omni.md at bb8735effca1cf1403092ccfbdec783738e72cd4

0 Fork 0

Files

T

tzhouamandClaude Opus 5 b36a3431bd Package this fork as vllm-embedding: README, docs, roadmap, harness

Turns a branch of accumulated edge/CPU work into something someone else can
read, reproduce and continue.

README.md states what the fork is for (multimodal models on edge CPUs rather
than servers), the headline memory and speed results with their costs, and why
the numbers are worth reading -- five retracted conclusions, all measurement
errors rather than coding errors. Upstream's README is preserved at
docs/README.vllm-omni.md and NOTICE records the Apache-2.0 derivation and the
files this fork changes.

docs/edge/:
  methodology.md  the measurement discipline, which is the most transferable
                  thing here: what each of the five retractions was, and the
                  rule it left behind. Includes the two traps specific to this
                  stack -- vLLM's AOT compile cache keying on config so two
                  arms sharing an artifact load the wrong graph, and allocated
                  not being resident until mincore(2) says so.
  results.md      every measured figure with its spread and baseline, and an
                  explicit note that the four memory levers overlap and must
                  not be added.
  cpu-4bit.md     the two 4-bit paths, the packed layout as measured rather
                  than assumed, why the path choice is a config field, and the
                  GPTQ output-head trap that invalidated a whole round of W4A8
                  numbers.
  harness.md      how to run a comparison, and the two arguments that matter
                  more than they look (--context-len, --prompt-len), each of
                  which has produced a wrong conclusion by being left at its
                  default.

ROADMAP.md orders by whether the verification method is correct rather than by
expected size, lists what is done with results, what is next with expected
gains and how each would be verified, and -- at equal length -- what was
considered and rejected, including the CPU megakernel with the measurements
that rule it out for now.

benchmarks/edge_harness/ brings the tooling into the repo so the numbers are
reproducible: the grid runner, the single-run benchmarks, the memory ledger,
the fidelity metric, the Android agent suite, the quantizers, and the tests
that pin the harness's judgement. 82 tests pass from the new location.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

2026-09-14 13:25:32 +08:00

8.3 KiB

Raw Blame History

Easy, fast, and cheap omni-modality model serving for everyone

Latest News 🔥

[2026/08] We released 0.28.0, featuring production-ready MiniMax H3 serving on GPU and NPU, a unified AR/DiT paged KV cache runtime, and enhanced realtime full-duplex serving for the MiniCPM-o series.
[2026/08] VeRL-Omni v0.2.0 is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the release notes.
[2026/08] We released 0.26.0 - aligned with the vLLM 0.26 release line, featuring MiniMax H3 joint video/audio generation, an experimental full-duplex realtime runtime for MiniCPM-o 4.5, distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
[2026/07] We released 0.24.0 - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
[2026/06] Starting with 0.14.0, vLLM-Omni publishes a stable release aligned with every even-numbered upstream vLLM minor version. 0.16.0, 0.18.0, 0.20.0, and 0.22.0 continued this cadence, expanding omni and world-model support with NVIDIA Cosmos3 and DreamZero, adding models such as MiniCPM-o 4.5, MOSS-TTS, and Lance, and advancing TTS, diffusion, distributed execution, quantization, RL integration through VeRL-Omni, and CUDA/ROCm/MUSA/NPU/XPU coverage.
[2026/03] Check out our first public project deepdive at the vLLM Hong Kong Meetup!
[2025/11] vLLM community officially released vllm-project/vllm-omni in order to support omni-modality models serving.

About

vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:

Omni-modality: Text, image, audio, video, and action data processing
Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
Heterogeneous outputs: from traditional text generation to multimodal and action outputs

vLLM-Omni is fast with:

State-of-the-art AR support by leveraging efficient KV cache management from vLLM
Pipelined stage execution overlapping for high throughput performance
Fully disaggregation based on OmniConnector and dynamic resource allocation across stages

vLLM-Omni is flexible and easy to use with:

Heterogeneous pipeline abstraction to manage complex model workflows
Seamless integration with popular Hugging Face models
Tensor, pipeline, data and expert parallelism support for distributed inference
Streaming outputs
OpenAI-compatible API server
Full-duplex realtime serving with streaming audio input and output

vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:

Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
Robot-policy and action models (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)

Getting Started

Visit our documentation to learn more.

Installation
Quickstart
Nightly CUDA Docker images built from the latest main
List of Supported Models
Deployment Recipes for vLLM-Omni model serving

Contributing

We welcome and value any contributions and collaborations. Please check out Contributing to vLLM-Omni for how to get involved.

Citation

If you use vLLM-Omni for your research, please cite our paper:

@article{yin2026vllmomni,
  title={vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models},
  author={Peiqi Yin, Jiangyun Zhu, Han Gao, Chenguang Zheng, Yongxiang Huang, Taichang Zhou, Ruirui Yang, Weizhi Liu, Weiqing Chen, Canlin Guo, Didan Deng, Zifeng Mo, Cong Wang, James Cheng, Roger Wang, Hongsheng Liu},
  journal={arXiv preprint arXiv:2602.02204},
  year={2026}
}

Join the Community

Feel free to ask questions, provide feedbacks and discuss with fellow users of vLLM-Omni in #sig-omni slack channel at slack.vllm.ai or vLLM user forum at discuss.vllm.ai.

Star History

License

Apache License 2.0, as found in the LICENSE file.

8.3 KiB Raw Blame History