tzhouamandClaude Opus 5
99ebfbbd04
The previous commit said "vLLM itself is not modified" and described the changes as going "through vLLM's own extension points". Challenged on it, and the challenge was right: those are two different claims and only one of them holds as written. What is true: no file in the vLLM tree or in site-packages/vllm is edited, the checkout is clean upstream and the wheel is stock 0.28.0+cpu, so there is no patch series to apply and no vLLM fork to maintain. What was wrong: 1. vllm_omni/patch.py does nine _patch_* functions performing ten direct attribute replacements -- CpuPlatform.check_and_update_config, CPUWorker.load_model, torch.compiler.load_compiled_function. That is monkey-patching, not an extension point, and calling it one made the arrangement sound more robust than it is. Only the registries are genuine hooks: register_quantization_config, ir.ops.*.register_impl, platform plugins, direct_register_custom_op. 2. "The changes survive a wheel upgrade rather than being lost on it" is true of the registrations and overstated for the patches. Each patch is keyed to a specific method on a specific vLLM class; an upstream rename or behaviour change can break it or silently turn it into a no-op wrapper. The failure mode is a missing optimisation, not a crash, which is worse -- so the doc now says to re-run the harness after upgrading. The section is rewritten to separate the two mechanisms, list what each patch works around, and note that two of the three exist for upstream bugs and should disappear when those are fixed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
hide
| hide | ||
|---|---|---|
|
Welcome to vLLM-Omni
Easy, fast, and cheap omni-modality model serving for everyone
<script async defer src="https://buttons.github.io/buttons.js"></script> Star Watch Fork
About
vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
- Omni-modality: Text, image, audio, video, and action data processing
- Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
- Heterogeneous outputs: from traditional text generation to multimodal and action outputs
vLLM-Omni is fast with:
- State-of-the-art AR support by leveraging efficient KV cache management from vLLM
- Pipelined stage execution overlapping for high throughput performance
- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages
vLLM-Omni is flexible and easy to use with:
- Heterogeneous pipeline abstraction to manage complex model workflows
- Seamless integration with popular Hugging Face models
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
- Robot-policy and action models (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
For more information, checkout the following: