tzhouamandClaude Fable 5.1
33de5d8629
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while our 0.28-based tree needs seven modules v0.29.0 no longer has (vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args, openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant, models.hunyuan_v1). Ten files conflicted. Upstream's structure is the base everywhere; the fork's features are re-attached to it: - deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config pops deploy_config (async_omni_engine, cli/serve); the deploy-profile selector and the resolve_model_config_path chain it needs stay in entrypoints/utils.py; stage_overrides takes rc1's dict type. - config_factory keeps the lazy diffusion output-type import; the diffusion registry import in rc1's new config/resolver.py is made lazy for the same WP6 reason (TTS path must not load diffusion). - requirements: our core/media split kept; rc1's transformers>=5.13 floor and kernels==0.16.1 applied in core.txt / media.txt. - TTS: rc1's split_granularity and bounded seed plus our output_mode= codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation; our private _conditioning_cache_salt is superseded by rc1's public tts_adapters/base.py::conditioning_cache_salt. - stage_runtime: rc1's single ExitStack G2 path (lock released before READY) replaces our two-path block, with the four worker_phase names the init profiler consumes still emitted; _cpu_placed / host admission kept. Not yet run against vLLM 0.29.0: the tests listed in the W6 record. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
hide
| hide | ||
|---|---|---|
|
Welcome to vLLM-Omni
Easy, fast, and cheap omni-modality model serving for everyone
<script async defer src="https://buttons.github.io/buttons.js"></script> Star Watch Fork
About
vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
- Omni-modality: Text, image, audio, video, and action data processing
- Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
- Heterogeneous outputs: from traditional text generation to multimodal and action outputs
vLLM-Omni is fast with:
- State-of-the-art AR support by leveraging efficient KV cache management from vLLM
- Pipelined stage execution overlapping for high throughput performance
- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages
vLLM-Omni is flexible and easy to use with:
- Heterogeneous pipeline abstraction to manage complex model workflows
- Seamless integration with popular Hugging Face models
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
- Robot-policy and action models (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
For more information, checkout the following: