hsliuustc0106
f4394f1d3d
Models whose request geometry opts out of the generic dummy run (H3: t2va needs an explicit aspect ratio and a bounded duration) can declare a dlo_dummy_run_recipe. Under distributed layerwise offload the engine's startup warmup runs it, so the learned cache-retention budget is primed before real traffic instead of on the first request. H3 declares a maximum-duration t2va generation: duration is the only workload variable at the fixed 768 short edge, so the learned peak dominates every production request and no real request re-learns. Validated on L20X: all cold-start releases happen during the startup warmup, zero during requests; first request 15.91 s vs 16.7 s unwarmed, steady 14.9 s; startup gains one max-length generation (~10-20 s). Non-DLO deployments keep the opted-out generic warmup unchanged. Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
hide
| hide | ||
|---|---|---|
|
Welcome to vLLM-Omni
Easy, fast, and cheap omni-modality model serving for everyone
<script async defer src="https://buttons.github.io/buttons.js"></script> Star Watch Fork
About
vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
- Omni-modality: Text, image, audio, video, and action data processing
- Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
- Heterogeneous outputs: from traditional text generation to multimodal and action outputs
vLLM-Omni is fast with:
- State-of-the-art AR support by leveraging efficient KV cache management from vLLM
- Pipelined stage execution overlapping for high throughput performance
- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages
vLLM-Omni is flexible and easy to use with:
- Heterogeneous pipeline abstraction to manage complex model workflows
- Seamless integration with popular Hugging Face models
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output (experimental)
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- TTS models (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX)
- Robot-policy and action models (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
For more information, checkout the following: