tzhouamandClaude Fable 5.1
49a839ecf9
* qwen3_tts_talker_only pipeline (single terminal LLM_AR stage, final_output_type "latent") + vllm_omni/deploy/qwen3_tts_talker_only.yaml; a terminal talker publishes its per-step codes.audio rows to the client (omni_client_multimodal_output_keys). * /v1/audio/speech/stream: session.config.output_mode="codec_tokens" streams codec.start (JSON) + binary frames (<IHB seq/n_frames/flags + int16 rows, vllm_omni/entrypoints/openai/codec_stream.py) + codec.done; new input.cancel message aborts an in-flight generation for both codec and PCM sessions (single reader task + session inbox replaces direct socket reads); optional session.config.seed; audio.done gains "cancelled" only when it happened. * serving_speech._generate_codec_chunks aligns the cumulative code tensor to output token ids (drops prefill placeholder/EOS rows) and emits deltas. * TTS adapters: supports_codec_stream / codec_stream_spec (Qwen3-TTS: 16 codebooks, 2048 ids, 12.5 Hz, 24 kHz, decoder qwen3_tts_tokenizer_v2_12hz). * examples/online_serving/text_to_speech/qwen3_tts/edge_decoder_client.py decodes locally with Qwen3TTSTokenizerV2Decoder (ramp schedule, 72-frame context, cancel/reset), codec_parity_check.py and benchmarks/tts/codec_stream_e2e.py compare against the two-stage server. * docs/serving/codec_token_streaming.md; CPU unit tests for the wire format, alignment, session config and handler (frames, unsupported adapter, cancel); L3 e2e tests/e2e/online_serving/test_qwen3_tts_codec_stream.py. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
hide
| hide | ||
|---|---|---|
|
Welcome to vLLM-Omni
Easy, fast, and cheap omni-modality model serving for everyone
<script async defer src="https://buttons.github.io/buttons.js"></script> Star Watch Fork
About
vLLM was originally designed to support large language models for text-based autoregressive generation tasks. vLLM-Omni is a framework that extends its support for omni-modality model inference and serving:
- Omni-modality: Text, image, audio, video, and action data processing
- Non-autoregressive Architectures: extend the AR support of vLLM to Diffusion Transformers (DiT) and other parallel generation models
- Heterogeneous outputs: from traditional text generation to multimodal and action outputs
vLLM-Omni is fast with:
- State-of-the-art AR support by leveraging efficient KV cache management from vLLM
- Pipelined stage execution overlapping for high throughput performance
- Fully disaggregation based on OmniConnector and dynamic resource allocation across stages
vLLM-Omni is flexible and easy to use with:
- Heterogeneous pipeline abstraction to manage complex model workflows
- Seamless integration with popular Hugging Face models
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- TTS models (e.g. Qwen3-TTS, IndexTTS 2.5, dots.tts, CosyVoice3)
- Diffusion models — image, video, and audio generation (e.g. MiniMax H3, LTX-2.5, SANA-Video, Wan2.2)
- Robot-policy and action models (e.g. π0, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
For more information, checkout the following: