Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fab29e2873 | ||
|
|
1ade14d147 | ||
|
|
b6313918d6 | ||
|
|
d77a35a32b | ||
|
|
f27e70dce7 | ||
|
|
c46abfef59 |
No files matched your search
+2
-2
@@ -43,6 +43,7 @@ nav:
|
|||||||
- Overview: features/README.md
|
- Overview: features/README.md
|
||||||
- Runtime and Stage Execution:
|
- Runtime and Stage Execution:
|
||||||
- Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
|
- Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
|
||||||
|
- Prefill-Decode Disaggregation (experimental): features/pd_disaggregation.md
|
||||||
- Sleep Mode: features/sleep_mode.md
|
- Sleep Mode: features/sleep_mode.md
|
||||||
- Quantization:
|
- Quantization:
|
||||||
- Overview: user_guide/quantization/overview.md
|
- Overview: user_guide/quantization/overview.md
|
||||||
@@ -163,8 +164,7 @@ nav:
|
|||||||
- CLI Reference: cli
|
- CLI Reference: cli
|
||||||
- Community:
|
- Community:
|
||||||
- Governance: community/governance.md
|
- Governance: community/governance.md
|
||||||
- Contact Us: community/contact_us.md
|
- community/*
|
||||||
- Community Meetings: community/meetings.md
|
|
||||||
- Slack: https://slack.vllm.ai
|
- Slack: https://slack.vllm.ai
|
||||||
- Blog: https://blog.vllm.ai
|
- Blog: https://blog.vllm.ai
|
||||||
- Forum: https://discuss.vllm.ai
|
- Forum: https://discuss.vllm.ai
|
||||||
@@ -1,9 +1,5 @@
|
|||||||
# Contact Us
|
# Contact Us
|
||||||
|
|
||||||
- For bugs and feature requests, search or open a GitHub [issue](https://github.com/vllm-project/vllm-omni/issues).
|
- For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm-omni/issues)
|
||||||
- For technical and usage questions, join the `sig-omni` channel in [vLLM Slack](https://slack.vllm.ai/) or use the [vLLM Forum](https://discuss.vllm.ai/).
|
- For coordinating contributions and development and discussing with other users and developers, please join `sig-omni` channel in our [Slack](https://slack.vllm.ai/) or use the [vLLM Forum](https://discuss.vllm.ai/)
|
||||||
- For coordinating contributions and development, use `sig-omni`, the forum, or join a [community meeting](meetings.md).
|
- For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm-omni/security/advisories) feature
|
||||||
|
|
||||||
## Security Disclosures
|
|
||||||
|
|
||||||
Do not share vulnerability details in public GitHub issues, pull requests, discussions, or Slack channels. To arrange a private report, send a direct message to one of the [lead maintainers](governance.md#lead-maintainers) in [vLLM Slack](https://slack.vllm.ai/). If you cannot send a direct message, you may post a detail-free request in `sig-omni` asking a lead maintainer to contact you privately; do not include any vulnerability details in that request.
|
|
||||||
@@ -52,11 +52,31 @@ Committers have write access and merge rights. They typically have deep expertis
|
|||||||
- [@ZJY0516](https://github.com/ZJY0516): Diffusion attention backend, kernel fusion, and CustomOp
|
- [@ZJY0516](https://github.com/ZJY0516): Diffusion attention backend, kernel fusion, and CustomOp
|
||||||
- [@yuanheng-zhao](https://github.com/yuanheng-zhao): Diffusion cache, offload, and Omni Support
|
- [@yuanheng-zhao](https://github.com/yuanheng-zhao): Diffusion cache, offload, and Omni Support
|
||||||
|
|
||||||
## Reviewer Routing
|
## Diffusion Workload Division
|
||||||
|
|
||||||
For path-based reviewer routing, consult [`.github/CODEOWNERS`](https://github.com/vllm-project/vllm-omni/blob/main/.github/CODEOWNERS). Detailed subsystem boundaries and safe-change guidance live in the [module design documents](../design/index.md#module-design-documents).
|
This section breaks down diffusion-related responsibilities. If you have PRs touching these areas, please ping the listed owners for review.
|
||||||
|
|
||||||
`CODEOWNERS` is operational routing metadata, not a governance or approval policy. Being listed does not grant committer status or merge rights, and it does not mean every listed reviewer must approve a change. The maintainer roster above is authoritative for project roles and merge rights.
|
### Runtime-related
|
||||||
|
|
||||||
|
- Diffusion models: @RuixiangMa, @wtomin, @xuechendi
|
||||||
|
|
||||||
|
### Optimization
|
||||||
|
|
||||||
|
- Parallel: @RuixiangMa, @wtomin
|
||||||
|
- Attention backend: @ZJY0516
|
||||||
|
- Cache: @yuanheng-zhao, @RuixiangMa, @SamitHuang
|
||||||
|
- Offload: @yuanheng-zhao
|
||||||
|
- Intel GPU support: @xuechendi
|
||||||
|
- Quantization: @lishunyang12, @david6666666
|
||||||
|
- Kernel fusion / communication-computation: @ZJY0516
|
||||||
|
|
||||||
|
### Docs & Test
|
||||||
|
|
||||||
|
- Diffusion docs: @RuixiangMa, @wtomin
|
||||||
|
|
||||||
|
## Meetings
|
||||||
|
|
||||||
|
Committers hold **bi-weekly meetings** to discuss future directions and collaborations of the project.
|
||||||
|
|
||||||
## Committer Nomination Process
|
## Committer Nomination Process
|
||||||
|
|
||||||
|
|||||||
@@ -1,8 +0,0 @@
|
|||||||
# Community Meetings
|
|
||||||
|
|
||||||
vLLM-Omni hosts developer-facing meetings for Chinese- and English-language audiences. Schedules and access details can change, so use these maintained documents as the source of truth:
|
|
||||||
|
|
||||||
- [Chinese-language meeting document](https://docs.google.com/document/d/1pdUBiS_7mdOUNDtdwy-9OUf7jsMWN324BIbps5olDME/edit?tab=t.0)
|
|
||||||
- [English-language meeting document](https://docs.google.com/document/d/19AlD08T8uJ7wqfs8VtYCi0iGMTwlTYVVlAxFZ3S7OIg/edit?tab=t.0)
|
|
||||||
|
|
||||||
Each document contains the current schedule, access details, agendas, and past meeting notes. For broader vLLM community events, see the [vLLM events calendar](https://vllm.ai/events/).
|
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
# Meetups
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
# Volunteers for Bugfix and CI
|
||||||
|
|
||||||
|
We encourage you to check current docs and [issues](https://github.com/vllm-project/vllm-omni/issues) to find possible solutions for your questions. If non of these can solve it, please propose an issue to describe your questions about bug or CI problems for developing.
|
||||||
|
|
||||||
|
If you have urgent need for locating and solving bugfix or CI problems, please find community volunteers below.
|
||||||
|
|
||||||
|
| Dec 4-Dec 12 | Dec 15-Dec 19 | Dec 22-Dec 26 | Dec 29- Jan 2, 2026| Jan 5-Jan 9 | Jan 12-Jan 16 |
|
||||||
|
|----------|----------|----------|----------|----------|----------|
|
||||||
|
| <a href="https://github.com/congw729">Conw729</a> | <a href="https://github.com/yinpeiqi">yinpeiqi</a> | <a href="https://github.com/tzhouam">tzhouam</a> | <a href="https://github.com/SamitHuang">SamitHuang</a> | <a href="https://github.com/gcanlin">gcanlin</a> | <a href="https://github.com/natureofnature">natureofnature</a> |
|
||||||
|
| <a href="https://github.com/david6666666">david6666666</a> | <a href="https://github.com/R2-Y">R2-Y</a> | <a href="https://github.com/hsliuustc0106">hsliuustc0106</a> | <a href="https://github.com/Gaohan123">Gaohan123</a> | <a href="https://github.com/ZJY0516">ZJY0516</a> | <a href="https://github.com/qibaoyuan">qibaoyuan</a> |
|
||||||
|
|
||||||
|
We kindly welcome more contributors to fix bugs and contribute new features!
|
||||||
@@ -14,10 +14,6 @@ For introduction, please check [Introduction for stage config](./stage_configs.m
|
|||||||
|
|
||||||
- **[GPU Memory Calculation and Configuration](./gpu_memory_utilization.md)** - Guide on how to calculate memory requirements and set up `gpu_memory_utilization` for optimal performance
|
- **[GPU Memory Calculation and Configuration](./gpu_memory_utilization.md)** - Guide on how to calculate memory requirements and set up `gpu_memory_utilization` for optimal performance
|
||||||
|
|
||||||
## Multi-Stage Recipes
|
|
||||||
|
|
||||||
- **[Prefill-Decode Disaggregation](./pd_disaggregation.md)** - How to derive a PD-aware Qwen3-Omni stage config from the default config without introducing another bundled YAML
|
|
||||||
|
|
||||||
## Optimization Features
|
## Optimization Features
|
||||||
|
|
||||||
- **[Diffusion Features Overview](../user_guide/diffusion_features.md)** - Complete overview of all diffusion model features and supported models
|
- **[Diffusion Features Overview](../user_guide/diffusion_features.md)** - Complete overview of all diffusion model features and supported models
|
||||||
@@ -1,171 +0,0 @@
|
|||||||
# Prefill-Decode (PD) Disaggregation
|
|
||||||
|
|
||||||
PD disaggregation splits the Qwen3-Omni thinker into separate prefill and decode
|
|
||||||
stages so prompt processing and token generation can run on different workers.
|
|
||||||
|
|
||||||
This is documented as a stage-config recipe instead of a bundled YAML because the
|
|
||||||
deployment-specific values usually change per environment:
|
|
||||||
|
|
||||||
- GPU placement
|
|
||||||
- `tensor_parallel_size`
|
|
||||||
- connector backend and connector ports
|
|
||||||
- connector IPs or bootstrap addresses
|
|
||||||
|
|
||||||
Start from the [default Qwen3-Omni stage config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
|
|
||||||
and copy it to your own file, for example `qwen3_omni_pd.yaml`. Then apply the
|
|
||||||
changes below.
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
- 3+ GPUs for a basic layout: prefill, decode, and talker+code2wav
|
|
||||||
- A KV connector supported by vLLM, such as `MooncakeConnector`
|
|
||||||
- Matching `tensor_parallel_size` on the prefill and decode thinker stages
|
|
||||||
|
|
||||||
## 1. Split the thinker into prefill and decode stages
|
|
||||||
|
|
||||||
Replace the original thinker stage with two stages:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
stage_args:
|
|
||||||
- stage_id: 0
|
|
||||||
stage_type: llm
|
|
||||||
is_prefill_only: true
|
|
||||||
runtime:
|
|
||||||
devices: "0"
|
|
||||||
engine_args:
|
|
||||||
max_num_seqs: 16
|
|
||||||
model_stage: thinker
|
|
||||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
|
||||||
worker_type: ar
|
|
||||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
|
||||||
gpu_memory_utilization: 0.9
|
|
||||||
enforce_eager: true
|
|
||||||
trust_remote_code: true
|
|
||||||
engine_output_type: latent
|
|
||||||
distributed_executor_backend: "mp"
|
|
||||||
enable_prefix_caching: false
|
|
||||||
max_num_batched_tokens: 32768
|
|
||||||
hf_config_name: thinker_config
|
|
||||||
tensor_parallel_size: 1
|
|
||||||
kv_transfer_config:
|
|
||||||
kv_connector: "MooncakeConnector"
|
|
||||||
kv_role: "kv_producer"
|
|
||||||
kv_rank: 0
|
|
||||||
kv_parallel_size: 2
|
|
||||||
kv_connector_extra_config:
|
|
||||||
mooncake_bootstrap_port: 25201
|
|
||||||
final_output: false
|
|
||||||
is_comprehension: true
|
|
||||||
default_sampling_params:
|
|
||||||
temperature: 0.4
|
|
||||||
top_p: 0.9
|
|
||||||
top_k: 1
|
|
||||||
max_tokens: 2048
|
|
||||||
seed: 42
|
|
||||||
detokenize: True
|
|
||||||
repetition_penalty: 1.05
|
|
||||||
|
|
||||||
- stage_id: 1
|
|
||||||
stage_type: llm
|
|
||||||
is_decode_only: true
|
|
||||||
runtime:
|
|
||||||
devices: "1"
|
|
||||||
engine_args:
|
|
||||||
max_num_seqs: 64
|
|
||||||
model_stage: thinker
|
|
||||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
|
||||||
worker_type: ar
|
|
||||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
|
||||||
gpu_memory_utilization: 0.9
|
|
||||||
enforce_eager: true
|
|
||||||
trust_remote_code: true
|
|
||||||
engine_output_type: latent
|
|
||||||
distributed_executor_backend: "mp"
|
|
||||||
enable_prefix_caching: false
|
|
||||||
max_num_batched_tokens: 32768
|
|
||||||
hf_config_name: thinker_config
|
|
||||||
tensor_parallel_size: 1
|
|
||||||
kv_transfer_config:
|
|
||||||
kv_connector: "MooncakeConnector"
|
|
||||||
kv_role: "kv_consumer"
|
|
||||||
kv_rank: 1
|
|
||||||
kv_parallel_size: 2
|
|
||||||
kv_connector_extra_config:
|
|
||||||
mooncake_bootstrap_port: 25202
|
|
||||||
engine_input_source: [0]
|
|
||||||
final_output: true
|
|
||||||
final_output_type: text
|
|
||||||
is_comprehension: true
|
|
||||||
default_sampling_params:
|
|
||||||
temperature: 0.4
|
|
||||||
top_p: 0.9
|
|
||||||
top_k: 1
|
|
||||||
max_tokens: 2048
|
|
||||||
seed: 42
|
|
||||||
detokenize: True
|
|
||||||
repetition_penalty: 1.05
|
|
||||||
```
|
|
||||||
|
|
||||||
Notes:
|
|
||||||
|
|
||||||
- `is_prefill_only: true` marks the thinker stage that only saves KV.
|
|
||||||
- `is_decode_only: true` marks the thinker stage that resumes from remote KV.
|
|
||||||
- `kv_transfer_config` is required on both stages.
|
|
||||||
- The orchestrator forces the prefill stage to run with `max_tokens=1`, so the
|
|
||||||
prefill side only processes the prompt and exports KV.
|
|
||||||
|
|
||||||
## 2. Shift the downstream stages by one index
|
|
||||||
|
|
||||||
After inserting the extra thinker stage, renumber the remaining stages:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
- stage_id: 2
|
|
||||||
runtime:
|
|
||||||
devices: "2"
|
|
||||||
engine_input_source: [1]
|
|
||||||
sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_token_only
|
|
||||||
|
|
||||||
- stage_id: 3
|
|
||||||
runtime:
|
|
||||||
devices: "2"
|
|
||||||
engine_args:
|
|
||||||
max_num_seqs: 1
|
|
||||||
engine_input_source: [2]
|
|
||||||
sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_token_only
|
|
||||||
```
|
|
||||||
|
|
||||||
Compared with the default Qwen3-Omni config:
|
|
||||||
|
|
||||||
- the talker becomes stage `2` instead of stage `1`
|
|
||||||
- the code2wav stage becomes stage `3` instead of stage `2`
|
|
||||||
- the talker now reads from decode stage `1`
|
|
||||||
|
|
||||||
## 3. Add runtime edges for the four-stage pipeline
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
runtime:
|
|
||||||
enabled: true
|
|
||||||
edges:
|
|
||||||
- from: 0
|
|
||||||
to: 1
|
|
||||||
- from: 1
|
|
||||||
to: 2
|
|
||||||
- from: 2
|
|
||||||
to: 3
|
|
||||||
```
|
|
||||||
|
|
||||||
## 4. Launch with your custom config
|
|
||||||
|
|
||||||
```bash
|
|
||||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
|
||||||
--deploy-config /path/to/qwen3_omni_pd.yaml
|
|
||||||
```
|
|
||||||
|
|
||||||
## Operational Notes
|
|
||||||
|
|
||||||
- `MooncakeConnector` does not support heterogeneous TP sizes across the PD
|
|
||||||
pair. Keep prefill and decode at the same `tensor_parallel_size`.
|
|
||||||
- If the thinker requires TP=2, both thinker stages must use TP=2 and be given
|
|
||||||
separate GPU sets, for example `"0,1"` for prefill and `"2,3"` for decode.
|
|
||||||
- Choose connector ports and addresses that match your deployment. The values
|
|
||||||
shown above are examples only.
|
|
||||||
@@ -3,7 +3,7 @@
|
|||||||
Thank you for your interest in contributing to vLLM-Omni! This document provides guidelines and instructions for contributing.
|
Thank you for your interest in contributing to vLLM-Omni! This document provides guidelines and instructions for contributing.
|
||||||
|
|
||||||
!!! note
|
!!! note
|
||||||
vLLM-Omni hosts developer-facing meetings for Chinese- and English-language audiences. See [Community Meetings](../community/meetings.md) for current schedules, access details, agendas, and past notes.
|
We host weekly developer-facing online meetings to discuss milestones and updates **every Tuesday at 19:30 PDT**. Meeting link as well as the past meeting notes can be found [here](https://tinyurl.com/vllm-omni-meeting).
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
@@ -122,7 +122,7 @@ Repository skills guide the agent, but they do not replace contributor judgment,
|
|||||||
If you encounter a bug or have a feature request, please search existing issues first to see if it has already been reported. If not, please file a new issue, providing as much relevant information as possible.
|
If you encounter a bug or have a feature request, please search existing issues first to see if it has already been reported. If not, please file a new issue, providing as much relevant information as possible.
|
||||||
|
|
||||||
!!! important
|
!!! important
|
||||||
Do not report suspected security vulnerabilities through a public issue, pull request, discussion, or Slack channel. Follow the [security disclosure instructions](../community/contact_us.md#security-disclosures) to arrange a private report.
|
If you discover a security vulnerability, please report it by creating a GitHub issue with the `security` label.
|
||||||
|
|
||||||
## Pull Requests & Code Reviews
|
## Pull Requests & Code Reviews
|
||||||
|
|
||||||
|
|||||||
@@ -20,13 +20,19 @@ an implementation contract has a user-facing workflow.
|
|||||||
| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
|
| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
|
||||||
| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
|
| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
|
||||||
|
|
||||||
|
The [Prefill-Decode Disaggregation design reference](pd_disaggregation.md)
|
||||||
|
describes the experimental Qwen3-Omni runtime topology and its current
|
||||||
|
integration boundaries. It is not yet a supported launch recipe.
|
||||||
|
|
||||||
Some runtime designs are deliberately not promoted as standalone User Guide
|
Some runtime designs are deliberately not promoted as standalone User Guide
|
||||||
features yet:
|
features yet:
|
||||||
|
|
||||||
- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and
|
- Generic, OmniConnector-based
|
||||||
|
[Disaggregated Inference](../design/feature/disaggregated_inference.md) is not
|
||||||
|
yet a standalone User Guide feature. Its
|
||||||
[OmniConnector implementations](../design/index.md#communication) describe
|
[OmniConnector implementations](../design/index.md#communication) describe
|
||||||
topology and transport contracts. Practical deployment configuration remains
|
transport contracts, while practical connector configuration remains under
|
||||||
under [Pipeline and deploy configurations](../configuration/stage_configs.md).
|
[Pipeline and deploy configurations](../configuration/stage_configs.md).
|
||||||
- [Async Chunk](../design/feature/async_chunk.md) and
|
- [Async Chunk](../design/feature/async_chunk.md) and
|
||||||
[Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
|
[Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
|
||||||
are model- and pipeline-dependent. Use the selected model's deploy
|
are model- and pipeline-dependent. Use the selected model's deploy
|
||||||
|
|||||||
@@ -0,0 +1,140 @@
|
|||||||
|
# Prefill-Decode (PD) Disaggregation (experimental)
|
||||||
|
|
||||||
|
!!! warning "Experimental design reference"
|
||||||
|
PD disaggregation is only partially integrated on the current `main` branch.
|
||||||
|
The runtime contains PD detection and routing scaffolding, but there is no
|
||||||
|
supported deploy overlay or end-to-end validated launch recipe yet. Treat this
|
||||||
|
page as a description of the current design, not as production guidance.
|
||||||
|
|
||||||
|
Prefill-decode disaggregation splits the Qwen3-Omni Thinker into two logical
|
||||||
|
stages: a prefill worker that processes the prompt and produces KV cache, and a
|
||||||
|
decode worker that imports that cache and generates tokens. The Talker and
|
||||||
|
Code2Wav stages remain downstream; PD does not disaggregate the entire
|
||||||
|
Qwen3-Omni pipeline.
|
||||||
|
|
||||||
|
## Current design
|
||||||
|
|
||||||
|
### Runtime topology
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
client["Client request"] --> orchestrator["Omni Orchestrator"]
|
||||||
|
orchestrator -->|"original prompt<br/>max_tokens = 1"| prefill["Thinker prefill<br/>KV producer"]
|
||||||
|
prefill -->|"KV blocks"| connector[("vLLM KV connector")]
|
||||||
|
prefill -->|"KV routing metadata"| orchestrator
|
||||||
|
orchestrator -->|"original prompt<br/>remote KV parameters"| decode["Thinker decode<br/>KV consumer"]
|
||||||
|
connector -->|"remote KV load"| decode
|
||||||
|
decode -->|"generated text"| client
|
||||||
|
```
|
||||||
|
|
||||||
|
The orchestrator owns the request lifecycle, while the vLLM KV connector owns
|
||||||
|
the bulk KV transfer. The original prompt is submitted to both Thinker stages:
|
||||||
|
the decode worker still needs the request tokens and multimodal features to
|
||||||
|
construct its request, while the imported KV cache prevents it from repeating
|
||||||
|
the completed prefill computation.
|
||||||
|
|
||||||
|
For the full Qwen3-Omni pipeline, splitting the Thinker adds one logical stage:
|
||||||
|
|
||||||
|
| Stage | Role | Output or handoff |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 0 | Thinker prefill | Saves prompt KV and prompt-side multimodal output |
|
||||||
|
| 1 | Thinker decode | Loads remote KV, generates text, and produces decode-side conditioning |
|
||||||
|
| 2 | Talker | Converts combined Thinker conditioning into codec codes |
|
||||||
|
| 3 | Code2Wav | Converts codec codes into audio |
|
||||||
|
|
||||||
|
Text remains a final output of the decode stage, and audio remains a final
|
||||||
|
output of Code2Wav.
|
||||||
|
|
||||||
|
### Request lifecycle
|
||||||
|
|
||||||
|
1. At startup, PD detection looks for one stage marked `is_prefill_only` and
|
||||||
|
one downstream stage marked `is_decode_only`. The decode stage must list the
|
||||||
|
prefill stage as an input source.
|
||||||
|
2. The orchestrator clones the Thinker sampling parameters for prefill, forces
|
||||||
|
`max_tokens=1`, clears stop conditions, and sets producer-side
|
||||||
|
`kv_transfer_params`. This makes prefill finish by length after exporting KV.
|
||||||
|
3. When prefill finishes, the orchestrator captures its KV routing metadata and
|
||||||
|
Qwen multimodal output under the original request ID.
|
||||||
|
4. The orchestrator submits the original prompt to decode with consumer-side
|
||||||
|
`kv_transfer_params`. Those parameters identify the transfer, prefill engine,
|
||||||
|
bootstrap endpoint, and, when required by the connector, remote request.
|
||||||
|
5. The decode engine imports the KV cache and continues generation. The normal
|
||||||
|
Omni stage path then carries Thinker conditioning through Talker and
|
||||||
|
Code2Wav.
|
||||||
|
|
||||||
|
### Control and data paths
|
||||||
|
|
||||||
|
PD uses two different connector layers. They solve different handoffs and are
|
||||||
|
not interchangeable.
|
||||||
|
|
||||||
|
| Path | Payload | Owner |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| PD control path | Request ID, original prompt, `transfer_id`, bootstrap address, engine ID, and connector-specific request metadata | Omni Orchestrator |
|
||||||
|
| PD data path | Thinker attention KV blocks | vLLM KV connector |
|
||||||
|
| Downstream model path | Thinker hidden states and embeddings, Talker codec codes, and audio tensors | vLLM-Omni stage connectors and processors |
|
||||||
|
|
||||||
|
### Detection and validation
|
||||||
|
|
||||||
|
The current runtime accepts one logical prefill/decode pair per pipeline. Each
|
||||||
|
stage may use tensor parallelism, but both sides of the pair must satisfy these
|
||||||
|
invariants:
|
||||||
|
|
||||||
|
- both stages define `kv_transfer_config`
|
||||||
|
- prefill uses `kv_producer` or `kv_both`
|
||||||
|
- decode uses `kv_consumer` or `kv_both`
|
||||||
|
- both stages use the same KV connector
|
||||||
|
- `kv_buffer_device` and `kv_buffer_size` agree when set on both stages
|
||||||
|
- both stages use the same `tensor_parallel_size`
|
||||||
|
|
||||||
|
These checks happen before request processing so an inconsistent PD topology
|
||||||
|
fails during engine initialization.
|
||||||
|
|
||||||
|
## What is implemented today
|
||||||
|
|
||||||
|
| Area | Current state |
|
||||||
|
| --- | --- |
|
||||||
|
| Pair detection and topology validation | Implemented in the shared PD helper |
|
||||||
|
| Prefill sampling preparation | Implemented for sync and async entrypoints |
|
||||||
|
| Prefill-to-decode routing | Implemented in the current Orchestrator |
|
||||||
|
| Deploy-based configuration | Not yet migrated; the supported `stages:` deploy schema cannot currently expand Qwen3-Omni into the four-stage PD topology |
|
||||||
|
| Qwen3 Thinker-to-Talker handoff | Bridge state and embedding-merge helpers exist, but the merge is not wired into the live stage processor |
|
||||||
|
| Connector compatibility | Incomplete; the Orchestrator currently requires `remote_request_id`, while the pinned Mooncake connector completes prefill without returning that metadata |
|
||||||
|
| Validation coverage | PD entrypoint tests and the PD deploy overlay in online end-to-end tests are temporarily disabled during migration |
|
||||||
|
| Performance | No non-regression or throughput/latency result is established for the current path |
|
||||||
|
|
||||||
|
Because these pieces are still converging, the previous legacy `stage_args` YAML
|
||||||
|
and `vllm serve` command have been removed. The serve CLI now accepts the
|
||||||
|
deploy-based `stages:` schema, so retaining the legacy recipe would imply a
|
||||||
|
launch path that current `main` rejects.
|
||||||
|
|
||||||
|
## Configuration contract under development
|
||||||
|
|
||||||
|
The intended deploy integration starts from the
|
||||||
|
[default Qwen3-Omni deploy config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
|
||||||
|
and expands its frozen three-stage pipeline into the four stages shown above.
|
||||||
|
A complete integration must:
|
||||||
|
|
||||||
|
- split the Thinker definition without duplicating the Talker or Code2Wav model
|
||||||
|
contracts
|
||||||
|
- remap downstream stage IDs, input sources, and stage connectors
|
||||||
|
- inject producer and consumer `kv_transfer_config` values into the two Thinker
|
||||||
|
stages
|
||||||
|
- allocate separate device sets for prefill and decode
|
||||||
|
- preserve both final outputs: decode text and Code2Wav audio
|
||||||
|
- provide connector-specific routing metadata without hard-coding one connector
|
||||||
|
contract
|
||||||
|
|
||||||
|
The required number of GPUs depends on tensor-parallel sizes and whether Talker
|
||||||
|
and Code2Wav share a device. A basic TP=1 layout uses one device for prefill, one
|
||||||
|
for decode, and one for the two downstream audio stages.
|
||||||
|
|
||||||
|
## Implementation map
|
||||||
|
|
||||||
|
- PD detection, validation, and sampling-parameter preparation:
|
||||||
|
[`PDDisaggregationMixin`](gh-file:vllm_omni/entrypoints/pd_utils.py)
|
||||||
|
- startup detection and connector metadata:
|
||||||
|
[`AsyncOmniEngine`](gh-file:vllm_omni/engine/async_omni_engine.py)
|
||||||
|
- request-scoped prefill-to-decode routing:
|
||||||
|
[`Orchestrator`](gh-file:vllm_omni/engine/orchestrator.py)
|
||||||
|
- Qwen3 downstream conditioning helpers:
|
||||||
|
[Qwen3-Omni stage input processor](gh-file:vllm_omni/model_executor/stage_input_processors/qwen3_omni.py)
|
||||||
+3
-3
@@ -10,12 +10,12 @@ A: Refer to [GPU memory calculation and configuration](../configuration/gpu_memo
|
|||||||
|
|
||||||
> Q: I encounter some bugs or CI problems, which is urgent. How can I solve it?
|
> Q: I encounter some bugs or CI problems, which is urgent. How can I solve it?
|
||||||
|
|
||||||
A: Search the current [issues](https://github.com/vllm-project/vllm-omni/issues) first. If no existing issue matches a bug or CI failure, open a [bug report](https://github.com/vllm-project/vllm-omni/issues/new?template=400-bug-report.yml) with a minimal reproducer, environment details, and relevant logs. For technical or usage questions, use the `sig-omni` channel in [vLLM Slack](https://slack.vllm.ai/) or the [vLLM Forum](https://discuss.vllm.ai/).
|
A: At first, you can check current [issues](https://github.com/vllm-project/vllm-omni/issues) to find possible solutions. If none of these satisfy your demand and it is urgent, please find these [volunteers](https://docs.vllm.ai/projects/vllm-omni/en/latest/community/volunteers/) for help.
|
||||||
|
|
||||||
> Q: Does vLLM-Omni support AWQ or any other quantization?
|
> Q: Does vLLM-Omni support AWQ or any other quantization?
|
||||||
|
|
||||||
A: AWQ is available through the inherited vLLM quantization registry, and vLLM-Omni supports several additional methods. Actual support and validation vary by model, component, hardware, and checkpoint. See the [quantization overview](../user_guide/quantization/overview.md) for validated combinations and method-specific guides.
|
A: We plan to introduce GGUF FP8 prequantized models and online FP8 quantization in version 0.16.0. Support for other quantization types will follow in future releases. For details, please see our [Q1 quantization roadmap](https://github.com/vllm-project/vllm-omni/issues/1057).
|
||||||
|
|
||||||
> Q: Does vLLM-Omni support multimodal streaming input and output?
|
> Q: Does vLLM-Omni support multimodal streaming input and output?
|
||||||
|
|
||||||
A: Yes, for specific models and serving paths. Current capabilities include [streaming video input](../serving/video_stream_api.md), [streaming diffusion output](../user_guide/diffusion/execution_modes.md#streaming-output), and experimental full-duplex real-time audio described in the [project overview](../README.md). Check the model and API documentation because input and output support is model-specific.
|
A: Not yet. We already put it on the [Roadmap](https://github.com/vllm-project/vllm-omni/issues/165). Please stay tuned!
|
||||||
+1
-2
@@ -73,12 +73,11 @@ plugins:
|
|||||||
- awesome-nav
|
- awesome-nav
|
||||||
- redirects:
|
- redirects:
|
||||||
redirect_maps:
|
redirect_maps:
|
||||||
community/meetups.md: community/meetings.md
|
|
||||||
community/volunteers.md: community/contact_us.md
|
|
||||||
user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
|
user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
|
||||||
user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
|
user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
|
||||||
user_guide/diffusion/cpu_offload_diffusion.md: user_guide/diffusion/cpu_offload.md
|
user_guide/diffusion/cpu_offload_diffusion.md: user_guide/diffusion/cpu_offload.md
|
||||||
user_guide/quantization/index.md: user_guide/quantization/overview.md
|
user_guide/quantization/index.md: user_guide/quantization/overview.md
|
||||||
|
configuration/pd_disaggregation.md: features/pd_disaggregation.md
|
||||||
design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
|
design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
|
||||||
design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
|
design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
|
||||||
- glightbox
|
- glightbox
|
||||||
|
|||||||
@@ -189,7 +189,7 @@ def _generate_single_stage_video(
|
|||||||
|
|
||||||
|
|
||||||
def _generate_bagel_image(
|
def _generate_bagel_image(
|
||||||
quantization_config: str | None = None,
|
diffusion_quantization_config: str | None = None,
|
||||||
num_inference_steps: int = 15,
|
num_inference_steps: int = 15,
|
||||||
) -> tuple[Any, float]:
|
) -> tuple[Any, float]:
|
||||||
"""Generate an image with BAGEL (multi-stage: LLM + Diffusion).
|
"""Generate an image with BAGEL (multi-stage: LLM + Diffusion).
|
||||||
@@ -202,8 +202,8 @@ def _generate_bagel_image(
|
|||||||
"stage_configs_path": config_path,
|
"stage_configs_path": config_path,
|
||||||
"stage_init_timeout": 300,
|
"stage_init_timeout": 300,
|
||||||
}
|
}
|
||||||
if quantization_config:
|
if diffusion_quantization_config:
|
||||||
omni_kwargs["quantization_config"] = quantization_config
|
omni_kwargs["diffusion_quantization_config"] = diffusion_quantization_config
|
||||||
|
|
||||||
model_name = omni_kwargs.pop("model")
|
model_name = omni_kwargs.pop("model")
|
||||||
with OmniRunner(model_name, **omni_kwargs) as runner:
|
with OmniRunner(model_name, **omni_kwargs) as runner:
|
||||||
@@ -380,21 +380,21 @@ def test_single_stage_ltx2_fp8_uses_less_memory():
|
|||||||
|
|
||||||
@hardware_test(res={"cuda": "H100"})
|
@hardware_test(res={"cuda": "H100"})
|
||||||
def test_bagel_fp8_generates_image():
|
def test_bagel_fp8_generates_image():
|
||||||
"""BAGEL with FP8 quantization_config generates a valid image.
|
"""BAGEL with diffusion-stage FP8 generates a valid image.
|
||||||
|
|
||||||
FP8 should only apply to the diffusion stage (Stage-1), not the
|
FP8 should only apply to the diffusion stage (Stage-1), not the
|
||||||
LLM stage (Stage-0). We verify this by checking:
|
LLM stage (Stage-0). We verify this by checking:
|
||||||
1. Image is generated successfully
|
1. Image is generated successfully
|
||||||
2. LLM stage finish_reason is 'stop' (not 'length' from garbled output)
|
2. LLM stage finish_reason is 'stop' (not 'length' from garbled output)
|
||||||
"""
|
"""
|
||||||
image, _ = _generate_bagel_image(quantization_config="fp8")
|
image, _ = _generate_bagel_image(diffusion_quantization_config="fp8")
|
||||||
image.save("test_bagel_fp8.png")
|
image.save("test_bagel_fp8.png")
|
||||||
|
|
||||||
|
|
||||||
@hardware_test(res={"cuda": "H100"})
|
@hardware_test(res={"cuda": "H100"})
|
||||||
def test_bagel_bf16_generates_image():
|
def test_bagel_bf16_generates_image():
|
||||||
"""BAGEL without quantization generates a valid image (baseline)."""
|
"""BAGEL without quantization generates a valid image (baseline)."""
|
||||||
image, _ = _generate_bagel_image(quantization_config=None)
|
image, _ = _generate_bagel_image(diffusion_quantization_config=None)
|
||||||
image.save("test_bagel_bf16.png")
|
image.save("test_bagel_bf16.png")
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user