Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b031feef06 | ||
|
|
c8bc80cb96 | ||
|
|
8ff87320c2 |
No files matched your search
+2
-2
@@ -43,7 +43,6 @@ nav:
|
||||
- Overview: features/README.md
|
||||
- Runtime and Stage Execution:
|
||||
- Execution Modes and Streaming: user_guide/diffusion/execution_modes.md
|
||||
- Prefill-Decode Disaggregation (experimental): features/pd_disaggregation.md
|
||||
- Sleep Mode: features/sleep_mode.md
|
||||
- Quantization:
|
||||
- Overview: user_guide/quantization/overview.md
|
||||
@@ -164,7 +163,8 @@ nav:
|
||||
- CLI Reference: cli
|
||||
- Community:
|
||||
- Governance: community/governance.md
|
||||
- community/*
|
||||
- Contact Us: community/contact_us.md
|
||||
- Community Meetings: community/meetings.md
|
||||
- Slack: https://slack.vllm.ai
|
||||
- Blog: https://blog.vllm.ai
|
||||
- Forum: https://discuss.vllm.ai
|
||||
@@ -1,5 +1,9 @@
|
||||
# Contact Us
|
||||
|
||||
- For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm-omni/issues)
|
||||
- For coordinating contributions and development and discussing with other users and developers, please join `sig-omni` channel in our [Slack](https://slack.vllm.ai/) or use the [vLLM Forum](https://discuss.vllm.ai/)
|
||||
- For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm-omni/security/advisories) feature
|
||||
- For bugs and feature requests, search or open a GitHub [issue](https://github.com/vllm-project/vllm-omni/issues).
|
||||
- For technical and usage questions, join the `sig-omni` channel in [vLLM Slack](https://slack.vllm.ai/) or use the [vLLM Forum](https://discuss.vllm.ai/).
|
||||
- For coordinating contributions and development, use `sig-omni`, the forum, or join a [community meeting](meetings.md).
|
||||
|
||||
## Security Disclosures
|
||||
|
||||
Do not share vulnerability details in public GitHub issues, pull requests, discussions, or Slack channels. To arrange a private report, send a direct message to one of the [lead maintainers](governance.md#lead-maintainers) in [vLLM Slack](https://slack.vllm.ai/). If you cannot send a direct message, you may post a detail-free request in `sig-omni` asking a lead maintainer to contact you privately; do not include any vulnerability details in that request.
|
||||
@@ -52,31 +52,11 @@ Committers have write access and merge rights. They typically have deep expertis
|
||||
- [@ZJY0516](https://github.com/ZJY0516): Diffusion attention backend, kernel fusion, and CustomOp
|
||||
- [@yuanheng-zhao](https://github.com/yuanheng-zhao): Diffusion cache, offload, and Omni Support
|
||||
|
||||
## Diffusion Workload Division
|
||||
## Reviewer Routing
|
||||
|
||||
This section breaks down diffusion-related responsibilities. If you have PRs touching these areas, please ping the listed owners for review.
|
||||
For path-based reviewer routing, consult [`.github/CODEOWNERS`](https://github.com/vllm-project/vllm-omni/blob/main/.github/CODEOWNERS). Detailed subsystem boundaries and safe-change guidance live in the [module design documents](../design/index.md#module-design-documents).
|
||||
|
||||
### Runtime-related
|
||||
|
||||
- Diffusion models: @RuixiangMa, @wtomin, @xuechendi
|
||||
|
||||
### Optimization
|
||||
|
||||
- Parallel: @RuixiangMa, @wtomin
|
||||
- Attention backend: @ZJY0516
|
||||
- Cache: @yuanheng-zhao, @RuixiangMa, @SamitHuang
|
||||
- Offload: @yuanheng-zhao
|
||||
- Intel GPU support: @xuechendi
|
||||
- Quantization: @lishunyang12, @david6666666
|
||||
- Kernel fusion / communication-computation: @ZJY0516
|
||||
|
||||
### Docs & Test
|
||||
|
||||
- Diffusion docs: @RuixiangMa, @wtomin
|
||||
|
||||
## Meetings
|
||||
|
||||
Committers hold **bi-weekly meetings** to discuss future directions and collaborations of the project.
|
||||
`CODEOWNERS` is operational routing metadata, not a governance or approval policy. Being listed does not grant committer status or merge rights, and it does not mean every listed reviewer must approve a change. The maintainer roster above is authoritative for project roles and merge rights.
|
||||
|
||||
## Committer Nomination Process
|
||||
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
# Community Meetings
|
||||
|
||||
vLLM-Omni hosts developer-facing meetings for Chinese- and English-language audiences. Schedules and access details can change, so use these maintained documents as the source of truth:
|
||||
|
||||
- [Chinese-language meeting document](https://docs.google.com/document/d/1pdUBiS_7mdOUNDtdwy-9OUf7jsMWN324BIbps5olDME/edit?tab=t.0)
|
||||
- [English-language meeting document](https://docs.google.com/document/d/19AlD08T8uJ7wqfs8VtYCi0iGMTwlTYVVlAxFZ3S7OIg/edit?tab=t.0)
|
||||
|
||||
Each document contains the current schedule, access details, agendas, and past meeting notes. For broader vLLM community events, see the [vLLM events calendar](https://vllm.ai/events/).
|
||||
@@ -1 +0,0 @@
|
||||
# Meetups
|
||||
@@ -1,12 +0,0 @@
|
||||
# Volunteers for Bugfix and CI
|
||||
|
||||
We encourage you to check current docs and [issues](https://github.com/vllm-project/vllm-omni/issues) to find possible solutions for your questions. If non of these can solve it, please propose an issue to describe your questions about bug or CI problems for developing.
|
||||
|
||||
If you have urgent need for locating and solving bugfix or CI problems, please find community volunteers below.
|
||||
|
||||
| Dec 4-Dec 12 | Dec 15-Dec 19 | Dec 22-Dec 26 | Dec 29- Jan 2, 2026| Jan 5-Jan 9 | Jan 12-Jan 16 |
|
||||
|----------|----------|----------|----------|----------|----------|
|
||||
| <a href="https://github.com/congw729">Conw729</a> | <a href="https://github.com/yinpeiqi">yinpeiqi</a> | <a href="https://github.com/tzhouam">tzhouam</a> | <a href="https://github.com/SamitHuang">SamitHuang</a> | <a href="https://github.com/gcanlin">gcanlin</a> | <a href="https://github.com/natureofnature">natureofnature</a> |
|
||||
| <a href="https://github.com/david6666666">david6666666</a> | <a href="https://github.com/R2-Y">R2-Y</a> | <a href="https://github.com/hsliuustc0106">hsliuustc0106</a> | <a href="https://github.com/Gaohan123">Gaohan123</a> | <a href="https://github.com/ZJY0516">ZJY0516</a> | <a href="https://github.com/qibaoyuan">qibaoyuan</a> |
|
||||
|
||||
We kindly welcome more contributors to fix bugs and contribute new features!
|
||||
@@ -14,6 +14,10 @@ For introduction, please check [Introduction for stage config](./stage_configs.m
|
||||
|
||||
- **[GPU Memory Calculation and Configuration](./gpu_memory_utilization.md)** - Guide on how to calculate memory requirements and set up `gpu_memory_utilization` for optimal performance
|
||||
|
||||
## Multi-Stage Recipes
|
||||
|
||||
- **[Prefill-Decode Disaggregation](./pd_disaggregation.md)** - How to derive a PD-aware Qwen3-Omni stage config from the default config without introducing another bundled YAML
|
||||
|
||||
## Optimization Features
|
||||
|
||||
- **[Diffusion Features Overview](../user_guide/diffusion_features.md)** - Complete overview of all diffusion model features and supported models
|
||||
@@ -0,0 +1,171 @@
|
||||
# Prefill-Decode (PD) Disaggregation
|
||||
|
||||
PD disaggregation splits the Qwen3-Omni thinker into separate prefill and decode
|
||||
stages so prompt processing and token generation can run on different workers.
|
||||
|
||||
This is documented as a stage-config recipe instead of a bundled YAML because the
|
||||
deployment-specific values usually change per environment:
|
||||
|
||||
- GPU placement
|
||||
- `tensor_parallel_size`
|
||||
- connector backend and connector ports
|
||||
- connector IPs or bootstrap addresses
|
||||
|
||||
Start from the [default Qwen3-Omni stage config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
|
||||
and copy it to your own file, for example `qwen3_omni_pd.yaml`. Then apply the
|
||||
changes below.
|
||||
|
||||
## Requirements
|
||||
|
||||
- 3+ GPUs for a basic layout: prefill, decode, and talker+code2wav
|
||||
- A KV connector supported by vLLM, such as `MooncakeConnector`
|
||||
- Matching `tensor_parallel_size` on the prefill and decode thinker stages
|
||||
|
||||
## 1. Split the thinker into prefill and decode stages
|
||||
|
||||
Replace the original thinker stage with two stages:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
stage_type: llm
|
||||
is_prefill_only: true
|
||||
runtime:
|
||||
devices: "0"
|
||||
engine_args:
|
||||
max_num_seqs: 16
|
||||
model_stage: thinker
|
||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
hf_config_name: thinker_config
|
||||
tensor_parallel_size: 1
|
||||
kv_transfer_config:
|
||||
kv_connector: "MooncakeConnector"
|
||||
kv_role: "kv_producer"
|
||||
kv_rank: 0
|
||||
kv_parallel_size: 2
|
||||
kv_connector_extra_config:
|
||||
mooncake_bootstrap_port: 25201
|
||||
final_output: false
|
||||
is_comprehension: true
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
top_k: 1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.05
|
||||
|
||||
- stage_id: 1
|
||||
stage_type: llm
|
||||
is_decode_only: true
|
||||
runtime:
|
||||
devices: "1"
|
||||
engine_args:
|
||||
max_num_seqs: 64
|
||||
model_stage: thinker
|
||||
model_arch: Qwen3OmniMoeForConditionalGeneration
|
||||
worker_type: ar
|
||||
scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
|
||||
gpu_memory_utilization: 0.9
|
||||
enforce_eager: true
|
||||
trust_remote_code: true
|
||||
engine_output_type: latent
|
||||
distributed_executor_backend: "mp"
|
||||
enable_prefix_caching: false
|
||||
max_num_batched_tokens: 32768
|
||||
hf_config_name: thinker_config
|
||||
tensor_parallel_size: 1
|
||||
kv_transfer_config:
|
||||
kv_connector: "MooncakeConnector"
|
||||
kv_role: "kv_consumer"
|
||||
kv_rank: 1
|
||||
kv_parallel_size: 2
|
||||
kv_connector_extra_config:
|
||||
mooncake_bootstrap_port: 25202
|
||||
engine_input_source: [0]
|
||||
final_output: true
|
||||
final_output_type: text
|
||||
is_comprehension: true
|
||||
default_sampling_params:
|
||||
temperature: 0.4
|
||||
top_p: 0.9
|
||||
top_k: 1
|
||||
max_tokens: 2048
|
||||
seed: 42
|
||||
detokenize: True
|
||||
repetition_penalty: 1.05
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `is_prefill_only: true` marks the thinker stage that only saves KV.
|
||||
- `is_decode_only: true` marks the thinker stage that resumes from remote KV.
|
||||
- `kv_transfer_config` is required on both stages.
|
||||
- The orchestrator forces the prefill stage to run with `max_tokens=1`, so the
|
||||
prefill side only processes the prompt and exports KV.
|
||||
|
||||
## 2. Shift the downstream stages by one index
|
||||
|
||||
After inserting the extra thinker stage, renumber the remaining stages:
|
||||
|
||||
```yaml
|
||||
- stage_id: 2
|
||||
runtime:
|
||||
devices: "2"
|
||||
engine_input_source: [1]
|
||||
sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_token_only
|
||||
|
||||
- stage_id: 3
|
||||
runtime:
|
||||
devices: "2"
|
||||
engine_args:
|
||||
max_num_seqs: 1
|
||||
engine_input_source: [2]
|
||||
sync_process_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_token_only
|
||||
```
|
||||
|
||||
Compared with the default Qwen3-Omni config:
|
||||
|
||||
- the talker becomes stage `2` instead of stage `1`
|
||||
- the code2wav stage becomes stage `3` instead of stage `2`
|
||||
- the talker now reads from decode stage `1`
|
||||
|
||||
## 3. Add runtime edges for the four-stage pipeline
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
enabled: true
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
- from: 1
|
||||
to: 2
|
||||
- from: 2
|
||||
to: 3
|
||||
```
|
||||
|
||||
## 4. Launch with your custom config
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
--deploy-config /path/to/qwen3_omni_pd.yaml
|
||||
```
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- `MooncakeConnector` does not support heterogeneous TP sizes across the PD
|
||||
pair. Keep prefill and decode at the same `tensor_parallel_size`.
|
||||
- If the thinker requires TP=2, both thinker stages must use TP=2 and be given
|
||||
separate GPU sets, for example `"0,1"` for prefill and `"2,3"` for decode.
|
||||
- Choose connector ports and addresses that match your deployment. The values
|
||||
shown above are examples only.
|
||||
@@ -3,7 +3,7 @@
|
||||
Thank you for your interest in contributing to vLLM-Omni! This document provides guidelines and instructions for contributing.
|
||||
|
||||
!!! note
|
||||
We host weekly developer-facing online meetings to discuss milestones and updates **every Tuesday at 19:30 PDT**. Meeting link as well as the past meeting notes can be found [here](https://tinyurl.com/vllm-omni-meeting).
|
||||
vLLM-Omni hosts developer-facing meetings for Chinese- and English-language audiences. See [Community Meetings](../community/meetings.md) for current schedules, access details, agendas, and past notes.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -122,7 +122,7 @@ Repository skills guide the agent, but they do not replace contributor judgment,
|
||||
If you encounter a bug or have a feature request, please search existing issues first to see if it has already been reported. If not, please file a new issue, providing as much relevant information as possible.
|
||||
|
||||
!!! important
|
||||
If you discover a security vulnerability, please report it by creating a GitHub issue with the `security` label.
|
||||
Do not report suspected security vulnerabilities through a public issue, pull request, discussion, or Slack channel. Follow the [security disclosure instructions](../community/contact_us.md#security-disclosures) to arrange a private report.
|
||||
|
||||
## Pull Requests & Code Reviews
|
||||
|
||||
|
||||
@@ -20,19 +20,13 @@ an implementation contract has a user-facing workflow.
|
||||
| Choose serial, batched, step-wise, or streaming diffusion execution | [Execution Modes and Streaming](../user_guide/diffusion/execution_modes.md) | [Diffusion Continuous Batching](../design/feature/diffusion_continuous_batching.md), [Async Diffusion Output](../design/feature/async_diffusion_output.md) |
|
||||
| Reclaim stage memory without restarting the server | [Sleep Mode](sleep_mode.md) | Runtime lifecycle behavior is documented in the user guide |
|
||||
|
||||
The [Prefill-Decode Disaggregation design reference](pd_disaggregation.md)
|
||||
describes the experimental Qwen3-Omni runtime topology and its current
|
||||
integration boundaries. It is not yet a supported launch recipe.
|
||||
|
||||
Some runtime designs are deliberately not promoted as standalone User Guide
|
||||
features yet:
|
||||
|
||||
- Generic, OmniConnector-based
|
||||
[Disaggregated Inference](../design/feature/disaggregated_inference.md) is not
|
||||
yet a standalone User Guide feature. Its
|
||||
- [Disaggregated Inference](../design/feature/disaggregated_inference.md) and
|
||||
[OmniConnector implementations](../design/index.md#communication) describe
|
||||
transport contracts, while practical connector configuration remains under
|
||||
[Pipeline and deploy configurations](../configuration/stage_configs.md).
|
||||
topology and transport contracts. Practical deployment configuration remains
|
||||
under [Pipeline and deploy configurations](../configuration/stage_configs.md).
|
||||
- [Async Chunk](../design/feature/async_chunk.md) and
|
||||
[Async Omni Output Materialization](../design/feature/omni_async_output_materialization.md)
|
||||
are model- and pipeline-dependent. Use the selected model's deploy
|
||||
|
||||
@@ -1,140 +0,0 @@
|
||||
# Prefill-Decode (PD) Disaggregation (experimental)
|
||||
|
||||
!!! warning "Experimental design reference"
|
||||
PD disaggregation is only partially integrated on the current `main` branch.
|
||||
The runtime contains PD detection and routing scaffolding, but there is no
|
||||
supported deploy overlay or end-to-end validated launch recipe yet. Treat this
|
||||
page as a description of the current design, not as production guidance.
|
||||
|
||||
Prefill-decode disaggregation splits the Qwen3-Omni Thinker into two logical
|
||||
stages: a prefill worker that processes the prompt and produces KV cache, and a
|
||||
decode worker that imports that cache and generates tokens. The Talker and
|
||||
Code2Wav stages remain downstream; PD does not disaggregate the entire
|
||||
Qwen3-Omni pipeline.
|
||||
|
||||
## Current design
|
||||
|
||||
### Runtime topology
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
client["Client request"] --> orchestrator["Omni Orchestrator"]
|
||||
orchestrator -->|"original prompt<br/>max_tokens = 1"| prefill["Thinker prefill<br/>KV producer"]
|
||||
prefill -->|"KV blocks"| connector[("vLLM KV connector")]
|
||||
prefill -->|"KV routing metadata"| orchestrator
|
||||
orchestrator -->|"original prompt<br/>remote KV parameters"| decode["Thinker decode<br/>KV consumer"]
|
||||
connector -->|"remote KV load"| decode
|
||||
decode -->|"generated text"| client
|
||||
```
|
||||
|
||||
The orchestrator owns the request lifecycle, while the vLLM KV connector owns
|
||||
the bulk KV transfer. The original prompt is submitted to both Thinker stages:
|
||||
the decode worker still needs the request tokens and multimodal features to
|
||||
construct its request, while the imported KV cache prevents it from repeating
|
||||
the completed prefill computation.
|
||||
|
||||
For the full Qwen3-Omni pipeline, splitting the Thinker adds one logical stage:
|
||||
|
||||
| Stage | Role | Output or handoff |
|
||||
| --- | --- | --- |
|
||||
| 0 | Thinker prefill | Saves prompt KV and prompt-side multimodal output |
|
||||
| 1 | Thinker decode | Loads remote KV, generates text, and produces decode-side conditioning |
|
||||
| 2 | Talker | Converts combined Thinker conditioning into codec codes |
|
||||
| 3 | Code2Wav | Converts codec codes into audio |
|
||||
|
||||
Text remains a final output of the decode stage, and audio remains a final
|
||||
output of Code2Wav.
|
||||
|
||||
### Request lifecycle
|
||||
|
||||
1. At startup, PD detection looks for one stage marked `is_prefill_only` and
|
||||
one downstream stage marked `is_decode_only`. The decode stage must list the
|
||||
prefill stage as an input source.
|
||||
2. The orchestrator clones the Thinker sampling parameters for prefill, forces
|
||||
`max_tokens=1`, clears stop conditions, and sets producer-side
|
||||
`kv_transfer_params`. This makes prefill finish by length after exporting KV.
|
||||
3. When prefill finishes, the orchestrator captures its KV routing metadata and
|
||||
Qwen multimodal output under the original request ID.
|
||||
4. The orchestrator submits the original prompt to decode with consumer-side
|
||||
`kv_transfer_params`. Those parameters identify the transfer, prefill engine,
|
||||
bootstrap endpoint, and, when required by the connector, remote request.
|
||||
5. The decode engine imports the KV cache and continues generation. The normal
|
||||
Omni stage path then carries Thinker conditioning through Talker and
|
||||
Code2Wav.
|
||||
|
||||
### Control and data paths
|
||||
|
||||
PD uses two different connector layers. They solve different handoffs and are
|
||||
not interchangeable.
|
||||
|
||||
| Path | Payload | Owner |
|
||||
| --- | --- | --- |
|
||||
| PD control path | Request ID, original prompt, `transfer_id`, bootstrap address, engine ID, and connector-specific request metadata | Omni Orchestrator |
|
||||
| PD data path | Thinker attention KV blocks | vLLM KV connector |
|
||||
| Downstream model path | Thinker hidden states and embeddings, Talker codec codes, and audio tensors | vLLM-Omni stage connectors and processors |
|
||||
|
||||
### Detection and validation
|
||||
|
||||
The current runtime accepts one logical prefill/decode pair per pipeline. Each
|
||||
stage may use tensor parallelism, but both sides of the pair must satisfy these
|
||||
invariants:
|
||||
|
||||
- both stages define `kv_transfer_config`
|
||||
- prefill uses `kv_producer` or `kv_both`
|
||||
- decode uses `kv_consumer` or `kv_both`
|
||||
- both stages use the same KV connector
|
||||
- `kv_buffer_device` and `kv_buffer_size` agree when set on both stages
|
||||
- both stages use the same `tensor_parallel_size`
|
||||
|
||||
These checks happen before request processing so an inconsistent PD topology
|
||||
fails during engine initialization.
|
||||
|
||||
## What is implemented today
|
||||
|
||||
| Area | Current state |
|
||||
| --- | --- |
|
||||
| Pair detection and topology validation | Implemented in the shared PD helper |
|
||||
| Prefill sampling preparation | Implemented for sync and async entrypoints |
|
||||
| Prefill-to-decode routing | Implemented in the current Orchestrator |
|
||||
| Deploy-based configuration | Not yet migrated; the supported `stages:` deploy schema cannot currently expand Qwen3-Omni into the four-stage PD topology |
|
||||
| Qwen3 Thinker-to-Talker handoff | Bridge state and embedding-merge helpers exist, but the merge is not wired into the live stage processor |
|
||||
| Connector compatibility | Incomplete; the Orchestrator currently requires `remote_request_id`, while the pinned Mooncake connector completes prefill without returning that metadata |
|
||||
| Validation coverage | PD entrypoint tests and the PD deploy overlay in online end-to-end tests are temporarily disabled during migration |
|
||||
| Performance | No non-regression or throughput/latency result is established for the current path |
|
||||
|
||||
Because these pieces are still converging, the previous legacy `stage_args` YAML
|
||||
and `vllm serve` command have been removed. The serve CLI now accepts the
|
||||
deploy-based `stages:` schema, so retaining the legacy recipe would imply a
|
||||
launch path that current `main` rejects.
|
||||
|
||||
## Configuration contract under development
|
||||
|
||||
The intended deploy integration starts from the
|
||||
[default Qwen3-Omni deploy config](gh-file:vllm_omni/deploy/qwen3_omni_moe.yaml)
|
||||
and expands its frozen three-stage pipeline into the four stages shown above.
|
||||
A complete integration must:
|
||||
|
||||
- split the Thinker definition without duplicating the Talker or Code2Wav model
|
||||
contracts
|
||||
- remap downstream stage IDs, input sources, and stage connectors
|
||||
- inject producer and consumer `kv_transfer_config` values into the two Thinker
|
||||
stages
|
||||
- allocate separate device sets for prefill and decode
|
||||
- preserve both final outputs: decode text and Code2Wav audio
|
||||
- provide connector-specific routing metadata without hard-coding one connector
|
||||
contract
|
||||
|
||||
The required number of GPUs depends on tensor-parallel sizes and whether Talker
|
||||
and Code2Wav share a device. A basic TP=1 layout uses one device for prefill, one
|
||||
for decode, and one for the two downstream audio stages.
|
||||
|
||||
## Implementation map
|
||||
|
||||
- PD detection, validation, and sampling-parameter preparation:
|
||||
[`PDDisaggregationMixin`](gh-file:vllm_omni/entrypoints/pd_utils.py)
|
||||
- startup detection and connector metadata:
|
||||
[`AsyncOmniEngine`](gh-file:vllm_omni/engine/async_omni_engine.py)
|
||||
- request-scoped prefill-to-decode routing:
|
||||
[`Orchestrator`](gh-file:vllm_omni/engine/orchestrator.py)
|
||||
- Qwen3 downstream conditioning helpers:
|
||||
[Qwen3-Omni stage input processor](gh-file:vllm_omni/model_executor/stage_input_processors/qwen3_omni.py)
|
||||
+3
-3
@@ -10,12 +10,12 @@ A: Refer to [GPU memory calculation and configuration](../configuration/gpu_memo
|
||||
|
||||
> Q: I encounter some bugs or CI problems, which is urgent. How can I solve it?
|
||||
|
||||
A: At first, you can check current [issues](https://github.com/vllm-project/vllm-omni/issues) to find possible solutions. If none of these satisfy your demand and it is urgent, please find these [volunteers](https://docs.vllm.ai/projects/vllm-omni/en/latest/community/volunteers/) for help.
|
||||
A: Search the current [issues](https://github.com/vllm-project/vllm-omni/issues) first. If no existing issue matches a bug or CI failure, open a [bug report](https://github.com/vllm-project/vllm-omni/issues/new?template=400-bug-report.yml) with a minimal reproducer, environment details, and relevant logs. For technical or usage questions, use the `sig-omni` channel in [vLLM Slack](https://slack.vllm.ai/) or the [vLLM Forum](https://discuss.vllm.ai/).
|
||||
|
||||
> Q: Does vLLM-Omni support AWQ or any other quantization?
|
||||
|
||||
A: We plan to introduce GGUF FP8 prequantized models and online FP8 quantization in version 0.16.0. Support for other quantization types will follow in future releases. For details, please see our [Q1 quantization roadmap](https://github.com/vllm-project/vllm-omni/issues/1057).
|
||||
A: AWQ is available through the inherited vLLM quantization registry, and vLLM-Omni supports several additional methods. Actual support and validation vary by model, component, hardware, and checkpoint. See the [quantization overview](../user_guide/quantization/overview.md) for validated combinations and method-specific guides.
|
||||
|
||||
> Q: Does vLLM-Omni support multimodal streaming input and output?
|
||||
|
||||
A: Not yet. We already put it on the [Roadmap](https://github.com/vllm-project/vllm-omni/issues/165). Please stay tuned!
|
||||
A: Yes, for specific models and serving paths. Current capabilities include [streaming video input](../serving/video_stream_api.md), [streaming diffusion output](../user_guide/diffusion/execution_modes.md#streaming-output), and experimental full-duplex real-time audio described in the [project overview](../README.md). Check the model and API documentation because input and output support is model-specific.
|
||||
+2
-1
@@ -73,11 +73,12 @@ plugins:
|
||||
- awesome-nav
|
||||
- redirects:
|
||||
redirect_maps:
|
||||
community/meetups.md: community/meetings.md
|
||||
community/volunteers.md: community/contact_us.md
|
||||
user_guide/diffusion/request_batching.md: user_guide/diffusion/execution_modes.md
|
||||
user_guide/diffusion/step_execution.md: user_guide/diffusion/execution_modes.md
|
||||
user_guide/diffusion/cpu_offload_diffusion.md: user_guide/diffusion/cpu_offload.md
|
||||
user_guide/quantization/index.md: user_guide/quantization/overview.md
|
||||
configuration/pd_disaggregation.md: features/pd_disaggregation.md
|
||||
design/feature/diffusion_request_level_batching.md: design/feature/diffusion_continuous_batching.md
|
||||
design/feature/diffusion_step_execution.md: design/feature/diffusion_continuous_batching.md
|
||||
- glightbox
|
||||
|
||||
@@ -189,7 +189,7 @@ def _generate_single_stage_video(
|
||||
|
||||
|
||||
def _generate_bagel_image(
|
||||
diffusion_quantization_config: str | None = None,
|
||||
quantization_config: str | None = None,
|
||||
num_inference_steps: int = 15,
|
||||
) -> tuple[Any, float]:
|
||||
"""Generate an image with BAGEL (multi-stage: LLM + Diffusion).
|
||||
@@ -202,8 +202,8 @@ def _generate_bagel_image(
|
||||
"stage_configs_path": config_path,
|
||||
"stage_init_timeout": 300,
|
||||
}
|
||||
if diffusion_quantization_config:
|
||||
omni_kwargs["diffusion_quantization_config"] = diffusion_quantization_config
|
||||
if quantization_config:
|
||||
omni_kwargs["quantization_config"] = quantization_config
|
||||
|
||||
model_name = omni_kwargs.pop("model")
|
||||
with OmniRunner(model_name, **omni_kwargs) as runner:
|
||||
@@ -380,21 +380,21 @@ def test_single_stage_ltx2_fp8_uses_less_memory():
|
||||
|
||||
@hardware_test(res={"cuda": "H100"})
|
||||
def test_bagel_fp8_generates_image():
|
||||
"""BAGEL with diffusion-stage FP8 generates a valid image.
|
||||
"""BAGEL with FP8 quantization_config generates a valid image.
|
||||
|
||||
FP8 should only apply to the diffusion stage (Stage-1), not the
|
||||
LLM stage (Stage-0). We verify this by checking:
|
||||
1. Image is generated successfully
|
||||
2. LLM stage finish_reason is 'stop' (not 'length' from garbled output)
|
||||
"""
|
||||
image, _ = _generate_bagel_image(diffusion_quantization_config="fp8")
|
||||
image, _ = _generate_bagel_image(quantization_config="fp8")
|
||||
image.save("test_bagel_fp8.png")
|
||||
|
||||
|
||||
@hardware_test(res={"cuda": "H100"})
|
||||
def test_bagel_bf16_generates_image():
|
||||
"""BAGEL without quantization generates a valid image (baseline)."""
|
||||
image, _ = _generate_bagel_image(diffusion_quantization_config=None)
|
||||
image, _ = _generate_bagel_image(quantization_config=None)
|
||||
image.save("test_bagel_bf16.png")
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user