[Docs] Remove stale stage config references (#6270)
Signed-off-by: zwhzzz0821 <2831474076@qq.com>
This commit is contained in:
1 parent
efaf4276f9
commit
6f55a06fab
85 files changed
+566
-794
No files matched your search
@@ -335,8 +335,8 @@ The single resolution boundary is implemented by
|
||||
`StageConfigFactory.create_from_model()` and
|
||||
`VllmOmniConfig.from_pipeline_config()` in
|
||||
[`vllm_omni/config`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/config).
|
||||
The legacy `stage_args` YAML path remains only for models that have not yet
|
||||
migrated to `PipelineConfig` and `DeployConfig`.
|
||||
The legacy `stage_args` YAML path has been removed. Model topology now resolves
|
||||
through `PipelineConfig`, with runtime overrides supplied by `DeployConfig`.
|
||||
|
||||
In the typed path, each stage configuration derives from
|
||||
`BaseVllmOmniStageConfig` and is specialized as
|
||||
|
||||
@@ -211,38 +211,39 @@ In sequential mode, each stage must wait for the previous stage to complete enti
|
||||
|
||||
## Configuration
|
||||
|
||||
Enable async_chunk in stage configuration YAML:
|
||||
Enable async chunk mode in the deploy YAML:
|
||||
|
||||
```yaml
|
||||
async_chunk: true
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
engine_args:
|
||||
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_async_chunk
|
||||
- stage_id: 1
|
||||
engine_args:
|
||||
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_async_chunk
|
||||
```
|
||||
|
||||
### Stage Configuration
|
||||
The registered `PipelineConfig` owns the async handoff processor functions.
|
||||
For Qwen3-Omni these are `thinker2talker_async_chunk` and
|
||||
`talker2code2wav_async_chunk`; deploy YAML only selects the mode and runtime
|
||||
sizing.
|
||||
|
||||
### Configuration ownership
|
||||
|
||||
- `async_chunk: bool`: Enable/disable async chunk mode
|
||||
- `custom_process_next_stage_input_func: str`: Path to custom chunk processing function; receives `(transfer_manager, pooling_output, request)`. For qwen3-omni: `thinker2talker_async_chunk`, `talker2code2wav_async_chunk`
|
||||
- `stage_connector_config: dict`: Connector configuration
|
||||
- `worker_type: str`: Model type, e.g. `"ar"` or `"generation"` (used by OmniChunkTransferAdapter for mode-specific payload handling)
|
||||
- `async_chunk_process_next_stage_input_func: str`: Pipeline-owned chunk processor path
|
||||
- `execution_type: StageExecutionType`: Pipeline-owned runtime family
|
||||
- `connectors: dict`: Deploy-owned connector definitions
|
||||
- `max_num_seqs: int`: Maximum number of sequences for concurrent processing in the stage
|
||||
|
||||
|
||||
### Connector Configuration
|
||||
|
||||
```yaml
|
||||
connectors:
|
||||
- from_stage: 0
|
||||
to_stage: 1
|
||||
spec:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
stage_id: 0
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_shared_memory
|
||||
- stage_id: 1
|
||||
input_connectors:
|
||||
from_stage_0: connector_of_shared_memory
|
||||
```
|
||||
|
||||
### Code2Wav Batch Configuration
|
||||
@@ -250,13 +251,10 @@ connectors:
|
||||
For optimal performance with async_chunk, the code2wav stage should be configured with batching:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 2 # code2wav stage
|
||||
runtime:
|
||||
devices: "1"
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
max_num_seqs: 64 # Enables batched audio generation
|
||||
devices: "1"
|
||||
max_num_seqs: 64 # Enables batched audio generation
|
||||
```
|
||||
|
||||
## Related Files
|
||||
|
||||
@@ -59,19 +59,18 @@ This `metadata` must be passed through the control plane so `get()` can locate t
|
||||
|
||||
## Configuration Model
|
||||
|
||||
Define connectors in runtime:
|
||||
Define connectors at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
```
|
||||
|
||||
Wire stages to connectors:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_shared_memory
|
||||
|
||||
@@ -32,26 +32,25 @@ mooncake_master \
|
||||
|
||||
## Configuration
|
||||
|
||||
Define the connector in runtime:
|
||||
Define the connector at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_mooncake:
|
||||
name: MooncakeStoreConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
metadata_server: "http://<MASTER_IP>:8080/metadata"
|
||||
master: "<MASTER_IP>:50051"
|
||||
segment: 512000000
|
||||
localbuf: 64000000
|
||||
proto: "tcp"
|
||||
connectors:
|
||||
connector_of_mooncake:
|
||||
name: MooncakeStoreConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
metadata_server: "http://<MASTER_IP>:8080/metadata"
|
||||
master: "<MASTER_IP>:50051"
|
||||
segment: 512000000
|
||||
localbuf: 64000000
|
||||
proto: "tcp"
|
||||
```
|
||||
|
||||
Wire stages to the connector:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_mooncake
|
||||
|
||||
@@ -21,26 +21,25 @@ InfiniBand/RoCE NICs).
|
||||
|
||||
## Configuration
|
||||
|
||||
Define the connector in runtime:
|
||||
Define the connector at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
rdma_connector:
|
||||
name: MooncakeTransferEngineConnector
|
||||
extra:
|
||||
host: "auto" # Auto-detect local RDMA IP
|
||||
zmq_port: 50051 # ZMQ base port (see "Port Offset Scheme" below)
|
||||
protocol: "rdma" # "rdma" or "tcp"
|
||||
device_name: "" # RDMA device (e.g., "mlx5_0"), empty for auto-detect
|
||||
memory_pool_size: 4294967296 # 4 GB (CPU); use 2147483648 (2 GB) for GPU
|
||||
memory_pool_device: "cpu" # "cpu" for pinned memory (recommended), "cuda" for GPUDirect RDMA
|
||||
connectors:
|
||||
rdma_connector:
|
||||
name: MooncakeTransferEngineConnector
|
||||
extra:
|
||||
host: "auto" # Auto-detect local RDMA IP
|
||||
zmq_port: 50051 # ZMQ base port (see "Port Offset Scheme" below)
|
||||
protocol: "rdma" # "rdma" or "tcp"
|
||||
device_name: "" # RDMA device (e.g., "mlx5_0"), empty for auto-detect
|
||||
memory_pool_size: 4294967296 # 4 GB (CPU); use 2147483648 (2 GB) for GPU
|
||||
memory_pool_device: "cpu" # "cpu" for pinned memory (recommended), "cuda" for GPUDirect RDMA
|
||||
```
|
||||
|
||||
Wire stages to the connector:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: rdma_connector
|
||||
@@ -415,7 +414,7 @@ A receiver connector:
|
||||
- allocates receive buffers from its own pool
|
||||
- requests metadata or transfer service from the sender
|
||||
|
||||
The role is not inferred dynamically. It is injected by the stage configuration layer:
|
||||
The role is not inferred dynamically. It is injected from the deploy connector wiring:
|
||||
|
||||
- incoming edge for a stage -> `role="receiver"`
|
||||
- outgoing edge for a stage -> `role="sender"`
|
||||
|
||||
@@ -13,10 +13,9 @@ segment name is returned in metadata.
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -65,23 +65,22 @@ dscli stop --worker_address "${WORKER_IP}:31501"
|
||||
|
||||
## Configuration
|
||||
|
||||
Define the connector in runtime:
|
||||
Define the connector at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_yuanrong:
|
||||
name: YuanrongConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
port: 31501
|
||||
get_sub_timeout_ms: 1000
|
||||
connectors:
|
||||
connector_of_yuanrong:
|
||||
name: YuanrongConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
port: 31501
|
||||
get_sub_timeout_ms: 1000
|
||||
```
|
||||
|
||||
Wire stages to the connector:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_yuanrong
|
||||
|
||||
@@ -94,26 +94,23 @@ For Ascend P2P, configure the connector with `protocol: "ascend"` and an NPU
|
||||
memory pool:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
connectors:
|
||||
yuanrong_te_connector:
|
||||
name: YuanrongTransferEngineConnector
|
||||
extra:
|
||||
host: "auto"
|
||||
zmq_port: 50051
|
||||
rpc_port: "auto"
|
||||
protocol: "ascend"
|
||||
device_name: "auto"
|
||||
memory_pool_size: 1073741824
|
||||
memory_pool_device: "npu"
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
max_inflight: 1
|
||||
connectors:
|
||||
yuanrong_te_connector:
|
||||
name: YuanrongTransferEngineConnector
|
||||
extra:
|
||||
host: "auto"
|
||||
zmq_port: 50051
|
||||
rpc_port: "auto"
|
||||
protocol: "ascend"
|
||||
device_name: "auto"
|
||||
memory_pool_size: 1073741824
|
||||
memory_pool_device: "npu"
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
```
|
||||
|
||||
Important fields:
|
||||
@@ -141,31 +138,27 @@ For AR 4-way TP sending KV cache to DiT 4-way TP, set the stage TP topology in
|
||||
both stage `omni_kv_config` blocks:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
runtime:
|
||||
devices: "0,1,2,3"
|
||||
engine_args:
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_send_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
devices: "0,1,2,3"
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_send_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
output_connectors:
|
||||
to_stage_1: yuanrong_te_connector
|
||||
|
||||
- stage_id: 1
|
||||
runtime:
|
||||
devices: "4,5,6,7"
|
||||
engine_args:
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
devices: "4,5,6,7"
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
input_connectors:
|
||||
from_stage_0: yuanrong_te_connector
|
||||
```
|
||||
|
||||
@@ -41,7 +41,7 @@ With this in mind, consider the set of blocks in a 2D layout, where the row repr
|
||||
|
||||
### Example
|
||||
!!! note "Note 3"
|
||||
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. It can be enabled/disabled per-stage via the `enable_prefix_caching` parameter in the model's stage config.
|
||||
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. Configure it with the pipeline-wide `enable_prefix_caching` field in the deploy config.
|
||||
|
||||
The way in which vLLM-Omni ties into vLLM's prefix caching is best understood by example. Say that we have the following:
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ and execution behavior to stable runtime contracts.
|
||||
### MODEL-INV-001: Registration is explicit
|
||||
|
||||
**Rule:** A model integration MUST declare how its model class, loader, input
|
||||
processor, and stage configuration are selected.
|
||||
processor, and pipeline configuration are selected.
|
||||
|
||||
### MODEL-INV-002: Model code does not route stages
|
||||
|
||||
|
||||
@@ -10,9 +10,8 @@ owners:
|
||||
primary_code_paths:
|
||||
- vllm_omni/config/**
|
||||
- vllm_omni/deploy/**
|
||||
- vllm_omni/model_executor/stage_configs/**
|
||||
related_code_paths:
|
||||
- vllm_omni/platforms/*/stage_configs/**
|
||||
- vllm_omni/model_executor/models/*/pipeline.py
|
||||
related_code_paths: []
|
||||
depends_on: []
|
||||
validation_paths:
|
||||
- tests/config/**
|
||||
@@ -95,7 +94,7 @@ parallel input is effective for a diffusion stage.
|
||||
## Safe-change guide while deferred
|
||||
|
||||
Changes should trace every affected producer to its runtime consumer and test
|
||||
the structured, legacy, CLI, and deployment paths that are actually supported.
|
||||
the structured pipeline, CLI, and deployment paths that are actually supported.
|
||||
For inherited vLLM fields, tests should separately cover constructor behavior
|
||||
and effective `EngineArgs` projection. Do not infer a stable contract from
|
||||
this placeholder page.
|
||||
|
||||
@@ -111,7 +111,7 @@ This post walks through each optimization in the same order they are typically e
|
||||
|
||||
For both Qwen3-Omni and Qwen3-TTS, batching is a pipeline-level optimization:
|
||||
|
||||
- Requests are grouped per stage using `runtime.max_batch_size`
|
||||
- Requests are grouped per stage using `stages[].max_num_seqs`
|
||||
- Each stage executes batch inference with its own scheduler/worker
|
||||
- Stage outputs are routed to downstream stages with per-request mapping preserved
|
||||
|
||||
@@ -171,7 +171,7 @@ At concurrency 10, batching alone brings Qwen3-TTS RTF from 2.19 (slower than re
|
||||
|
||||
In decode-heavy serving, repeatedly launching many small kernels from CPU can become a visible overhead. CUDA Graph reduces this overhead by capturing and replaying stable execution graphs.
|
||||
|
||||
In stage configs, this is represented by `enforce_eager: false` for stages where graph capture is desired (Thinker/Talker), while Code2Wav keeps eager mode depending on stage behavior.
|
||||
In deploy configs, this is represented by `enforce_eager: false` for stages where graph capture is desired (Thinker/Talker), while Code2Wav keeps eager mode depending on stage behavior.
|
||||
|
||||
### CUDA Graph results on top of batching
|
||||
|
||||
@@ -320,7 +320,7 @@ In the async chunk pipeline, the standard `codec_chunk_frames` is 25 (each chunk
|
||||
|
||||
**Dynamic initial chunk sizing (default behavior):**
|
||||
|
||||
Rather than using a fixed initial chunk size, vLLM-Omni dynamically selects it based on current server load. The initial chunk size is chosen from power-of-2 steps [2, 4, 8, 16] based on load factor (`active_requests / max_batch_size`):
|
||||
Rather than using a fixed initial chunk size, vLLM-Omni dynamically selects it based on current server load. The initial chunk size is chosen from power-of-2 steps [2, 4, 8, 16] based on load factor (`active_requests / max_num_seqs`):
|
||||
|
||||
| Server load | Initial chunk frames | Rationale |
|
||||
| --- | --- | --- |
|
||||
@@ -349,16 +349,15 @@ This overrides the dynamic calculation for that request.
|
||||
**Config (server-side):**
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
codec_streaming: true
|
||||
codec_chunk_frames: 25 # standard chunk size (~2s of audio)
|
||||
codec_left_context_frames: 25
|
||||
# initial chunk is computed dynamically by default
|
||||
# set initial_codec_chunk_frames: 2 to force a fixed value
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
codec_streaming: true
|
||||
codec_chunk_frames: 25 # standard chunk size (~2s of audio)
|
||||
codec_left_context_frames: 25
|
||||
# initial chunk is computed dynamically by default
|
||||
# set initial_codec_chunk_frames: 2 to force a fixed value
|
||||
```
|
||||
|
||||
The 64 ms TTFP result reported above for Qwen3-TTS at concurrency 1 uses the dynamic initial chunk, which picks `initial_codec_chunk_frames=2` at low load. At higher concurrency the dynamic sizing increases the initial chunk to maintain decode efficiency.
|
||||
@@ -408,7 +407,7 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
|
||||
|
||||
Notes:
|
||||
|
||||
- `runtime.max_batch_size` controls stage-level batching.
|
||||
- `stages[].max_num_seqs` controls stage-level batching.
|
||||
- Thinker/Talker commonly use `enforce_eager: false` for CUDA Graph paths.
|
||||
- Code2Wav often remains eager (`enforce_eager: true`) depending on runtime behavior.
|
||||
- Qwen3-Omni defaults `VLLM_USE_FLASHINFER_MOE_FP16=0`. The Triton has been more stable & faster
|
||||
@@ -426,33 +425,26 @@ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
|
||||
|
||||
```yaml
|
||||
async_chunk: true
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0 # thinker
|
||||
runtime:
|
||||
max_batch_size: 64
|
||||
engine_args:
|
||||
enforce_eager: false
|
||||
max_num_batched_tokens: 32768
|
||||
custom_process_next_stage_input_func: >-
|
||||
vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_async_chunk
|
||||
max_num_seqs: 64
|
||||
enforce_eager: false
|
||||
max_num_batched_tokens: 32768
|
||||
|
||||
- stage_id: 1 # talker
|
||||
runtime:
|
||||
max_batch_size: 64
|
||||
engine_args:
|
||||
enforce_eager: false
|
||||
max_num_batched_tokens: 32768
|
||||
custom_process_next_stage_input_func: >-
|
||||
vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_async_chunk
|
||||
max_num_seqs: 64
|
||||
enforce_eager: false
|
||||
max_num_batched_tokens: 32768
|
||||
|
||||
- stage_id: 2 # code2wav
|
||||
runtime:
|
||||
max_batch_size: 64
|
||||
engine_args:
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 51200
|
||||
max_num_seqs: 64
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 51200
|
||||
```
|
||||
|
||||
The async handoff processors are fixed by the registered Qwen3-Omni
|
||||
`PipelineConfig`; deploy YAML only selects `async_chunk` and runtime sizing.
|
||||
|
||||
#### Reproduce Qwen3-Omni benchmarks
|
||||
|
||||
```bash
|
||||
@@ -483,7 +475,7 @@ vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
|
||||
The default config (`qwen3_tts.yaml`) enables the full optimization stack:
|
||||
|
||||
- Batching with `max_batch_size: 10` on the Talker stage
|
||||
- Batching with `max_num_seqs: 10` on the Talker stage
|
||||
- CUDA Graph on the Talker (`enforce_eager: false`)
|
||||
- Async chunk with streaming transport
|
||||
|
||||
@@ -500,33 +492,29 @@ vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
|
||||
```yaml
|
||||
async_chunk: true
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0 # Talker (AR decoder)
|
||||
runtime:
|
||||
max_batch_size: 10
|
||||
engine_args:
|
||||
enforce_eager: false
|
||||
max_num_batched_tokens: 512
|
||||
custom_process_next_stage_input_func: >-
|
||||
vllm_omni.model_executor.stage_input_processors.qwen3_tts.talker2code2wav_async_chunk
|
||||
max_num_seqs: 10
|
||||
enforce_eager: false
|
||||
max_num_batched_tokens: 512
|
||||
|
||||
- stage_id: 1 # Code2Wav (vocoder)
|
||||
runtime:
|
||||
max_batch_size: 1
|
||||
engine_args:
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 8192
|
||||
max_num_seqs: 1
|
||||
enforce_eager: true
|
||||
max_num_batched_tokens: 8192
|
||||
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
codec_streaming: true
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 25
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
codec_streaming: true
|
||||
codec_chunk_frames: 25
|
||||
codec_left_context_frames: 25
|
||||
```
|
||||
|
||||
The async Talker-to-Code2Wav processor is defined by the registered Qwen3-TTS
|
||||
pipeline rather than the deploy YAML.
|
||||
|
||||
#### Reproduce Qwen3-TTS benchmarks
|
||||
|
||||
```bash
|
||||
|
||||
Reference in new issue
Block a user