[Docs] Remove stale stage config references (#6270)
Signed-off-by: zwhzzz0821 <2831474076@qq.com>
This commit is contained in:
1 parent
efaf4276f9
commit
6f55a06fab
85 files changed
+566
-794
No files matched your search
@@ -211,38 +211,39 @@ In sequential mode, each stage must wait for the previous stage to complete enti
|
||||
|
||||
## Configuration
|
||||
|
||||
Enable async_chunk in stage configuration YAML:
|
||||
Enable async chunk mode in the deploy YAML:
|
||||
|
||||
```yaml
|
||||
async_chunk: true
|
||||
stage_args:
|
||||
- stage_id: 0
|
||||
engine_args:
|
||||
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.thinker2talker_async_chunk
|
||||
- stage_id: 1
|
||||
engine_args:
|
||||
custom_process_next_stage_input_func: vllm_omni.model_executor.stage_input_processors.qwen3_omni.talker2code2wav_async_chunk
|
||||
```
|
||||
|
||||
### Stage Configuration
|
||||
The registered `PipelineConfig` owns the async handoff processor functions.
|
||||
For Qwen3-Omni these are `thinker2talker_async_chunk` and
|
||||
`talker2code2wav_async_chunk`; deploy YAML only selects the mode and runtime
|
||||
sizing.
|
||||
|
||||
### Configuration ownership
|
||||
|
||||
- `async_chunk: bool`: Enable/disable async chunk mode
|
||||
- `custom_process_next_stage_input_func: str`: Path to custom chunk processing function; receives `(transfer_manager, pooling_output, request)`. For qwen3-omni: `thinker2talker_async_chunk`, `talker2code2wav_async_chunk`
|
||||
- `stage_connector_config: dict`: Connector configuration
|
||||
- `worker_type: str`: Model type, e.g. `"ar"` or `"generation"` (used by OmniChunkTransferAdapter for mode-specific payload handling)
|
||||
- `async_chunk_process_next_stage_input_func: str`: Pipeline-owned chunk processor path
|
||||
- `execution_type: StageExecutionType`: Pipeline-owned runtime family
|
||||
- `connectors: dict`: Deploy-owned connector definitions
|
||||
- `max_num_seqs: int`: Maximum number of sequences for concurrent processing in the stage
|
||||
|
||||
|
||||
### Connector Configuration
|
||||
|
||||
```yaml
|
||||
connectors:
|
||||
- from_stage: 0
|
||||
to_stage: 1
|
||||
spec:
|
||||
name: SharedMemoryConnector
|
||||
extra:
|
||||
stage_id: 0
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_shared_memory
|
||||
- stage_id: 1
|
||||
input_connectors:
|
||||
from_stage_0: connector_of_shared_memory
|
||||
```
|
||||
|
||||
### Code2Wav Batch Configuration
|
||||
@@ -250,13 +251,10 @@ connectors:
|
||||
For optimal performance with async_chunk, the code2wav stage should be configured with batching:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 2 # code2wav stage
|
||||
runtime:
|
||||
devices: "1"
|
||||
engine_args:
|
||||
model_stage: code2wav
|
||||
max_num_seqs: 64 # Enables batched audio generation
|
||||
devices: "1"
|
||||
max_num_seqs: 64 # Enables batched audio generation
|
||||
```
|
||||
|
||||
## Related Files
|
||||
|
||||
@@ -59,19 +59,18 @@ This `metadata` must be passed through the control plane so `get()` can locate t
|
||||
|
||||
## Configuration Model
|
||||
|
||||
Define connectors in runtime:
|
||||
Define connectors at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
```
|
||||
|
||||
Wire stages to connectors:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_shared_memory
|
||||
|
||||
@@ -32,26 +32,25 @@ mooncake_master \
|
||||
|
||||
## Configuration
|
||||
|
||||
Define the connector in runtime:
|
||||
Define the connector at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_mooncake:
|
||||
name: MooncakeStoreConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
metadata_server: "http://<MASTER_IP>:8080/metadata"
|
||||
master: "<MASTER_IP>:50051"
|
||||
segment: 512000000
|
||||
localbuf: 64000000
|
||||
proto: "tcp"
|
||||
connectors:
|
||||
connector_of_mooncake:
|
||||
name: MooncakeStoreConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
metadata_server: "http://<MASTER_IP>:8080/metadata"
|
||||
master: "<MASTER_IP>:50051"
|
||||
segment: 512000000
|
||||
localbuf: 64000000
|
||||
proto: "tcp"
|
||||
```
|
||||
|
||||
Wire stages to the connector:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_mooncake
|
||||
|
||||
@@ -21,26 +21,25 @@ InfiniBand/RoCE NICs).
|
||||
|
||||
## Configuration
|
||||
|
||||
Define the connector in runtime:
|
||||
Define the connector at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
rdma_connector:
|
||||
name: MooncakeTransferEngineConnector
|
||||
extra:
|
||||
host: "auto" # Auto-detect local RDMA IP
|
||||
zmq_port: 50051 # ZMQ base port (see "Port Offset Scheme" below)
|
||||
protocol: "rdma" # "rdma" or "tcp"
|
||||
device_name: "" # RDMA device (e.g., "mlx5_0"), empty for auto-detect
|
||||
memory_pool_size: 4294967296 # 4 GB (CPU); use 2147483648 (2 GB) for GPU
|
||||
memory_pool_device: "cpu" # "cpu" for pinned memory (recommended), "cuda" for GPUDirect RDMA
|
||||
connectors:
|
||||
rdma_connector:
|
||||
name: MooncakeTransferEngineConnector
|
||||
extra:
|
||||
host: "auto" # Auto-detect local RDMA IP
|
||||
zmq_port: 50051 # ZMQ base port (see "Port Offset Scheme" below)
|
||||
protocol: "rdma" # "rdma" or "tcp"
|
||||
device_name: "" # RDMA device (e.g., "mlx5_0"), empty for auto-detect
|
||||
memory_pool_size: 4294967296 # 4 GB (CPU); use 2147483648 (2 GB) for GPU
|
||||
memory_pool_device: "cpu" # "cpu" for pinned memory (recommended), "cuda" for GPUDirect RDMA
|
||||
```
|
||||
|
||||
Wire stages to the connector:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: rdma_connector
|
||||
@@ -415,7 +414,7 @@ A receiver connector:
|
||||
- allocates receive buffers from its own pool
|
||||
- requests metadata or transfer service from the sender
|
||||
|
||||
The role is not inferred dynamically. It is injected by the stage configuration layer:
|
||||
The role is not inferred dynamically. It is injected from the deploy connector wiring:
|
||||
|
||||
- incoming edge for a stage -> `role="receiver"`
|
||||
- outgoing edge for a stage -> `role="sender"`
|
||||
|
||||
@@ -13,10 +13,9 @@ segment name is returned in metadata.
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
connectors:
|
||||
connector_of_shared_memory:
|
||||
name: SharedMemoryConnector
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
@@ -65,23 +65,22 @@ dscli stop --worker_address "${WORKER_IP}:31501"
|
||||
|
||||
## Configuration
|
||||
|
||||
Define the connector in runtime:
|
||||
Define the connector at the top level of the deploy YAML:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
connectors:
|
||||
connector_of_yuanrong:
|
||||
name: YuanrongConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
port: 31501
|
||||
get_sub_timeout_ms: 1000
|
||||
connectors:
|
||||
connector_of_yuanrong:
|
||||
name: YuanrongConnector
|
||||
extra:
|
||||
host: "127.0.0.1"
|
||||
port: 31501
|
||||
get_sub_timeout_ms: 1000
|
||||
```
|
||||
|
||||
Wire stages to the connector:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
output_connectors:
|
||||
to_stage_1: connector_of_yuanrong
|
||||
|
||||
@@ -94,26 +94,23 @@ For Ascend P2P, configure the connector with `protocol: "ascend"` and an NPU
|
||||
memory pool:
|
||||
|
||||
```yaml
|
||||
runtime:
|
||||
enabled: true
|
||||
defaults:
|
||||
connectors:
|
||||
yuanrong_te_connector:
|
||||
name: YuanrongTransferEngineConnector
|
||||
extra:
|
||||
host: "auto"
|
||||
zmq_port: 50051
|
||||
rpc_port: "auto"
|
||||
protocol: "ascend"
|
||||
device_name: "auto"
|
||||
memory_pool_size: 1073741824
|
||||
memory_pool_device: "npu"
|
||||
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
max_inflight: 1
|
||||
connectors:
|
||||
yuanrong_te_connector:
|
||||
name: YuanrongTransferEngineConnector
|
||||
extra:
|
||||
host: "auto"
|
||||
zmq_port: 50051
|
||||
rpc_port: "auto"
|
||||
protocol: "ascend"
|
||||
device_name: "auto"
|
||||
memory_pool_size: 1073741824
|
||||
memory_pool_device: "npu"
|
||||
edges:
|
||||
- from: 0
|
||||
to: 1
|
||||
window_size: -1
|
||||
```
|
||||
|
||||
Important fields:
|
||||
@@ -141,31 +138,27 @@ For AR 4-way TP sending KV cache to DiT 4-way TP, set the stage TP topology in
|
||||
both stage `omni_kv_config` blocks:
|
||||
|
||||
```yaml
|
||||
stage_args:
|
||||
stages:
|
||||
- stage_id: 0
|
||||
runtime:
|
||||
devices: "0,1,2,3"
|
||||
engine_args:
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_send_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
devices: "0,1,2,3"
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_send_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
output_connectors:
|
||||
to_stage_1: yuanrong_te_connector
|
||||
|
||||
- stage_id: 1
|
||||
runtime:
|
||||
devices: "4,5,6,7"
|
||||
engine_args:
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
devices: "4,5,6,7"
|
||||
parallel_config:
|
||||
tensor_parallel_size: 4
|
||||
omni_kv_config:
|
||||
need_recv_cache: true
|
||||
rank_mapping:
|
||||
from_tp: 4
|
||||
to_tp: 4
|
||||
input_connectors:
|
||||
from_stage_0: yuanrong_te_connector
|
||||
```
|
||||
|
||||
@@ -41,7 +41,7 @@ With this in mind, consider the set of blocks in a 2D layout, where the row repr
|
||||
|
||||
### Example
|
||||
!!! note "Note 3"
|
||||
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. It can be enabled/disabled per-stage via the `enable_prefix_caching` parameter in the model's stage config.
|
||||
Prefix caching in vLLM-Omni currently is only supported on AutoRegressive stages with one kv-cache group. Configure it with the pipeline-wide `enable_prefix_caching` field in the deploy config.
|
||||
|
||||
The way in which vLLM-Omni ties into vLLM's prefix caching is best understood by example. Say that we have the following:
|
||||
|
||||
|
||||
Reference in new issue
Block a user