[Bugfix] Remove duplicated config keyword max batch size (#1851)
Signed-off-by: tzhouam <tzhouam@connect.ust.hk> Signed-off-by: Zhou Taichang <tzhouam@connect.ust.hk>
This commit is contained in:
1 parent
4c2293743b
commit
05d8de813e
67 files changed
+226
-159
No files matched your search
@@ -130,7 +130,7 @@ The default yaml configuration deploys Thinker and DiT on the same GPU. You can
|
||||
| :------------------------------- | :------------------------------ | :----------------------- |
|
||||
| `stage_type` | `llm` | Stage type |
|
||||
| `devices` | `"0"` | GPU device ID |
|
||||
| `max_batch_size` | `1` | Maximum batch size |
|
||||
| `max_num_seqs` | `1` | Maximum batch size |
|
||||
| `model_stage` | `thinker` | Model stage identifier |
|
||||
| `model_arch` | `BagelForConditionalGeneration` | Model architecture |
|
||||
| `gpu_memory_utilization` | `0.4` | GPU memory utilization |
|
||||
@@ -146,7 +146,7 @@ The default yaml configuration deploys Thinker and DiT on the same GPU. You can
|
||||
| :------------------------------- | :---------- | :-------------------------- |
|
||||
| `stage_type` | `diffusion` | Stage type |
|
||||
| `devices` | `"0"` | GPU device ID |
|
||||
| `max_batch_size` | `1` | Maximum batch size |
|
||||
| `max_num_seqs` | `1` | Maximum batch size |
|
||||
| `model_stage` | `dit` | Model stage identifier |
|
||||
| `gpu_memory_utilization` | `0.4` | GPU memory utilization |
|
||||
| `omni_kv_config.need_recv_cache` | `true` | Whether to receive KV cache |
|
||||
|
||||
@@ -107,7 +107,7 @@ completes. This demonstrates that audio data is available progressively rather t
|
||||
|
||||
## Batched Decoding
|
||||
|
||||
The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, provide a stage config with `max_batch_size > 1` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`.
|
||||
The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, provide a stage config with `max_num_seqs > 1` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`.
|
||||
|
||||
```
|
||||
python end2end.py --query-type CustomVoice \
|
||||
@@ -116,7 +116,7 @@ python end2end.py --query-type CustomVoice \
|
||||
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_batch.yaml
|
||||
```
|
||||
|
||||
**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_batch_size >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_batch_size`, it won't help — stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently.
|
||||
**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_num_seqs >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_num_seqs`, it won't help — stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently.
|
||||
|
||||
## Notes
|
||||
|
||||
|
||||
@@ -49,7 +49,7 @@ if __name__ == "__main__":
|
||||
|
||||
!!! info
|
||||
|
||||
For diffusion pipelines, the stage config field `stage_args.[].runtime.max_batch_size` is 1 by default, and the input
|
||||
For diffusion pipelines, the stage config field `stage_args.[].engine_args.max_num_seqs` is 1 by default, and the input
|
||||
list is sliced into single-item requests before feeding into the diffusion pipeline. For models that do internally support
|
||||
batched inputs, you can [modify this configuration](https://github.com/vllm-project/vllm-omni/tree/main/configuration/stage_configs.md) to let the model accept a longer batch of prompts.
|
||||
|
||||
|
||||
Reference in new issue
Block a user