anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[Clean] Remove multi-replica Bagel CI and related docs/configs (#3407)

Browse Source 
Signed-off-by: Chenguang ZHENG <645327136@qq.com>
This commit is contained in:
Chenguang Zheng authored and GitHub committed 2026-05-08 09:28:52 +08:00
1 parent e6466cf0e6
commit eeb7e698c9
4 files changed
+1 -154
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.buildkite/test-nightly.yml
-41
View File
@@ -336,9 +336,6 @@ steps:
timeout_in_minutes: 120
commands:
- pytest -sv tests/e2e/ -k "not test_wan and not test_bagel_expansion and not hunyuan" -m "full_model and diffusion and H100" --run-level "full_model" --ignore=tests/e2e/accuracy
# revoke after bagel optimization finished
# Keep Bagel expansion and multi-replica tests in their dedicated H100 jobs below.
# - pytest -sv tests/e2e/ -k "not test_wan and not test_bagel_expansion and not hunyuan" -m "full_model and diffusion and H100" --run-level "full_model" --ignore=tests/e2e/accuracy --ignore=tests/e2e/online_serving/test_bagel_multi_replicas.py
agents:
queue: "mithril-h100-pool"
plugins:
@@ -411,44 +408,6 @@ steps:
path: /mnt/hf-cache
type: DirectoryOrCreate
# revoke after bagel optimization finished
# - label: ":full_moon: Diffusion X2I(&A&T) · BAGEL Multi-Replica with H100"
# timeout_in_minutes: 120
# commands:
# - pytest -s -v tests/e2e/online_serving/test_bagel_multi_replicas.py -m "full_model and diffusion and H100" --run-level "full_model"
# agents:
# queue: "mithril-h100-pool"
# plugins:
# - kubernetes:
# podSpec:
# containers:
# - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
# resources:
# limits:
# nvidia.com/gpu: 4
# volumeMounts:
# - name: devshm
# mountPath: /dev/shm
# - name: hf-cache
# mountPath: /root/.cache/huggingface
# env:
# - name: HF_HOME
# value: /root/.cache/huggingface
# - name: HF_TOKEN
# valueFrom:
# secretKeyRef:
# name: hf-token-secret
# key: token
# nodeSelector:
# node.kubernetes.io/instance-type: gpu-h100-sxm
# volumes:
# - name: devshm
# emptyDir:
# medium: Memory
# - name: hf-cache
# hostPath:
# path: /mnt/hf-cache
# type: DirectoryOrCreate
- label: ":full_moon: Diffusion X2I(&A&T) · Function Test with L4"
timeout_in_minutes: 60
benchmarks/diffusion/README.md
-31
View File
@@ -32,8 +32,6 @@ python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
- The benchmark talks to `http://<host>:<port>/v1/chat/completions`.
- If you run the server on another host or port, pass `--base-url` accordingly.
- To compare diffusion multi-replica serving, restart the server with different
`runtime.num_replicas` values and keep the benchmark parameters unchanged.
## 2. Supported Datasets
@@ -95,35 +93,6 @@ You can point to your own trace using `--dataset-path`.
## 3. Benchmark Parameters
### Multi-replica sweep
For replica scaling, use the same model, prompt count, image/video size, and
step count for every run. Change only the serving stage config:
```yaml
runtime:
devices: "0,1,2,3"
num_replicas: 4
```
Then run a fixed benchmark and write one JSON file per replica count:
```bash
python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
--base-url http://localhost:8099 \
--model Qwen/Qwen-Image \
--task t2i \
--dataset random \
--num-prompts 32 \
--max-concurrency 8 \
--request-rate inf \
--num-inference-steps 10 \
--width 512 --height 512 \
--output-file diffusion_replica_4.json
```
Repeat for `num_replicas=1`, `2`, `3`, and `4`, using matching `runtime.devices`.
### Basic flags
- `--base-url`: Server address (the script calls `.../v1/chat/completions`).
docs/user_guide/diffusion/multi_replica_serving.md
-80
View File
@@ -1,80 +0,0 @@
# Diffusion Multi-Replica Serving
Diffusion stages can be replicated in online serving by setting `runtime.num_replicas` on a `stage_type: diffusion` stage. Each replica owns an independent diffusion engine process and receives requests through round-robin routing with request affinity.
## When to Use
Use multi-replica serving when one diffusion process does not saturate the available GPUs, or when you want higher request throughput for many independent image/video generation requests. It improves concurrency throughput; it does not make a single request faster.
## Configuration
A replica consumes the devices required by that diffusion stage. For single-device diffusion pipelines, set `runtime.devices` to one device per replica:
```yaml
stages:
- stage_id: 0
stage_type: diffusion
runtime:
devices: "0,1,2,3"
num_replicas: 4
engine_args:
model: Qwen/Qwen-Image
parallel_config:
world_size: 1
```
For multi-device diffusion pipelines, `parallel_config.world_size` is treated as the number of devices per replica. For example, four 2-GPU replicas need eight devices:
```yaml
stages:
- stage_id: 0
stage_type: diffusion
runtime:
devices: "0,1,2,3,4,5,6,7"
num_replicas: 4
engine_args:
model: Wan-AI/Wan2.2-T2V-A14B-Diffusers
parallel_config:
world_size: 2
```
The runtime splits the device list into contiguous chunks per replica. With the second example, replica 0 uses `0,1`, replica 1 uses `2,3`, replica 2 uses `4,5`, and replica 3 uses `6,7`.
## Serving
Start the server with the stage config:
```bash
vllm serve Qwen/Qwen-Image \
--omni \
--port 8099 \
--stage-configs-path /path/to/diffusion_multi_replica.yaml
```
Then send normal OpenAI-compatible diffusion chat requests. The orchestrator routes new requests across replicas and keeps updates/abort calls on the same replica for each request.
## Benchmarking
Benchmark each replica count by changing `runtime.num_replicas` and `runtime.devices`, restarting the server, then running:
```bash
python benchmarks/diffusion/diffusion_benchmark_serving.py \
--base-url http://localhost:8099 \
--model Qwen/Qwen-Image \
--task t2i \
--dataset random \
--num-prompts 8 \
--request-rate inf \
--num-inference-steps 2 \
--width 256 --height 256 \
--output-file diffusion_replica_4.json
```
For a quick scaling sweep, test `num_replicas` values `1`, `2`, `3`, and `4` with the same prompt count and generation parameters. Compare throughput and end-to-end latency from the JSON outputs.
## Notes and Limitations
- Multi-replica serving is currently supported for diffusion stages. Single-stage mode still rejects non-diffusion replicas.
- `runtime.devices` must provide exactly `num_replicas * devices_per_replica` devices.
- Request-level latency may stay similar or slightly increase because each request still runs on one replica.
- Throughput improves only when enough concurrent requests are issued and each replica has enough GPU memory.
docs/user_guide/diffusion_features.md
+1 -2
View File
@@ -15,7 +15,7 @@ vLLM-Omni supports various advanced features for diffusion models:
- Acceleration: **cache methods**, **parallelism methods**, **startup optimizations**
- Memory optimization: **cpu offloading**, **quantization**
- Extensions: **LoRA inference**, **frame interpolation**
- Execution modes: **step execution**, **multi-replica serving**
- Execution modes: **step execution**
## Supported Features
@@ -43,7 +43,6 @@ Parallelism methods distribute computation across GPUs without quality loss (mat
| **[Tensor Parallelism](diffusion/parallelism/tensor_parallel.md)** | Shards model weights across devices | Large models that don't fit in single GPU, with 2+ GPUs |
| **[HSDP](diffusion/parallelism/hsdp.md)** | Weight sharding via FSDP2, redistributed on-demand at runtime | Very large models (14B+) on limited VRAM, combinable with SP |
| **[Expert Parallelism](diffusion/parallelism/expert_parallel.md)** | Shards MoE expert MLP blocks across devices | MoE diffusion models (e.g., HunyuanImage3.0) |
| **[Multi-Replica Serving](diffusion/multi_replica_serving.md)** | Runs multiple independent diffusion stage replicas and routes requests across them | Online serving throughput with many concurrent requests |
#### Startup Optimization