diff --git a/.buildkite/test-nightly.yml b/.buildkite/test-nightly.yml index f2a765dcc..6dcc303dd 100644 --- a/.buildkite/test-nightly.yml +++ b/.buildkite/test-nightly.yml @@ -336,9 +336,6 @@ steps: timeout_in_minutes: 120 commands: - pytest -sv tests/e2e/ -k "not test_wan and not test_bagel_expansion and not hunyuan" -m "full_model and diffusion and H100" --run-level "full_model" --ignore=tests/e2e/accuracy - # revoke after bagel optimization finished - # Keep Bagel expansion and multi-replica tests in their dedicated H100 jobs below. - # - pytest -sv tests/e2e/ -k "not test_wan and not test_bagel_expansion and not hunyuan" -m "full_model and diffusion and H100" --run-level "full_model" --ignore=tests/e2e/accuracy --ignore=tests/e2e/online_serving/test_bagel_multi_replicas.py agents: queue: "mithril-h100-pool" plugins: @@ -411,44 +408,6 @@ steps: path: /mnt/hf-cache type: DirectoryOrCreate - # revoke after bagel optimization finished - # - label: ":full_moon: Diffusion X2I(&A&T) · BAGEL Multi-Replica with H100" - # timeout_in_minutes: 120 - # commands: - # - pytest -s -v tests/e2e/online_serving/test_bagel_multi_replicas.py -m "full_model and diffusion and H100" --run-level "full_model" - # agents: - # queue: "mithril-h100-pool" - # plugins: - # - kubernetes: - # podSpec: - # containers: - # - image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT - # resources: - # limits: - # nvidia.com/gpu: 4 - # volumeMounts: - # - name: devshm - # mountPath: /dev/shm - # - name: hf-cache - # mountPath: /root/.cache/huggingface - # env: - # - name: HF_HOME - # value: /root/.cache/huggingface - # - name: HF_TOKEN - # valueFrom: - # secretKeyRef: - # name: hf-token-secret - # key: token - # nodeSelector: - # node.kubernetes.io/instance-type: gpu-h100-sxm - # volumes: - # - name: devshm - # emptyDir: - # medium: Memory - # - name: hf-cache - # hostPath: - # path: /mnt/hf-cache - # type: DirectoryOrCreate - label: ":full_moon: Diffusion X2I(&A&T) · Function Test with L4" timeout_in_minutes: 60 diff --git a/benchmarks/diffusion/README.md b/benchmarks/diffusion/README.md index a8b11fe90..16a86c88f 100644 --- a/benchmarks/diffusion/README.md +++ b/benchmarks/diffusion/README.md @@ -32,8 +32,6 @@ python3 benchmarks/diffusion/diffusion_benchmark_serving.py \ - The benchmark talks to `http://:/v1/chat/completions`. - If you run the server on another host or port, pass `--base-url` accordingly. -- To compare diffusion multi-replica serving, restart the server with different - `runtime.num_replicas` values and keep the benchmark parameters unchanged. ## 2. Supported Datasets @@ -95,35 +93,6 @@ You can point to your own trace using `--dataset-path`. ## 3. Benchmark Parameters -### Multi-replica sweep - -For replica scaling, use the same model, prompt count, image/video size, and -step count for every run. Change only the serving stage config: - -```yaml -runtime: - devices: "0,1,2,3" - num_replicas: 4 -``` - -Then run a fixed benchmark and write one JSON file per replica count: - -```bash -python3 benchmarks/diffusion/diffusion_benchmark_serving.py \ - --base-url http://localhost:8099 \ - --model Qwen/Qwen-Image \ - --task t2i \ - --dataset random \ - --num-prompts 32 \ - --max-concurrency 8 \ - --request-rate inf \ - --num-inference-steps 10 \ - --width 512 --height 512 \ - --output-file diffusion_replica_4.json -``` - -Repeat for `num_replicas=1`, `2`, `3`, and `4`, using matching `runtime.devices`. - ### Basic flags - `--base-url`: Server address (the script calls `.../v1/chat/completions`). diff --git a/docs/user_guide/diffusion/multi_replica_serving.md b/docs/user_guide/diffusion/multi_replica_serving.md deleted file mode 100644 index 2da8c698f..000000000 --- a/docs/user_guide/diffusion/multi_replica_serving.md +++ /dev/null @@ -1,80 +0,0 @@ -# Diffusion Multi-Replica Serving - -Diffusion stages can be replicated in online serving by setting `runtime.num_replicas` on a `stage_type: diffusion` stage. Each replica owns an independent diffusion engine process and receives requests through round-robin routing with request affinity. - -## When to Use - -Use multi-replica serving when one diffusion process does not saturate the available GPUs, or when you want higher request throughput for many independent image/video generation requests. It improves concurrency throughput; it does not make a single request faster. - -## Configuration - -A replica consumes the devices required by that diffusion stage. For single-device diffusion pipelines, set `runtime.devices` to one device per replica: - -```yaml -stages: - - stage_id: 0 - stage_type: diffusion - runtime: - devices: "0,1,2,3" - num_replicas: 4 - engine_args: - model: Qwen/Qwen-Image - parallel_config: - world_size: 1 -``` - -For multi-device diffusion pipelines, `parallel_config.world_size` is treated as the number of devices per replica. For example, four 2-GPU replicas need eight devices: - -```yaml -stages: - - stage_id: 0 - stage_type: diffusion - runtime: - devices: "0,1,2,3,4,5,6,7" - num_replicas: 4 - engine_args: - model: Wan-AI/Wan2.2-T2V-A14B-Diffusers - parallel_config: - world_size: 2 -``` - -The runtime splits the device list into contiguous chunks per replica. With the second example, replica 0 uses `0,1`, replica 1 uses `2,3`, replica 2 uses `4,5`, and replica 3 uses `6,7`. - -## Serving - -Start the server with the stage config: - -```bash -vllm serve Qwen/Qwen-Image \ - --omni \ - --port 8099 \ - --stage-configs-path /path/to/diffusion_multi_replica.yaml -``` - -Then send normal OpenAI-compatible diffusion chat requests. The orchestrator routes new requests across replicas and keeps updates/abort calls on the same replica for each request. - -## Benchmarking - -Benchmark each replica count by changing `runtime.num_replicas` and `runtime.devices`, restarting the server, then running: - -```bash -python benchmarks/diffusion/diffusion_benchmark_serving.py \ - --base-url http://localhost:8099 \ - --model Qwen/Qwen-Image \ - --task t2i \ - --dataset random \ - --num-prompts 8 \ - --request-rate inf \ - --num-inference-steps 2 \ - --width 256 --height 256 \ - --output-file diffusion_replica_4.json -``` - -For a quick scaling sweep, test `num_replicas` values `1`, `2`, `3`, and `4` with the same prompt count and generation parameters. Compare throughput and end-to-end latency from the JSON outputs. - -## Notes and Limitations - -- Multi-replica serving is currently supported for diffusion stages. Single-stage mode still rejects non-diffusion replicas. -- `runtime.devices` must provide exactly `num_replicas * devices_per_replica` devices. -- Request-level latency may stay similar or slightly increase because each request still runs on one replica. -- Throughput improves only when enough concurrent requests are issued and each replica has enough GPU memory. diff --git a/docs/user_guide/diffusion_features.md b/docs/user_guide/diffusion_features.md index 643301d1b..69c81737d 100644 --- a/docs/user_guide/diffusion_features.md +++ b/docs/user_guide/diffusion_features.md @@ -15,7 +15,7 @@ vLLM-Omni supports various advanced features for diffusion models: - Acceleration: **cache methods**, **parallelism methods**, **startup optimizations** - Memory optimization: **cpu offloading**, **quantization** - Extensions: **LoRA inference**, **frame interpolation** -- Execution modes: **step execution**, **multi-replica serving** +- Execution modes: **step execution** ## Supported Features @@ -43,7 +43,6 @@ Parallelism methods distribute computation across GPUs without quality loss (mat | **[Tensor Parallelism](diffusion/parallelism/tensor_parallel.md)** | Shards model weights across devices | Large models that don't fit in single GPU, with 2+ GPUs | | **[HSDP](diffusion/parallelism/hsdp.md)** | Weight sharding via FSDP2, redistributed on-demand at runtime | Very large models (14B+) on limited VRAM, combinable with SP | | **[Expert Parallelism](diffusion/parallelism/expert_parallel.md)** | Shards MoE expert MLP blocks across devices | MoE diffusion models (e.g., HunyuanImage3.0) | -| **[Multi-Replica Serving](diffusion/multi_replica_serving.md)** | Runs multiple independent diffusion stage replicas and routes requests across them | Online serving throughput with many concurrent requests | #### Startup Optimization