From 64f5ef615c1d7f47cac8be95842b2da4bc38dfde Mon Sep 17 00:00:00 2001 From: Alicia <115451386+congw729@users.noreply.github.com> Date: Fri, 27 Feb 2026 17:28:35 +0800 Subject: [PATCH] [Doc] [skip ci]Sync. (#1363) Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com> --- .github/PULL_REQUEST_TEMPLATE.md | 6 +- docs/.nav.yml | 6 + .../examples/offline_inference/bagel.md | 86 ++++++ .../examples/offline_inference/glm_image.md | 156 ++++++++++ .../offline_inference/image_to_video.md | 5 + .../examples/offline_inference/qwen3_tts.md | 32 ++- .../offline_inference/text_to_image.md | 34 ++- .../examples/online_serving/bagel.md | 148 +++++++++- .../examples/online_serving/glm_image.md | 269 ++++++++++++++++++ .../examples/online_serving/image_to_video.md | 97 +++++++ .../examples/online_serving/qwen2_5_omni.md | 2 +- .../examples/online_serving/qwen3_omni.md | 2 +- .../examples/online_serving/qwen3_tts.md | 49 ++-- .../examples/online_serving/text_to_video.md | 130 +++++++++ examples/online_serving/qwen3_tts/README.md | 241 +++++++++------- 15 files changed, 1122 insertions(+), 141 deletions(-) create mode 100644 docs/user_guide/examples/offline_inference/glm_image.md create mode 100644 docs/user_guide/examples/online_serving/glm_image.md create mode 100644 docs/user_guide/examples/online_serving/image_to_video.md create mode 100644 docs/user_guide/examples/online_serving/text_to_video.md diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index c67a9c7f4..1b4762794 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -12,10 +12,10 @@ PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTT Essential Elements of an Effective PR Description Checklist - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". -- [ ] The test plan. Please providing the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the [test style doc](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/tests_style/) -- [ ] The test results. Please pasting the results comparison before and after, or e2e results. +- [ ] The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the [test style doc](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/ci/tests_style/) +- [ ] The test results. Please paste the results comparison before and after, or the e2e results. - [ ] (Optional) The necessary documentation update, such as updating `supported_models.md` and `examples` for a new model. **Please run `mkdocs serve` to sync the documentation editions to `./docs`.** -- [ ] (Optional) Release notes update. If your change is user facing, please update the release notes draft. +- [ ] (Optional) Release notes update. If your change is user-facing, please update the release notes draft. **BEFORE SUBMITTING, PLEASE READ ** (anything written below this line will be removed by GitHub Actions) diff --git a/docs/.nav.yml b/docs/.nav.yml index 7fe8f988a..c504478a5 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -13,6 +13,7 @@ nav: - examples/README.md - Offline Inference: - BAGEL-7B-MoT: user_guide/examples/offline_inference/bagel.md + - GLM-Image Multistage End-to-End Inference: user_guide/examples/offline_inference/glm_image.md - Image-To-Image: user_guide/examples/offline_inference/image_to_image.md - Image-To-Video: user_guide/examples/offline_inference/image_to_video.md - Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md @@ -23,6 +24,7 @@ nav: - Text-To-Video: user_guide/examples/offline_inference/text_to_video.md - Online Serving: - BAGEL-7B-MoT: user_guide/examples/online_serving/bagel.md + - GLM-Image Online Serving: user_guide/examples/online_serving/glm_image.md - Image-To-Image: user_guide/examples/online_serving/image_to_image.md - Image-To-Video: user_guide/examples/online_serving/image_to_video.md - Qwen2.5-Omni: user_guide/examples/online_serving/qwen2_5_omni.md @@ -50,10 +52,13 @@ nav: - Parallelism Acceleration: user_guide/diffusion/parallelism_acceleration.md - CPU Offloading: user_guide/diffusion/cpu_offload_diffusion.md - LoRA: user_guide/diffusion/lora.md + - Hybrid Sharded Data Parallel: design/feature/hsdp.md + - Custom Pipeline: features/custom_pipeline.md - ComfyUI: features/comfyui.md - Developer Guide: - General: - contributing/README.md + - pr_reviewer.md - glob: contributing/* flatten_single_child_sections: true - Model Implementation: @@ -73,6 +78,7 @@ nav: - design/feature/tensor_parallel.md - design/feature/cache_dit.md - design/feature/teacache.md + - design/feature/async_chunk_design.md - Module Design: - design/module/ar_module.md - design/module/dit_module.md diff --git a/docs/user_guide/examples/offline_inference/bagel.md b/docs/user_guide/examples/offline_inference/bagel.md index 8bab92977..cd245aa8d 100644 --- a/docs/user_guide/examples/offline_inference/bagel.md +++ b/docs/user_guide/examples/offline_inference/bagel.md @@ -154,6 +154,24 @@ The default yaml configuration deploys Thinker and DiT on the same GPU. You can ------ +#### Tensor Parallelism (TP) + +For larger models or multi-GPU environments, you can enable Tensor Parallelism (TP) by modifying the stage configuration (e.g., [`bagel.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/bagel.yaml)). + +1. **Set `tensor_parallel_size`**: Increase this value (e.g., to `2` or `4`). +2. **Set `devices`**: Specify the comma-separated GPU IDs to be used for the stage (e.g., `"0,1"`). + +Example configuration for TP=2 on GPUs 0 and 1: +```yaml + engine_args: + tensor_parallel_size: 2 + ... + runtime: + devices: "0,1" +``` + +------ + #### ๐Ÿ”— Runtime Configuration | Parameter | Value | Description | @@ -162,6 +180,74 @@ The default yaml configuration deploys Thinker and DiT on the same GPU. You can | `max_inflight` | `1` | Maximum inflight requests | | `shm_threshold_bytes` | `65536` | Shared memory threshold (64KB) | +## Using Mooncake Connector + +[Mooncake](https://github.com/kvcache-ai/Mooncake) is a high-performance distributed KV cache transfer engine that enables efficient cross-node data movement via TCP or RDMA, making it ideal for multi-node disaggregated inference. + +By default, BAGEL uses `SharedMemoryConnector` for inter-stage communication. You can switch to the Mooncake connector for better performance on multi-GPU setups and to enable multi-node deployment. + +### Prerequisites + +Install the Mooncake transfer engine: + +```bash +# For CUDA-enabled systems (recommended) +pip install mooncake-transfer-engine + +# For non-CUDA systems +pip install mooncake-transfer-engine-non-cuda +``` + +### Step 1: Start the Mooncake Master + +On the **primary node**, start the Mooncake master service (run in a separate terminal or background with `&`): + +```bash +# Optional: enable disk-backed storage by creating a directory and passing --root_fs_dir. +# Without it, Mooncake runs in memory-only mode, which is sufficient for KV cache transfer. +mkdir -p ./mc_storage + +mooncake_master \ + --rpc_port=50051 \ + --enable_http_metadata_server=true \ + --http_metadata_server_host=0.0.0.0 \ + --http_metadata_server_port=8080 \ + --metrics_port=9003 \ + --root_fs_dir=./mc_storage/ \ + --cluster_id=mc-local-1 & +``` + +### Step 2: Run Offline Inference with Mooncake + +Use the provided Mooncake stage config [`bagel_multiconnector.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml). Before launching, update the `metadata_server` and `master` addresses in the YAML to match your Mooncake master node's IP (use `127.0.0.1` for single-node testing). + +```bash +cd examples/offline_inference/bagel + +# Text to Image with Mooncake +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2img \ + --prompts "A cute cat" \ + --stage-configs-path ../../../vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml + +# Image to Text with Mooncake +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality img2text \ + --image-path /path/to/image.jpg \ + --prompts "Describe this image" \ + --stage-configs-path ../../../vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml + +# Text to Text with Mooncake +python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ + --modality text2text \ + --prompts "What is the capital of France?" \ + --stage-configs-path ../../../vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml +``` + +For more details on the Mooncake connector and multi-node setup, see the [Mooncake Store Connector documentation](https://github.com/vllm-project/vllm-omni/tree/main/docs/design/feature/omni_connectors/mooncake_store_connector.md). + +------ + ## FAQ - If you encounter an error about the backend of librosa, try to install ffmpeg with the command below. diff --git a/docs/user_guide/examples/offline_inference/glm_image.md b/docs/user_guide/examples/offline_inference/glm_image.md new file mode 100644 index 000000000..7b5ab25a0 --- /dev/null +++ b/docs/user_guide/examples/offline_inference/glm_image.md @@ -0,0 +1,156 @@ +# GLM-Image Multistage End-to-End Inference + +Source . + + +This example demonstrates how to run GLM-Image with the vLLM-Omni multistage architecture. + +## Architecture + +GLM-Image uses a 2-stage pipeline: + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ GLM-Image Pipeline โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ โ”‚ +โ”‚ Stage 0 (AR Model) Stage 1 (Diffusion) โ”‚ +โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ +โ”‚ โ”‚ vLLM-optimized โ”‚ โ”‚ GlmImagePipeline โ”‚ โ”‚ +โ”‚ โ”‚ GlmImageFor โ”‚ prior โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ +โ”‚ โ”‚ Conditional โ”‚โ”€โ”€tokensโ”€โ”€โ”€โ–บโ”‚ โ”‚ DiT Denoiser โ”‚ โ”‚ โ”‚ +โ”‚ โ”‚ Generation โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ +โ”‚ โ”‚ (9B AR model) โ”‚ โ”‚ โ”‚ โ”‚ โ”‚ +โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ–ผ โ”‚ โ”‚ +โ”‚ โ–ฒ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ VAE Decode โ”‚โ”€โ”€โ”ผโ”€โ”€โ–บ Image +โ”‚ Text/Image โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”‚ +โ”‚ Input โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ +โ”‚ โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` + +## Features + +- **vLLM-optimized AR**: Uses PagedAttention and tensor parallelism for faster prior token generation +- **Flexible deployment**: AR and Diffusion stages can run on different GPUs +- **Text-to-Image**: Generate images from text descriptions +- **Image-to-Image**: Edit existing images with text prompts + +## Usage + +### Text-to-Image + +```bash +python end2end.py \ + --model-path /path/to/glm-image \ + --config-path ../../vllm_omni/model_executor/stage_configs/glm_image.yaml \ + --prompt "A beautiful sunset over the ocean with sailing boats" \ + --height 1024 \ + --width 1024 \ + --output output_t2i.png +``` + +### Image-to-Image (Image Editing) + +```bash +python end2end.py \ + --model-path /path/to/glm-image \ + --config-path ../../vllm_omni/model_executor/stage_configs/glm_image.yaml \ + --prompt "Transform this scene into a winter wonderland" \ + --image input.png \ + --output output_i2i.png +``` + +### With Custom Parameters + +```bash +python end2end.py \ + --model-path /path/to/glm-image \ + --config-path ../../vllm_omni/model_executor/stage_configs/glm_image.yaml \ + --prompt "A photorealistic cat sitting on a window sill" \ + --height 1024 \ + --width 1024 \ + --num-inference-steps 50 \ + --guidance-scale 1.5 \ + --seed 42 \ + --output output.png +``` + +## Shell Scripts + +### Run Text-to-Image + +```bash +./run_t2i.sh +``` + +### Run Image-to-Image + +```bash +./run_i2i.sh --image /path/to/input.png +``` + +## Stage Configuration + +The stage config (`glm_image.yaml`) defines: + +- **Stage 0 (AR)**: Uses `GPUARWorker` with vLLM engine + + - Model: `GlmImageForConditionalGeneration` + - Output: `token_ids` (prior tokens) + +- **Stage 1 (Diffusion)**: Uses diffusion engine + - Model: `GlmImagePipeline` + - Output: Generated image + +See `vllm_omni/model_executor/stage_configs/glm_image.yaml` for full configuration. + +## Comparison with Single-Stage + +| Aspect | Single-Stage (transformers) | Multistage (vLLM) | +| ----------- | --------------------------- | ------------------- | +| AR Model | transformers native | vLLM PagedAttention | +| Memory | Higher (no KV cache opt) | Lower (optimized) | +| Throughput | Lower | Higher | +| Flexibility | Single GPU | Multi-GPU support | + +## Troubleshooting + +### OOM Error + +Try reducing memory usage: + +```bash +# In glm_image.yaml, adjust: +gpu_memory_utilization: 0.5 # Reduce from 0.6 +``` + +### Slow Initialization + +The first run loads model weights. Subsequent runs are faster: + +```bash +--stage-init-timeout 900 # Increase timeout for slow storage +``` + +## Requirements + +- vLLM-Omni with GLM-Image support +- CUDA-capable GPU (recommended: H100/A100 with 80GB) +- GLM-Image model weights + +## Example materials + +??? abstract "end2end.py" + ``````py + --8<-- "examples/offline_inference/glm_image/end2end.py" + `````` +??? abstract "run_i2i.sh" + ``````sh + --8<-- "examples/offline_inference/glm_image/run_i2i.sh" + `````` +??? abstract "run_t2i.sh" + ``````sh + --8<-- "examples/offline_inference/glm_image/run_t2i.sh" + `````` diff --git a/docs/user_guide/examples/offline_inference/image_to_video.md b/docs/user_guide/examples/offline_inference/image_to_video.md index cfcadb702..8052c07af 100644 --- a/docs/user_guide/examples/offline_inference/image_to_video.md +++ b/docs/user_guide/examples/offline_inference/image_to_video.md @@ -69,6 +69,11 @@ Key arguments: - `--vae-use-tiling`: Enable VAE tiling for memory optimization. - `--cfg-parallel-size`: set it to 2 to enable CFG Parallel. See more examples in [`user_guide`](https://github.com/vllm-project/vllm-omni/tree/main/docs/user_guide/diffusion/parallelism_acceleration.md#cfg-parallel). - `--enable-cpu-offload`: enable CPU offloading for diffusion models. +- `--use-hsdp`: Enable Hybrid Sharded Data Parallel to shard model weights across GPUs. +- `--hsdp-shard-size`: Number of GPUs to shard model weights across within each replica group. -1 (default) auto-calculates as world_size / replicate_size. +- `--hsdp-replicate-size`: Number of replica groups for HSDP. Each replica holds a full sharded copy. Default 1 means pure sharding (no replication). + + > โ„น๏ธ If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage. diff --git a/docs/user_guide/examples/offline_inference/qwen3_tts.md b/docs/user_guide/examples/offline_inference/qwen3_tts.md index 878696a6b..a5acf32d6 100644 --- a/docs/user_guide/examples/offline_inference/qwen3_tts.md +++ b/docs/user_guide/examples/offline_inference/qwen3_tts.md @@ -90,13 +90,43 @@ Examples: python end2end.py --query-type Base --mode-tag icl ``` +## Streaming Mode + +Add `--streaming` to stream audio chunks progressively via `AsyncOmni` (requires `async_chunk: true` in the stage config): + +```bash +python end2end.py --query-type CustomVoice --streaming --output-dir /tmp/out_stream +``` + +Each 25-frame Code2Wav chunk is logged as it arrives. The final WAV file is written once generation +completes. This demonstrates that audio data is available progressively rather than only at the end. + +> **Note:** Streaming uses `AsyncOmni` internally. The non-streaming path (`Omni`) is unchanged. + +## Batched Decoding + +The Code2Wav stage (stage 1) supports batched decoding, where multiple requests are decoded in a single forward pass through the SpeechTokenizer. To use it, provide a stage config with `max_batch_size > 1` and pass multiple prompts via `--txt-prompts` with a matching `--batch-size`. + +``` +python end2end.py --query-type CustomVoice \ + --txt-prompts benchmark_prompts.txt \ + --batch-size 4 \ + --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_batch.yaml +``` + +**Important:** `--batch-size` must match a CUDA graph capture size (1, 2, 4, 8, 16...) because the Talker's code predictor KV cache is sized to `max_num_seqs`, and CUDA graphs pad the batch to the next capture size. Both stages need `max_batch_size >= batch_size` in the stage config for batching to take effect. If only stage 1 has a higher `max_batch_size`, it won't help โ€” stage 1 can only batch chunks from requests that are in-flight simultaneously, which requires stage 0 to also process multiple requests concurrently. + ## Notes - The script uses the model paths embedded in `end2end.py`. Update them if your local cache path differs. -- Use `--output-dir` (preferred) or `--output-wav` to change the output folder. +- Use `--output-dir` to change the output folder. ## Example materials +??? abstract "benchmark_prompts.txt" + ``````txt + --8<-- "examples/offline_inference/qwen3_tts/benchmark_prompts.txt" + `````` ??? abstract "end2end.py" ``````py --8<-- "examples/offline_inference/qwen3_tts/end2end.py" diff --git a/docs/user_guide/examples/offline_inference/text_to_image.md b/docs/user_guide/examples/offline_inference/text_to_image.md index a45108cf4..8b6dadc8e 100644 --- a/docs/user_guide/examples/offline_inference/text_to_image.md +++ b/docs/user_guide/examples/offline_inference/text_to_image.md @@ -3,7 +3,7 @@ Source . -This folder provides several entrypoints for experimenting with `Qwen/Qwen-Image` `Qwen/Qwen-Image-2512` `Tongyi-MAI/Z-Image-Turbo` using vLLM-Omni: +This folder provides several entrypoints for experimenting with `Qwen/Qwen-Image` `Qwen/Qwen-Image-2512` `Tongyi-MAI/Z-Image-Turbo` `stepfun-ai/NextStep-1.1` using vLLM-Omni, note that NextStep-1.1 has different architecture so we treat it differently regarding running arguments and pipeline. - `text_to_image.py`: command-line script for single image generation with advanced options. - `web_demo.py`: lightweight Gradio UI for interactive prompt/seed/CFG exploration. @@ -77,6 +77,8 @@ if __name__ == "__main__": ## Local CLI Usage +### Qwen/Tongyi Models + ```bash python text_to_image.py \ --model Tongyi-MAI/Z-Image-Turbo \ @@ -90,7 +92,26 @@ python text_to_image.py \ --output outputs/coffee.png ``` -Key arguments: +### NextStep Models + +NextStep-1.1 can have extra arguments +```bash +python text_to_image.py \ + --model stepfun-ai/NextStep-1.1 \ + --prompt "A baby panda wearing an Iron Man mask, holding a board with 'NextStep-1' written on it" \ + --height 512 \ + --width 512 \ + --num-inference-steps 28 \ + --guidance-scale 7.5 \ + --guidance-scale-2 1.0 \ + --cfg-schedule constant \ + --output nextstep_output.png \ + --seed 42 +``` + +### Key Arguments + +**Common arguments:** - `--prompt`: text description (string). - `--seed`: integer seed for deterministic sampling. @@ -101,8 +122,15 @@ Key arguments: - `--output`: path to save the generated PNG. - `--vae-use-slicing`: enable VAE slicing for memory optimization. - `--vae-use-tiling`: enable VAE tiling for memory optimization. -- `--cfg-parallel-size`: set it to 2 to enable CFG Parallel. See more examples in [`user_guide`](https://github.com/vllm-project/vllm-omni/tree/main/docs/user_guide/diffusion/parallelism_acceleration.md#cfg-parallel). +- `--cfg-parallel-size`: set it to 2 to enable CFG Parallel. See more examples in [`user_guide`](https://github.com/vllm-project/vllm-omni/tree/main/docs/user_guide/diffusion_acceleration.md#using-cfg-parallel). - `--enable-cpu-offload`: enable CPU offloading for diffusion models. +- `--guidance-scale`: classifier-free guidance scale. + +**NextStep-1.1 specific:** +- `--guidance-scale-2`: secondary guidance scale, e.g. image-level CFG (default: 1.0). +- `--timesteps-shift`: timesteps shift parameter for sampling (default: 1.0). +- `--cfg-schedule`: CFG schedule type, "constant" or "linear" (default: "constant"). +- `--use-norm`: apply layer normalization to sampled tokens. > โ„น๏ธ If you encounter OOM errors, try using `--vae-use-slicing` and `--vae-use-tiling` to reduce memory usage. diff --git a/docs/user_guide/examples/online_serving/bagel.md b/docs/user_guide/examples/online_serving/bagel.md index f29c095f4..4a6094c08 100644 --- a/docs/user_guide/examples/online_serving/bagel.md +++ b/docs/user_guide/examples/online_serving/bagel.md @@ -25,12 +25,152 @@ cd /workspace/vllm-omni/examples/online_serving/bagel bash run_server.sh ``` -If you have a custom stage configs file, launch the server with the command below: - ```bash vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/stage_configs_file ``` +#### ๐Ÿš€ Tensor Parallelism (TP) + +For larger models or multi-GPU environments, you can enable Tensor Parallelism (TP) for the server. + +1. **Modify Stage Config**: Create or modify a stage configuration yaml (e.g., [`bagel.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/bagel.yaml)). Set `tensor_parallel_size` to `2` (or more) and update `devices` to include multiple GPU IDs (e.g., `"0,1"`). + +```yaml + engine_args: + tensor_parallel_size: 2 + ... + runtime: + devices: "0,1" +``` + +2. **Launch Server**: +```bash +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/your/custom_bagel.yaml +``` + +#### Using Mooncake Connector + +By default, BAGEL uses `SharedMemoryConnector` for inter-stage communication. You can use the [Mooncake](https://github.com/kvcache-ai/Mooncake) connector to transfer KV cache between stages, which also enables multi-node deployment. + +**1. Install Mooncake** + +```bash +# For CUDA-enabled systems (recommended) +pip install mooncake-transfer-engine + +# For non-CUDA systems +pip install mooncake-transfer-engine-non-cuda +``` + +**2. Start Mooncake Master** on the primary node: + +```bash +# Optional: enable disk-backed storage by creating a directory and passing --root_fs_dir. +# Without it, Mooncake runs in memory-only mode, which is sufficient for KV cache transfer. +mkdir -p ./mc_storage + +mooncake_master \ + --rpc_port=50051 \ + --enable_http_metadata_server=true \ + --http_metadata_server_host=0.0.0.0 \ + --http_metadata_server_port=8080 \ + --metrics_port=9003 \ + --root_fs_dir=./mc_storage/ \ + --cluster_id=mc-local-1 & +``` + +**3. Launch the server** with the Mooncake stage config: + +```bash +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 \ + --stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml +``` + +> **Note**: Before launching, edit [`bagel_multiconnector.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml) and replace the `metadata_server` and `master` addresses with your Mooncake master node's actual IP. For single-node testing, `127.0.0.1` works. + +The client-side usage is identical to the default setup -- the Mooncake connector is transparent to the API. See the requests section below. + +For more details on the Mooncake connector configuration, see the [Mooncake Store Connector documentation](https://github.com/vllm-project/vllm-omni/tree/main/docs/design/feature/omni_connectors/mooncake_store_connector.md). + +#### Multi-Node Deployment + +You can deploy each stage on a **separate node** for better resource utilization. In this example, the orchestrator (Stage 0 / Thinker) and Stage 1 (DiT) run on different machines, connected via Mooncake. + +Replace `` below with the actual IP address of your orchestrator node (e.g., `10.244.227.244`). + +> [!WARNING] +> **Before launching**, edit [`bagel_multiconnector.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml) and replace the `metadata_server` and `master` addresses with your Mooncake master node's actual IP. Mismatched addresses will cause silent connection failures. + +**1. Start Mooncake Master** (on the orchestrator node): + +```bash +mooncake_master \ + --rpc_port=50051 \ + --enable_http_metadata_server=true \ + --http_metadata_server_host= \ + --http_metadata_server_port=8080 \ + --metrics_port=9003 +``` + +**2. Launch Stage 0 (Thinker / Orchestrator)** on the orchestrator node: + +```bash +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \ + --port 8000 \ # API server port for client requests + --stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \ + --stage-id 0 \ + -oma \ + -omp 8091 +``` + +**3. Launch Stage 1 (DiT)** on the remote node in headless mode: + +```bash +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni \ + --stage-configs-path vllm_omni/model_executor/stage_configs/bagel_multiconnector.yaml \ + --stage-id 1 \ + --headless \ + -oma \ + -omp 8091 +``` + +**Mooncake Master arguments:** + +| Argument | Description | +| :------- | :---------- | +| `--rpc_port` | Mooncake RPC port for control-plane coordination between stages | +| `--enable_http_metadata_server` | Enable the HTTP metadata server for service discovery | +| `--http_metadata_server_host` | IP address to bind the metadata server (use the orchestrator node's IP) | +| `--http_metadata_server_port` | Port for the HTTP metadata server | +| `--metrics_port` | Port for Prometheus-compatible metrics endpoint | + +**vllm serve arguments:** + +| Argument | Description | +| :------- | :---------- | +| `--stage-id` | Which stage this process runs (0 = Thinker, 1 = DiT) | +| `--headless` | Run without the API server (worker-only mode) | +| `-oma` | Orchestrator master address | +| `-omp` | Orchestrator master port for Stage 1 to connect to Stage 0 for task coordination | + +> [!IMPORTANT] +> **Startup Order**: Stage 0 (orchestrator) must be launched **before** Stage 1 (headless). +> Stage 0 will appear to hang on startup until Stage 1 (worker) connects โ€” this is expected behavior. + +**Network Requirements** + +All nodes must have network connectivity to each other. Ensure the following ports are open **between all participating nodes**: + +| Port | Protocol | Service | Direction | +| :--- | :------- | :------ | :-------- | +| 50051 | TCP | Mooncake Master RPC | Worker โ†’ Orchestrator | +| 8080 | TCP | Mooncake HTTP Metadata Server | Worker โ†’ Orchestrator | +| 8091 | TCP | Orchestrator Master (`-omp`) | Worker โ†’ Orchestrator | +| 8000 | TCP | API Server (`--port`) | Client โ†’ Orchestrator | +| 9003 | TCP | Metrics (optional) | Monitoring โ†’ Orchestrator | + +> **Tip**: If nodes are behind a firewall or in different VPCs/security groups, make sure the above ports are allowed in ingress/egress rules. All nodes should be reachable via their IP addresses (no NAT). Using nodes on the same subnet or VPC is recommended to minimize latency for Mooncake KV cache transfers. + ### Send Multi-modal Request Get into the bagel folder: @@ -242,3 +382,7 @@ sudo apt install ffmpeg ``````sh --8<-- "examples/online_serving/bagel/run_server.sh" `````` +??? abstract "run_server_stage_cli.sh" + ``````sh + --8<-- "examples/online_serving/bagel/run_server_stage_cli.sh" + `````` diff --git a/docs/user_guide/examples/online_serving/glm_image.md b/docs/user_guide/examples/online_serving/glm_image.md new file mode 100644 index 000000000..c0d176480 --- /dev/null +++ b/docs/user_guide/examples/online_serving/glm_image.md @@ -0,0 +1,269 @@ +# GLM-Image Online Serving + +Source . + + +This example demonstrates how to deploy GLM-Image for online image generation using vLLM-Omni. + +## ๐Ÿ› ๏ธ Installation + +Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md) + +## Run examples (GLM-Image) + +**Note**: These examples work with the default configuration on **2ร— NVIDIA A100 (80GB)** or equivalent. Stage 0 (AR) and Stage 1 (Diffusion) each use one GPU by default. For single-GPU setups, modify the stage configuration to share the same device. + +### Launch the Server + +```bash +# Use default configuration +vllm serve zai-org/GLM-Image --omni --port 8091 +``` + +Or use the convenience script: + +```bash +cd examples/online_serving/glm_image +bash run_server.sh +``` + +If you have a custom stage configs file: + +```bash +vllm serve zai-org/GLM-Image --omni --port 8091 --stage-configs-path /path/to/glm_image.yaml +``` + +### Send Requests + +Get into the glm_image folder: + +```bash +cd examples/online_serving/glm_image +``` + +Send request via Python: + +```bash +python openai_chat_client.py --prompt "A cute cat sitting on a window sill" +``` + +The Python client supports the following command-line arguments: + +- `--prompt` (or `-p`): Text prompt for generation (default: `A beautiful sunset over the ocean with sailing boats`) +- `--output` (or `-o`): Output file path (default: `glm_image_output.png`) +- `--server` (or `-s`): Server URL (default: `http://localhost:8091`) +- `--image` (or `-i`): Input image path (for image-to-image editing) +- `--height`: Image height in pixels (default: 1024) +- `--width`: Image width in pixels (default: 1024) +- `--steps`: Number of inference steps (default: 50) +- `--guidance-scale`: Classifier-free guidance scale (default: 1.5) +- `--seed`: Random seed (default: 42) +- `--negative`: Negative prompt + +## Modality Control + +GLM-Image supports **text-to-image** and **image-to-image** modes. + +The default yaml configuration deploys AR on GPU 0 and DiT on GPU 1. You can use the default configuration file: [`glm_image.yaml`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/model_executor/stage_configs/glm_image.yaml) + +| Mode | Input | Output | Description | +| -------------- | ------------ | ------ | ---------------------------------- | +| Text-to-Image | Text | Image | Generate images from text prompts | +| Image-to-Image | Image + Text | Image | Edit images with text instructions | + +### Text-to-Image + +Generate images from text prompts: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "A photorealistic mountain landscape at sunset" \ + --height 1024 \ + --width 1024 \ + --output landscape.png +``` + +**Using curl** + +```bash +curl -s http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "messages": [ + {"role": "user", "content": "A beautiful sunset over the ocean with sailing boats"} + ], + "extra_body": { + "height": 1024, + "width": 1024, + "num_inference_steps": 50, + "guidance_scale": 1.5, + "seed": 42 + } + }' | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png +``` + +Or use the script: + +```bash +bash run_curl_text_to_image.sh "A futuristic city skyline at night" +``` + +### Image-to-Image (Image Editing) + +Edit images with text instructions: + +**Using Python client** + +```bash +python openai_chat_client.py \ + --prompt "Convert this image to watercolor style" \ + --image input.png \ + --output watercolor.png +``` + +**Using curl** + +```bash +IMG_B64=$(base64 < input.png | tr -d '\n') + +curl -s http://localhost:8091/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d @- < output.png +{ + "messages": [{ + "role": "user", + "content": [ + {"type": "text", "text": "Convert this image to watercolor style"}, + {"type": "image_url", "image_url": {"url": "data:image/png;base64,'$IMG_B64'"}} + ] + }], + "extra_body": { + "height": 1024, + "width": 1024, + "num_inference_steps": 50, + "guidance_scale": 1.5, + "seed": 42 + } +} +EOF +``` + +Or use the script: + +```bash +bash run_curl_image_edit.sh input.png "Convert to watercolor style" +``` + +## Generation Parameters (extra_body) + +| Parameter | Type | Default | Description | +| --------------------- | ----- | ------- | ----------------------------------- | +| `height` | int | 1024 | Image height in pixels | +| `width` | int | 1024 | Image width in pixels | +| `num_inference_steps` | int | 50 | Number of diffusion denoising steps | +| `guidance_scale` | float | 1.5 | Classifier-free guidance scale | +| `seed` | int | 42 | Random seed for reproducibility | +| `negative_prompt` | str | None | Negative prompt | + +## Response Format + +```json +{ + "id": "chatcmpl-xxx", + "created": 1234567890, + "model": "zai-org/GLM-Image", + "choices": [ + { + "index": 0, + "message": { + "role": "assistant", + "content": [ + { + "type": "image_url", + "image_url": { + "url": "data:image/png;base64,..." + } + } + ] + }, + "finish_reason": "stop" + } + ], + "usage": {} +} +``` + +## Extract Image + +```bash +# From a saved JSON response +cat response.json | jq -r '.choices[0].message.content[0].image_url.url' | cut -d',' -f2- | base64 -d > output.png +``` + +## Architecture + +GLM-Image uses a 2-stage multistage pipeline: + +``` +Stage 0 (AR Model) Stage 1 (Diffusion) +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ vLLM-optimized โ”‚ prior โ”‚ GlmImagePipeline โ”‚ +โ”‚ GlmImageFor โ”‚โ”€โ”€tokensโ”€โ”€โ–บโ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ +โ”‚ Conditional โ”‚ โ”‚ โ”‚ DiT Denoiser โ”‚ โ”‚ +โ”‚ Generation โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ +โ”‚ (9B AR model) โ”‚ โ”‚ โ–ผ โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ + โ–ฒ โ”‚ โ”‚ VAE Decode โ”‚โ”€โ”€โ”ผโ”€โ”€โ–บ Image + โ”‚ โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ + Text / Image โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + Input +``` + +## VRAM Requirements + +| Stage | VRAM | +| :---------------- | :--------------------- | +| Stage-0 (AR) | **~18 GiB + KV Cache** | +| Stage-1 (DiT+VAE) | **~20 GiB** | +| Total | **~38 GiB + KV Cache** | + +## File Description + +| File | Description | +| --------------------------- | ------------------------------------- | +| `run_server.sh` | Server startup script | +| `run_curl_text_to_image.sh` | Text-to-image curl example | +| `run_curl_image_edit.sh` | Image-to-image (editing) curl example | +| `openai_chat_client.py` | Python client (t2i + i2i) | + +## FAQ + +- If you encounter OOM errors, adjust `gpu_memory_utilization` in the stage config: + +```yaml +# In glm_image.yaml, reduce from default 0.6: +gpu_memory_utilization: 0.5 +``` + +- The first request may be slow due to model warmup. Subsequent requests will be faster. + +## Example materials + +??? abstract "openai_chat_client.py" + ``````py + --8<-- "examples/online_serving/glm_image/openai_chat_client.py" + `````` +??? abstract "run_curl_image_edit.sh" + ``````sh + --8<-- "examples/online_serving/glm_image/run_curl_image_edit.sh" + `````` +??? abstract "run_curl_text_to_image.sh" + ``````sh + --8<-- "examples/online_serving/glm_image/run_curl_text_to_image.sh" + `````` +??? abstract "run_server.sh" + ``````sh + --8<-- "examples/online_serving/glm_image/run_server.sh" + `````` diff --git a/docs/user_guide/examples/online_serving/image_to_video.md b/docs/user_guide/examples/online_serving/image_to_video.md new file mode 100644 index 000000000..dcbb10067 --- /dev/null +++ b/docs/user_guide/examples/online_serving/image_to_video.md @@ -0,0 +1,97 @@ +# Image-To-Video + +Source . + + +This example demonstrates how to deploy the Wan2.2 image-to-video model for online video generation using vLLM-Omni. + +## Start Server + +### Basic Start + +```bash +vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --port 8091 +``` + +### Start with Parameters + +Or use the startup script: + +```bash +bash run_server.sh +``` + +The script allows overriding: +- `MODEL` (default: `Wan-AI/Wan2.2-I2V-A14B-Diffusers`) +- `PORT` (default: `8091`) +- `BOUNDARY_RATIO` (default: `0.875`) +- `FLOW_SHIFT` (default: `12.0`) +- `CACHE_BACKEND` (default: `none`) +- `ENABLE_CACHE_DIT_SUMMARY` (default: `0`) + +## API Calls + +### Method 1: Using curl + +```bash +# Basic image-to-video generation +bash run_curl_image_to_video.sh + +# Or execute directly (OpenAI-style multipart) +curl -X POST http://localhost:8091/v1/videos \ + -H "Accept: application/json" \ + -F "prompt=A bear playing with yarn, smooth motion" \ + -F "negative_prompt=low quality, blurry, static" \ + -F "input_reference=@/path/to/qwen-bear.png" \ + -F "width=832" \ + -F "height=480" \ + -F "num_frames=33" \ + -F "fps=16" \ + -F "num_inference_steps=40" \ + -F "guidance_scale=1.0" \ + -F "guidance_scale_2=1.0" \ + -F "boundary_ratio=0.875" \ + -F "flow_shift=12.0" \ + -F "seed=42" | jq -r '.data[0].b64_json' | base64 -d > wan22_i2v_output.mp4 +``` + +## Request Format + +### Required Fields + +```bash +curl -X POST http://localhost:8091/v1/videos \ + -F "prompt=A bear playing with yarn, smooth motion" \ + -F "negative_prompt=low quality, blurry, static" \ + -F "input_reference=@/path/to/qwen-bear.png" +``` + +### Generation with Parameters + +```bash +curl -X POST http://localhost:8091/v1/videos \ + -F "prompt=A bear playing with yarn, smooth motion" \ + -F "negative_prompt=low quality, blurry, static" \ + -F "input_reference=@/path/to/qwen-bear.png" \ + -F "width=832" \ + -F "height=480" \ + -F "num_frames=33" \ + -F "fps=16" \ + -F "num_inference_steps=40" \ + -F "guidance_scale=1.0" \ + -F "guidance_scale_2=1.0" \ + -F "boundary_ratio=0.875" \ + -F "flow_shift=12.0" \ + -F "seed=42" +``` + +## Example materials + +??? abstract "run_curl_image_to_video.sh" + ``````sh + --8<-- "examples/online_serving/image_to_video/run_curl_image_to_video.sh" + `````` +??? abstract "run_server.sh" + ``````sh + --8<-- "examples/online_serving/image_to_video/run_server.sh" + `````` diff --git a/docs/user_guide/examples/online_serving/qwen2_5_omni.md b/docs/user_guide/examples/online_serving/qwen2_5_omni.md index 361044ed8..976d39a96 100644 --- a/docs/user_guide/examples/online_serving/qwen2_5_omni.md +++ b/docs/user_guide/examples/online_serving/qwen2_5_omni.md @@ -30,7 +30,7 @@ cd examples/online_serving/qwen2_5_omni #### Send request via python ```bash -python openai_chat_completion_client_for_multimodal_generation.py --query-type mixed_modalities +python openai_chat_completion_client_for_multimodal_generation.py --query-type mixed_modalities --port 8091 --host "localhost" ``` The Python client supports the following command-line arguments: diff --git a/docs/user_guide/examples/online_serving/qwen3_omni.md b/docs/user_guide/examples/online_serving/qwen3_omni.md index d26246533..c903215f2 100644 --- a/docs/user_guide/examples/online_serving/qwen3_omni.md +++ b/docs/user_guide/examples/online_serving/qwen3_omni.md @@ -36,7 +36,7 @@ cd examples/online_serving/qwen3_omni #### Send request via python ```bash -python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image +python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image --port 8091 --host "localhost" ``` The Python client supports the following command-line arguments: diff --git a/docs/user_guide/examples/online_serving/qwen3_tts.md b/docs/user_guide/examples/online_serving/qwen3_tts.md index 83253491b..82f187b25 100644 --- a/docs/user_guide/examples/online_serving/qwen3_tts.md +++ b/docs/user_guide/examples/online_serving/qwen3_tts.md @@ -9,13 +9,13 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/ ## Supported Models -| Model | Task Type | Description | -|-------|-----------|-------------| +| Model | Task Type | Description | +| -------------------------------------- | ----------- | ----------------------------------------------------- | | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control | -| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description | -| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio | -| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | -| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | +| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description | +| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio | +| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | +| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | ## Run examples (Qwen3-TTS) @@ -237,32 +237,33 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w ### OpenAI Standard Parameters -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `input` | string | **required** | Text to synthesize | -| `model` | string | server's model | Model to use (optional, should match server if specified) | -| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) | -| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | -| `speed` | float | 1.0 | Playback speed (0.25-4.0) | +| Parameter | Type | Default | Description | +| ----------------- | ------ | -------------- | ----------------------------------------------------------- | +| `input` | string | **required** | Text to synthesize | +| `model` | string | server's model | Model to use (optional, should match server if specified) | +| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) | +| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | +| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) | ### vLLM-Omni Extension Parameters -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | -| `language` | string | "Auto" | Language (see supported languages below) | -| `instructions` | string | "" | Voice style/emotion instructions | -| `max_new_tokens` | int | 2048 | Maximum tokens to generate | +| Parameter | Type | Default | Description | +| ---------------- | ------ | ------------- | ---------------------------------------------------------------------------- | +| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | +| `language` | string | "Auto" | Language (see supported languages below) | +| `instructions` | string | "" | Voice style/emotion instructions | +| `max_new_tokens` | int | 2048 | Maximum tokens to generate | +| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | **Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian ### Voice Clone Parameters (Base task) -| Parameter | Type | Required | Description | -|-----------|------|----------|-------------| -| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) | -| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | -| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | +| Parameter | Type | Required | Description | +| -------------------- | ------ | -------- | -------------------------------------------- | +| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) | +| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | +| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | ## Streaming diff --git a/docs/user_guide/examples/online_serving/text_to_video.md b/docs/user_guide/examples/online_serving/text_to_video.md new file mode 100644 index 000000000..a632c49a7 --- /dev/null +++ b/docs/user_guide/examples/online_serving/text_to_video.md @@ -0,0 +1,130 @@ +# Text-To-Video + +Source . + + +This example demonstrates how to deploy the Wan2.2 text-to-video model for online video generation using vLLM-Omni. + +## Start Server + +### Basic Start + +```bash +vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091 +``` + +### Start with Parameters + +Or use the startup script: + +```bash +bash run_server.sh +``` + +The script allows overriding: +- `MODEL` (default: `Wan-AI/Wan2.2-T2V-A14B-Diffusers`) +- `PORT` (default: `8091`) +- `BOUNDARY_RATIO` (default: `0.875`) +- `FLOW_SHIFT` (default: `5.0`) +- `CACHE_BACKEND` (default: `none`) +- `ENABLE_CACHE_DIT_SUMMARY` (default: `0`) + +## API Calls + +### Method 1: Using curl + +```bash +# Basic text-to-video generation +bash run_curl_text_to_video.sh + +# Or execute directly (OpenAI-style multipart) +curl -s http://localhost:8091/v1/videos \ + -H "Accept: application/json" \ + -F "prompt=Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." \ + -F "width=832" \ + -F "height=480" \ + -F "num_frames=33" \ + -F "negative_prompt=่‰ฒ่ฐƒ่‰ณไธฝ ๏ผŒ่ฟ‡ๆ›๏ผŒ้™ๆ€๏ผŒ็ป†่Š‚ๆจก็ณŠไธๆธ…๏ผŒๅญ—ๅน•๏ผŒ้ฃŽๆ ผ๏ผŒไฝœๅ“๏ผŒ็”ปไฝœ๏ผŒ็”ป้ข๏ผŒ้™ๆญข๏ผŒๆ•ดไฝ“ๅ‘็ฐ๏ผŒๆœ€ๅทฎ่ดจ้‡๏ผŒไฝŽ่ดจ้‡๏ผŒJPEGๅŽ‹็ผฉๆฎ‹็•™๏ผŒไธ‘้™‹็š„๏ผŒๆฎ‹็ผบ็š„๏ผŒๅคšไฝ™็š„ๆ‰‹ๆŒ‡๏ผŒ็”ปๅพ—ไธๅฅฝ็š„ๆ‰‹้ƒจ๏ผŒ็”ปๅพ—ไธๅฅฝ็š„่„ธ้ƒจ๏ผŒ็•ธๅฝข็š„๏ผŒๆฏๅฎน็š„๏ผŒๅฝขๆ€็•ธๅฝข็š„่‚ขไฝ“๏ผŒๆ‰‹ๆŒ‡่žๅˆ๏ผŒ้™ๆญขไธๅŠจ็š„็”ป้ข๏ผŒๆ‚ไนฑ็š„่ƒŒๆ™ฏ๏ผŒไธ‰ๆก่…ฟ๏ผŒ่ƒŒๆ™ฏไบบๅพˆๅคš๏ผŒๅ€’็€่ตฐ" \ + -F "fps=16" \ + -F "num_inference_steps=40" \ + -F "guidance_scale=4.0" \ + -F "guidance_scale_2=4.0" \ + -F "boundary_ratio=0.875" \ + -F "seed=42" | jq -r '.data[0].b64_json' | base64 -d > wan22_output.mp4 +``` + +## Request Format + +### Simple Text-to-Video Generation + +```bash +curl -X POST http://localhost:8091/v1/videos \ + -F "prompt=A cinematic view of a futuristic city at sunset" +``` + +### Generation with Parameters + +```bash +curl -X POST http://localhost:8091/v1/videos \ + -F "prompt=A cinematic view of a futuristic city at sunset" \ + -F "width=832" \ + -F "height=480" \ + -F "num_frames=33" \ + -F "negative_prompt=low quality, blurry, static" \ + -F "fps=16" \ + -F "num_inference_steps=40" \ + -F "guidance_scale=4.0" \ + -F "guidance_scale_2=4.0" \ + -F "boundary_ratio=0.875" \ + -F "flow_shift=5.0" \ + -F "seed=42" +``` + +## Generation Parameters + +| Parameter | Type | Default | Description | +| --------------------- | ------ | ------- | ------------------------------------------------ | +| `prompt` | str | - | Text description of the desired video | +| `negative_prompt` | str | None | Negative prompt | +| `n` | int | 1 | Number of videos to generate | +| `width` | int | None | Video width in pixels | +| `height` | int | None | Video height in pixels | +| `num_frames` | int | None | Number of frames to generate | +| `fps` | int | None | Frames per second for output video | +| `num_inference_steps` | int | None | Number of denoising steps | +| `guidance_scale` | float | None | CFG guidance scale (low-noise stage) | +| `guidance_scale_2` | float | None | CFG guidance scale (high-noise stage, Wan2.2) | +| `boundary_ratio` | float | None | Boundary split ratio for low/high DiT (Wan2.2) | +| `flow_shift` | float | None | Scheduler flow shift (Wan2.2) | +| `seed` | int | None | Random seed (reproducible) | +| `lora` | object | None | LoRA configuration | +| `extra_body` | object | None | Model-specific extra parameters | + +## Response Format + +```json +{ + "created": 1234567890, + "data": [ + { "b64_json": "" } + ] +} +``` + +## Extract Video + +```bash +# Extract base64 from response and decode to video +cat response.json | jq -r '.data[0].b64_json' | base64 -d > wan22_output.mp4 +``` + +## Example materials + +??? abstract "run_curl_text_to_video.sh" + ``````sh + --8<-- "examples/online_serving/text_to_video/run_curl_text_to_video.sh" + `````` +??? abstract "run_server.sh" + ``````sh + --8<-- "examples/online_serving/text_to_video/run_server.sh" + `````` diff --git a/examples/online_serving/qwen3_tts/README.md b/examples/online_serving/qwen3_tts/README.md index 71a3960f0..d50baa289 100644 --- a/examples/online_serving/qwen3_tts/README.md +++ b/examples/online_serving/qwen3_tts/README.md @@ -1,40 +1,75 @@ # Qwen3-TTS -This directory contains examples for running Qwen3-TTS models with vLLM-Omni's online serving API. +## ๐Ÿ› ๏ธ Installation + +Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md) ## Supported Models -| Model | Task Type | Description | -|-------|-----------|-------------| +| Model | Task Type | Description | +| -------------------------------------- | ----------- | ----------------------------------------------------- | | `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` | CustomVoice | Predefined speaker voices with optional style control | -| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description | -| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio | -| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | -| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | +| `Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign` | VoiceDesign | Natural language voice style description | +| `Qwen/Qwen3-TTS-12Hz-1.7B-Base` | Base | Voice cloning from reference audio | +| `Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice` | CustomVoice | Smaller/faster variant | +| `Qwen/Qwen3-TTS-12Hz-0.6B-Base` | Base | Smaller/faster variant for voice cloning | -## Quick Start +## Run examples (Qwen3-TTS) -### 1. Start the Server - -```bash -# CustomVoice model (default) -./run_server.sh - -# Or specify task type -./run_server.sh CustomVoice -./run_server.sh VoiceDesign -./run_server.sh Base -``` - -Or launch directly with vllm serve: +### Launch the Server ```bash +# CustomVoice model (predefined speakers) vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \ - --omni --port 8091 --trust-remote-code --enforce-eager + --omni \ + --port 8091 \ + --trust-remote-code \ + --enforce-eager + +# VoiceDesign model +vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \ + --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \ + --omni \ + --port 8091 \ + --trust-remote-code \ + --enforce-eager + +# Base model (voice cloning) +vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base \ + --stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts.yaml \ + --omni \ + --port 8091 \ + --trust-remote-code \ + --enforce-eager ``` -### 2. Run the Client +If you have custom stage configs file, launch the server with command below +```bash +vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ + --stage-configs-path /path/to/stage_configs_file \ + --omni \ + --port 8091 \ + --trust-remote-code \ + --enforce-eager +``` + +Alternatively, use the convenience script: +```bash +./run_server.sh # Default: CustomVoice model +./run_server.sh CustomVoice # CustomVoice model +./run_server.sh VoiceDesign # VoiceDesign model +./run_server.sh Base # Base (voice clone) model +``` + +### Send TTS Request + +Get into the example folder +```bash +cd examples/online_serving/qwen3_tts +``` + +#### Send request via python ```bash # CustomVoice: Use predefined speaker @@ -65,7 +100,21 @@ python openai_speech_client.py \ --ref-text "Original transcript of the reference audio" ``` -### 3. Using curl +The Python client supports the following command-line arguments: + +- `--api-base`: API base URL (default: `http://localhost:8091`) +- `--model` (or `-m`): Model name/path (default: `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice`) +- `--task-type` (or `-t`): TTS task type. Options: `CustomVoice`, `VoiceDesign`, `Base` +- `--text`: Text to synthesize (required) +- `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc. +- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian` +- `--instructions`: Voice style/emotion instructions +- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task) +- `--ref-text`: Reference audio transcript for voice cloning (Base task) +- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus` +- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`) + +#### Send request via curl ```bash # Simple TTS request @@ -90,24 +139,48 @@ curl -X POST http://localhost:8091/v1/audio/speech \ curl http://localhost:8091/v1/audio/voices ``` -## Server Configuration +### Using OpenAI SDK -### TTS-Specific CLI Parameters +```python +from openai import OpenAI -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `--tts-max-instructions-length` | int | 500 | Maximum character length for voice style instructions | +client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") -**Example with custom instructions length:** +response = client.audio.speech.create( + model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", + voice="vivian", + input="Hello, how are you?", +) -```bash -vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ - --omni \ - --port 8000 \ - --tts-max-instructions-length 1000 +response.stream_to_file("output.wav") ``` -This is useful when you need longer voice style descriptions, especially for the VoiceDesign task type. +### Using Python httpx + +```python +import httpx + +response = httpx.post( + "http://localhost:8091/v1/audio/speech", + json={ + "input": "Hello, how are you?", + "voice": "vivian", + "language": "English", + }, + timeout=300.0, +) + +with open("output.wav", "wb") as f: + f.write(response.content) +``` + +### FAQ + +If you encounter error about backend of librosa, try to install ffmpeg with command below. +``` +sudo apt update +sudo apt install ffmpeg +``` ## API Reference @@ -115,6 +188,7 @@ This is useful when you need longer voice style descriptions, especially for the ``` POST /v1/audio/speech +Content-Type: application/json ``` This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/docs/api-reference/audio/createSpeech) format with additional Qwen3-TTS parameters. @@ -146,8 +220,7 @@ Lists available voices for the loaded model: "ref_audio": "URL or base64 for voice cloning", "ref_text": "Reference audio transcript", "x_vector_only_mode": false, - "max_new_tokens": 2048, - "stream": false + "max_new_tokens": 2048 } ``` @@ -155,84 +228,46 @@ Lists available voices for the loaded model: ### Response -Returns audio data in the requested format (default: WAV). +Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/wav`). ## Parameters ### OpenAI Standard Parameters -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `input` | string | **required** | Text to synthesize | -| `model` | string | server's model | Model to use (optional, should match server if specified) | -| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) | -| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | -| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) | +| Parameter | Type | Default | Description | +| ----------------- | ------ | -------------- | ----------------------------------------------------------- | +| `input` | string | **required** | Text to synthesize | +| `model` | string | server's model | Model to use (optional, should match server if specified) | +| `voice` | string | "vivian" | Speaker name (e.g., vivian, ryan, aiden) | +| `response_format` | string | "wav" | Audio format: wav, mp3, flac, pcm, aac, opus | +| `speed` | float | 1.0 | Playback speed (0.25-4.0, not supported with `stream=true`) | ### vLLM-Omni Extension Parameters -| Parameter | Type | Default | Description | -|-----------|------|---------|-------------| -| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | -| `language` | string | "Auto" | Language (see supported languages below) | -| `instructions` | string | "" | Voice style/emotion instructions | -| `max_new_tokens` | int | 2048 | Maximum tokens to generate | -| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | +| Parameter | Type | Default | Description | +| ---------------- | ------ | ------------- | ---------------------------------------------------------------------------- | +| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | +| `language` | string | "Auto" | Language (see supported languages below) | +| `instructions` | string | "" | Voice style/emotion instructions | +| `max_new_tokens` | int | 2048 | Maximum tokens to generate | +| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | **Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian ### Voice Clone Parameters (Base task) -| Parameter | Type | Default | Description | -|-----------|------|----------|-------------| -| `ref_audio` | string | null | Reference audio (URL or base64 data URL) | -| `ref_text` | string | null | Transcript of reference audio | -| `x_vector_only_mode` | bool | null | Use speaker embedding only (no ICL) | - -## Python Usage - -### Using OpenAI SDK - -```python -from openai import OpenAI - -client = OpenAI(base_url="http://localhost:8091/v1", api_key="none") - -response = client.audio.speech.create( - model="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice", - voice="vivian", - input="Hello, how are you?", -) - -response.stream_to_file("output.wav") -``` - -### Using httpx - -```python -import httpx - -response = httpx.post( - "http://localhost:8091/v1/audio/speech", - json={ - "input": "Hello, how are you?", - "voice": "vivian", - "language": "English", - }, - timeout=300.0, -) - -with open("output.wav", "wb") as f: - f.write(response.content) -``` +| Parameter | Type | Required | Description | +| -------------------- | ------ | -------- | -------------------------------------------- | +| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) | +| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | +| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | ## Streaming Set `stream=true` with `response_format="pcm"` to receive raw PCM audio chunks as they are decoded -(one chunk per 25-frame Code2Wav window, ~2 s latency to first byte): +(one chunk per 25-frame Code2Wav window): ```bash -# Stream raw PCM and pipe directly to a player curl -X POST http://localhost:8091/v1/audio/speech \ -H "Content-Type: application/json" \ -d '{ @@ -242,18 +277,12 @@ curl -X POST http://localhost:8091/v1/audio/speech \ "stream": true, "response_format": "pcm" }' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 - - -# Or save the raw PCM for offline processing -curl -X POST http://localhost:8091/v1/audio/speech \ - -H "Content-Type: application/json" \ - -d '{"input": "Hello!", "stream": true, "response_format": "pcm"}' \ - --output output.pcm ``` **Constraints:** - `stream=true` requires `response_format="pcm"` (raw 16-bit signed PCM, 24 kHz mono). -- `speed` adjustment is not supported when streaming (`speed` must be 1.0 or omitted). -- Requires the server to be launched with a stage config that sets `async_chunk: true`. +- `speed` adjustment is not supported when streaming. +- Requires the server stage config to have `async_chunk: true` (default in `qwen3_tts.yaml`). ## Limitations @@ -261,7 +290,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \ ## Troubleshooting -1. **"TTS model did not produce audio output"**: Ensure you're using the correct model variant for your task type (CustomVoice task โ†’ CustomVoice model, etc.) +1. **TTS model did not produce audio output**: Ensure you're using the correct model variant for your task type (CustomVoice task โ†’ CustomVoice model, etc.) 2. **Connection refused**: Make sure the server is running on the correct port 3. **Out of memory**: Use smaller model variant (`Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice`) or reduce `--gpu-memory-utilization` 4. **Unsupported speaker**: Use `/v1/audio/voices` to list available voices for the loaded model