diff --git a/docs/user_guide/examples/offline_inference/mimo_audio.md b/docs/user_guide/examples/offline_inference/mimo_audio.md index 6af7bc72e..1a3be15d6 100644 --- a/docs/user_guide/examples/offline_inference/mimo_audio.md +++ b/docs/user_guide/examples/offline_inference/mimo_audio.md @@ -13,7 +13,7 @@ MiMo-Audio provides multiple task variants for audio understanding and generatio - **tts_sft_with_instruct**: TTS generation with explicit voice style instructions. - **tts_sft_with_audio**: TTS generation with audio reference for voice cloning. - **tts_sft_with_natural_instruction**: TTS generation from natural language descriptions embedded in text. -- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text). +- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text). (note: the upstream task name uses the spelling 'trancribing', don't fix it) - **audio_understanding_sft**: Understand and analyze audio content with text queries. - **audio_understanding_sft_with_thinking**: Audio understanding with reasoning chain. - **spoken_dialogue_sft_multiturn**: Multi-turn spoken dialogue with audio input/output. diff --git a/docs/user_guide/examples/offline_inference/qwen3_tts.md b/docs/user_guide/examples/offline_inference/qwen3_tts.md index 15a5fac23..19fea4132 100644 --- a/docs/user_guide/examples/offline_inference/qwen3_tts.md +++ b/docs/user_guide/examples/offline_inference/qwen3_tts.md @@ -11,7 +11,7 @@ Qwen3 TTS provides multiple task variants for speech generation: - **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction. - **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice. -- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. +- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL. ## Setup Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. diff --git a/docs/user_guide/examples/online_serving/qwen3_tts.md b/docs/user_guide/examples/online_serving/qwen3_tts.md index 65023258e..ee293131d 100644 --- a/docs/user_guide/examples/online_serving/qwen3_tts.md +++ b/docs/user_guide/examples/online_serving/qwen3_tts.md @@ -21,10 +21,10 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/ Two interactive Gradio demos are available, both supporting all 3 task types: -| Demo | File | Transport | Streaming Quality | -| ---- | ---- | --------- | ----------------- | -| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | -| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | +| Demo | File | Transport | Streaming Quality | +| -------- | ------------------------ | ------------ | -------------------------------------------------- | +| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | +| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | ```bash # Option 1: Launch server + Standard Gradio together @@ -137,8 +137,8 @@ The Python client supports the following command-line arguments: - `--speaker`: Speaker name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc. - `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian` - `--instructions`: Voice style/emotion instructions -- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task) -- `--ref-text`: Reference audio transcript for voice cloning (Base task) +- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server. +- `--ref-text`: Reference audio transcript for voice cloning (Base task). - `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus` - `--output` (or `-o`): Output audio file path (default: `tts_output.wav`) @@ -284,7 +284,7 @@ This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/ "task_type": "CustomVoice", "language": "Auto", "instructions": "Optional style instructions", - "ref_audio": "URL or base64 for voice cloning", + "ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning", "ref_text": "Reference audio transcript", "x_vector_only_mode": false, "max_new_tokens": 2048 @@ -317,24 +317,24 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w ### vLLM-Omni Extension Parameters -| Parameter | Type | Default | Description | -| ---------------- | ------ | ------------- | ---------------------------------------------------------------------------- | -| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | -| `language` | string | "Auto" | Language (see supported languages below) | -| `instructions` | string | "" | Voice style/emotion instructions | -| `max_new_tokens` | int | 2048 | Maximum tokens to generate | -| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. | -| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | +| Parameter | Type | Default | Description | +| ---------------------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------- | +| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | +| `language` | string | "Auto" | Language (see supported languages below) | +| `instructions` | string | "" | Voice style/emotion instructions | +| `max_new_tokens` | int | 2048 | Maximum tokens to generate | +| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. | +| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | **Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian ### Voice Clone Parameters (Base task) -| Parameter | Type | Required | Description | -| -------------------- | ------ | -------- | -------------------------------------------- | -| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) | -| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | -| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | +| Parameter | Type | Required | Description | +| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- | +| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) | +| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | +| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | ## Streaming diff --git a/docs/user_guide/examples/online_serving/text_to_image.md b/docs/user_guide/examples/online_serving/text_to_image.md index be8c7d894..39be6e5f0 100644 --- a/docs/user_guide/examples/online_serving/text_to_image.md +++ b/docs/user_guide/examples/online_serving/text_to_image.md @@ -23,6 +23,29 @@ Or use the startup script: bash run_server.sh ``` +### Start with Parallelism Acceleration + +Enable Tensor Parallelism and VAE Patch Parallelism for faster inference: + +```bash +# With Tensor Parallelism (requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2 + +# With Tensor Parallelism and VAE Patch Parallelism (requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2 --vae-patch-parallel-size 2 --vae-use-tiling + +# With Sequence Parallelism (Ulysses-SP, requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 + +# With Ring-Attention (requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --ring 2 + +# Combined: Ulysses + Ring (requires >= 4 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 --ring 2 +``` + +For more details on parallelism acceleration, see the [Parallelism Acceleration Guide](https://github.com/vllm-project/vllm-omni/tree/main/examples/diffusion/parallelism_acceleration.md). + ## API Calls ### Method 1: Using curl @@ -97,12 +120,12 @@ curl -X POST http://localhost:8091/v1/images/generations \ ### LoRA Parameters -| Parameter | Type | Description | -| ------------- | ------ | ---------------------------------------------------------------- | -| `name` | str | LoRA adapter name (optional, defaults to path stem) | -| `local_path` | str | Server-local path to LoRA adapter folder (PEFT format, required) | -| `scale` | float | LoRA scale factor (default: 1.0) | -| `int_id` | int | LoRA integer ID for caching (optional, derived from path if not provided) | +| Parameter | Type | Description | +| ------------ | ----- | ------------------------------------------------------------------------- | +| `name` | str | LoRA adapter name (optional, defaults to path stem) | +| `local_path` | str | Server-local path to LoRA adapter folder (PEFT format, required) | +| `scale` | float | LoRA scale factor (default: 1.0) | +| `int_id` | int | LoRA integer ID for caching (optional, derived from path if not provided) | ### LoRA Adapter Format @@ -162,16 +185,16 @@ Use `extra_body` to pass generation parameters: ## Generation Parameters (extra_body) -| Parameter | Type | Default | Description | -| ------------------------ | ----- | ------- | ------------------------------ | -| `height` | int | None | Image height in pixels | -| `width` | int | None | Image width in pixels | -| `size` | str | None | Image size (e.g., "1024x1024") | -| `num_inference_steps` | int | 50 | Number of denoising steps | -| `true_cfg_scale` | float | 4.0 | Qwen-Image CFG scale | -| `seed` | int | None | Random seed (reproducible) | -| `negative_prompt` | str | None | Negative prompt | -| `num_outputs_per_prompt` | int | 1 | Number of images to generate | +| Parameter | Type | Default | Description | +| ------------------------ | ----- | ------- | ---------------------------------- | +| `height` | int | None | Image height in pixels | +| `width` | int | None | Image width in pixels | +| `size` | str | None | Image size (e.g., "1024x1024") | +| `num_inference_steps` | int | 50 | Number of denoising steps | +| `true_cfg_scale` | float | 4.0 | Qwen-Image CFG scale | +| `seed` | int | None | Random seed (reproducible) | +| `negative_prompt` | str | None | Negative prompt | +| `num_outputs_per_prompt` | int | 1 | Number of images to generate | | `--cfg-parallel-size`. | int | 1 | Number of GPUs for CFG parallelism | ## Response Format diff --git a/examples/offline_inference/mimo_audio/README.md b/examples/offline_inference/mimo_audio/README.md index dd5f87217..55f7dca0f 100644 --- a/examples/offline_inference/mimo_audio/README.md +++ b/examples/offline_inference/mimo_audio/README.md @@ -10,7 +10,7 @@ MiMo-Audio provides multiple task variants for audio understanding and generatio - **tts_sft_with_instruct**: TTS generation with explicit voice style instructions. - **tts_sft_with_audio**: TTS generation with audio reference for voice cloning. - **tts_sft_with_natural_instruction**: TTS generation from natural language descriptions embedded in text. -- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text). +- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text). (note: the upstream task name uses the spelling 'trancribing', don't fix it) - **audio_understanding_sft**: Understand and analyze audio content with text queries. - **audio_understanding_sft_with_thinking**: Audio understanding with reasoning chain. - **spoken_dialogue_sft_multiturn**: Multi-turn spoken dialogue with audio input/output. diff --git a/examples/offline_inference/qwen3_tts/README.md b/examples/offline_inference/qwen3_tts/README.md index 90a2541bc..bf59dc9ba 100644 --- a/examples/offline_inference/qwen3_tts/README.md +++ b/examples/offline_inference/qwen3_tts/README.md @@ -8,7 +8,7 @@ Qwen3 TTS provides multiple task variants for speech generation: - **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction. - **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice. -- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. +- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL. ## Setup Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. diff --git a/examples/online_serving/qwen3_tts/README.md b/examples/online_serving/qwen3_tts/README.md index 4709c9d42..3718f8e59 100644 --- a/examples/online_serving/qwen3_tts/README.md +++ b/examples/online_serving/qwen3_tts/README.md @@ -18,10 +18,10 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/ Two interactive Gradio demos are available, both supporting all 3 task types: -| Demo | File | Transport | Streaming Quality | -| ---- | ---- | --------- | ----------------- | -| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | -| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | +| Demo | File | Transport | Streaming Quality | +| -------- | ------------------------ | ------------ | -------------------------------------------------- | +| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks | +| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) | ```bash # Option 1: Launch server + Standard Gradio together @@ -134,8 +134,8 @@ The Python client supports the following command-line arguments: - `--voice`: Speaker/voice name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc. - `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian` - `--instructions`: Voice style/emotion instructions -- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task) -- `--ref-text`: Reference audio transcript for voice cloning (Base task) +- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server. +- `--ref-text`: Reference audio transcript for voice cloning (Base task). - `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus` - `--output` (or `-o`): Output audio file path (default: `tts_output.wav`) @@ -281,7 +281,7 @@ This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/ "task_type": "CustomVoice", "language": "Auto", "instructions": "Optional style instructions", - "ref_audio": "URL or base64 for voice cloning", + "ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning", "ref_text": "Reference audio transcript", "x_vector_only_mode": false, "max_new_tokens": 2048 @@ -308,24 +308,24 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w ### vLLM-Omni Extension Parameters -| Parameter | Type | Default | Description | -| ---------------- | ------ | ------------- | ---------------------------------------------------------------------------- | -| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | -| `language` | string | "Auto" | Language (see supported languages below) | -| `instructions` | string | "" | Voice style/emotion instructions | -| `max_new_tokens` | int | 2048 | Maximum tokens to generate | -| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. | -| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | +| Parameter | Type | Default | Description | +| ---------------------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------- | +| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base | +| `language` | string | "Auto" | Language (see supported languages below) | +| `instructions` | string | "" | Voice style/emotion instructions | +| `max_new_tokens` | int | 2048 | Maximum tokens to generate | +| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. | +| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) | **Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian ### Voice Clone Parameters (Base task) -| Parameter | Type | Required | Description | -| -------------------- | ------ | -------- | -------------------------------------------- | -| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) | -| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | -| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | +| Parameter | Type | Required | Description | +| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- | +| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) | +| `ref_text` | string | No | Transcript of reference audio (for ICL mode) | +| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) | ## Streaming diff --git a/examples/online_serving/text_to_image/README.md b/examples/online_serving/text_to_image/README.md index 140036d00..52091045f 100644 --- a/examples/online_serving/text_to_image/README.md +++ b/examples/online_serving/text_to_image/README.md @@ -20,6 +20,29 @@ Or use the startup script: bash run_server.sh ``` +### Start with Parallelism Acceleration + +Enable Tensor Parallelism and VAE Patch Parallelism for faster inference: + +```bash +# With Tensor Parallelism (requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2 + +# With Tensor Parallelism and VAE Patch Parallelism (requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2 --vae-patch-parallel-size 2 --vae-use-tiling + +# With Sequence Parallelism (Ulysses-SP, requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 + +# With Ring-Attention (requires >= 2 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --ring 2 + +# Combined: Ulysses + Ring (requires >= 4 GPUs) +vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 --ring 2 +``` + +For more details on parallelism acceleration, see the [Parallelism Acceleration Guide](../../diffusion/parallelism_acceleration.md). + ## API Calls ### Method 1: Using curl @@ -94,12 +117,12 @@ curl -X POST http://localhost:8091/v1/images/generations \ ### LoRA Parameters -| Parameter | Type | Description | -| ------------- | ------ | ---------------------------------------------------------------- | -| `name` | str | LoRA adapter name (optional, defaults to path stem) | -| `local_path` | str | Server-local path to LoRA adapter folder (PEFT format, required) | -| `scale` | float | LoRA scale factor (default: 1.0) | -| `int_id` | int | LoRA integer ID for caching (optional, derived from path if not provided) | +| Parameter | Type | Description | +| ------------ | ----- | ------------------------------------------------------------------------- | +| `name` | str | LoRA adapter name (optional, defaults to path stem) | +| `local_path` | str | Server-local path to LoRA adapter folder (PEFT format, required) | +| `scale` | float | LoRA scale factor (default: 1.0) | +| `int_id` | int | LoRA integer ID for caching (optional, derived from path if not provided) | ### LoRA Adapter Format @@ -159,16 +182,16 @@ Use `extra_body` to pass generation parameters: ## Generation Parameters (extra_body) -| Parameter | Type | Default | Description | -| ------------------------ | ----- | ------- | ------------------------------ | -| `height` | int | None | Image height in pixels | -| `width` | int | None | Image width in pixels | -| `size` | str | None | Image size (e.g., "1024x1024") | -| `num_inference_steps` | int | 50 | Number of denoising steps | -| `true_cfg_scale` | float | 4.0 | Qwen-Image CFG scale | -| `seed` | int | None | Random seed (reproducible) | -| `negative_prompt` | str | None | Negative prompt | -| `num_outputs_per_prompt` | int | 1 | Number of images to generate | +| Parameter | Type | Default | Description | +| ------------------------ | ----- | ------- | ---------------------------------- | +| `height` | int | None | Image height in pixels | +| `width` | int | None | Image width in pixels | +| `size` | str | None | Image size (e.g., "1024x1024") | +| `num_inference_steps` | int | 50 | Number of denoising steps | +| `true_cfg_scale` | float | 4.0 | Qwen-Image CFG scale | +| `seed` | int | None | Random seed (reproducible) | +| `negative_prompt` | str | None | Negative prompt | +| `num_outputs_per_prompt` | int | 1 | Number of images to generate | | `--cfg-parallel-size`. | int | 1 | Number of GPUs for CFG parallelism | ## Response Format