anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[skip ci] Keep the latest version. (#2112)

Browse Source 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
This commit is contained in:
Alicia authored and GitHub committed 2026-03-24 14:26:20 +08:00
1 parent 3cdf28828e
commit b2b061a9b3
8 files changed
+122 -76
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/user_guide/examples/offline_inference/mimo_audio.md
+1 -1
View File
@@ -13,7 +13,7 @@ MiMo-Audio provides multiple task variants for audio understanding and generatio
- **tts_sft_with_instruct**: TTS generation with explicit voice style instructions.
- **tts_sft_with_audio**: TTS generation with audio reference for voice cloning.
- **tts_sft_with_natural_instruction**: TTS generation from natural language descriptions embedded in text.
- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text).
- **audio_trancribing_sft**: Transcribe audio to text (speech-to-text). (note: the upstream task name uses the spelling 'trancribing', don't fix it)
- **audio_understanding_sft**: Understand and analyze audio content with text queries.
- **audio_understanding_sft_with_thinking**: Audio understanding with reasoning chain.
- **spoken_dialogue_sft_multiturn**: Multi-turn spoken dialogue with audio input/output.
docs/user_guide/examples/offline_inference/qwen3_tts.md
+1 -1
View File
@@ -11,7 +11,7 @@ Qwen3 TTS provides multiple task variants for speech generation:
- **CustomVoice**: Generate speech with a known speaker identity (speaker ID) and optional instruction.
- **VoiceDesign**: Generate speech from text plus a descriptive instruction that designs a new voice.
- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection.
- **Base**: Voice cloning using a reference audio + reference transcript, with optional mode selection. The `ref_audio` field accepts a local file path, HTTP URL, or base64 data URL.
## Setup
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
docs/user_guide/examples/online_serving/qwen3_tts.md
+20 -20
View File
@@ -21,10 +21,10 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
Two interactive Gradio demos are available, both supporting all 3 task types:
| Demo | File | Transport | Streaming Quality |
| ---- | ---- | --------- | ----------------- |
| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
| Demo | File | Transport | Streaming Quality |
| -------- | ------------------------ | ------------ | -------------------------------------------------- |
| Standard | `gradio_demo.py` | HTTP chunked | May have small gaps between chunks |
| FastRTC | `gradio_fastrtc_demo.py` | WebRTC | Gapless streaming (requires `pip install fastrtc`) |
```bash
# Option 1: Launch server + Standard Gradio together
@@ -137,8 +137,8 @@ The Python client supports the following command-line arguments:
- `--speaker`: Speaker name (default: `vivian`). Options: `vivian`, `ryan`, `aiden`, etc.
- `--language`: Language. Options: `Auto`, `Chinese`, `English`, `Japanese`, `Korean`, `German`, `French`, `Russian`, `Portuguese`, `Spanish`, `Italian`
- `--instructions`: Voice style/emotion instructions
- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task)
- `--ref-text`: Reference audio transcript for voice cloning (Base task)
- `--ref-audio`: Reference audio file path or URL for voice cloning (Base task). Local paths are automatically base64-encoded by the client before sending to the server.
- `--ref-text`: Reference audio transcript for voice cloning (Base task).
- `--response-format`: Audio output format (default: `wav`). Options: `wav`, `mp3`, `flac`, `pcm`, `aac`, `opus`
- `--output` (or `-o`): Output audio file path (default: `tts_output.wav`)
@@ -284,7 +284,7 @@ This endpoint follows the [OpenAI Audio Speech API](https://platform.openai.com/
"task_type": "CustomVoice",
"language": "Auto",
"instructions": "Optional style instructions",
"ref_audio": "URL or base64 for voice cloning",
"ref_audio": "HTTP URL, base64 data URL, or file:// URI for voice cloning",
"ref_text": "Reference audio transcript",
"x_vector_only_mode": false,
"max_new_tokens": 2048
@@ -317,24 +317,24 @@ Returns binary audio data with appropriate `Content-Type` header (e.g., `audio/w
### vLLM-Omni Extension Parameters
| Parameter | Type | Default | Description |
| ---------------- | ------ | ------------- | ---------------------------------------------------------------------------- |
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
| `language` | string | "Auto" | Language (see supported languages below) |
| `instructions` | string | "" | Voice style/emotion instructions |
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
| Parameter | Type | Default | Description |
| ---------------------------- | ------ | ------------- | -------------------------------------------------------------------------------------------------------------------- |
| `task_type` | string | "CustomVoice" | Task: CustomVoice, VoiceDesign, or Base |
| `language` | string | "Auto" | Language (see supported languages below) |
| `instructions` | string | "" | Voice style/emotion instructions |
| `max_new_tokens` | int | 2048 | Maximum tokens to generate |
| `initial_codec_chunk_frames` | int | null | Per-request initial chunk size override for TTFA tuning. When null, IC is computed dynamically based on server load. |
| `stream` | bool | false | Stream raw PCM chunks as they are decoded (requires `response_format="pcm"`) |
**Supported languages:** Auto, Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian
### Voice Clone Parameters (Base task)
| Parameter | Type | Required | Description |
| -------------------- | ------ | -------- | -------------------------------------------- |
| `ref_audio` | string | **Yes** | Reference audio (URL or base64 data URL) |
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) |
| Parameter | Type | Required | Description |
| -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------- |
| `ref_audio` | string | **Yes** | Reference audio (HTTP URL, base64 data URL, or `file://` URI with `--allowed-local-media-path`) |
| `ref_text` | string | No | Transcript of reference audio (for ICL mode) |
| `x_vector_only_mode` | bool | No | Use speaker embedding only (no ICL) |
## Streaming
docs/user_guide/examples/online_serving/text_to_image.md
+39 -16
View File
@@ -23,6 +23,29 @@ Or use the startup script:
bash run_server.sh
```
### Start with Parallelism Acceleration
Enable Tensor Parallelism and VAE Patch Parallelism for faster inference:
```bash
# With Tensor Parallelism (requires >= 2 GPUs)
vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2
# With Tensor Parallelism and VAE Patch Parallelism (requires >= 2 GPUs)
vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2 --vae-patch-parallel-size 2 --vae-use-tiling
# With Sequence Parallelism (Ulysses-SP, requires >= 2 GPUs)
vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2
# With Ring-Attention (requires >= 2 GPUs)
vllm serve Qwen/Qwen-Image --omni --port 8091 --ring 2
# Combined: Ulysses + Ring (requires >= 4 GPUs)
vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 --ring 2
```
For more details on parallelism acceleration, see the [Parallelism Acceleration Guide](https://github.com/vllm-project/vllm-omni/tree/main/examples/diffusion/parallelism_acceleration.md).
## API Calls
### Method 1: Using curl
@@ -97,12 +120,12 @@ curl -X POST http://localhost:8091/v1/images/generations \
### LoRA Parameters
| Parameter | Type | Description |
| ------------- | ------ | ---------------------------------------------------------------- |
| `name` | str | LoRA adapter name (optional, defaults to path stem) |
| `local_path` | str | Server-local path to LoRA adapter folder (PEFT format, required) |
| `scale` | float | LoRA scale factor (default: 1.0) |
| `int_id` | int | LoRA integer ID for caching (optional, derived from path if not provided) |
| Parameter | Type | Description |
| ------------ | ----- | ------------------------------------------------------------------------- |
| `name` | str | LoRA adapter name (optional, defaults to path stem) |
| `local_path` | str | Server-local path to LoRA adapter folder (PEFT format, required) |
| `scale` | float | LoRA scale factor (default: 1.0) |
| `int_id` | int | LoRA integer ID for caching (optional, derived from path if not provided) |
### LoRA Adapter Format
@@ -162,16 +185,16 @@ Use `extra_body` to pass generation parameters:
## Generation Parameters (extra_body)
| Parameter | Type | Default | Description |
| ------------------------ | ----- | ------- | ------------------------------ |
| `height` | int | None | Image height in pixels |
| `width` | int | None | Image width in pixels |
| `size` | str | None | Image size (e.g., "1024x1024") |
| `num_inference_steps` | int | 50 | Number of denoising steps |
| `true_cfg_scale` | float | 4.0 | Qwen-Image CFG scale |
| `seed` | int | None | Random seed (reproducible) |
| `negative_prompt` | str | None | Negative prompt |
| `num_outputs_per_prompt` | int | 1 | Number of images to generate |
| Parameter | Type | Default | Description |
| ------------------------ | ----- | ------- | ---------------------------------- |
| `height` | int | None | Image height in pixels |
| `width` | int | None | Image width in pixels |
| `size` | str | None | Image size (e.g., "1024x1024") |
| `num_inference_steps` | int | 50 | Number of denoising steps |
| `true_cfg_scale` | float | 4.0 | Qwen-Image CFG scale |
| `seed` | int | None | Random seed (reproducible) |
| `negative_prompt` | str | None | Negative prompt |
| `num_outputs_per_prompt` | int | 1 | Number of images to generate |
| `--cfg-parallel-size`. | int | 1 | Number of GPUs for CFG parallelism |
## Response Format