[Doc] Sync and fix. (#2110)

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
2026-03-24 03:20:50 +00:00
1 parent 9bf8f8ce79
commit b0efb6ede0
@@ -0,0 +1,226 @@
+# vLLM-Omni Helm Chart
+
+Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/chart-helm>.
+
+
+Helm chart for deploying [vLLM-Omni](https://github.com/vllm-project/vllm-omni) on Kubernetes. vLLM-Omni extends vLLM with omni-modality model serving, supporting text-to-image, multimodal chat, text-to-speech, and more.
+
+## Prerequisites
+
+- Kubernetes 1.24+
+- Helm 3.x
+- NVIDIA GPU nodes with [NVIDIA Device Plugin](https://github.com/NVIDIA/k8s-device-plugin)
+
+## Quick Start
+
+```bash
+helm install my-release ./chart-helm \
+  --set model=Tongyi-MAI/Z-Image-Turbo
+```
+
+## Configuration
+
+### Model Selection
+
+Set the `model` value to any supported HuggingFace model ID:
+
+| Model | Type | GPUs | Notes |
+|-------|------|------|-------|
+| `Tongyi-MAI/Z-Image-Turbo` | text-to-image | 1 | Small, fast (default) |
+| `stabilityai/stable-diffusion-3.5-medium` | text-to-image | 1 | ~6GB VRAM |
+| `Qwen/Qwen-Image` | text-to-image | 1 | Large, ~40GB+ VRAM |
+| `Qwen/Qwen2.5-Omni-7B` | multimodal | 2 | Text + audio + image + video |
+| `Qwen/Qwen3-Omni-7B-Chat` | multimodal | 2 | Latest omni model |
+| `Qwen/Qwen3-TTS` | text-to-speech | 1 | TTS |
+
+### HuggingFace Token
+
+For gated models that require authentication:
+
+```bash
+helm install my-release ./chart-helm \
+  --set model=Qwen/Qwen2.5-Omni-7B \
+  --set hfToken=hf_xxxxx \
+  --set resources.requests."nvidia\.com/gpu"=2 \
+  --set resources.limits."nvidia\.com/gpu"=2
+```
+
+### Omni-Specific Flags
+
+Enable VAE memory optimizations for diffusion models:
+
+```bash
+helm install my-release ./chart-helm \
+  --set model=Qwen/Qwen-Image \
+  --set omniArgs.vaeUseSlicing=true \
+  --set omniArgs.vaeUseTiling=true
+```
+
+Enable CPU offloading:
+
+```bash
+helm install my-release ./chart-helm \
+  --set model=Qwen/Qwen-Image \
+  --set omniArgs.enableCpuOffload=true
+```
+
+Pass additional raw CLI flags:
+
+```bash
+helm install my-release ./chart-helm \
+  --set model=Qwen/Qwen-Image \
+  --set omniArgs.extraArgs[0]="--enable-layerwise-offload"
+```
+
+### Model Cache
+
+By default, a PersistentVolumeClaim is created for the HuggingFace model cache to avoid re-downloading models on pod restarts:
+
+```yaml
+modelCache:
+  enabled: true
+  storageSize: "50Gi"
+  storageClassName: ""
+```
+
+To use an ephemeral volume instead:
+
+```bash
+helm install my-release ./chart-helm \
+  --set modelCache.enabled=false
+```
+
+### Custom Command Override
+
+To fully override the container command:
+
+```bash
+helm install my-release ./chart-helm \
+  --set image.command[0]=vllm \
+  --set image.command[1]=serve \
+  --set image.command[2]=my-model \
+  --set image.command[3]=--omni \
+  --set image.command[4]=--host \
+  --set image.command[5]=0.0.0.0
+```
+
+## API Endpoints
+
+Once deployed, vLLM-Omni exposes the following OpenAI-compatible endpoints:
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | Health check |
+| `/v1/models` | GET | List available models |
+| `/v1/chat/completions` | POST | Chat completions (text/multimodal) |
+| `/v1/images/generations` | POST | Image generation |
+| `/v1/images/edits` | POST | Image editing |
+| `/v1/audio/speech` | POST | Text-to-speech |
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `Chart.yaml` | Chart metadata (name, version, maintainers) |
+| `values.yaml` | Default configuration values |
+| `values.schema.json` | JSON schema for validating values |
+| `templates/_helpers.tpl` | Helper templates for common configurations |
+| `templates/deployment.yaml` | Kubernetes Deployment |
+| `templates/service.yaml` | Kubernetes Service (ClusterIP) |
+| `templates/secrets.yaml` | Secrets (generic + HuggingFace token) |
+| `templates/pvc.yaml` | PersistentVolumeClaim for model cache |
+| `templates/configmap.yaml` | Optional ConfigMap |
+| `templates/hpa.yaml` | HorizontalPodAutoscaler |
+| `templates/poddisruptionbudget.yaml` | PodDisruptionBudget |
+| `templates/custom-objects.yaml` | Custom Kubernetes objects |
+
+## Running Tests
+
+This chart includes unit tests using [helm-unittest](https://github.com/helm-unittest/helm-unittest). Install the plugin and run tests:
+
+```bash
+# Install plugin
+helm plugin install https://github.com/helm-unittest/helm-unittest
+
+# Run tests
+helm unittest .
+```
+
+## Example materials
+
+??? abstract ".helmignore"
+    ``````
+    --8<-- "examples/online_serving/chart-helm/.helmignore"
+    ``````
+??? abstract "Chart.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/Chart.yaml"
+    ``````
+??? abstract "ct.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/ct.yaml"
+    ``````
+??? abstract "lintconf.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/lintconf.yaml"
+    ``````
+??? abstract "templates/_helpers.tpl"
+    ``````tpl
+    --8<-- "examples/online_serving/chart-helm/templates/_helpers.tpl"
+    ``````
+??? abstract "templates/configmap.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/configmap.yaml"
+    ``````
+??? abstract "templates/custom-objects.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/custom-objects.yaml"
+    ``````
+??? abstract "templates/deployment.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/deployment.yaml"
+    ``````
+??? abstract "templates/hpa.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/hpa.yaml"
+    ``````
+??? abstract "templates/ingress.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/ingress.yaml"
+    ``````
+??? abstract "templates/poddisruptionbudget.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/poddisruptionbudget.yaml"
+    ``````
+??? abstract "templates/pvc.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/pvc.yaml"
+    ``````
+??? abstract "templates/secrets.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/secrets.yaml"
+    ``````
+??? abstract "templates/service.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/templates/service.yaml"
+    ``````
+??? abstract "tests/deployment_test.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/tests/deployment_test.yaml"
+    ``````
+??? abstract "tests/ingress_test.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/tests/ingress_test.yaml"
+    ``````
+??? abstract "tests/pvc_test.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/tests/pvc_test.yaml"
+    ``````
+??? abstract "tests/secrets_test.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/tests/secrets_test.yaml"
+    ``````
+??? abstract "values.yaml"
+    ``````yaml
+    --8<-- "examples/online_serving/chart-helm/values.yaml"
+    ``````
@@ -144,7 +144,7 @@ curl -X POST http://localhost:8091/v1/videos \

 ## Create Response Format

-`POST /v1/videos` returns a job record.
+`POST /v1/videos` returns a job record, not inline base64 video data.

 ```json
 {
@@ -201,6 +201,10 @@ done

 ## Example materials

+??? abstract "run_curl_hunyuan_video_15.sh"
+    ``````sh
+    --8<-- "examples/online_serving/image_to_video/run_curl_hunyuan_video_15.sh"
+    ``````
 ??? abstract "run_curl_image_to_video.sh"
    ``````sh
    --8<-- "examples/online_serving/image_to_video/run_curl_image_to_video.sh"
@@ -209,3 +213,7 @@ done
    ``````sh
    --8<-- "examples/online_serving/image_to_video/run_server.sh"
    ``````
+??? abstract "run_server_hunyuan_video_15.sh"
+    ``````sh
+    --8<-- "examples/online_serving/image_to_video/run_server_hunyuan_video_15.sh"
+    ``````
@@ -0,0 +1,72 @@
+# Online serving Example of vLLM-Omni for MiMo-Audio
+
+Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/mimo_audio>.
+
+
+## 🛠️ Installation
+
+Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
+
+## Run examples (MiMo-Audio)
+
+### Launch the Server
+```bash
+export MIMO_AUDIO_TOKENIZER_PATH="XiaomiMiMo/MiMo-Audio-Tokenizer"
+
+vllm-omni serve XiaomiMiMo/MiMo-Audio-7B-Instruct --omni \
+--served-model-name "MiMo-Audio-7B-Instruct"  \
+--port 18091 --stage-configs-path ./vllm_omni/model_executor/stage_configs/mimo_audio.yaml \
+--chat-template ./examples/online_serving/mimo_audio/chat_template.jinja
+```
+> ⚠️ **Important**  
+> **MiMo-Audio is not compatible with the default chat template.**  
+> The provided `chat_template.jinja` implements MiMo-specific role, audio token, and instruction formatting and **must be used for all inference**.
+
+
+### Send Multi-modal Request
+
+Get into the example folder
+```bash
+cd examples/online_serving/mimo_audio
+```
+
+####  Send request via python
+
+```bash
+# Audio dialogue task
+python openai_chat_completion_client_for_multimodal_generation.py \
+--query-type multi_audios \
+--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
+```
+
+The Python client supports the following command-line arguments:
+
+- `--query-type` (or `-q`): Query type (default: `multi_audios`)
+  - Options: `multi_audios`, `text`
+- `--message-json` (or `-m`): Path to `base64` multi rounds audio messages json file
+  - Do not pass any value for "text" query type
+  - Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs, only for "Are these two audio clips the same?" task
+  - Example: `---message-json ./examples/offline_inference/mimo_audio/message_base64_wav.json`
+- `--prompt` (or `-p`): Custom text prompt/question, only for query type is "text"(TTS task)
+  - Attention! Do not pass any value for "multi_audios" query type
+  - Example: `--prompt "What are the main activities shown in this video?"`
+
+
+For example, to use multi rounds audios with local files:
+
+```bash
+python openai_chat_completion_client_for_multimodal_generation.py \
+--query-type multi_audios \
+--message-json ../../offline_inference/mimo_audio/message_base64_wav.json
+```
+
+## Example materials
+
+??? abstract "chat_template.jinja"
+    ``````jinja
+    --8<-- "examples/online_serving/mimo_audio/chat_template.jinja"
+    ``````
+??? abstract "openai_chat_completion_client_for_multimodal_generation.py"
+    ``````py
+    --8<-- "examples/online_serving/mimo_audio/openai_chat_completion_client_for_multimodal_generation.py"
+    ``````
@@ -352,6 +352,42 @@ curl -X POST http://localhost:8091/v1/audio/speech \
 - `speed` adjustment is not supported when streaming.
 - Requires the server stage config to have `async_chunk: true` (default in `qwen3_tts.yaml`).

+## Streaming Text Input (WebSocket)
+
+The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally, buffers and splits it at sentence boundaries, and generates audio per sentence.
+
+When `stream_audio=true`, each sentence is emitted as `audio.start`, one or more binary PCM frames, and `audio.done`.
+
+### Quick Start
+
+```bash
+python streaming_speech_client.py \
+    --text "Hello world. How are you? I am fine."
+
+python streaming_speech_client.py \
+    --text "Hello world. How are you? I am fine." \
+    --simulate-stt --stt-delay 0.1
+```
+
+### WebSocket Protocol
+
+Client -> Server:
+
+```jsonc
+{"type": "session.config", "voice": "Vivian", "task_type": "CustomVoice", "language": "Auto", "split_granularity": "sentence", "stream_audio": true, "response_format": "pcm"}
+{"type": "input.text", "text": "Hello, how are you? "}
+{"type": "input.done"}
+```
+
+Server -> Client:
+
+```jsonc
+{"type": "audio.start", "sentence_index": 0, "sentence_text": "Hello, how are you?", "format": "pcm", "sample_rate": 24000}
+// binary PCM frame(s)
+{"type": "audio.done", "sentence_index": 0, "total_bytes": 96000, "error": false}
+{"type": "session.done", "total_sentences": 1}
+```
+
 ## Limitations

 - **Single request**: Batch processing is not yet optimized for online serving.
@@ -386,6 +422,10 @@ curl -X POST http://localhost:8091/v1/audio/speech \
    ``````sh
    --8<-- "examples/online_serving/qwen3_tts/run_server.sh"
    ``````
+??? abstract "streaming_speech_client.py"
+    ``````py
+    --8<-- "examples/online_serving/qwen3_tts/streaming_speech_client.py"
+    ``````
 ??? abstract "tts_common.py"
    ``````py
    --8<-- "examples/online_serving/qwen3_tts/tts_common.py"
@@ -23,29 +23,6 @@ Or use the startup script:
 bash run_server.sh
 ```

-### Start with Parallelism Acceleration
-
-Enable Tensor Parallelism and VAE Patch Parallelism for faster inference:
-
-```bash
-# With Tensor Parallelism (requires >= 2 GPUs)
-vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2
-
-# With Tensor Parallelism and VAE Patch Parallelism (requires >= 2 GPUs)
-vllm serve Qwen/Qwen-Image --omni --port 8091 --tensor-parallel-size 2 --vae-patch-parallel-size 2 --vae-use-tiling
-
-# With Sequence Parallelism (Ulysses-SP, requires >= 2 GPUs)
-vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2
-
-# With Ring-Attention (requires >= 2 GPUs)
-vllm serve Qwen/Qwen-Image --omni --port 8091 --ring 2
-
-# Combined: Ulysses + Ring (requires >= 4 GPUs)
-vllm serve Qwen/Qwen-Image --omni --port 8091 --usp 2 --ring 2
-```
-
-For more details on parallelism acceleration, see the [Parallelism Acceleration Guide](../../diffusion/parallelism_acceleration.md).
-
 ## API Calls

 ### Method 1: Using curl
@@ -46,6 +46,7 @@ The main endpoints are:
 ## Storage

 Generated video files are stored on local disk by the async video API.
+Local file storage behavior can be controlled via the following environment variables:

 - `VLLM_OMNI_STORAGE_PATH`: directory used for generated files (default: `/tmp/storage`)
 - `VLLM_OMNI_STORAGE_MAX_CONCURRENCY`: max concurrent save/delete operations (default: `4`)
@@ -147,7 +148,7 @@ curl -X POST http://localhost:8091/v1/videos \

 ## Create Response Format

-`POST /v1/videos` returns a job record.
+`POST /v1/videos` returns a job record, not inline base64 video data.

 ```json
 {
@@ -204,6 +205,10 @@ done

 ## Example materials

+??? abstract "run_curl_hunyuan_video_15.sh"
+    ``````sh
+    --8<-- "examples/online_serving/text_to_video/run_curl_hunyuan_video_15.sh"
+    ``````
 ??? abstract "run_curl_text_to_video.sh"
    ``````sh
    --8<-- "examples/online_serving/text_to_video/run_curl_text_to_video.sh"
@@ -212,3 +217,7 @@ done
    ``````sh
    --8<-- "examples/online_serving/text_to_video/run_server.sh"
    ``````
+??? abstract "run_server_hunyuan_video_15.sh"
+    ``````sh
+    --8<-- "examples/online_serving/text_to_video/run_server_hunyuan_video_15.sh"
+    ``````