[Docs] Unify recipe serve commands on vllm serve --omni (#6221)
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
This commit is contained in:
1 parent
68bf8c2c42
commit
9798bc3a8d
6 files changed
+11
-8
No files matched your search
@@ -45,7 +45,7 @@ MusicGen-style delay pattern).
|
||||
**Online serving:**
|
||||
|
||||
```bash
|
||||
vllm-omni serve bosonai/higgs-audio-v3-tts-4b \
|
||||
vllm serve bosonai/higgs-audio-v3-tts-4b \
|
||||
--host 0.0.0.0 --port 8095 \
|
||||
--trust-remote-code --omni
|
||||
```
|
||||
|
||||
@@ -51,7 +51,7 @@ NemotronH (~3B active parameters). Pick the pipeline by task:
|
||||
`model_type` auto-resolves `audex_tts.yaml`):
|
||||
|
||||
```bash
|
||||
vllm-omni serve nvidia/Nemotron-Labs-Audex-2B \
|
||||
vllm serve nvidia/Nemotron-Labs-Audex-2B \
|
||||
--host 0.0.0.0 --port 8097 \
|
||||
--trust-remote-code --omni
|
||||
# other modes: add --stage-configs-path vllm_omni/deploy/audex_{tta,thinker_only,s2s}.yaml
|
||||
@@ -109,7 +109,7 @@ The 30B REQUIRES its explicit deploy yaml (the shared HF `model_type`
|
||||
auto-resolves to the 2B-tuned config otherwise):
|
||||
|
||||
```bash
|
||||
vllm-omni serve nvidia/Nemotron-Labs-Audex-30B-A3B \
|
||||
vllm serve nvidia/Nemotron-Labs-Audex-30B-A3B \
|
||||
--host 0.0.0.0 --port 8097 \
|
||||
--trust-remote-code --omni \
|
||||
--stage-configs-path vllm_omni/deploy/audex_tts_30b.yaml
|
||||
|
||||
@@ -78,7 +78,7 @@ python examples/offline_inference/text_to_audio/text_to_audio.py \
|
||||
Start the online serving endpoint:
|
||||
|
||||
```bash
|
||||
vllm-omni serve /path/to/stable-audio-open-1.0 \
|
||||
vllm serve /path/to/stable-audio-open-1.0 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8091 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
|
||||
@@ -39,6 +39,9 @@ platform section, document one or more tested hardware configurations.
|
||||
|
||||
#### Command
|
||||
|
||||
Serve commands use `vllm serve <model> --omni`. The `--omni` flag is what
|
||||
selects the Omni pipeline, so it is required and cannot be dropped.
|
||||
|
||||
```bash
|
||||
# Add the exact command(s) here
|
||||
```
|
||||
|
||||
@@ -39,7 +39,7 @@ Other hardware is welcome as community validation lands.
|
||||
Launch the two-stage talker:
|
||||
|
||||
```bash
|
||||
vllm-omni serve inclusionAI/Ming-omni-tts-0.5B \
|
||||
vllm serve inclusionAI/Ming-omni-tts-0.5B \
|
||||
--deploy-config vllm_omni/deploy/ming_tts.yaml \
|
||||
--omni \
|
||||
--port 8091 \
|
||||
|
||||
@@ -44,20 +44,20 @@ conditioned on reference audio and its transcript.
|
||||
Start the server from the repository root:
|
||||
|
||||
```bash
|
||||
vllm-omni serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091
|
||||
vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091
|
||||
```
|
||||
|
||||
Async chunking is enabled by default in the bundled deployment config. For
|
||||
the sync (non-streaming) path:
|
||||
|
||||
```bash
|
||||
vllm-omni serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091 --no-async-chunk
|
||||
vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091 --no-async-chunk
|
||||
```
|
||||
|
||||
Use a custom deploy config for advanced cases:
|
||||
|
||||
```bash
|
||||
vllm-omni serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091 \
|
||||
vllm serve zai-org/GLM-TTS --omni --trust-remote-code --port 8091 \
|
||||
--deploy-config /path/to/your_glm_tts_overrides.yaml
|
||||
```
|
||||
|
||||
|
||||
Reference in new issue
Block a user