5014fea4a0
Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com> Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
90 lines
3.2 KiB
Markdown
90 lines
3.2 KiB
Markdown
# Audex (Nemotron-Labs-Audex-2B) online serving
|
|
|
|
One checkpoint, four deployment modes. `run_server.sh` starts the server
|
|
for a given `MODE`; `client.py --mode <mode>` tests it. The capability
|
|
matrix:
|
|
|
|
| mode (`vllm_omni/deploy/audex_<mode>.yaml`) | audio in | text out | speech out | general audio out | endpoint |
|
|
|---|---|---|---|---|---|
|
|
| `tts` | ❌ | ❌ | ✅ | ❌ | `/v1/audio/speech` |
|
|
| `tta` | ❌ | ❌ | ❌ | ✅ | `/v1/audio/speech` |
|
|
| `thinker_only` | ✅ | ✅ | ❌ | ❌ | `/v1/chat/completions` |
|
|
| `s2s` | ✅ | ✅ | ✅ | ❌ | both |
|
|
|
|
Audio-input modes of `client.py` fall back to vLLM's public
|
|
`mary_had_lamb` asset when `--audio-file` is omitted; CFG defaults follow
|
|
the official settings per mode (tts/s2s 1.5, tta 3.0; `--cfg-scale 1.0`
|
|
disables).
|
|
|
|
## tts — text → speech
|
|
|
|
Text-only thinker (`checkpoint_folder_audiogen`) emits `<speechcodec_N>`
|
|
tokens; the streaming causal decoder produces 16 kHz WAVs. This is also
|
|
the DEFAULT pipeline when the repo root is served without a deploy config.
|
|
|
|
```bash
|
|
./run_server.sh # MODE=tts, port 8097
|
|
|
|
python client.py --mode tts --text "Hello there." --output hello.wav
|
|
# or raw curl:
|
|
curl -s http://localhost:8097/v1/audio/speech \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"model": "nvidia/Nemotron-Labs-Audex-2B", "input": "Hello there.", "response_format": "wav", "extra_params": {"cfg_scale": 1.5}}' \
|
|
-o hello.wav
|
|
```
|
|
|
|
## tta — caption → general audio
|
|
|
|
Same thinker over the 4-codebook `<audiocodec_N>` RVQ block, decoded by
|
|
the external XCodec1 checkpoint (auto-downloaded; override with
|
|
`XCODEC1_PATH`). Clips are capped at 10 s.
|
|
|
|
```bash
|
|
MODE=tta ./run_server.sh
|
|
|
|
python client.py --mode tta --caption "Heavy rain falling on a tin roof." \
|
|
--output rain.wav
|
|
```
|
|
|
|
## thinker_only — audio (+ instruction) → text
|
|
|
|
Single-stage audio understanding on the full checkpoint (NV-Whisper
|
|
encoder + projector + LM). Send the clip as `input_audio` chat content.
|
|
|
|
```bash
|
|
MODE=thinker_only ./run_server.sh
|
|
|
|
python client.py --mode thinker_only # transcribe the demo asset
|
|
python client.py --mode thinker_only --audio-file a.wav \
|
|
--question "What language is being spoken?"
|
|
```
|
|
|
|
## s2s — spoken question → spoken answer
|
|
|
|
The audio-capable thinker plus the speech decoder in one deployment.
|
|
`client.py --mode s2s` runs the official three passes: ASR (chat with
|
|
`input_audio`) → chat on the transcript → TTS on the answer. Text-final
|
|
passes carry `"modalities": ["text"]`; only the TTS pass streams through
|
|
the decoder.
|
|
|
|
```bash
|
|
MODE=s2s PORT=8098 ./run_server.sh
|
|
|
|
python client.py --mode s2s --port 8098 \
|
|
--audio-file question.wav --output answer.wav
|
|
```
|
|
|
|
Offline counterparts (no HTTP, one `Omni` engine per script) live in
|
|
`examples/offline_inference/audex/`.
|
|
|
|
## 30B-A3B (nvidia/Nemotron-Labs-Audex-30B-A3B)
|
|
|
|
`SIZE=30b` switches run_server.sh to the 30B yamls and model id:
|
|
|
|
SIZE=30b ./run_server.sh # 30B tts on port 8097
|
|
SIZE=30b MODE=s2s PORT=8098 ./run_server.sh
|
|
|
|
Pass `--model nvidia/Nemotron-Labs-Audex-30B-A3B` to the client — the server
|
|
validates the model id and 404s on a mismatch. Single-H100 defaults; see the
|
|
yaml comments for the TP2 fallback. First launch downloads ~60 GB.
|