vllm-omni/examples/online_serving/audex/README.md

Audex (Nemotron-Labs-Audex-2B) online serving

One checkpoint, four deployment modes. run_server.sh starts the server for a given MODE; client.py --mode <mode> tests it. The capability matrix:

mode (vllm_omni/deploy/audex_<mode>.yaml)	audio in	text out	speech out	general audio out	endpoint
tts	❌	❌	✅	❌	/v1/audio/speech
tta	❌	❌	❌	✅	/v1/audio/speech
thinker_only	✅	✅	❌	❌	/v1/chat/completions
s2s	✅	✅	✅	❌	both

Audio-input modes of client.py fall back to vLLM's public mary_had_lamb asset when --audio-file is omitted; CFG defaults follow the official settings per mode (tts/s2s 1.5, tta 3.0; --cfg-scale 1.0 disables).

tts — text → speech

Text-only thinker (checkpoint_folder_audiogen) emits <speechcodec_N> tokens; the streaming causal decoder produces 16 kHz WAVs. This is also the DEFAULT pipeline when the repo root is served without a deploy config.

./run_server.sh                       # MODE=tts, port 8097

python client.py --mode tts --text "Hello there." --output hello.wav
# or raw curl:
curl -s http://localhost:8097/v1/audio/speech \
    -H 'Content-Type: application/json' \
    -d '{"model": "nvidia/Nemotron-Labs-Audex-2B", "input": "Hello there.", "response_format": "wav", "extra_params": {"cfg_scale": 1.5}}' \
    -o hello.wav

tta — caption → general audio

Same thinker over the 4-codebook <audiocodec_N> RVQ block, decoded by the external XCodec1 checkpoint (auto-downloaded; override with XCODEC1_PATH). Clips are capped at 10 s.

MODE=tta ./run_server.sh

python client.py --mode tta --caption "Heavy rain falling on a tin roof." \
    --output rain.wav

thinker_only — audio (+ instruction) → text

Single-stage audio understanding on the full checkpoint (NV-Whisper encoder + projector + LM). Send the clip as input_audio chat content.

MODE=thinker_only ./run_server.sh

python client.py --mode thinker_only                      # transcribe the demo asset
python client.py --mode thinker_only --audio-file a.wav \
    --question "What language is being spoken?"

s2s — spoken question → spoken answer

The audio-capable thinker plus the speech decoder in one deployment. client.py --mode s2s runs the official three passes: ASR (chat with input_audio) → chat on the transcript → TTS on the answer. Text-final passes carry "modalities": ["text"]; only the TTS pass streams through the decoder.

MODE=s2s PORT=8098 ./run_server.sh

python client.py --mode s2s --port 8098 \
    --audio-file question.wav --output answer.wav

Offline counterparts (no HTTP, one Omni engine per script) live in examples/offline_inference/audex/.

30B-A3B (nvidia/Nemotron-Labs-Audex-30B-A3B)

SIZE=30b switches run_server.sh to the 30B yamls and model id:

SIZE=30b ./run_server.sh                 # 30B tts on port 8097
SIZE=30b MODE=s2s PORT=8098 ./run_server.sh

Pass --model nvidia/Nemotron-Labs-Audex-30B-A3B to the client — the server validates the model id and 404s on a mismatch. Single-H100 defaults; see the yaml comments for the TP2 fallback. First launch downloads ~60 GB.