[Model] Integrate AURA into Full-Duplex Runtime (#7633)
This commit is contained in:
1 parent
d397a2f547
commit
62b15142bd
77 files changed
+7695
-457
No files matched your search
@@ -6,29 +6,29 @@
|
||||
ASR -> AURA -> Qwen3-TTS Talker -> Code2Wav
|
||||
```
|
||||
|
||||
Qwen3-TTS remains two engine stages so the pipeline reuses the existing native
|
||||
Talker and Code2Wav implementation.
|
||||
**Supported online path:** Realtime duplex at `/v1/realtime?duplex=1`.
|
||||
|
||||
The default deploy profile (`vllm_omni/deploy/aura_omni.yaml`) sets
|
||||
`session_mode: duplex`. The `aura_omni` pipeline declares `duplex_plugin`, and
|
||||
`DuplexOmniEngine` requires that mode (same pattern as MiniCPM-o duplex
|
||||
deploys). Turn-based `chat/completions` is not a supported AURA online mode
|
||||
against this profile.
|
||||
|
||||
`async_chunk` defaults to `false` in that file (safer if a turn-based
|
||||
`OmniOrchestrator` ever loads it). Duplex Realtime Stage2→3 still needs
|
||||
async chunks — use the smoke deploy below, or set `async_chunk: true`
|
||||
yourself.
|
||||
|
||||
```bash
|
||||
vllm serve aurateam/AURA \
|
||||
--omni \
|
||||
--deploy-config vllm_omni/deploy/aura_omni.yaml \
|
||||
--served-model-name aurateam/AURA \
|
||||
--trust-remote-code
|
||||
bash examples/online_serving/aura_omni/run_duplex_smoke_serve.sh
|
||||
python examples/online_serving/aura_omni/smoke_duplex_realtime_client.py
|
||||
```
|
||||
|
||||
Configure local checkpoints by editing per-stage `model` values in
|
||||
`vllm_omni/deploy/aura_omni.yaml`. The deploy file sets
|
||||
`pipeline: aura_omni`, so the four-stage topology is used even if the
|
||||
command-line `--model` points at one of the component checkpoints.
|
||||
Configure local checkpoints by editing per-stage `model` values in the deploy
|
||||
or smoke YAML. The file sets `pipeline: aura_omni`, so the four-stage topology
|
||||
is used even if the command-line `--model` points at one component checkpoint.
|
||||
|
||||
Send requests with `"model": "aurateam/AURA"`. The ASR, AURA, and Qwen3-TTS
|
||||
checkpoint paths are internal stage models from the deploy YAML, not the
|
||||
OpenAI-facing served model name.
|
||||
|
||||
The AURA stage can emit `<|silent|>`. Silent outputs are treated as a gate:
|
||||
they produce no Qwen3-TTS Talker input, so no audio is synthesized for that
|
||||
turn.
|
||||
Silent Stage1 outputs (`<|silent|>` / id `151669`) skip TTS for that turn.
|
||||
|
||||
## GPU Utilization Recommendation
|
||||
|
||||
@@ -40,26 +40,21 @@ VRAM each stage can reserve. Start with this split for a single GPU:
|
||||
- Stage 2 (Qwen3-TTS Talker): `0.20`
|
||||
- Stage 3 (Qwen3-TTS Code2Wav): `0.20`
|
||||
|
||||
## TTS Modes
|
||||
## TTS modes (stage extras)
|
||||
|
||||
`aura_omni` can pass AURA text to Qwen3-TTS in two task modes:
|
||||
When the duplex session supplies TTS extras, AURA text can feed Qwen3-TTS as:
|
||||
|
||||
- `Base`: voice clone from `tts_ref_audio` with ICL enabled in the AURA
|
||||
pipeline. Provide both `tts_ref_audio` and `tts_ref_text`. Set
|
||||
`tts_x_vector_only_mode=true` to disable ICL and use speaker embedding only.
|
||||
- `CustomVoice`: predefined speaker mode. Use a Qwen3-TTS CustomVoice
|
||||
checkpoint for stages 2 and 3 in `aura_omni.yaml`, then pass
|
||||
`tts_task_type=CustomVoice` and `tts_speaker`.
|
||||
- `Base`: voice clone from `tts_ref_audio` (optional x-vector-only mode)
|
||||
- `CustomVoice`: predefined speaker (`tts_speaker`) with a CustomVoice checkpoint
|
||||
on stages 2 and 3
|
||||
|
||||
By default, AURA responses are passed to Qwen3-TTS as text. Set
|
||||
`additional_information.tts_pass_token_ids=true` to pass AURA-generated
|
||||
assistant token ids directly instead. Even when token passthrough is disabled,
|
||||
the stage processor uses AURA token ids when available to estimate the Talker
|
||||
prefill length, so scheduling does not rely on raw character length.
|
||||
Optional `tts_pass_token_ids` passes AURA assistant token ids into Talker
|
||||
instead of detokenized text.
|
||||
|
||||
The example client exposes this as:
|
||||
## Old chat / Gradio / curl scripts (unsupported)
|
||||
|
||||
```bash
|
||||
python examples/online_serving/aura_omni/openai_chat_completion_client.py \
|
||||
--tts-pass-token-ids
|
||||
```
|
||||
`examples/online_serving/aura_omni/` still contains chat-completions, curl, and
|
||||
Gradio helpers from an older turn-based path. They do **not** work against the
|
||||
shipped duplex profile. That path was an incomplete stand-in before streaming
|
||||
I/O and session history were available; it is not a supported AURA mode. Use
|
||||
the Realtime duplex smoke client for online verification.
|
||||
Reference in new issue
Block a user