[Model] Accelerate Breeze-TTS-2 non-streaming inference (#7752)

This commit is contained in:
Yueqian Lin authored and GitHub committed 2026-09-22 17:10:15 -07:00
1 parent d2aa07159c
commit 3f473a358b
84 files changed
+8191 -5622

No files matched your search

@@ -42,7 +42,8 @@ Scheduler / current input batch
prefill / other spans single-token decode
preprocess preprocess or preprocess_decode_batch
| |
prompt embedding runner-managed talker_mtp
prompt embedding decode embeddings
| talker_mtp (if present)
\ /
model forward
```
@@ -96,9 +97,24 @@ computed per-row fields are not promised at that earlier boundary.
### Routing guarantee
With `has_preprocess=True`, the batched decode hook is eligible only when it is
callable, the runner has `talker_mtp`, the scheduled span is one token, and
callable, the scheduled span is one token, and
`_omni_is_prefill` is false. Otherwise the normal `preprocess` hook runs.
The hook's return format depends on whether the runner manages MTP:
| Runner capability | `preprocess_decode_batch` return tuple |
| --- | --- |
| Without `talker_mtp` | `(input_ids, inputs_embeds, updates)` |
| With `talker_mtp` | `(input_ids, inputs_embeds, hidden_states, text_step, updates)` |
Each tensor's batch dimension and each entry in the `updates` list correspond
to one physical request row, including separate CFG branches. Non-MTP batches
contain contiguous single-token decode spans; the runner flushes a batch before
processing any other span. It copies embeddings into its existing input buffer
and applies each update dictionary to the corresponding request. Returning the
original `input_ids` object lets it skip an unchanged token-ID copy. The existing
five-result MTP format remains supported.
Runner-managed MTP is likewise restricted to single-token decode rows. A prefill
row, including a one-token tail, is never added to that MTP batch: the runner
does not consume its `mtp_inputs`, generate codes for it, or overwrite its
@@ -130,6 +146,8 @@ formalizes that existing behavior. Prefer upgrading the matching vLLM and
vLLM-Omni release pair and consuming the explicit phase keys. Do not silently
fall back to `input_ids.numel() > 1` when the model requires chunking correctness.
For a custom or backported runtime, verify both metadata and MTP guards.
Older runners can restrict `preprocess_decode_batch` to MTP models; a non-MTP
model must retain its scalar `preprocess` implementation for those runtimes.
## Correctness and testing plan
+1 -1
View File
@@ -72,7 +72,7 @@ th {
| `AuKForConditionalGeneration` | AuK / AuK-Flash (instruction-driven speech generation and editing) | `tencent/AuK`, `tencent/AuK-Flash` (with `Qwen/Qwen2.5-Omni-3B`; assemble with `tools/prepare_auk_checkpoint.py`) | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/AuK-H100.md) |
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | ✅︎ | — |
| `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | ✅︎ | — |
| `BreezeForConditionalGeneration` | Breeze-TTS-2 (TTS, voice clone, voice direction) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) |
| `BreezeForConditionalGeneration` | Breeze-TTS-2 (voice design, voice clone, voice direction, CFG) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | | — |
| `MammothModa2ForConditionalGeneration` | MammothModa2-Dev (AR-only image understanding) | `bytedance-research/MammothModa2-Dev` | ✅︎ | | | | — |
| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
Binary file not shown.

After

Width:  |  Height:  |  Size: 234 KiB