[Model] Accelerate Breeze-TTS-2 non-streaming inference (#7752)
This commit is contained in:
1 parent
d2aa07159c
commit
3f473a358b
84 files changed
+8191
-5622
No files matched your search
@@ -42,7 +42,8 @@ Scheduler / current input batch
|
||||
prefill / other spans single-token decode
|
||||
preprocess preprocess or preprocess_decode_batch
|
||||
| |
|
||||
prompt embedding runner-managed talker_mtp
|
||||
prompt embedding decode embeddings
|
||||
| talker_mtp (if present)
|
||||
\ /
|
||||
model forward
|
||||
```
|
||||
@@ -96,9 +97,24 @@ computed per-row fields are not promised at that earlier boundary.
|
||||
### Routing guarantee
|
||||
|
||||
With `has_preprocess=True`, the batched decode hook is eligible only when it is
|
||||
callable, the runner has `talker_mtp`, the scheduled span is one token, and
|
||||
callable, the scheduled span is one token, and
|
||||
`_omni_is_prefill` is false. Otherwise the normal `preprocess` hook runs.
|
||||
|
||||
The hook's return format depends on whether the runner manages MTP:
|
||||
|
||||
| Runner capability | `preprocess_decode_batch` return tuple |
|
||||
| --- | --- |
|
||||
| Without `talker_mtp` | `(input_ids, inputs_embeds, updates)` |
|
||||
| With `talker_mtp` | `(input_ids, inputs_embeds, hidden_states, text_step, updates)` |
|
||||
|
||||
Each tensor's batch dimension and each entry in the `updates` list correspond
|
||||
to one physical request row, including separate CFG branches. Non-MTP batches
|
||||
contain contiguous single-token decode spans; the runner flushes a batch before
|
||||
processing any other span. It copies embeddings into its existing input buffer
|
||||
and applies each update dictionary to the corresponding request. Returning the
|
||||
original `input_ids` object lets it skip an unchanged token-ID copy. The existing
|
||||
five-result MTP format remains supported.
|
||||
|
||||
Runner-managed MTP is likewise restricted to single-token decode rows. A prefill
|
||||
row, including a one-token tail, is never added to that MTP batch: the runner
|
||||
does not consume its `mtp_inputs`, generate codes for it, or overwrite its
|
||||
@@ -130,6 +146,8 @@ formalizes that existing behavior. Prefer upgrading the matching vLLM and
|
||||
vLLM-Omni release pair and consuming the explicit phase keys. Do not silently
|
||||
fall back to `input_ids.numel() > 1` when the model requires chunking correctness.
|
||||
For a custom or backported runtime, verify both metadata and MTP guards.
|
||||
Older runners can restrict `preprocess_decode_batch` to MTP models; a non-MTP
|
||||
model must retain its scalar `preprocess` implementation for those runtimes.
|
||||
|
||||
## Correctness and testing plan
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@ th {
|
||||
| `AuKForConditionalGeneration` | AuK / AuK-Flash (instruction-driven speech generation and editing) | `tencent/AuK`, `tencent/AuK-Flash` (with `Qwen/Qwen2.5-Omni-3B`; assemble with `tools/prepare_auk_checkpoint.py`) | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Tencent/AuK-H100.md) |
|
||||
| `OmniVoiceModel` | OmniVoice | `k2-fsa/OmniVoice` | ✅︎ | | | ✅︎ | — |
|
||||
| `VoxCPM2TalkerForConditionalGeneration` | VoxCPM2 | `openbmb/VoxCPM2` | ✅︎ | | | ✅︎ | — |
|
||||
| `BreezeForConditionalGeneration` | Breeze-TTS-2 (TTS, voice clone, voice direction) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) |
|
||||
| `BreezeForConditionalGeneration` | Breeze-TTS-2 (voice design, voice clone, voice direction, CFG) | `BreezeBlue/Breeze-TTS-2` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/BreezeBlue/Breeze-TTS-2.md) |
|
||||
| `MammothModa2ForConditionalGeneration` | MammothModa2-Preview | `bytedance-research/MammothModa2-Preview` | ✅︎ | ✅︎ | | | — |
|
||||
| `MammothModa2ForConditionalGeneration` | MammothModa2-Dev (AR-only image understanding) | `bytedance-research/MammothModa2-Dev` | ✅︎ | | | | — |
|
||||
| `Flux2KleinPipeline` | FLUX.2-klein | `black-forest-labs/FLUX.2-klein-4B`, `black-forest-labs/FLUX.2-klein-9B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 234 KiB |
Reference in new issue
Block a user