Merge main and preserve multi-API engine and duplex guards
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
Sy03 committed
commit
daf91b4d7a
329 files changed
+36428
-35780
No files matched your search
@@ -1,24 +1,33 @@
|
||||
# FastVideo VSA
|
||||
|
||||
FastVideo Variable Sparse Attention (VSA) accelerates the `FastVideo/FastWan2.2-TI2V-5B-Diffusers` model's
|
||||
self-attention by partitioning the post-patch latent grid into spatiotemporal
|
||||
FastVideo Variable Sparse Attention (VSA) partitions the post-patch latent grid into spatiotemporal
|
||||
blocks. For every query block, VSA scores the key/value blocks and computes
|
||||
attention only against the selected top-k blocks.
|
||||
|
||||
The supported checkpoint provides both text-to-video and image-to-video modes through `Wan22Pipeline`; the separate Wan I2V-14B, S2V, and VACE pipelines are outside this backend's supported scope.
|
||||
## Supported models
|
||||
|
||||
MiniMax-H3 served with a FastH3 VSA adapter uses a second, H3-specific route
|
||||
through the same backend. See the
|
||||
[MiniMax-H3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) for its geometry,
|
||||
supported topologies, and commands.
|
||||
| Model / checkpoint | Required adapter | Tasks | Sequence parallelism |
|
||||
| --- | --- | --- | --- |
|
||||
| `FastVideo/FastWan2.2-TI2V-5B-Diffusers` | None | T2V, I2V through `Wan22Pipeline` | Disabled |
|
||||
| `MiniMaxAI/MiniMax-H3` | [FastH3 VSA (recipe)](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter) | T2VA | Disabled or pure Ulysses |
|
||||
| Wan I2V-14B, S2V, VACE | — | Unsupported | — |
|
||||
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
## Installation
|
||||
|
||||
Install vLLM-Omni from this checkout with the `vsa` extra:
|
||||
|
||||
```bash
|
||||
uv pip install -e '.[vsa]'
|
||||
```
|
||||
|
||||
The extra installs the tested kernel dependency automatically. Prebuilt kernels require
|
||||
Linux, Python 3.12, and glibc 2.34 or newer (x86-64 or aarch64). The full
|
||||
FastVideo framework and provider environment variables are not required.
|
||||
|
||||
`FASTVIDEO_VSA` selects the attention algorithm. On SM120, the H3 integration
|
||||
currently executes FastVideo's 64-token Triton block-sparse kernel; it does not
|
||||
dispatch to FlashInfer. The `vsa` extra installs `fastvideo-kernel==0.3.4` for
|
||||
this execution path.
|
||||
|
||||
## Enable the backend
|
||||
|
||||
@@ -27,24 +36,18 @@ flag. Use `--fastvideo-vsa-topk` to set the number of key/value blocks retained
|
||||
for every query block:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-backend FASTVIDEO_VSA \
|
||||
--fastvideo-vsa-topk 64
|
||||
```
|
||||
|
||||
The backwards-compatible environment variable selects the backend with the
|
||||
default `topk=64`:
|
||||
<details markdown="1">
|
||||
<summary>Alternative configuration formats</summary>
|
||||
|
||||
Equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
export DIFFUSION_ATTENTION_BACKEND=FASTVIDEO_VSA
|
||||
vllm-omni serve <model>
|
||||
```
|
||||
|
||||
To tune top-k, pass the CLI backend and top-k flags together as shown above,
|
||||
or use the equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-config \
|
||||
'{"default":{"backend":"FASTVIDEO_VSA","fastvideo_vsa_topk":64}}'
|
||||
```
|
||||
@@ -73,8 +76,15 @@ stages:
|
||||
fastvideo_vsa_topk: 64
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Choose top-k
|
||||
|
||||
H3 uses 64-token video blocks and keeps
|
||||
all prefix blocks; see the [FastH3 VSA recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
### Wan top-k behavior
|
||||
|
||||
At runtime the backend logs the sequence shape and derived block count:
|
||||
|
||||
```text
|
||||
@@ -117,8 +127,16 @@ selecting VSA does not turn a native checkpoint into a distilled model.
|
||||
|
||||
## Verify routing and fallback
|
||||
|
||||
Check the startup and first-forward logs instead of assuming that selecting
|
||||
the backend guarantees sparse execution:
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
H3 accelerator faults propagate instead of attempting dense recovery.
|
||||
|
||||
On the Wan route, check the startup and first-forward logs:
|
||||
|
||||
- `route=VSA` means top-k block selection is active.
|
||||
- `route=VSA_ALL_BLOCKS` means the FastVideo DMD checkpoint retained all
|
||||
@@ -126,6 +144,10 @@ the backend guarantees sparse execution:
|
||||
- `route=SDPA` or `FASTVIDEO_VSA falling back to SDPA: ...` means dense SDPA
|
||||
executed; the warning includes the reason.
|
||||
|
||||
For H3, check `FastH3 adapter active` at startup and
|
||||
`FASTVIDEO_VSA H3 routing` during DiT execution, as described in the
|
||||
[model recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
The Wan route requires CUDA tensors in FP16 or BF16, 256-token blocks,
|
||||
standard `head_size**-0.5` scaling, equal Q/K/V head counts, no attention mask,
|
||||
and no active sequence-parallel context. The MiniMax-H3 route uses 64-token
|
||||
|
||||
@@ -18,6 +18,7 @@ For the full list of supported architectures across all modalities, see
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | — |
|
||||
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
|
||||
| Gepard-1.0 | `nineninesix/gepard-1.0` | — (zero-shot default voice) | ✓ (PCM / WAV stream) | `default` only | — |
|
||||
| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✓ (`ref_audio`+`ref_text`) | ✓ (codec_streaming) | — | — |
|
||||
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
|
||||
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
|
||||
@@ -92,7 +93,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
|
||||
```
|
||||
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard, 24 kHz for the others).
|
||||
|
||||
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md).
|
||||
|
||||
@@ -212,6 +213,61 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server
|
||||
|
||||
---
|
||||
|
||||
## Gepard-1.0
|
||||
|
||||
Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit `voice` or pass `"default"`. Voice cloning from reference audio is not available yet.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Same NeMo NanoCodec install as the [offline Gepard section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#gepard-10). On a host whose CUDA toolkit cannot build kernels — no `nvcc`/`ninja`, or a consumer Blackwell (`sm_120`) card — also `export VLLM_USE_FLASHINFER_SAMPLER=0` before launch. That env is not a deploy-YAML field.
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
# or:
|
||||
./gepard/run_server.sh
|
||||
```
|
||||
|
||||
`--stage-init-timeout 900` matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.
|
||||
|
||||
The packaged `vllm_omni/deploy/gepard.yaml` must be passed with `--deploy-config`. The checkpoint self-identifies as `qwen3_5_text`, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets `async_chunk: false`, `max_num_seqs: 4`, and currently pins `seed: 42`, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request `seed` in tests and clients rather than depending on either default.
|
||||
|
||||
### Sending requests
|
||||
|
||||
```bash
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking."
|
||||
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm
|
||||
```
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`, `stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields include `speed`, `language`, `instructions`, `task_type`, `ref_audio`, `ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 22.05 kHz mono. `max_new_tokens` is a **frame** budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
|
||||
- Supported request fields: `input` (required), `voice` (`default` only), `response_format` (`wav` default; `wav/pcm/flac/mp3` non-streaming; `opus` 400 because 22.05 kHz is not an Opus sample rate; streaming `pcm/wav` only), `stream` / `stream_format`, `max_new_tokens`, `seed`.
|
||||
- Unsupported: `speed`, `extra_params` (including `temperature`/`top_p`/`top_k`), `ref_audio`, `ref_text`, `speaker_embedding`, `task_type`, `instructions`, `language`, and `word_timestamps`.
|
||||
- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
|
||||
- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
|
||||
- For the request-field table, see the [Speech API Gepard section](https://github.com/vllm-project/vllm-omni/blob/main/docs/serving/speech_api.md#gepard-10).
|
||||
|
||||
---
|
||||
|
||||
## higgs-audio v2
|
||||
|
||||
2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.
|
||||
|
||||
Reference in new issue
Block a user