Merge remote-tracking branch 'origin/main' into qwen-image-2.1
# Conflicts: # vllm_omni/diffusion/executor/multiproc_executor.py # vllm_omni/diffusion/offloader/sequential_backend.py
This commit is contained in:
Isotr0py committed
commit
9174554e40
1332 files changed
+160051
-60504
No files matched your search
@@ -1,24 +1,33 @@
|
||||
# FastVideo VSA
|
||||
|
||||
FastVideo Variable Sparse Attention (VSA) accelerates the `FastVideo/FastWan2.2-TI2V-5B-Diffusers` model's
|
||||
self-attention by partitioning the post-patch latent grid into spatiotemporal
|
||||
FastVideo Variable Sparse Attention (VSA) partitions the post-patch latent grid into spatiotemporal
|
||||
blocks. For every query block, VSA scores the key/value blocks and computes
|
||||
attention only against the selected top-k blocks.
|
||||
|
||||
The supported checkpoint provides both text-to-video and image-to-video modes through `Wan22Pipeline`; the separate Wan I2V-14B, S2V, and VACE pipelines are outside this backend's supported scope.
|
||||
## Supported models
|
||||
|
||||
MiniMax-H3 served with a FastH3 VSA adapter uses a second, H3-specific route
|
||||
through the same backend. See the
|
||||
[MiniMax-H3 recipe](https://recipes.vllm.ai/MiniMaxAI/MiniMax-H3) for its geometry,
|
||||
supported topologies, and commands.
|
||||
| Model / checkpoint | Required adapter | Tasks | Sequence parallelism |
|
||||
| --- | --- | --- | --- |
|
||||
| `FastVideo/FastWan2.2-TI2V-5B-Diffusers` | None | T2V, I2V through `Wan22Pipeline` | Disabled |
|
||||
| `MiniMaxAI/MiniMax-H3` | [FastH3 VSA (recipe)](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter) | T2VA | Disabled or pure Ulysses |
|
||||
| Wan I2V-14B, S2V, VACE | — | Unsupported | — |
|
||||
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
## Installation
|
||||
|
||||
Install vLLM-Omni from this checkout with the `vsa` extra:
|
||||
|
||||
```bash
|
||||
uv pip install -e '.[vsa]'
|
||||
```
|
||||
|
||||
The extra installs the tested kernel dependency automatically. Prebuilt kernels require
|
||||
Linux, Python 3.12, and glibc 2.34 or newer (x86-64 or aarch64). The full
|
||||
FastVideo framework and provider environment variables are not required.
|
||||
|
||||
`FASTVIDEO_VSA` selects the attention algorithm. On SM120, the H3 integration
|
||||
currently executes FastVideo's 64-token Triton block-sparse kernel; it does not
|
||||
dispatch to FlashInfer. The `vsa` extra installs `fastvideo-kernel==0.3.4` for
|
||||
this execution path.
|
||||
|
||||
## Enable the backend
|
||||
|
||||
@@ -27,24 +36,18 @@ flag. Use `--fastvideo-vsa-topk` to set the number of key/value blocks retained
|
||||
for every query block:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-backend FASTVIDEO_VSA \
|
||||
--fastvideo-vsa-topk 64
|
||||
```
|
||||
|
||||
The backwards-compatible environment variable selects the backend with the
|
||||
default `topk=64`:
|
||||
<details markdown="1">
|
||||
<summary>Alternative configuration formats</summary>
|
||||
|
||||
Equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
export DIFFUSION_ATTENTION_BACKEND=FASTVIDEO_VSA
|
||||
vllm-omni serve <model>
|
||||
```
|
||||
|
||||
To tune top-k, pass the CLI backend and top-k flags together as shown above,
|
||||
or use the equivalent structured configuration:
|
||||
|
||||
```bash
|
||||
vllm-omni serve <model> \
|
||||
vllm serve <model> --omni \
|
||||
--diffusion-attention-config \
|
||||
'{"default":{"backend":"FASTVIDEO_VSA","fastvideo_vsa_topk":64}}'
|
||||
```
|
||||
@@ -73,8 +76,15 @@ stages:
|
||||
fastvideo_vsa_topk: 64
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Choose top-k
|
||||
|
||||
H3 uses 64-token video blocks and keeps
|
||||
all prefix blocks; see the [FastH3 VSA recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
### Wan top-k behavior
|
||||
|
||||
At runtime the backend logs the sequence shape and derived block count:
|
||||
|
||||
```text
|
||||
@@ -117,8 +127,16 @@ selecting VSA does not turn a native checkpoint into a distilled model.
|
||||
|
||||
## Verify routing and fallback
|
||||
|
||||
Check the startup and first-forward logs instead of assuming that selecting
|
||||
the backend guarantees sparse execution:
|
||||
VSA is a CUDA-only, explicitly selected backend. It requires the
|
||||
`fastvideo-kernel` package and currently supports non-causal self-attention
|
||||
with equal query and key/value sequence lengths. Unsupported shapes, masks,
|
||||
dtypes, sequence-parallel execution, or kernel failures fall back to
|
||||
`TORCH_SDPA` and emit a warning with the reason. On the Wan route, an active
|
||||
sequence-parallel context is one of those fallbacks; the H3 route supports pure
|
||||
Ulysses and rejects ring or all-gather sequence parallelism at startup.
|
||||
H3 accelerator faults propagate instead of attempting dense recovery.
|
||||
|
||||
On the Wan route, check the startup and first-forward logs:
|
||||
|
||||
- `route=VSA` means top-k block selection is active.
|
||||
- `route=VSA_ALL_BLOCKS` means the FastVideo DMD checkpoint retained all
|
||||
@@ -126,6 +144,10 @@ the backend guarantees sparse execution:
|
||||
- `route=SDPA` or `FASTVIDEO_VSA falling back to SDPA: ...` means dense SDPA
|
||||
executed; the warning includes the reason.
|
||||
|
||||
For H3, check `FastH3 adapter active` at startup and
|
||||
`FASTVIDEO_VSA H3 routing` during DiT execution, as described in the
|
||||
[model recipe](https://github.com/vllm-project/vllm-omni/blob/main/recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter).
|
||||
|
||||
The Wan route requires CUDA tensors in FP16 or BF16, 256-token blocks,
|
||||
standard `head_size**-0.5` scaling, equal Q/K/V head counts, no attention mask,
|
||||
and no active sequence-parallel context. The MiniMax-H3 route uses 64-token
|
||||
|
||||
@@ -78,3 +78,35 @@ Protected video tails require a compatible MindIE-SD release.
|
||||
For common configuration and selector behavior, see the
|
||||
[attention backend overview](../attention_backends.md) and the
|
||||
[backend selection design](../../../design/feature/attention_backend_selection.md).
|
||||
|
||||
## Wan2.2 T2V quantization on Ascend
|
||||
|
||||
`RAINFUSION_ATTN` accepts `diffusion_kv_cache_dtype: fp8` or `mxfp4` for
|
||||
Wan2.2 T2V A14B.
|
||||
MindIE-SD must expose `sparse_attention` with `rf_v3` and an explicit `precision`
|
||||
argument. Omni passes the selected precision directly to this API; the installed
|
||||
native operators must support that precision. Unsupported inputs or an
|
||||
incompatible Python API raise an actionable error; Omni does not change BSA
|
||||
precision automatically. Update `diffusion_kv_cache_dtype` to a precision
|
||||
supported by the installed MindIE-SD, or unset it. Native execution errors also
|
||||
propagate without retry.
|
||||
BSA FP8 and MXFP4 retain MindIE-SD's default rotation policy.
|
||||
|
||||
`diffusion_kv_cache_skip_layers`/`diffusion_kv_cache_skip_steps` explicitly
|
||||
select floating-point BSA and keep attention sparse. The existing `block_sparse`
|
||||
warmup/layer exclusions and short-sequence threshold instead select Dense
|
||||
attention, using the configured Dense precision. Leave `block_sparse.precision`
|
||||
at its default when using `diffusion_kv_cache_dtype` as the shared Dense/BSA
|
||||
precision setting.
|
||||
|
||||
Configure BSA quantization in the model's existing deployment YAML with
|
||||
`backend: RAINFUSION_ATTN` and stage-level `diffusion_kv_cache_dtype: fp8` or
|
||||
`mxfp4`. The precision and skip selectors are siblings of
|
||||
`diffusion_attention_config`; they are not per-role `quant` fields.
|
||||
For 40 denoising steps, `diffusion_kv_cache_skip_steps: "0,1,38,39"` and
|
||||
`diffusion_kv_cache_skip_layers: "0,39"` are recommended starting points. These
|
||||
fixed step indices do not automatically follow a changed step count. Selected
|
||||
forwards remain floating-point BSA; unsupported BSA precision is an error rather
|
||||
than an automatic fallback. See
|
||||
[quantized attention](../../quantization/quantized_kvcache.md#wan22-t2v-quantized-attention-on-ascend)
|
||||
for the shared configuration structure and index semantics.
|
||||
@@ -0,0 +1,72 @@
|
||||
# Device-side video postprocessing
|
||||
|
||||
vLLM-Omni can convert decoded WAN2.2 video from normalized floating-point
|
||||
`[B, C, T, H, W]` tensors to contiguous `uint8` `[B, T, H, W, C]` frames before
|
||||
the worker-to-engine device-to-host transfer. This reduces the transported video
|
||||
payload by four times when the original IPC path widens bfloat16 to float32.
|
||||
|
||||
The optimization is disabled by default. Enable it with a structured diffusion
|
||||
configuration:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
engine = Omni(
|
||||
model="Wan-AI/Wan2.2-TI2V-5B-Diffusers",
|
||||
video_output_transport={"enable_device_postprocess": True},
|
||||
)
|
||||
```
|
||||
|
||||
For the server CLI, pass the same object as JSON:
|
||||
|
||||
```bash
|
||||
vllm serve Wan-AI/Wan2.2-TI2V-5B-Diffusers --omni \
|
||||
--video-output-transport '{"enable_device_postprocess": true}'
|
||||
```
|
||||
|
||||
For a deploy configuration, place it on the diffusion stage:
|
||||
|
||||
```yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
video_output_transport:
|
||||
enable_device_postprocess: true
|
||||
```
|
||||
|
||||
## Runtime contract
|
||||
|
||||
WAN2.2 emits decoded video through the typed `DiffusionOutput.media` field. The
|
||||
model runner validates and splits the batch into request-local tensors before
|
||||
preparing each video for transport. The worker then performs the D2H copy and
|
||||
shared-memory IPC using the prepared representation. The engine uses the generic
|
||||
media finalizer instead of a model-specific postprocessor.
|
||||
|
||||
The runtime leaves the video in normalized floating-point form when:
|
||||
|
||||
- `enable_device_postprocess` is false;
|
||||
- the requested output type is not `np`; or
|
||||
- frame interpolation still requires the floating-point tensor.
|
||||
|
||||
These are policy fallbacks, not malformed-contract errors. If the device cannot
|
||||
allocate the temporary conversion buffer, the runner logs a warning and prepares
|
||||
the request-local float representation instead. Invalid tensor layout, encoding,
|
||||
value range, lifecycle state, and non-memory runtime failures still fail before
|
||||
IPC.
|
||||
|
||||
## Precision
|
||||
|
||||
Device preparation converts to float32 before denormalization and quantization.
|
||||
For bfloat16 WAN output this is more precise than the historical host path,
|
||||
which denormalizes in bfloat16. Relative to that path, values may differ by at
|
||||
most one level out of 255. Float32 inputs remain byte-identical after
|
||||
quantization.
|
||||
|
||||
## Scope
|
||||
|
||||
WAN2.2 is the reference migration for the first contract version. Other video
|
||||
pipelines continue using their existing output paths until they can declare the
|
||||
same typed media contract without bypassing model-specific float consumers such
|
||||
as safety checks or audio/video packaging.
|
||||
|
||||
See [RFC #6541](https://github.com/vllm-project/vllm-omni/issues/6541) for the
|
||||
contract, lifecycle, batching rules, and migration plan.
|
||||
@@ -28,7 +28,7 @@ and their optimal Triton tile configurations:
|
||||
|
||||
Config loading order (3-tier, see ops/mot_gemm.py):
|
||||
1. $VLLM_TUNED_CONFIG_FOLDER/<filename> (env override)
|
||||
2. This directory: vllm_omni/diffusion/layers/mot/configs/<filename>
|
||||
2. This directory: vllm_omni/diffusion/models/bagel/mot/configs/<filename>
|
||||
3. Conservative default config (compiles everywhere, sub-optimal perf)
|
||||
|
||||
If no config file matches the current device, a warning is printed with
|
||||
@@ -39,7 +39,7 @@ To generate configs for your hardware:
|
||||
python benchmarks/kernels/mot_linear_benchmarks.py \
|
||||
--model ByteDance-Seed/BAGEL-7B-MoT \
|
||||
--tp-size 1 --dtype w16a16 --tune \
|
||||
--save-dir vllm_omni/diffusion/layers/mot/configs/
|
||||
--save-dir vllm_omni/diffusion/models/bagel/mot/configs/
|
||||
|
||||
For multi-GPU tuning (uses Ray for parallel search):
|
||||
|
||||
|
||||
@@ -353,9 +353,14 @@ modes can resolve the same `None` value to different schedules.
|
||||
Cache-DiT on any selected component. Prompt-embedding cache is also
|
||||
incompatible when the text encoder uses `allgather`. Rank-local cache hits
|
||||
could otherwise make ranks enter different weight collectives.
|
||||
- Per-tensor online FP8 linears use the ordinary loader and can run with either
|
||||
DiT transfer path. With DiT `allgather`, every rank temporarily materializes
|
||||
the complete FP8 model in host memory before DLO retains only its shard.
|
||||
- Per-tensor online FP8, INT8, and MXFP8 linears use the ordinary loader and
|
||||
can run with either DiT transfer path. With DiT `allgather`, every rank
|
||||
temporarily materializes the complete quantized model in host memory before
|
||||
DLO retains only its shard. Layers too wide for the NPU quantization kernel
|
||||
(`npu_quant_matmul` rejects an output dimension past 65535) stay
|
||||
unquantized and load straight into host memory under offload-after-quant —
|
||||
their runtime layout is a plain contiguous bf16 weight, the same layout DLO
|
||||
shards on the ordinary path, so they are allowed under `allgather` too.
|
||||
Other online quantization methods require rank-local transfer for the
|
||||
affected component until their runtime layouts are validated.
|
||||
- Resident leading layers require DiT `rank-local` transfer and a model
|
||||
|
||||
@@ -19,6 +19,12 @@ CFG-Parallel accelerates diffusion models by distributing positive and negative
|
||||
|
||||
See supported models list in [Supported Models](../../diffusion_features.md#supported-models).
|
||||
|
||||
Distilled Cosmos3 checkpoints, including `nvidia/Cosmos3-Super-Image2Video-4Step`,
|
||||
run without CFG and reject `--cfg-parallel-size` greater than 1 at startup.
|
||||
Use `--ulysses-degree 2` for two-GPU inference. These checkpoints always use
|
||||
`guidance_scale=1.0`; other explicitly requested values produce a warning, and
|
||||
`negative_prompt` does not affect generation.
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
@@ -34,5 +34,34 @@ that provide their own `setup_compile()` implementation manage their compilation
|
||||
policy independently. Compilation is lazy, so backend or graph errors can first
|
||||
surface on the initial request.
|
||||
|
||||
## Pinning one packed shape (MiniMax-H3)
|
||||
|
||||
Compiled regions are keyed by input shape. MiniMax-H3 packs a request into a
|
||||
row count that follows the request -- a longer prompt, more frames, a larger
|
||||
frame -- and rounds it up to the next 64-row boundary, so two requests can land
|
||||
on two different packed lengths.
|
||||
|
||||
Under the default `diffusion_compile_dynamic: true` a new packed length is
|
||||
absorbed by the dynamic shape, so it is not by itself a recompilation. It still
|
||||
hands the backend a shape it has not autotuned, which is the risk this knob
|
||||
exists to remove: a fixed-shape deployment (`--no-diffusion-compile-dynamic`)
|
||||
recompiles the transformer blocks per shape, and a first-seen large shape can
|
||||
push autotuning into its worst-case memory use.
|
||||
|
||||
A request pins the length with `extra_args["pad_seq_len"]`:
|
||||
|
||||
```python
|
||||
sampling_params = SamplingParams(extra_args={"pad_seq_len": 54080})
|
||||
```
|
||||
|
||||
The value must be a positive multiple of 64 and must cover the rows the request
|
||||
actually uses; the packed sequence is then padded to it instead of to the next
|
||||
64-row boundary. Pick a bucket that covers every request dimension the
|
||||
deployment accepts -- prompt length, frame count, frame size and reference
|
||||
blocks all change the used row count. The server logs the effective length as
|
||||
`MiniMax H3 packed sequence: ... pad_seq_len=... used=... seq_len=...` whenever
|
||||
a request pins it. The padding rows are masked out, so their cost is the
|
||||
attention and feed-forward work on those rows.
|
||||
|
||||
Use `--enforce-eager` to disable the model runner's generic compile setup.
|
||||
Pipelines that compile internally define their own eager-mode behavior.
|
||||
@@ -1,35 +0,0 @@
|
||||
# HunyuanImage-3.0 Image-to-Text Inference
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/hunyuan_image3>.
|
||||
|
||||
|
||||
This example demonstrates how to run HunyuanImage-3.0 Image-to-Text with the vLLM-Omni.
|
||||
|
||||
## Local CLI Usage
|
||||
|
||||
Download the example image:
|
||||
|
||||
```bash
|
||||
wget https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg
|
||||
```
|
||||
|
||||
Run example:
|
||||
|
||||
```bash
|
||||
python image_to_text.py \
|
||||
--image cherry_blossom.jpg \
|
||||
--prompt "<|startoftext|>You are an assistant that understands images and outputs text.<img>Describe the content of the picture."
|
||||
```
|
||||
|
||||
Key arguments:
|
||||
|
||||
- `--model`: Model used. Default is: tencent/HunyuanImage-3.0-Instruct (Optional).
|
||||
- `--image`: Path to input image (required).
|
||||
- `--prompt`: Text description used to guide image understanding (required).
|
||||
|
||||
## Example materials
|
||||
|
||||
??? abstract "image_to_text.py"
|
||||
``````py
|
||||
--8<-- "examples/offline_inference/hunyuan_image3/image_to_text.py"
|
||||
``````
|
||||
@@ -2,7 +2,6 @@
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni>.
|
||||
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/README.md)
|
||||
@@ -15,6 +14,19 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
For a local deployment with multiple API frontend processes sharing one set
|
||||
of stage engines, add `--api-server-count`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--api-server-count 2 --port 8091
|
||||
```
|
||||
|
||||
This mode supports local EngineCore stages with one local process group per
|
||||
replica, including stages configured with multiple `num_replicas`. Headless,
|
||||
remote, diffusion, intra-replica data-parallel, and Ray stage deployments
|
||||
remain single-frontend.
|
||||
|
||||
The default deployment configuration situated at `vllm_omni/deploy/qwen3_omni_moe.yaml` is resolved and loaded
|
||||
automatically via the model registry, obviating the necessity for the `--deploy-config` flag in standard deployment topologies.
|
||||
The bundled Qwen3-Omni setup explicitly sets `moe_backend: triton` for the Thinker and Talker stages. This keeps
|
||||
@@ -22,7 +34,12 @@ both stages on vLLM's Triton unquantized MoE path and avoids the unstable FlashI
|
||||
some GPUs.
|
||||
Asynchronous chunk streaming is **enabled by default** within the bundled configuration.
|
||||
|
||||
For Thinker pipeline parallelism, configure `pipeline_parallel_size` and `devices`
|
||||
on the Thinker stage. Keep Talker and Code2Wav at `pipeline_parallel_size: 1`. The
|
||||
default asynchronous scheduling and audio chunk streaming can remain enabled.
|
||||
|
||||
To explicitly utilize a custom deployment YAML, specify the configuration path:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
|
||||
--deploy-config /path/to/deploy_config_file
|
||||
@@ -46,7 +63,7 @@ Adopt the stage-based CLI architecture to independently instantiate execution pr
|
||||
The example below pins Stage 0 to GPU 0 and Stage 1/2 to GPU 1 via
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
**1. Stage 0 (Thinker + API server)**
|
||||
#### 1. Stage 0 (Thinker + API server)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
@@ -56,7 +73,7 @@ CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**2. Stage 1 (Talker)**
|
||||
#### 2. Stage 1 (Talker)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
@@ -66,7 +83,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--omni-master-port 26000
|
||||
```
|
||||
|
||||
**3. Stage 2 (Code2Wav)**
|
||||
#### 3. Stage 2 (Code2Wav)
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
@@ -91,6 +108,9 @@ To experiment with another supported MoE backend, set `moe_backend` in the
|
||||
deployment YAML for both the Thinker and Talker stages. For the stage-based
|
||||
CLI, pass `--moe-backend` directly on each stage command.
|
||||
|
||||
For other stage settings, pass the option on that stage command. For example,
|
||||
to change its GPU memory utilization:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
--stage-id 1 \
|
||||
@@ -103,11 +123,12 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
### Send Multi-modal Request
|
||||
|
||||
Get into the example folder
|
||||
|
||||
```bash
|
||||
cd examples/online_serving/qwen3_omni
|
||||
```
|
||||
|
||||
#### Send request via python
|
||||
#### Send request via python
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py --query-type use_image --port 8091 --host "localhost"
|
||||
@@ -122,7 +143,6 @@ The Python client supports the following command-line arguments:
|
||||
- `--audio-path` (or `-a`): Path to local audio file or URL. If not provided and query-type is `use_audio`, uses default audio URL. Supports local file paths (automatically encoded to base64) or HTTP/HTTPS URLs and common audio formats: MP3, WAV, OGG, FLAC, M4A. Example: `--audio-path /path/to/audio.wav` or `--audio-path https://example.com/audio.mp3`
|
||||
- `--prompt` (or `-p`): Custom text prompt/question. If not provided, uses default prompt for the selected query type. Example: `--prompt "What are the main activities shown in this video?"`
|
||||
|
||||
|
||||
For example, to use a local video file with custom prompt:
|
||||
|
||||
```bash
|
||||
@@ -132,19 +152,20 @@ python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--prompt "What are the main activities shown in this video?"
|
||||
```
|
||||
|
||||
#### Send request via curl
|
||||
#### Send request via curl
|
||||
|
||||
```bash
|
||||
bash run_curl_multimodal_generation.sh use_image
|
||||
```
|
||||
|
||||
## Modality control
|
||||
|
||||
You can control output modalities to specify which types of output the model should generate. This is useful when you only need text output and want to skip audio generation stages for better performance.
|
||||
|
||||
### Supported modalities
|
||||
|
||||
| Modalities | Output |
|
||||
|------------|--------|
|
||||
| ------------ | -------- |
|
||||
| `["text"]` | Text only |
|
||||
| `["audio"]` | Audio only |
|
||||
| `["text", "audio"]` | Text + Audio |
|
||||
@@ -222,7 +243,9 @@ print(response.choices[1].message.audio) # Audio response
|
||||
```
|
||||
|
||||
## Streaming Output
|
||||
|
||||
If you want to enable streaming output, please set the argument as below. The final output will be obtained just after generated by corresponding stage. We support both text streaming output and audio streaming output. Other modalities can output normally.
|
||||
|
||||
```bash
|
||||
python openai_chat_completion_client_for_multimodal_generation.py \
|
||||
--query-type use_image \
|
||||
@@ -255,12 +278,14 @@ The convenience script launches both the vLLM server and Gradio demo together:
|
||||
```
|
||||
|
||||
This script will:
|
||||
|
||||
1. Start the vLLM server in the background
|
||||
2. Wait for the server to be ready
|
||||
3. Launch the Gradio demo
|
||||
4. Handle cleanup when you press Ctrl+C
|
||||
|
||||
The script supports the following arguments:
|
||||
|
||||
- `--model`: Model name/path (default: Qwen/Qwen3-Omni-30B-A3B-Instruct)
|
||||
- `--server-port`: Port for vLLM server (default: 8091)
|
||||
- `--gradio-port`: Port for Gradio demo (default: 7861)
|
||||
@@ -271,18 +296,19 @@ The script supports the following arguments:
|
||||
|
||||
#### Option 2: Manual Launch (Two-Step Process)
|
||||
|
||||
**Step 1: Launch the vLLM API server**
|
||||
##### Step 1: Launch the vLLM API server
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
|
||||
```
|
||||
|
||||
If you have a custom deploy config file:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 --deploy-config /path/to/deploy_config_file
|
||||
```
|
||||
|
||||
**Step 2: Run the Gradio demo**
|
||||
##### Step 2: Run the Gradio demo
|
||||
|
||||
In a separate terminal:
|
||||
|
||||
@@ -295,7 +321,7 @@ Then open `http://localhost:7861/` on your local browser to interact with the we
|
||||
The gradio script supports the following arguments:
|
||||
|
||||
- `--model`: Model name/path (should match the server model)
|
||||
- `--api-base`: Base URL for the vLLM API server (default: http://localhost:8091/v1)
|
||||
- `--api-base`: Base URL for the vLLM API server (default: <http://localhost:8091/v1>)
|
||||
- `--ip`: Host/IP for Gradio server (default: 127.0.0.1)
|
||||
- `--port`: Port for Gradio server (default: 7861)
|
||||
- `--share`: Share the Gradio demo publicly (creates a public link)
|
||||
|
||||
@@ -18,6 +18,7 @@ For the full list of supported architectures across all modalities, see
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | — |
|
||||
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
|
||||
| Gepard-1.0 | `nineninesix/gepard-1.0` | — (zero-shot default voice) | ✓ (PCM / WAV stream) | `default` only | — |
|
||||
| higgs-audio v2 | `bosonai/higgs-audio-v2-generation-3B-base` | ✓ (`ref_audio`+`ref_text`) | ✓ (codec_streaming) | — | — |
|
||||
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
|
||||
| OmniVoice | `k2-fsa/OmniVoice` | (offline only) | — | — | — |
|
||||
@@ -92,7 +93,7 @@ curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
}' --no-buffer | play -t raw -r 24000 -e signed -b 16 -c 1 -
|
||||
```
|
||||
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 24 kHz for the others).
|
||||
Adjust the player's sample rate to match the model (44.1 kHz for Fish Speech, 48 kHz for VoxCPM2, 22.05 kHz for Gepard, 24 kHz for the others).
|
||||
|
||||
For full request-shape documentation (all parameters, response formats, error codes), see the [Speech API reference](https://github.com/vllm-project/vllm-omni/tree/main/docs/serving/speech_api.md).
|
||||
|
||||
@@ -120,6 +121,15 @@ vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remot
|
||||
|
||||
Streaming is on by default via `async_chunk: true` in `vllm_omni/deploy/cosyvoice3.yaml`. Pass `--no-async-chunk` (or `NO_ASYNC_CHUNK=1 ./cosyvoice3/run_server.sh`) for the legacy synchronous path.
|
||||
|
||||
Cross-request Stage-1 flow batching is opt-in. Enable it in the server environment when concurrent requests should share a flow-estimator call:
|
||||
|
||||
```bash
|
||||
export COSYVOICE3_BATCH_FLOW=1
|
||||
vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code
|
||||
```
|
||||
|
||||
Batching preserves output lengths and streaming cache alignment, but the different GEMM shapes can produce small waveform differences compared with processing each request separately. Leave `COSYVOICE3_BATCH_FLOW` unset (or set it to `0`) when request-independent numerical behavior is required. Set `COSYVOICE3_BATCH_FLOW_DEBUG=1` to log the observed group-size distribution and enable detailed Stage-1 profiler scopes; diagnostics are disabled by default to avoid per-step profiling overhead.
|
||||
|
||||
### CLI client
|
||||
|
||||
The client defaults to the official upstream zero-shot prompt, so it runs without extra flags:
|
||||
@@ -203,6 +213,61 @@ python fish_speech/gradio_demo.py --api-base http://localhost:8091 # if server
|
||||
|
||||
---
|
||||
|
||||
## Gepard-1.0
|
||||
|
||||
Single-stage native AR TTS at 22.05 kHz mono. Zero-shot only: omit `voice` or pass `"default"`. Voice cloning from reference audio is not available yet.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
Same NeMo NanoCodec install as the [offline Gepard section](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/text_to_speech/README.md#gepard-10). On a host whose CUDA toolkit cannot build kernels — no `nvcc`/`ninja`, or a consumer Blackwell (`sm_120`) card — also `export VLLM_USE_FLASHINFER_SAMPLER=0` before launch. That env is not a deploy-YAML field.
|
||||
|
||||
### Launch
|
||||
|
||||
```bash
|
||||
vllm-omni serve nineninesix/gepard-1.0 --omni --port 8091 --trust-remote-code \
|
||||
--stage-init-timeout 900 \
|
||||
--deploy-config vllm_omni/deploy/gepard.yaml
|
||||
# or:
|
||||
./gepard/run_server.sh
|
||||
```
|
||||
|
||||
`--stage-init-timeout 900` matches the online e2e fixture and the offline example. Serve defaults to 300s, which is often too short for a cold download of the talker plus NanoCodec.
|
||||
|
||||
The packaged `vllm_omni/deploy/gepard.yaml` must be passed with `--deploy-config`. The checkpoint self-identifies as `qwen3_5_text`, so omitting the YAML launches a diffusion fallback instead of the Gepard pipeline. The YAML sets `async_chunk: false`, `max_num_seqs: 4`, and currently pins `seed: 42`, so serving is deterministic by default until that YAML seed is removed. Pass an explicit per-request `seed` in tests and clients rather than depending on either default.
|
||||
|
||||
### Sending requests
|
||||
|
||||
```bash
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking."
|
||||
|
||||
python examples/online_serving/text_to_speech/gepard/speech_client.py \
|
||||
--text "Hello, this is Gepard speaking." --seed 7 --stream --output output.pcm
|
||||
```
|
||||
|
||||
Gepard accepts `input` (required), `voice` (`"default"`), `response_format`, `stream` / `stream_format`, `max_new_tokens`, and `seed`. Unsupported fields include `speed`, `language`, `instructions`, `task_type`, `ref_audio`, `ref_text`, `extra_params`, and `word_timestamps`.
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8091/v1/audio/speech \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"input": "Hello, this is Gepard speaking.",
|
||||
"voice": "default",
|
||||
"seed": 7
|
||||
}' --output output.wav
|
||||
```
|
||||
|
||||
### Notes
|
||||
|
||||
- Output: 22.05 kHz mono. `max_new_tokens` is a **frame** budget (1 token = 1 frame = 1024 samples ≈ 46.4 ms at 21.5 fps; adapter bounds 1..4096).
|
||||
- Supported request fields: `input` (required), `voice` (`default` only), `response_format` (`wav` default; `wav/pcm/flac/mp3` non-streaming; `opus` 400 because 22.05 kHz is not an Opus sample rate; streaming `pcm/wav` only), `stream` / `stream_format`, `max_new_tokens`, `seed`.
|
||||
- Unsupported: `speed`, `extra_params` (including `temperature`/`top_p`/`top_k`), `ref_audio`, `ref_text`, `speaker_embedding`, `task_type`, `instructions`, `language`, and `word_timestamps`.
|
||||
- Concurrent requests at `max_num_seqs: 4` are supported. Native-AR recompute preemption is a known limitation of this architecture (a request that is preempted mid-generation can resume incorrectly); keep concurrency at or below `max_num_seqs` and treat preemption as out of scope until the platform fix lands.
|
||||
- Optional comparison against the upstream Gepard reference server needs Blackwell/Hopper + CUDA 13 + Postgres and is not part of CI.
|
||||
- For the request-field table, see the [Speech API Gepard section](https://github.com/vllm-project/vllm-omni/blob/main/docs/serving/speech_api.md#gepard-10).
|
||||
|
||||
---
|
||||
|
||||
## higgs-audio v2
|
||||
|
||||
2-stage TTS at 24 kHz: a vLLM-native Llama-3.2-3B talker with a DualFFN audio expert (Stage 0) feeding a HiggsAudio codec decoder (Stage 1) that streams chunks back to the client.
|
||||
@@ -348,6 +413,22 @@ vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091
|
||||
./qwen3_tts/run_server.sh Base
|
||||
```
|
||||
|
||||
For a local deployment with multiple API frontend processes sharing one set
|
||||
of TTS stage engines, add `--api-server-count`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
|
||||
--omni --api-server-count 2 --port 8091
|
||||
```
|
||||
|
||||
This mode supports local EngineCore stages with one local process group per
|
||||
replica, including stages configured with multiple `num_replicas`. Headless,
|
||||
remote, intra-replica data-parallel, and Ray stage deployments are not
|
||||
supported. Runtime voice upload and deletion are disabled with multiple
|
||||
API frontends because their registries are process-local. Built-in voices,
|
||||
inline `ref_audio`, and voices restored from `custom_voice_dir` at startup are
|
||||
supported.
|
||||
|
||||
### Executor backend
|
||||
|
||||
Single-GPU serves now default to the uniproc executor (lower IPC overhead, the Base cloning use case from [#2603](https://github.com/vllm-project/vllm-omni/issues/2603) / [#2604](https://github.com/vllm-project/vllm-omni/pull/2604)). `vllm_omni/deploy/qwen3_tts.yaml` is the only Qwen3-TTS deploy config; pass `--deploy-config <path>` to override.
|
||||
@@ -413,6 +494,10 @@ checkpoint. When a request names an uploaded voice, the server infers
|
||||
`task_type="Base"`. Built-in presets such as `vivian` and `ryan` remain
|
||||
CustomVoice speakers and require a CustomVoice checkpoint.
|
||||
|
||||
The runtime upload and delete routes require a single API frontend; with
|
||||
`--api-server-count > 1`, use inline `ref_audio` or restore precomputed voices
|
||||
from `custom_voice_dir` at startup.
|
||||
|
||||
### Precomputed custom voices
|
||||
|
||||
For reused Base voice-cloning speakers, precompute the reference artifacts once
|
||||
|
||||
@@ -204,6 +204,91 @@ omni = Omni(
|
||||
| `--linear-backend cutlass` | str | auto | Select the validated CUTLASS linear backend for supported ModelOpt NVFP4 or mixed FP8/NVFP4 diffusion stages |
|
||||
| `--moe-backend cutlass` | str | auto | Select the validated CUTLASS MoE backend for supported ModelOpt mixed MoE checkpoints |
|
||||
|
||||
## Cosmos3 Mixed-Precision Schedule
|
||||
|
||||
The experimental Cosmos3 schedule can use the checkpoint-native quantized GEMM for middle denoising
|
||||
steps and dense 16-bit activation execution for selected first and last steps.
|
||||
The schedule supports serialized ModelOpt FP8 and NVFP4 checkpoints:
|
||||
|
||||
| Detected weight format | Native middle steps | A16 boundary steps |
|
||||
|------------------------|---------------------|--------------------|
|
||||
| FP8 | W8A8 | W8A16 |
|
||||
| NVFP4 | W4A4 | W4A16 |
|
||||
|
||||
The native and A16 paths share one quantized weight representation. The A16
|
||||
path dequantizes the live backend weight and calls `F.linear`; it does not keep
|
||||
a second checkpoint-weight snapshot. Scheduled FP8 currently requires
|
||||
serialized tensorwise scales and a backend that retains canonical FP8 weights.
|
||||
Scheduled NVFP4 requires a supported CUTLASS-compatible native or FlashInfer
|
||||
layout. SmoothQuant, Marlin-repacked FP8, and per-channel/per-token FP8
|
||||
schedules fail closed.
|
||||
|
||||
The reasoner uses dense A16 execution by default when its weights are FP8 or
|
||||
NVFP4. Set the nested `reasoner` field to `native` to retain the checkpoint-native
|
||||
W8A8/W4A4 path. A BF16 reasoner remains BF16 in either mode. The schedule
|
||||
currently requires tensor parallel size 1 and one active request per worker,
|
||||
does not support HSDP, and does not support block-scaled FP8. Its live-weight
|
||||
design is compatible with model-level and standard layer-wise offload, but
|
||||
those paths still require GPU end-to-end validation. Distributed layer-wise
|
||||
offload is rejected because its direct loader bypasses ModelOpt post-load
|
||||
transformations.
|
||||
|
||||
A checkpoint can opt in by adding a versioned policy to its authoritative
|
||||
`transformer/config.json`. The same policy schema is used for FP8 and NVFP4:
|
||||
|
||||
```json
|
||||
{
|
||||
"quantization_config": {
|
||||
"quant_method": "modelopt",
|
||||
"quant_algo": "FP8",
|
||||
"runtime": {
|
||||
"diffusion_step_policy": {
|
||||
"schema_version": 1,
|
||||
"type": "first_last_n",
|
||||
"index_space": "denoising_loop_iteration",
|
||||
"scope": ["transformer"],
|
||||
"default_mode": "native",
|
||||
"first_steps": {"count": 3, "mode": "a16"},
|
||||
"last_steps": {"count": 3, "mode": "a16"},
|
||||
"overlap": "a16",
|
||||
"reasoner": "a16"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The example preserves the checkpoint's existing `quant_method` and
|
||||
`quant_algo`; only `runtime` is added.
|
||||
|
||||
When present and valid, the checkpoint policy is the default. An explicit
|
||||
runtime object overrides it. To retain ordinary checkpoint-native execution
|
||||
for every step, disable only the schedule:
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/Cosmos3-Nano-modelopt \
|
||||
--omni \
|
||||
--additional-config '{"cosmos3_mixed_precision":{"enabled":false}}'
|
||||
```
|
||||
|
||||
This does not disable checkpoint quantization; it prevents installation of the
|
||||
mixed-step runtime.
|
||||
|
||||
```bash
|
||||
vllm serve /path/to/Cosmos3-Nano-modelopt \
|
||||
--omni \
|
||||
--additional-config \
|
||||
'{"cosmos3_mixed_precision":{"first_steps":3,"last_steps":3,"reasoner":"a16"}}'
|
||||
```
|
||||
|
||||
The presence of `cosmos3_mixed_precision` supplies an explicit runtime
|
||||
override; an empty object uses the defaults shown above.
|
||||
The schedule uses vLLM's selected native backend and fails during loading if
|
||||
its live weight layout cannot support A16 dequantization.
|
||||
The runtime selects FP8 or NVFP4 from the checkpoint's ModelOpt configuration.
|
||||
BF16 linears are untouched. Mixed FP8/NVFP4 checkpoints are not supported by
|
||||
this schedule.
|
||||
|
||||
## Validation and Notes
|
||||
|
||||
1. Compare the ModelOpt checkpoint against the BF16 baseline with the same
|
||||
|
||||
@@ -6,22 +6,25 @@ W4A4 MXFP4 (Microscaling FP4) quantizes both weights and activations to FP4
|
||||
(`float4_e2m1fn_x2`, packed 2 values per byte) using the OCP MX format: groups
|
||||
of 32 K-dimension elements share a single `float8_e8m0fnu` exponent scale.
|
||||
|
||||
On Ascend, single-scale MXFP4 also supports Universal Optimal Scaling
|
||||
(UOS, `Qmax=7.25`), described in the single-scale scaling section below.
|
||||
|
||||
vLLM-Omni provides two quantization methods with different scale structures:
|
||||
|
||||
| Method | Scale structure | Mode | Use case |
|
||||
|--------|----------------|------|----------|
|
||||
| `mxfp4` | Single-scale (per-32 fine only) | Online only | Quick accuracy baseline; no checkpoint prep needed |
|
||||
| `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | Production; better accuracy; offline recommended |
|
||||
| -------- | ---------------- | ------ | ---------- |
|
||||
| `mxfp4` | Single-scale (per-32 fine only) | Online + pre-quantized single-scale weights | Online baseline, or an existing single-scale checkpoint in the layout below |
|
||||
| `mxfp4_dualscale` | Dual-scale (fine per-32 + coarse per-512 + per-channel `mul_scale`) | Online + Offline | W4A4 with dual-level scales; offline mode loads calibrated Smooth tensors |
|
||||
|
||||
!!! tip "Recommended: `mxfp4_dualscale` offline"
|
||||
For production deployments, use the `mxfp4_dualscale` offline mode with a
|
||||
!!! tip "Existing W4A4-only DualScale deployments"
|
||||
For W4A4-only deployments, the existing `mxfp4_dualscale` offline mode uses a
|
||||
pre-quantized checkpoint produced by msModelSlim. Offline checkpoints load
|
||||
calibrated `mul_scale` tensors from disk, providing measurably better accuracy
|
||||
than any online method. The one-time preprocessing cost amortises across all
|
||||
subsequent inference runs.
|
||||
calibrated `mul_scale` tensors from disk. Quality depends on the model,
|
||||
calibration recipe and BF16 fallback policy; evaluate the checkpoint against
|
||||
your quality target before deployment.
|
||||
|
||||
Use `mxfp4` online only for quick experimentation where preprocessing time
|
||||
is not acceptable and accuracy loss is tolerable.
|
||||
W4A8 layer/step fallback requires single-scale `mxfp4`, online or offline.
|
||||
DualScale supports W4A4 and BF16 routing only.
|
||||
|
||||
!!! warning "Online single-scale ≠ Offline dual-scale"
|
||||
`mxfp4_dualscale` offline mode uses `NPUMxfp4DualScaleLinearMethod`:
|
||||
@@ -34,61 +37,244 @@ vLLM-Omni provides two quantization methods with different scale structures:
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | Support |
|
||||
|--------|---------|
|
||||
| NVIDIA Blackwell GPU (SM 100+) | ⭕ |
|
||||
| NVIDIA Ada/Hopper GPU (SM 89+) | ⭕ |
|
||||
| NVIDIA Ampere GPU (SM 80+) | ⭕ |
|
||||
| AMD ROCm (gfx950 / MI355X) | ✅ |
|
||||
| Intel XPU | ⭕ |
|
||||
| Ascend NPU (Atlas 950 A5) | ✅ |
|
||||
The following matrix describes these vLLM-Omni implementations, rather than
|
||||
hardware support for the MXFP4 format in general.
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported, `⭕` not verified in this guide.
|
||||
| Device | Single-scale online OCP | Single-scale offline | UOS / W4A8 fallback | DualScale |
|
||||
| -------- | -------- | -------- | -------- | -------- |
|
||||
| Ascend NPU (Atlas 950 A5) | ✅ | ✅ | ✅ | ✅ |
|
||||
| AMD ROCm (gfx950 / MI355X) | ✅ | ❌ | ❌ | ❌ |
|
||||
| Other AMD architectures | ❌ | ❌ | ❌ | ❌ |
|
||||
| NVIDIA GPU | ❌ | ❌ | ❌ | ❌ |
|
||||
| Intel XPU | ❌ | ❌ | ❌ | ❌ |
|
||||
|
||||
Legend: `✅` supported by the implementation, `❌` unsupported.
|
||||
ROCm uses the existing AITER online path and requires `mxfp4_scale_alg=0`.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
### Diffusion Model (Wan2.2)
|
||||
|
||||
| Model | Online | Offline | Notes |
|
||||
|-------|--------|---------|-------|
|
||||
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade (`transformer` + `transformer_2`); both transformers quantized with the same config |
|
||||
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4_dualscale` | MoE cascade; same scheme as T2V-A14B |
|
||||
| Wan2.2-TI2V-5B | ❌ | ❌ | Parameter count too small; W4A4 causes unacceptable accuracy loss |
|
||||
| ------- | -------- | --------- | ------- |
|
||||
| Wan2.2-T2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4` / `mxfp4_dualscale` | Cascade experts resolve checkpoint metadata independently; explicit global runtime policy applies to both |
|
||||
| Wan2.2-I2V-A14B | `mxfp4` / `mxfp4_dualscale` | `mxfp4` / `mxfp4_dualscale` | Per-expert configuration uses the same interface as T2V; validate I2V checkpoints separately |
|
||||
| Wan2.2-TI2V-5B | Requires validation | Requires validation | This guide does not establish checkpoint compatibility or generation quality for TI2V-5B |
|
||||
|
||||
The choice between `mxfp4` and `mxfp4_dualscale` in **online mode** is about
|
||||
quantization quality, not model compatibility — both work on cascade (A14B) and
|
||||
single-transformer models alike, the same as `mxfp8` online:
|
||||
Both methods can be integrated with single-transformer and cascade models.
|
||||
This alone does not establish support or quality for every model variant:
|
||||
|
||||
- `mxfp4`: single-scale, lower overhead, simpler compute, online only
|
||||
- `mxfp4_dualscale`: dual-scale + optional BF16 fallback, better accuracy, online **and** offline
|
||||
- `mxfp4`: single-scale; online and pre-quantized weights, with optional W4A8 fallback on NPU
|
||||
- `mxfp4_dualscale`: dual-scale; online and offline, with BF16 fallback
|
||||
|
||||
**Offline** checkpoints for A14B are always in `mxfp4_dualscale` format (produced
|
||||
by the merge script); there is no offline `mxfp4` single-scale format.
|
||||
The existing `merge_mxfp4_dualscale_checkpoint.py` produces **offline
|
||||
`mxfp4_dualscale`** checkpoints. Single-scale weights must satisfy the separate
|
||||
layout described below; changing a DualScale checkpoint's method name to
|
||||
`mxfp4` does not convert its weights. Prepare single-scale checkpoints with
|
||||
external tooling. Real exported checkpoint compatibility and model quality
|
||||
require validation of that checkpoint; the synthetic tests do not establish
|
||||
UOS export or video acceptance.
|
||||
|
||||
!!! note "Per-layer BF16 fallback in offline cascade models"
|
||||
The A14B offline checkpoint uses `quant_method: mxfp4_dualscale`. Most
|
||||
An A14B offline DualScale checkpoint uses `quant_method: mxfp4_dualscale`. Most
|
||||
linear layers are stored as W4A4 MXFP4 DualScale; precision-sensitive layers
|
||||
retain their original BF16 weights and are listed in `ignored_layers` inside
|
||||
each transformer's `config.json`. The two transformers may have different
|
||||
`ignored_layers` sets — the pipeline reads each transformer's own `config.json`
|
||||
and rebuilds the config locally when they differ, so routing is always
|
||||
per-transformer-accurate.
|
||||
|
||||
!!! warning "TI2V-5B not supported"
|
||||
Wan2.2-TI2V-5B is excluded from W4A4 quantization. Its smaller parameter
|
||||
count makes it significantly more sensitive to 4-bit quantization noise,
|
||||
resulting in unacceptable accuracy loss. Use [MXFP8](mxfp8.md) for TI2V-5B.
|
||||
and resolves the storage format and BF16 layer policy independently.
|
||||
|
||||
## Configuration
|
||||
|
||||
### W4A8 fallback at selected layers and denoising steps (Ascend NPU)
|
||||
|
||||
For single-scale `mxfp4`, set `w4a8_fallback_steps` to a list of zero-based
|
||||
request denoising loop indices (not scheduler timestep values).
|
||||
Set `w4a8_fallback_layers` to exact runtime Linear paths, for example
|
||||
`["blocks.10.attn1.to_qkv", "blocks.12.ffn.net_2"]`. A selected layer uses W4A8
|
||||
at every step; at a selected step, all other quantized layers also use W4A8.
|
||||
The rules are combined with **OR**. W4A8 uses MXFP8 activations with MXFP4
|
||||
weights. Omitting both lists or passing `[]` keeps the existing W4A4 behavior. Existing
|
||||
`ignored_layers` take priority: **BF16 > (layer OR step) W4A8 > W4A4**.
|
||||
DualScale's existing leading BF16 block rule is unchanged; it does not enable
|
||||
W4A8.
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
|
||||
omni = Omni(
|
||||
model="<Wan2.2-T2V-A14B-Diffusers>",
|
||||
quantization_config={
|
||||
"method": "mxfp4",
|
||||
"w4a8_fallback_layers": ["blocks.10.attn1.to_qkv"],
|
||||
"w4a8_fallback_steps": [0, 1, 38, 39],
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
Both policies also work with offline single-scale `mxfp4`. Any nonempty
|
||||
W4A8 list with `mxfp4_dualscale`, online or offline, raises an explicit error
|
||||
at configuration construction. Empty lists preserve its existing W4A4 path.
|
||||
Each expert's checkpoint still determines its serialized
|
||||
format and BF16 layer routing. A serialized checkpoint config is the complete
|
||||
storage contract: `ignored_layers`, or its `modules_to_not_convert` alias,
|
||||
selects the BF16 layers, and omitting both means no explicit BF16 layers. A
|
||||
method-only checkpoint value such as `"mxfp4"` or
|
||||
`{"quant_method": "mxfp4"}` carries no storage policy and leaves a caller's
|
||||
online `ignored_layers` unchanged. An explicitly supplied quantization config
|
||||
retains its W4A8 layer and step lists when the loader rebuilds either expert's
|
||||
config from disk, including empty lists that disable checkpoint policies.
|
||||
Within an explicit runtime config, an omitted W4A8 list defaults to `[]`;
|
||||
individual omitted fields do not inherit saved policies. Without an explicit
|
||||
runtime config, auto-detection uses the saved checkpoint policies.
|
||||
|
||||
Layer names are relative to each expert (`blocks.10...`, without a
|
||||
`transformer.` or `transformer_2.` prefix). A global config applies the same
|
||||
list to both experts; per-component `transformer`/`transformer_2` configs can
|
||||
select different lists. Wan selects these expert names exactly: a `transformer`
|
||||
entry does not match `transformer_2`; an omitted expert uses `default` if set.
|
||||
A component mapped to `null` (or an unmatched component
|
||||
without a default) stays unquantized and requires a BF16 checkpoint; loading a
|
||||
quantized checkpoint for that disabled component raises an error. A top-level
|
||||
`quantization_config=None` retains checkpoint auto-detection.
|
||||
Names are exact, with no regex, wildcard, block expansion,
|
||||
or checkpoint-name remapping. Wan validates them against its runtime Linear
|
||||
modules and logs the selected paths/count and BF16 overrides per PP rank.
|
||||
Use `attn1.to_qkv` for fused self-attention; `attn1.to_q`, `to_k`, or `to_v`
|
||||
checkpoint names raise an error. Cross-attention `attn2.to_q/to_k/to_v` remains
|
||||
separate. Layer-only fallback does not require a denoise ForwardContext.
|
||||
|
||||
The text-to-video example accepts the JSON directly:
|
||||
|
||||
```bash
|
||||
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 \
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model /path/to/Wan2.2-T2V-A14B-Diffusers \
|
||||
--quantization-config '{"method":"mxfp4","w4a8_fallback_steps":[0,2]}' \
|
||||
--tensor-parallel-size 1 --ulysses-degree 2 --cfg-parallel-size 2 \
|
||||
--enforce-eager --height 480 --width 832 --num-frames 81 \
|
||||
--num-inference-steps 40 --seed 42 --output wan22-w4a8.mp4
|
||||
```
|
||||
|
||||
Use either `--quantization` or `--quantization-config` in this example.
|
||||
Serving accepts the same JSON through `--diffusion-quantization-config`.
|
||||
|
||||
The list uses one request's loop index across both high- and low-noise experts,
|
||||
and both CFG branches see the same index. A new request starts at step zero.
|
||||
Indices outside the request's step range are never selected. A pipeline must
|
||||
publish `ForwardContext.denoise_step_idx`; requesting step fallback without
|
||||
that context raises an error instead of silently ignoring the list.
|
||||
|
||||
Single-scale W4A4 and W4A8 reuse the same prepared `weight` and `weight_scale`.
|
||||
Online loading quantizes the original BF16/FP16 weight once; offline loading
|
||||
packs the numeric FP4 values and reshapes the saved E8M0 scale without
|
||||
requantizing the weight. Only activation precision changes during forward.
|
||||
There is no DualScale decoding, second W4 representation or `w4a8_weight` /
|
||||
`w4a8_weight_scale` cache. No weight quantization runs inside the denoising loop.
|
||||
|
||||
The mixed FP8/FP4 GEMM uses BF16 output and a matching BF16 row bias for
|
||||
compatibility with the target A5 runtime. An FP16 caller receives that result
|
||||
converted back to FP16; its output shape and dtype are preserved.
|
||||
|
||||
The single-scale fallback validation target is eager Wan2.2 T2V A14B on A5.
|
||||
Quality scores, optimal fallback lists, real UOS exports and additional
|
||||
compile/offload combinations require separate evaluation. Existing W4A4-only
|
||||
DualScale still needs complete 512-channel input groups; use TP=1 with
|
||||
sequence/CFG parallelism for its initial multi-card run.
|
||||
|
||||
### `mxfp4` — Pre-Quantized Single-Scale Weights (Ascend NPU)
|
||||
|
||||
Set the following fields in each transformer's `config.json` when a
|
||||
single-scale checkpoint uses calibrated Smooth scaling:
|
||||
|
||||
```json
|
||||
{
|
||||
"quantization_config": {
|
||||
"quant_method": "mxfp4",
|
||||
"is_checkpoint_mxfp4_serialized": true,
|
||||
"require_smooth_scale": true,
|
||||
"ignored_layers": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Each quantized Linear must provide tensors under its model parameter name:
|
||||
|
||||
| Tensor | Checkpoint layout | Meaning |
|
||||
| -------- | ------------------- | --------- |
|
||||
| `weight` | `(N, K)` numeric FP4 values | Packed to FP4 after loading |
|
||||
| `weight_scale` | `(N, ceil(K / 32))`, `uint8` | Raw E8M0 exponent bytes, required |
|
||||
| `mul_scale` | `(K,)`, FP32 | Smooth pre-scale, required when declared above |
|
||||
|
||||
Both W4A4 and W4A8 quantize `x * mul_scale`. The Smooth tensor is converted
|
||||
to the layer's activation dtype after loading. Row-parallel layers load the
|
||||
corresponding input-channel shard. Pre-fused QKV weights must use one shared
|
||||
Smooth tensor for the fused projection.
|
||||
|
||||
For backward compatibility, `require_smooth_scale` defaults to `false`:
|
||||
an absent tensor uses identity scaling, while a present tensor is still loaded.
|
||||
Without a declaration, an absent Smooth tensor cannot be distinguished from a
|
||||
legacy unsmoothed checkpoint. Calibrated checkpoints should declare
|
||||
`require_smooth_scale: true` so missing or misnamed Smooth tensors fail loading;
|
||||
incorrect tensor shapes, non-floating tensors, NaN/Inf and nonpositive values
|
||||
also fail. Values that overflow or underflow to zero in the activation dtype
|
||||
are rejected during weight processing. The flag requires serialized single-scale
|
||||
weights and is rejected for online quantization. It does not make DualScale
|
||||
scales optional or change missing-parameter policies for other backends.
|
||||
Either the checkpoint or an explicit caller can require Smooth: a `true`
|
||||
declaration is retained when per-expert metadata rebuilds the configuration,
|
||||
including when the other source omits the flag or sets it to `false`.
|
||||
|
||||
Prepare the canonical numeric FP4 checkpoint before inference using external
|
||||
tooling. Runtime loading does not parse producer-specific export formats.
|
||||
Retain the producer recipe to identify the weight-generation algorithm and
|
||||
Smooth calibration; tensor storage alone does not establish UOS provenance.
|
||||
|
||||
### `mxfp4` — Single-Scale Block Scaling (OCP / UOS)
|
||||
|
||||
`mxfp4_scale_alg` selects the MXFP4 block-scaling method on NPU.
|
||||
UOS is a scaling rule, not a separate activation dtype or an activation-only
|
||||
algorithm. Both choices use FP4 E2M1 values with shared E8M0 scales.
|
||||
|
||||
| Mode | Weight preparation | A4 activation quantization | W4A8 fallback |
|
||||
| --- | --- | --- | --- |
|
||||
| Online | Selected scaling method, once at loading | Selected scaling method | Same W4; OCP MXFP8 A8 |
|
||||
| Offline | Load existing W4 and scales unchanged | Selected scaling method | Same W4; OCP MXFP8 A8 |
|
||||
|
||||
The available settings are:
|
||||
|
||||
| Setting | MXFP4 scaling method | NPU parameters |
|
||||
| --- | --- | --- |
|
||||
| `0` (default) | OCP MX | `scale_alg=0` |
|
||||
| `2` | UOS (`Qmax=7.25`) | `scale_alg=2, dst_type_max=7.25` |
|
||||
|
||||
Universal Optimal Scaling (UOS, `Qmax=7.25`) is derived in
|
||||
[MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention](https://arxiv.org/abs/2607.24377),
|
||||
Section 4.1. The value `7.25` is the boundary used to select the shared
|
||||
power-of-two scale; the maximum finite E2M1 value remains `6`.
|
||||
This option applies UOS to Linear quantization. It does not implement the
|
||||
paper's full MXAttention pipeline or Pre-Normalization Quantization (PNQ).
|
||||
|
||||
Both use FP4 E2M1 with E8M0 scales, `axis=-1`, `block_size=32`,
|
||||
`round_mode="rint"`. W4A8 always quantizes the activation to
|
||||
`float8_e4m3fn` with `scale_alg=0` and the same grouping/rounding, without
|
||||
UOS parameters. A4 and A8 share the prepared W4 and scale. UOS is currently
|
||||
NPU-only; ROCm retains its existing AITER online algorithm.
|
||||
|
||||
An explicit UOS online configuration is
|
||||
`{"method": "mxfp4", "mxfp4_scale_alg": 2}`. For offline checkpoints the
|
||||
setting affects **only A4 activations**. Existing weights are never requantized:
|
||||
a UOS checkpoint can run UOS W4 + UOS A4 or UOS W4 + OCP A4, and both fall back to
|
||||
UOS W4 + OCP A8. Record the weight-generation algorithm separately from the
|
||||
runtime activation setting. Smooth remains active with either setting. The
|
||||
UOS configuration explicitly selects `2`; attention computation is unchanged.
|
||||
|
||||
### `mxfp4` — Single-Scale Online Mode
|
||||
|
||||
Online mode requires no pre-processing. vLLM-Omni quantizes BF16 weights to
|
||||
MXFP4 at load time using `npu_dynamic_mx_quant`. A single block scale
|
||||
(`float8_e8m0fnu`, one per 32 K elements) is computed on the fly; no
|
||||
calibration `mul_scale` is available. Applies equally to single-transformer
|
||||
and cascade (A14B) models — both transformers in a cascade receive the same
|
||||
quantization config automatically.
|
||||
Online mode quantizes BF16/FP16 weights once at loading and computes activation
|
||||
scales during forward. Each block of 32 K elements shares one E8M0 scale;
|
||||
no calibrated `mul_scale` is available. An explicit global quantization config
|
||||
applies to both cascade experts. Per-component configs can select different
|
||||
policies; checkpoint auto-detection resolves each expert independently.
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
@@ -103,7 +289,7 @@ outputs = omni.generate(
|
||||
|
||||
```bash
|
||||
# Single-transformer or cascade model — same command
|
||||
python text_to_video.py --model <your-model> --quantization mxfp4
|
||||
python examples/offline_inference/text_to_video/text_to_video.py --model <your-model> --quantization mxfp4
|
||||
|
||||
# Online serving
|
||||
vllm serve <your-model> --omni --quantization mxfp4
|
||||
@@ -113,19 +299,19 @@ vllm serve <your-model> --omni --quantization mxfp4
|
||||
|
||||
Online DualScale mode computes both fine and coarse scales on the fly from BF16
|
||||
weights using `npu_dynamic_dual_level_mx_quant`. Applies equally to
|
||||
single-transformer and cascade (A14B) models. Compared to `mxfp4` online,
|
||||
DualScale provides better quantization accuracy at higher compute cost.
|
||||
single-transformer and cascade (A14B) models with compatible Linear layers.
|
||||
Compare quality and performance using the intended model and workload.
|
||||
|
||||
The default configuration keeps the leading 5 transformer blocks in BF16
|
||||
(`num_bf16_fallback_layers=5`). Accuracy evaluation on Wan2.2-A14B shows this
|
||||
is sufficient to meet quality requirements and is the recommended setting.
|
||||
(`num_bf16_fallback_layers=5`). This is an implementation default; select and
|
||||
validate the BF16 policy against your own quality target.
|
||||
|
||||
```python
|
||||
omni = Omni(model="<your-model>", quantization="mxfp4_dualscale")
|
||||
```
|
||||
|
||||
```bash
|
||||
python text_to_video.py --model <your-model> --quantization mxfp4_dualscale
|
||||
python examples/offline_inference/text_to_video/text_to_video.py --model <your-model> --quantization mxfp4_dualscale
|
||||
```
|
||||
|
||||
If accuracy debugging identifies additional precision-sensitive layers, they
|
||||
@@ -136,21 +322,22 @@ omni = Omni(
|
||||
model="<your-model>",
|
||||
quantization_config={
|
||||
"method": "mxfp4_dualscale",
|
||||
"ignored_layers": ["blocks.10.attn1.to_q"], # explicit per-layer override
|
||||
"ignored_layers": ["blocks.10.attn1.to_qkv"], # explicit per-layer override
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
BF16 fallback routing in online mode applies two rules in priority order:
|
||||
|
||||
1. **`ignored_layers`** (explicit per-layer override): any layer whose prefix
|
||||
matches is kept in BF16 regardless of block index.
|
||||
1. **`ignored_layers`** (explicit per-layer override): a listed runtime Linear
|
||||
module is kept in BF16 regardless of block index. Use exact module paths,
|
||||
not arbitrary block prefixes or names ending in `.weight`.
|
||||
2. **`num_bf16_fallback_layers`** (coarse leading-block rule): the first N
|
||||
transformer blocks (`blocks.0` … `blocks.N-1`) fall back to BF16. Defaults
|
||||
to `5` (recommended). Layers outside `blocks.N.*`
|
||||
(e.g. `condition_embedder`) are always quantized.
|
||||
to `5`. Layers outside `blocks.N.*` are quantized unless explicitly listed
|
||||
in `ignored_layers`.
|
||||
|
||||
### `mxfp4_dualscale` — DualScale Offline Mode (Recommended)
|
||||
### `mxfp4_dualscale` — DualScale Offline Mode
|
||||
|
||||
Offline mode loads a pre-quantized DualScale checkpoint from msModelSlim. A
|
||||
preprocessing step converts the raw quantized output to the diffusers format
|
||||
@@ -169,8 +356,8 @@ inside `config.json`. At runtime, each layer's prefix is matched against
|
||||
Each quantized linear layer stores four tensors:
|
||||
|
||||
| Tensor | Shape | dtype | Description |
|
||||
|--------|-------|-------|-------------|
|
||||
| `weight` | `(N, K)` | float8_e4m3fn | FP4 packed (2 values per byte) |
|
||||
| -------- | ------- | ------- | ------------- |
|
||||
| `weight` | `(N, K)` | float8_e4m3fn | Numeric FP4 values; packed into two values per byte after loading |
|
||||
| `weight_scale` | `(N, K//32)` | uint8 | Fine block scale (`float8_e8m0fnu` bit pattern) |
|
||||
| `weight_dual_scale` | `(N, K//512, 1)` | float32 | Coarse block scale |
|
||||
| `mul_scale` | `(K,)` | float32 | Per-input-channel smooth pre-scale (from calibration) |
|
||||
@@ -223,7 +410,7 @@ python vllm_omni/quantization/tools/merge_mxfp4_dualscale_checkpoint.py \
|
||||
```
|
||||
|
||||
| Argument | Description |
|
||||
|----------|-------------|
|
||||
| ---------- | ------------- |
|
||||
| `--model-type` | Model variant: `Wan2.2-T2V-A14B` or `Wan2.2-I2V-A14B` |
|
||||
| `--original-model` | Root directory of the original BF16 diffusers model |
|
||||
| `--quant-path` | Root directory of the msModelSlim quantized output |
|
||||
@@ -251,14 +438,14 @@ The `quantization_config` injected into `config.json` for each transformer:
|
||||
```
|
||||
|
||||
`ignored_layers` lists every linear layer that retains its original BF16 weight,
|
||||
using vllm-omni model parameter names (QKV-fused, FFN underscored, `to_out`
|
||||
using vllm-omni runtime module paths (QKV-fused, FFN underscored, `to_out`
|
||||
unindexed). The exact entries are determined by the quantization tool (msModelSlim)
|
||||
and may differ between `transformer` and `transformer_2` in a cascade model.
|
||||
|
||||
#### Step 3 — Serve
|
||||
|
||||
```bash
|
||||
python text_to_video.py --model /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale
|
||||
python examples/offline_inference/text_to_video/text_to_video.py --model /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale
|
||||
|
||||
# Online serving
|
||||
vllm serve /path/to/Wan2.2-T2V-A14B-MXFP4-DualScale --omni
|
||||
@@ -275,25 +462,32 @@ omni = Omni(model="/path/to/Wan2.2-T2V-A14B-MXFP4-DualScale")
|
||||
|
||||
## Parameters
|
||||
|
||||
### `mxfp4` (single-scale, online only)
|
||||
### `mxfp4` (single-scale, online + offline)
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| ----------- | ------ | --------- | ------------- |
|
||||
| `method` | str | — | `"mxfp4"` |
|
||||
| `ignored_layers` | list[str] | `[]` | Layer prefixes to keep in BF16 |
|
||||
| `mxfp4_scale_alg` | int | `0` | MXFP4 block scaling: `0` OCP MX, `2` UOS (`Qmax=7.25`, NPU only). Online: W4 and A4; offline: A4 only. A8 remains OCP MXFP8 |
|
||||
| `is_checkpoint_mxfp4_serialized` | bool | `false` | Load pre-quantized single-scale weights and scales; declared in checkpoint `config.json` |
|
||||
| `require_smooth_scale` | bool | `false` | Require calibrated Smooth tensors for offline single-scale weights |
|
||||
| `ignored_layers` | list[str] | `[]` | Exact runtime Linear module paths to keep in BF16 |
|
||||
| `w4a8_fallback_layers` | list[str] | `[]` | Exact runtime Linear paths that always use A8; NPU only |
|
||||
| `w4a8_fallback_steps` | list[int] | `[]` | Zero-based denoise steps that use A8; NPU only |
|
||||
|
||||
### `mxfp4_dualscale` (dual-scale, online + offline)
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| ----------- | ------ | --------- | ------------- |
|
||||
| `method` | str | — | `"mxfp4_dualscale"` |
|
||||
| `is_checkpoint_serialized` | bool | `False` | `True` for offline DualScale checkpoints; auto-set from `config.json` when using the preprocessing script |
|
||||
| `ignored_layers` | list[str] | `[]` | Layer prefixes to keep in BF16. **Works in both modes**: offline — populated by the merge script for interleaved sensitive layers; online — user-supplied for explicit per-layer precision override |
|
||||
| `num_bf16_fallback_layers` | int | `5` | **Online mode only**: leading N transformer blocks (`blocks.0` … `blocks.N-1`) kept in BF16. Applied after `ignored_layers`; ignored in offline mode. Default of `5` is the evaluated recommended value for Wan2.2-A14B |
|
||||
| `ignored_layers` | list[str] | `[]` | Exact runtime Linear module paths to keep in BF16. **Works in both modes**: offline — populated by the merge script for interleaved sensitive layers; online — user-supplied for explicit per-layer precision override |
|
||||
| `num_bf16_fallback_layers` | int | `5` | **Online mode only**: leading N transformer blocks (`blocks.0` … `blocks.N-1`) kept in BF16. Applied after `ignored_layers`; ignored in offline mode. Validate the default of `5` against the intended model and quality target |
|
||||
| `w4a8_fallback_layers` | list[str] | `[]` | Must remain empty; DualScale does not support W4A8 fallback |
|
||||
| `w4a8_fallback_steps` | list[int] | `[]` | Must remain empty; DualScale does not support W4A8 fallback |
|
||||
|
||||
#### BF16 fallback priority (online mode)
|
||||
|
||||
```
|
||||
```text
|
||||
for each linear layer:
|
||||
if prefix in ignored_layers → BF16 (explicit override, highest priority)
|
||||
elif block_idx < num_bf16_fallback_layers → BF16 (coarse leading-block rule)
|
||||
@@ -307,8 +501,9 @@ unless they appear in `ignored_layers`.
|
||||
|
||||
1. **Online single-scale (`mxfp4`)** quantizes BF16 weights at load time using
|
||||
`npu_dynamic_mx_quant` (single-scale). No calibration `mul_scale` is
|
||||
available — all output partitions receive an identity pre-scale. No offline
|
||||
checkpoint format exists for this method.
|
||||
available — all output partitions receive an identity pre-scale. Offline
|
||||
single-scale checkpoints use the canonical numeric FP4 layout
|
||||
described above.
|
||||
|
||||
2. **Online dual-scale (`mxfp4_dualscale`, `is_checkpoint_serialized=False`)**
|
||||
quantizes BF16 weights using `npu_dynamic_dual_level_mx_quant` (fine + coarse
|
||||
@@ -316,7 +511,7 @@ unless they appear in `ignored_layers`.
|
||||
explicit `ignored_layers` stay in BF16 for accuracy.
|
||||
|
||||
3. **Offline dual-scale (`mxfp4_dualscale`, `is_checkpoint_serialized=True`)** —
|
||||
**recommended for production** — loads four tensors per quantized layer: FP4
|
||||
loads four tensors per quantized layer: FP4
|
||||
weight, fine scale (`uint8` reinterpreted as `float8_e8m0fnu`), coarse scale
|
||||
(`float32`), and per-input-channel `mul_scale` (`float32`). BF16 fallback
|
||||
layers have no quantization tensors and are routed via `ignored_layers`.
|
||||
@@ -325,21 +520,21 @@ unless they appear in `ignored_layers`.
|
||||
layout as `float8_e8m0fnu`) and reinterpreted at load time without a lossy
|
||||
float32 round-trip.
|
||||
|
||||
5. **Cascade model config propagation**: in a cascade model (transformer +
|
||||
transformer_2), vLLM-Omni reads each transformer's own `config.json` and
|
||||
rebuilds the quant config locally when `ignored_layers` differs between
|
||||
transformers, ensuring per-layer routing is accurate for each. The first
|
||||
transformer's config is propagated to `od_config` so the second transformer
|
||||
can reuse it as a starting point.
|
||||
5. **Cascade model configuration**: checkpoint auto-detection resolves each
|
||||
expert's own `config.json` independently; the first expert's detected config
|
||||
is not reused as the second expert's policy. An explicit global runtime
|
||||
config applies to both experts, while per-component configs select policies
|
||||
by the exact `transformer` / `transformer_2` names. Serialized checkpoint
|
||||
metadata still determines each expert's storage format and BF16 layers.
|
||||
|
||||
6. **Self-attention QKV fusion**: Q, K, V projection weights are fused into a
|
||||
single `QKVParallelLinear` layer at runtime. `ignored_layers` entries use the
|
||||
fused name (`attn1.to_qkv`), written automatically by the merge script.
|
||||
|
||||
7. W4A4 carries higher quantization noise than W8A8 (16 vs 256 levels). The
|
||||
DualScale offline method mitigates this with calibrated `mul_scale` smooth
|
||||
quantization. Use `ignored_layers` and `num_bf16_fallback_layers` to trade
|
||||
off compression vs. accuracy for precision-sensitive layers.
|
||||
7. FP4 has fewer representable values than FP8, so evaluate quantization
|
||||
error and generation quality for the chosen recipe. Offline Smooth scaling
|
||||
and BF16 fallback can help with sensitive layers.
|
||||
`num_bf16_fallback_layers` applies only to online DualScale.
|
||||
|
||||
## Adapting MXFP4 for a New Model
|
||||
|
||||
@@ -350,27 +545,26 @@ a merge script for offline checkpoints.
|
||||
|
||||
### Step 1 — Discover runtime layer names
|
||||
|
||||
`ignored_layers` entries must match the **runtime parameter names** used inside
|
||||
vllm-omni, which may differ from the names stored in the diffusers checkpoint.
|
||||
The canonical source of truth is the model's own `named_parameters()`.
|
||||
`ignored_layers` entries use exact **runtime Linear module paths**, without
|
||||
`.weight`. Inspect the actual transformer instance in model initialization or
|
||||
a developer harness; `Omni` does not expose an `omni.pipeline.transformer` API.
|
||||
For a transformer instance named `transformer`:
|
||||
|
||||
```python
|
||||
from vllm_omni import Omni
|
||||
from vllm.model_executor.layers.linear import LinearBase
|
||||
|
||||
# Load the model without quantization to inspect parameter names.
|
||||
omni = Omni(model="/path/to/your-model") # no --quantization flag
|
||||
for name, _ in omni.pipeline.transformer.named_parameters():
|
||||
if "weight" in name and "scale" not in name:
|
||||
for name, module in transformer.named_modules():
|
||||
if isinstance(module, LinearBase):
|
||||
print(name)
|
||||
```
|
||||
|
||||
Compare the printed names against the diffusers checkpoint keys
|
||||
Compare these module paths against the corresponding diffusers checkpoint keys
|
||||
(`safetensors.safe_open` or `torch.load`) to identify any renames your model
|
||||
applies. Common patterns that differ in Wan2.2 (and may appear in other
|
||||
models):
|
||||
|
||||
| Diffusers checkpoint name | vllm-omni runtime name | Reason |
|
||||
|---------------------------|------------------------|--------|
|
||||
| --------------------------- | ------------------------ | -------- |
|
||||
| `attn1.to_q`, `attn1.to_k`, `attn1.to_v` | `attn1.to_qkv` | Self-attention Q/K/V fused into `QKVParallelLinear` |
|
||||
| `ffn.net.0.proj` | `ffn.net_0.proj` | Dots in sub-module names replaced with underscores |
|
||||
| `ffn.net.2` | `ffn.net_2` | Same underscore rule |
|
||||
@@ -382,8 +576,9 @@ fused runtime parameters.
|
||||
|
||||
!!! warning "Partial QKV fallback is not allowed"
|
||||
If your model fuses Q, K, V into a single layer, `ignored_layers` must
|
||||
include **all three or none**. A partial fallback (e.g. `to_q` in BF16 but
|
||||
`to_k`, `to_v` quantized) cannot be expressed at runtime because they share
|
||||
name the fused runtime module (for Wan, `attn1.to_qkv`). The source
|
||||
checkpoint must keep **all three or none** in BF16. A partial fallback
|
||||
(e.g. `to_q` in BF16 but `to_k`, `to_v` quantized) cannot be expressed at runtime because they share
|
||||
one `QKVParallelLinear`. The merge script enforces this and raises an error
|
||||
if only some of the trio appear as non-quantized.
|
||||
|
||||
@@ -392,12 +587,13 @@ fused runtime parameters.
|
||||
#### Online mode
|
||||
|
||||
Pass `ignored_layers` directly in the quantization config using the **runtime
|
||||
names** discovered in Step 1. No code changes to the model are required.
|
||||
names** discovered in Step 1. This assumes the model already passes its
|
||||
quantization config and module prefixes to compatible vLLM Linear layers.
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="/path/to/your-model",
|
||||
quantization={
|
||||
quantization_config={
|
||||
"method": "mxfp4_dualscale",
|
||||
"ignored_layers": [
|
||||
"blocks.0.attn1.to_qkv", # runtime name, not diffusers name
|
||||
@@ -409,9 +605,14 @@ omni = Omni(
|
||||
```
|
||||
|
||||
```bash
|
||||
# CLI does not support list-typed ignored_layers directly.
|
||||
# Use the Python API or set ignored_layers in config.json (offline).
|
||||
python your_script.py --model /path/to/your-model --quantization mxfp4_dualscale
|
||||
# The text-to-video example accepts list-valued fields through JSON.
|
||||
python examples/offline_inference/text_to_video/text_to_video.py \
|
||||
--model /path/to/your-model \
|
||||
--quantization-config '{"method":"mxfp4_dualscale","ignored_layers":["blocks.0.attn1.to_qkv"]}'
|
||||
|
||||
# Serving accepts the same configuration.
|
||||
vllm serve /path/to/your-model --omni \
|
||||
--diffusion-quantization-config '{"method":"mxfp4_dualscale","ignored_layers":["blocks.0.attn1.to_qkv"]}'
|
||||
```
|
||||
|
||||
The `num_bf16_fallback_layers` coarse rule is an alternative to listing layers
|
||||
@@ -436,9 +637,13 @@ the merge script is correct. The injected block:
|
||||
}
|
||||
```
|
||||
|
||||
To add a layer manually (e.g. to pin an additional layer to BF16 without
|
||||
re-running the merge script), edit `config.json` inside the transformer
|
||||
subfolder. Use runtime names, not diffusers checkpoint names.
|
||||
Adding an offline BF16 fallback layer requires both the original BF16 weight
|
||||
from the base model and a matching `ignored_layers` entry. Regenerate the
|
||||
checkpoint with the revised storage policy, restoring the original weight
|
||||
(and bias, if present) and removing that layer's quantization tensors.
|
||||
Editing `config.json` alone does not reconstruct BF16 weights from stored FP4
|
||||
values. For fused QKV, restore all three projections together. Use runtime
|
||||
module paths in `ignored_layers`.
|
||||
|
||||
### Step 3 — Write a merge script for offline mode
|
||||
|
||||
@@ -474,6 +679,7 @@ renaming rules and collect the results.
|
||||
!!! tip "Validate before serving"
|
||||
After producing the offline checkpoint, load it without a `--quantization`
|
||||
flag and verify that vLLM-Omni auto-detects the correct method. Check that
|
||||
the layer count reported in the startup log matches expectations: quantized
|
||||
layer count + `ignored_layers` count should equal total linear layer count.
|
||||
Any mismatch indicates a name-mapping bug in the merge script.
|
||||
each intended runtime Linear module resolves to the expected quantized or
|
||||
BF16 method and that every BF16 entry has original BF16 weights. Account
|
||||
for fused modules and, in online DualScale, the leading-block fallback rule;
|
||||
the raw length of `ignored_layers` alone is not a reliable layer count.
|
||||
@@ -13,7 +13,7 @@ For the internal architecture and backend extension points, see the
|
||||
| Mode | Guide | Description | Methods |
|
||||
| ------ | ------- | ------------- | --------- |
|
||||
| Online quantization | [Online Quantization](online.md) | vLLM-Omni computes quantized weights and scales while loading the model. | FP8 W8A8, Int8 W8A8, BitsAndBytes W4, MXFP8 W8A8, MXFP4 W4A4 |
|
||||
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion Flash Attention tensors during inference. | FP8 FA |
|
||||
| Runtime attention quantization | [Quantized KV Cache](quantized_kvcache.md) | vLLM-Omni dynamically quantizes eligible diffusion attention tensors during inference. | FP8, MXFP8, MXFP4 |
|
||||
| Pre-quantized checkpoints | Method-specific guides | The checkpoint or an offline quantizer provides quantized weights and scales before serving. | ModelOpt, AutoRound, TorchAO, msModelSlim, serialized Int8, offline MXFP8, offline MXFP4 DualScale |
|
||||
|
||||
## Hardware Support
|
||||
|
||||
@@ -2,41 +2,36 @@
|
||||
|
||||
## Overview
|
||||
|
||||
In DiT-based image and video generation, Flash Attention can take a large share
|
||||
of denoising time, especially for high-resolution or long-frame workloads.
|
||||
vLLM-Omni supports online FP8 quantization for eligible diffusion Flash
|
||||
Attention (FA) to reduce FA latency while keeping model weights in their
|
||||
original dtype.
|
||||
In DiT-based image and video generation, attention can take a large share of
|
||||
denoising time, especially for high-resolution or long-frame workloads.
|
||||
vLLM-Omni supports online Q/K/V quantization for eligible diffusion attention
|
||||
paths while keeping model weights in their original dtype.
|
||||
|
||||
This feature is configured through `diffusion_kv_cache_dtype` on
|
||||
`OmniDiffusionConfig` (CLI: `--diffusion-kv-cache-dtype`). It is intentionally
|
||||
**not** the same as vLLM's `--kv-cache-dtype`, which controls autoregressive
|
||||
language-model KV cache storage and defaults to `"auto"`. Diffusion FA
|
||||
quantization uses the dedicated diffusion flags so omni serve does not inherit
|
||||
that default.
|
||||
Runtime attention precision is configured through `diffusion_kv_cache_dtype`
|
||||
on `OmniDiffusionConfig` (CLI: `--diffusion-kv-cache-dtype`). This setting is
|
||||
separate from vLLM's `--kv-cache-dtype`, which controls autoregressive
|
||||
language-model KV cache storage.
|
||||
|
||||
In vLLM-Omni diffusion pipelines, this is a runtime FA path: Q/K/V tensors are
|
||||
dynamically quantized before the attention operator. It does not quantize model
|
||||
weights and is separate from [FP8 W8A8](fp8.md), [Int8 W8A8](int8.md), or
|
||||
pre-quantized checkpoint formats.
|
||||
|
||||
If `diffusion_kv_cache_dtype` is not set, behavior is unchanged and attention
|
||||
runs in the native dtype.
|
||||
If `diffusion_kv_cache_dtype` is not set, attention runs in the native dtype.
|
||||
|
||||
## Hardware Support
|
||||
|
||||
| Device | FP8 FA |
|
||||
|--------|--------|
|
||||
| Ascend NPU | ✅ |
|
||||
| NVIDIA GPU | ❌ |
|
||||
| AMD ROCm | ❌ |
|
||||
| Intel XPU | ❌ |
|
||||
| Device | Dense FP8 | Dense MXFP8/MXFP4 | BSA FP8/MXFP4 |
|
||||
|--------|------------|-------------------|----------------|
|
||||
| Ascend NPU | ✅ | ✅ | ✅ |
|
||||
| NVIDIA GPU | ❌ | ❌ | ❌ |
|
||||
| AMD ROCm | ❌ | ❌ | ❌ |
|
||||
| Intel XPU | ❌ | ❌ | ❌ |
|
||||
|
||||
Legend: `✅` supported, `❌` unsupported.
|
||||
|
||||
FP8 FA is currently implemented only for the NPU Flash Attention backend. Other
|
||||
backends do not support `diffusion_kv_cache_dtype="fp8"` for diffusion attention
|
||||
and fall back to native dtype execution.
|
||||
These runtime methods are implemented only by the NPU Flash Attention and BSA
|
||||
backends. Other backends reject an incompatible explicit configuration.
|
||||
|
||||
## Model Type Support
|
||||
|
||||
@@ -44,7 +39,7 @@ and fall back to native dtype execution.
|
||||
|
||||
| Model | Scope | Status | Notes |
|
||||
|-------|-------|--------|-------|
|
||||
| Wan2.2 | Eligible DiT full-attention FA on Ascend NPU | Tested | Compare quality and latency against a BF16 baseline before production use |
|
||||
| Wan2.2 | Self-attention on Ascend NPU | NPU interface and T2V smoke tested | Compare quality and latency against a BF16 baseline before production use |
|
||||
| Other diffusion models | Eligible DiT full-attention FA on Ascend NPU | Not tested | You can try `diffusion_kv_cache_dtype="fp8"`; tune `diffusion_kv_cache_skip_steps` and `diffusion_kv_cache_skip_layers` when higher precision is needed |
|
||||
|
||||
### Multi-Stage Omni/TTS Model (Qwen3-Omni, Qwen3-TTS)
|
||||
@@ -105,12 +100,13 @@ The legacy keyword aliases `kv_cache_dtype`, `kv_cache_skip_steps`, and
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `diffusion_kv_cache_dtype` | str \| None | `None` | Set to `"fp8"` to enable dynamic FP8 FA on supported attention backends |
|
||||
| `diffusion_kv_cache_dtype` | str \| None | `None` | Runtime attention method. Dense FA supports `fp8`, `mxfp8`, and `mxfp4`; BSA supports `fp8` and `mxfp4` |
|
||||
| `diffusion_kv_cache_skip_steps` | str \| None | `None` | Denoising step selector to keep in native dtype, for example `"0,1,4-6"` |
|
||||
| `diffusion_kv_cache_skip_layers` | str \| None | `None` | Transformer layer selector to keep in native dtype, for example `"0-2,10"` |
|
||||
|
||||
Selectors use comma-separated integers and inclusive ranges. Listed steps or
|
||||
layers skip FP8 FA; all other eligible full-attention forwards use the FP8 path.
|
||||
layers run floating-point attention through the selected Dense or BSA backend;
|
||||
all other eligible forwards use `diffusion_kv_cache_dtype`.
|
||||
|
||||
## Validation and Notes
|
||||
|
||||
@@ -123,3 +119,60 @@ layers skip FP8 FA; all other eligible full-attention forwards use the FP8 path.
|
||||
4. Report both latency and quality results when enabling this option for a new
|
||||
model. For image or video models, include visual comparison and quantitative
|
||||
metrics when available, such as PSNR or SSIM.
|
||||
|
||||
## Wan2.2 T2V quantized attention on Ascend
|
||||
|
||||
Wan2.2 T2V A14B supports quantized self-attention through MindIE-SD. Model
|
||||
weights are unchanged, and model-level opt-out keeps cross-attention in its
|
||||
native dtype.
|
||||
|
||||
| Attention path | self-attention `backend` | Supported `diffusion_kv_cache_dtype` |
|
||||
| --- | --- | --- |
|
||||
| Dense FA | `FLASH_ATTN` | `fp8`, `mxfp8`, `mxfp4` |
|
||||
| BSA | `RAINFUSION_ATTN` | `fp8`, `mxfp4` |
|
||||
|
||||
Configure the method and fallback selectors once at the diffusion stage. The
|
||||
per-role attention config only selects the self-attention backend:
|
||||
|
||||
```yaml
|
||||
stages:
|
||||
- stage_id: 0
|
||||
diffusion_kv_cache_dtype: "mxfp8"
|
||||
diffusion_kv_cache_skip_layers: "0,39"
|
||||
diffusion_kv_cache_skip_steps: "0,1,38,39"
|
||||
diffusion_attention_config:
|
||||
per_role:
|
||||
self:
|
||||
backend: FLASH_ATTN # Use RAINFUSION_ATTN with fp8 or mxfp4 for BSA.
|
||||
```
|
||||
|
||||
For 40 denoising steps, `diffusion_kv_cache_skip_steps: "0,1,38,39"` and
|
||||
`diffusion_kv_cache_skip_layers: "0,39"` are recommended starting points. Steps
|
||||
are zero-based across the complete request and do not reset when Wan switches
|
||||
transformers; layer indices are zero-based and local to each transformer.
|
||||
Selectors accept comma-separated indices and inclusive ranges such as
|
||||
`"0,3-5"`. A selected forward uses floating-point attention while preserving
|
||||
the Dense or BSA path.
|
||||
|
||||
Unsupported precision/input combinations raise an error and must be corrected
|
||||
in the configuration; operator errors are not retried with another precision.
|
||||
|
||||
For BSA, change the backend to `RAINFUSION_ATTN` and set
|
||||
`diffusion_kv_cache_dtype` to `fp8` or `mxfp4`.
|
||||
See [RainFusion attention](../diffusion/attention_backends/rainfusion.md) for
|
||||
sparse-path configuration and behavior.
|
||||
|
||||
### Requirement and interface test
|
||||
|
||||
Until these APIs are available in a MindIE-SD release, use revision
|
||||
[`8637b5333b0225381b215390fd09a8732e671cc4`](https://gitcode.com/zqxu/MindIE-SD/commit/8637b5333b0225381b215390fd09a8732e671cc4).
|
||||
Build the Python package, PyTorch plugin, and custom operators from that same
|
||||
checkout. With CANN sourced, run the NPU interface tests from the Omni checkout:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/platforms/npu/quant/test_kv_quant_npu.py \
|
||||
-k real_npu -vv -s -o addopts=''
|
||||
```
|
||||
|
||||
A skipped NPU test is not a pass. Validate generated video quality against a
|
||||
floating-point run with the same prompt, seed, resolution, and step count.
|
||||
Reference in new issue
Block a user