vllm-omni/docs/user_guide/quantization/svdquant.md

SVDQuant W4A4

Overview

SVDQuant combines four-bit weights and activations with a small low-rank branch that corrects part of the quantization error. vLLM-Omni consumes an offline-quantized checkpoint; it does not calibrate the model while loading.

Checkpoint contract

Place the following entry in the diffusion transformer's config.json:

{
  "quantization_config": {
    "quant_method": "svdquant",
    "rank": 32,
    "precision": "nvfp4",
    "act_unsigned": false,
    "modules_to_not_convert": []
  }
}

For a quantized linear with input size K, output size N, and correction rank R, the checkpoint stores:

Suffix	Shape	dtype
qweight	(N, K / 2)	int8 (two packed FP4 values per byte)
wscales	(K / 16, N)	float8_e4m3fn
proj_down	(K, R)	bfloat16
proj_up	(N, R)	bfloat16
smooth_factor	(K,)	bfloat16
wcscales	(N,)	bfloat16
wtscale	(1,)	bfloat16

K must be divisible by 16 on every tensor-parallel rank. Set wcscales to ones when no per-output correction is needed. Modules listed in modules_to_not_convert keep their checkpoint precision.

Runtime support

The compatibility path accepts BF16 inputs and executes an NVFP4 GEMM followed by the BF16 rank correction. It supports vLLM's FlashInfer, CUTLASS, and FBGEMM NVFP4 tensor layouts; incompatible forced backends fail during model loading. SM103 is the currently validated and enabled hardware target. Native fusion of the NVFP4 GEMM and rank correction is separate from this checkpoint-loading contract.