[Quantization] Add BitsAndBytes W4 online quantization for diffusion transformer (#5037)
Signed-off-by: diyapeng <50388704+diyapeng@users.noreply.github.com> Co-authored-by: diyapeng <50388704+diyapeng@users.noreply.github.com>
This commit is contained in:
1 parent
b20ebc8f9b
commit
e0570a0859
7 files changed
+590
-2
No files matched your search
@@ -202,9 +202,10 @@ def parse_args() -> argparse.Namespace:
|
||||
"--quantization",
|
||||
type=str,
|
||||
default=None,
|
||||
choices=["fp8", "int8"],
|
||||
choices=["fp8", "int8", "bitsandbytes"],
|
||||
help="Quantization method for the transformer. "
|
||||
"Options: 'fp8' (FP8 W8A8 on Ada/Hopper, weight-only on older GPUs), 'int8' (Int8 W8A8). "
|
||||
"Options: 'fp8' (FP8 W8A8 on Ada/Hopper, weight-only on older GPUs), "
|
||||
"'int8' (Int8 W8A8), 'bitsandbytes' (NF4 4-bit weight-only, CUDA SM 75+). "
|
||||
"Default: None (no quantization, uses BF16).",
|
||||
)
|
||||
parser.add_argument(
|
||||
|
||||
Reference in new issue
Block a user