vllm-omni

3,038 Commits 235 Branches 31 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
tzhouamandClaude Opus 5	6d7b2bfc1c	cpu_int4: repack weights in n-blocks, to bound the load-time peak I0 of the reviewed plan: the peak, not the settled figure, is what an edge device has to survive. Measured earlier this session, W4A16 loads at 5722-5837 MB and settles near 4480 after the allocator is asked for its arenas back -- a 1.2-1.3 GB gap, and a device that cannot hold the peak never reaches the release. A large part of that gap is in our own repack path. unpack_nibbles expands 4-bit codes to int32, which is 4 bytes per weight and eight times the packed form, and the kernel's packer wants int32. Done for a whole matrix that transient is the largest object in the process: gate_up 13312 x 2048 104 MiB down 2048 x 6656 52 MiB q_k_v 3072 x 2048 24 MiB embedding 131072 x 2048 1024 MiB <- tied, so it is also the head against 24 MiB of packed weights for an entire layer. The embedding alone is a ~1 GB spike, which is about the size of the whole peak-vs-settled gap. The packed layout is [N/64][K][32 bytes], so n-blocks are independent and the expansion can cover one chunk at a time. Verified byte-identical to packing the whole matrix at 13312x2048, 2048x6656, 3072x2048 and 131072x2048, plus a ragged last chunk. The int32 transient drops from 1024 MiB to 4 MiB for the embedding and from 104 to 4 MiB for gate_up. to_split_halves is chunked the same way for the fallback path. Not yet claimed: the end-to-end peak reduction. The arithmetic says ~1 GB, but this study has been wrong often enough about memory that the number goes in only after RssAnon is sampled through a real load. Tests: 21 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 18:47:47 +08:00
tzhouamandClaude Opus 5	f3629320e7	cpu_int4: dequantize prefill weights from the tinygemm layout, not a copy The W4A16 path kept two packed weight layouts because the tinygemm layout was treated as opaque: the prefill path needed something row-major to dequantize from, so `_finalize_int4` built `weight_halves` plus cloned scale/zero beside the kernel's own buffer. That duplicate is the 740 MB the report attributes to "the dual weight layout". The layout is not opaque. Measured on this host by packing an index pattern and reading back where each nibble landed: it is [N/64][K][32 bytes] -- K is the middle axis, so a K range is contiguous inside each n-block -- and within every 32-byte run, byte d carries output channel d in its low nibble and d+32 in its high one, the same permutation for every k and every block. Decoding it is a deinterleave. _dequantize_from_int4pack does that, bit-exact against the old dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048, and the duplicate is no longer allocated. Measured with the two paths toggled in otherwise identical code, 3 arms x 2 interleaved passes, RssAnon summed over the process tree, 1536-token context: 615 MB of resident memory and ~915 MB of peak, landing statistically indistinguishable from prefill_dequant: false (the harness refused to rank them, 0.4% apart inside the spread). Config arithmetic predicts 764 MB; ~150 MB is allocator retention malloc_trim does not return. It is not free, and the docstring says so. The duplicate was not only a copy: to_split_halves de-interleaves once at load, so each prefill call read two contiguous planes, while this redoes the de-interleave every call. pp512 drops 2272 -> 1210 tok/s. Dequantizing to [K, N] and using matmul to dodge the transpose was tried and is worse (20.3 ms vs 14.4 per gate_up at 8 threads); recovering the prefill wants a C++ de-interleave, not an ATen expression chain. So it is a dial. Against prefill_dequant: false it is strictly better -- same memory, 2.7x the prefill, and dequant_threshold stays in force so multi-row batches never reach the tinygemm kernel and the documented SIGILL hazard. That setting is now dominated. Against keeping both layouts it trades 615 MB for 1.9x prefill; default on, because this path exists for devices chosen for their memory ceiling. supports_direct_dequant gates the reshape on N tiling the 64-channel block, which every Spark linear does (2048, 3072, 13312, 131072). An N like 80 packs as 64 + a 16 tail the reshape would mis-read, so those layers keep the copy rather than being quietly wrong. The choice is a `direct_dequant` field of the checkpoint's quantization_config, not only an env var, and that matters: the two paths build different graphs (one passes weight_halves into the custom op, the other passes None), so they must not share a compiled artifact. ModelConfig.compute_hash hashes config fields, so a config field reaches vLLM's compile-cache key; an env var does not. Toggling the env var against a warm cache loaded the wrong graph and killed three benchmark passes with `KeyError: 'weight_halves'` inside the AOT-loaded forward -- found because the harness records failures instead of dropping them. VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT remains as an experiment override, documented as needing its own VLLM_CACHE_ROOT. Tests: 18 passed, including bit-exactness at three shapes, a pin on the packed layout itself (so a torch release that repacks differently fails loudly rather than returning wrong weights), the N=80 fallback, and both paths producing identical output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-13 01:21:08 +08:00
tzhouamandClaude Opus 5	7869a8dc0d	Add cpu_int4, an asymmetric 4-bit weight path for vLLM's CPU backend Single-stream CPU decode reads every weight to emit one token, so bits per weight set the token rate. This registers a W4A16 group-wise method out-of-tree through register_quantization_config -- vLLM itself is untouched -- using PyTorch's _weight_int4pack_mm_for_cpu tinygemm kernel. Two paths over one set of weights, because that kernel dequantizes once per row of activations and so is linear in the batch (12.9x slower than bf16 at 512 rows). Single-row steps take tinygemm; wider batches unpack the weight to bf16 once and use the ordinary oneDNN/AMX GEMM. Shipping only the first measured pp512 = 557 tok/s against bf16's 4455; with both, 3589. The batch threshold is 2 rather than the ~8 rows where the unpack path starts winning on speed, because running tinygemm on a multi-row batch and then vLLM's cpu_attention_with_kv_cache in the same forward dies with SIGILL inside the attention kernel -- an AMX instruction with no valid tile config loaded. The same prompt is fine in bf16 and fine once every multi-row GEMM takes the unpack path. Positioning, measured on a Xeon 8480C with 32 pinned cores and a warm cache (analysis/spark_cpu_int4.md): vLLM's own GPTQ W4A8 AMX path is faster -- 88.7 tok/s decode and 4355 prefill against 85.5 and 3589 here -- but its config rejects sym=False at 4 bits, so it can only be fed a symmetric grid. This one is asymmetric and costs +12.5% perplexity against that path's +27.2%, and needs no AMX. llama.cpp Q4_K_M, for reference, is 98.0 tok/s decode and 1356 prefill at +7.6% and 5.15 bits per weight. Expanding a quantized checkpoint back to bf16 offline and scoring it reproduces the quantized run to within 0.2% perplexity, so both GEMM paths, the nibble packing and the quantized tied embedding are faithful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GXhbAfkSNRaeqZrus5Z2fx	2026-09-12 03:48:04 +08:00

Author

SHA1

Message

Date

tzhouamandClaude Opus 5

6d7b2bfc1c

cpu_int4: repack weights in n-blocks, to bound the load-time peak

I0 of the reviewed plan: the peak, not the settled figure, is what an edge
device has to survive. Measured earlier this session, W4A16 loads at 5722-5837
MB and settles near 4480 after the allocator is asked for its arenas back -- a
1.2-1.3 GB gap, and a device that cannot hold the peak never reaches the
release.

A large part of that gap is in our own repack path. unpack_nibbles expands
4-bit codes to int32, which is 4 bytes per weight and eight times the packed
form, and the kernel's packer wants int32. Done for a whole matrix that
transient is the largest object in the process:

    gate_up   13312 x 2048    104 MiB
    down       2048 x 6656     52 MiB
    q_k_v      3072 x 2048     24 MiB
    embedding 131072 x 2048  1024 MiB   <- tied, so it is also the head

against 24 MiB of packed weights for an entire layer. The embedding alone is a
~1 GB spike, which is about the size of the whole peak-vs-settled gap.

The packed layout is [N/64][K][32 bytes], so n-blocks are independent and the
expansion can cover one chunk at a time. Verified byte-identical to packing the
whole matrix at 13312x2048, 2048x6656, 3072x2048 and 131072x2048, plus a ragged
last chunk. The int32 transient drops from 1024 MiB to 4 MiB for the embedding
and from 104 to 4 MiB for gate_up. to_split_halves is chunked the same way for
the fallback path.

Not yet claimed: the end-to-end peak reduction. The arithmetic says ~1 GB, but
this study has been wrong often enough about memory that the number goes in
only after RssAnon is sampled through a real load. Tests: 21 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-13 18:47:47 +08:00

tzhouamandClaude Opus 5

f3629320e7

cpu_int4: dequantize prefill weights from the tinygemm layout, not a copy

The W4A16 path kept two packed weight layouts because the tinygemm layout was
treated as opaque: the prefill path needed something row-major to dequantize
from, so `_finalize_int4` built `weight_halves` plus cloned scale/zero beside
the kernel's own buffer. That duplicate is the 740 MB the report attributes to
"the dual weight layout".

The layout is not opaque. Measured on this host by packing an index pattern and
reading back where each nibble landed: it is [N/64][K][32 bytes] -- K is the
middle axis, so a K range is contiguous inside each n-block -- and within every
32-byte run, byte d carries output channel d in its low nibble and d+32 in its
high one, the same permutation for every k and every block. Decoding it is a
deinterleave. _dequantize_from_int4pack does that, bit-exact against the old
dequantizer at 2048x2048, 3072x2048 and gate_up's real 13312x2048, and the
duplicate is no longer allocated.

Measured with the two paths toggled in otherwise identical code, 3 arms x 2
interleaved passes, RssAnon summed over the process tree, 1536-token context:
615 MB of resident memory and ~915 MB of peak, landing statistically
indistinguishable from prefill_dequant: false (the harness refused to rank
them, 0.4% apart inside the spread). Config arithmetic predicts 764 MB; ~150 MB
is allocator retention malloc_trim does not return.

It is not free, and the docstring says so. The duplicate was not only a copy:
to_split_halves de-interleaves once at load, so each prefill call read two
contiguous planes, while this redoes the de-interleave every call. pp512 drops
2272 -> 1210 tok/s. Dequantizing to [K, N] and using matmul to dodge the
transpose was tried and is worse (20.3 ms vs 14.4 per gate_up at 8 threads);
recovering the prefill wants a C++ de-interleave, not an ATen expression chain.

So it is a dial. Against prefill_dequant: false it is strictly better -- same
memory, 2.7x the prefill, and dequant_threshold stays in force so multi-row
batches never reach the tinygemm kernel and the documented SIGILL hazard. That
setting is now dominated. Against keeping both layouts it trades 615 MB for
1.9x prefill; default on, because this path exists for devices chosen for
their memory ceiling.

supports_direct_dequant gates the reshape on N tiling the 64-channel block,
which every Spark linear does (2048, 3072, 13312, 131072). An N like 80 packs
as 64 + a 16 tail the reshape would mis-read, so those layers keep the copy
rather than being quietly wrong.

The choice is a `direct_dequant` field of the checkpoint's quantization_config,
not only an env var, and that matters: the two paths build different graphs
(one passes weight_halves into the custom op, the other passes None), so they
must not share a compiled artifact. ModelConfig.compute_hash hashes config
fields, so a config field reaches vLLM's compile-cache key; an env var does
not. Toggling the env var against a warm cache loaded the wrong graph and
killed three benchmark passes with `KeyError: 'weight_halves'` inside the
AOT-loaded forward -- found because the harness records failures instead of
dropping them. VLLM_OMNI_CPU_INT4_DIRECT_DEQUANT remains as an experiment
override, documented as needing its own VLLM_CACHE_ROOT.

Tests: 18 passed, including bit-exactness at three shapes, a pin on the packed
layout itself (so a torch release that repacks differently fails loudly rather
than returning wrong weights), the N=80 fallback, and both paths producing
identical output.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-13 01:21:08 +08:00

tzhouamandClaude Opus 5

7869a8dc0d

Add cpu_int4, an asymmetric 4-bit weight path for vLLM's CPU backend

Single-stream CPU decode reads every weight to emit one token, so bits per
weight set the token rate. This registers a W4A16 group-wise method
out-of-tree through register_quantization_config -- vLLM itself is untouched
-- using PyTorch's _weight_int4pack_mm_for_cpu tinygemm kernel.

Two paths over one set of weights, because that kernel dequantizes once per
row of activations and so is linear in the batch (12.9x slower than bf16 at
512 rows). Single-row steps take tinygemm; wider batches unpack the weight to
bf16 once and use the ordinary oneDNN/AMX GEMM. Shipping only the first
measured pp512 = 557 tok/s against bf16's 4455; with both, 3589.

The batch threshold is 2 rather than the ~8 rows where the unpack path starts
winning on speed, because running tinygemm on a multi-row batch and then
vLLM's cpu_attention_with_kv_cache in the same forward dies with SIGILL
inside the attention kernel -- an AMX instruction with no valid tile config
loaded. The same prompt is fine in bf16 and fine once every multi-row GEMM
takes the unpack path.

Positioning, measured on a Xeon 8480C with 32 pinned cores and a warm cache
(analysis/spark_cpu_int4.md): vLLM's own GPTQ W4A8 AMX path is faster --
88.7 tok/s decode and 4355 prefill against 85.5 and 3589 here -- but its
config rejects sym=False at 4 bits, so it can only be fed a symmetric grid.
This one is asymmetric and costs +12.5% perplexity against that path's
+27.2%, and needs no AMX. llama.cpp Q4_K_M, for reference, is 98.0 tok/s
decode and 1356 prefill at +7.6% and 5.15 bits per weight.

Expanding a quantized checkpoint back to bf16 offline and scoring it
reproduces the quantized run to within 0.2% perplexity, so both GEMM paths,
the nibble packing and the quantized tied embedding are faithful.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GXhbAfkSNRaeqZrus5Z2fx

2026-09-12 03:48:04 +08:00