tzhouamandClaude Opus 5 289c13df68 cpu_int4: tune the repack chunk to 2048 rows from a measured sweep ...

512 was a first guess. Repacking the 131072-row tied embedding at 24 threads:

    rows        chunks    time      int32 transient
    whole            1   208 ms         1024 MiB
    512            256   116 ms            4 MiB
    2048            64    79 ms           16 MiB
    8192            16    89 ms           64 MiB
    32768            4   129 ms          256 MiB

2048 is both the fastest and nearly the leanest. The sweep also settles a
suspicion: chunking is faster than packing the whole matrix, because the
working set stays in cache, so this costs nothing at load. The one pass that
appeared to show a 30 s init regression was the pass the harness had flagged
as contended.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

2026-09-14 13:19:36 +08:00

..

__init__.py

Add cpu_int4, an asymmetric 4-bit weight path for vLLM's CPU backend

2026-09-12 03:48:04 +08:00

cpu_int4.py

cpu_int4: tune the repack chunk to 2048 rows from a measured sweep

2026-09-14 13:19:36 +08:00

gptq_cpu_export.py

[Edge][CPU] W4A8 via vLLM's own AMX kernel, a per-device weight path, and stop recompiling on every start

2026-09-12 13:39:19 +08:00