466 lines
23 KiB
Markdown
466 lines
23 KiB
Markdown
# Qwen3.8-27B and Spark-X2.5-4B on a Ryzen AI laptop
|
||||
|
|
|
|||
|
|
Target machine: **AMD Ryzen AI 9 HX 370** (Zen 5, 12 cores / 24 threads,
|
|||
|
|
AVX-512 with VNNI and BF16, **no AMX**), 30.9 GiB LPDDR5X, **RTX 5090 Laptop**
|
|||
|
|
(24463 MiB, sm_120, PCIe 4.0 x8), **Radeon 890M** iGPU and an **XDNA2 NPU**,
|
|||
|
|
running under **WSL2** (kernel 6.18.33.2).
|
|||
|
|
|
|||
|
|
Two models were added and both run:
|
|||
|
|
|
|||
|
|
| model | status | evidence |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `XHToken/Spark-X2.5-4B` | runs; **24/24 greedy tokens identical to the HF reference** | `analysis/experiments/spark_edge/parity_hf_vs_vllm.py` |
|
|||
|
|
| `Qwen/Qwen3.8-27B-FP8` | runs; correct output; 1.05 tok/s at the tightest fit | offload sweep below |
|
|||
|
|
|
|||
|
|
Neither needed a new model class. Spark-4B is the same `Spark2_5ForCausalLM`
|
|||
|
|
the 1.7B uses and the implementation was already config-driven; Qwen3.8 is
|
|||
|
|
`Qwen3_5ForConditionalGeneration`, which vLLM 0.28 already carries. What both
|
|||
|
|
needed was the *sizing* and *placement* code around them, which is what this
|
|||
|
|
document is about, and four environment faults, which are in
|
|||
|
|
[§7](#7-what-was-actually-broken).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 1. Three accelerators, and what each one costs to reach
|
|||
|
|
|
|||
|
|
The request was to use the CPU, the NPU and the GPU together. An earlier
|
|||
|
|
version of this document said the iGPU and the NPU were simply unreachable
|
|||
|
|
under WSL2. That was true of the default setup and false as a statement about
|
|||
|
|
the machine: both have a route, the routes are not equivalent, and finding them
|
|||
|
|
changed one of the two answers.
|
|||
|
|
|
|||
|
|
| device | reachable from WSL? | route | state |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| RTX 5090 Laptop | yes | `/dev/dxg` → `torch.cuda` | working |
|
|||
|
|
| Radeon 890M iGPU | **yes** | DirectML over `/dev/dxg` (`torch-directml`) | **working, and slower than the CPU** |
|
|||
|
|
| XDNA2 NPU | no | Windows-side ONNX Runtime + **VitisAI** EP | **working** — 2.6–5.3× the CPU on GEMMs |
|
|||
|
|
|
|||
|
|
### The iGPU: reachable, and measured not worth it
|
|||
|
|
|
|||
|
|
There is no `/dev/dri` under WSL2, and there does not need to be. Mesa ships a
|
|||
|
|
D3D12 Gallium driver and Microsoft ships `libd3d12.so` / `libdxcore.so` in
|
|||
|
|
`/usr/lib/wsl/lib`, so any WDDM adapter is addressable through `/dev/dxg`.
|
|||
|
|
`glxinfo` already reports `D3D12 (AMD Radeon(TM) 890M Graphics)`, accelerated,
|
|||
|
|
OpenGL 4.6. For compute, `torch-directml` offers it as a torch device:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
DirectML device_count: 2
|
|||
|
|
[0] NVIDIA GeForce RTX 5090 Laptop GPU
|
|||
|
|
[1] AMD Radeon(TM) 890M Graphics
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
It computes correctly — a 512×512 matmul agrees with the CPU to **3.6e-07**.
|
|||
|
|
Then it was benchmarked, which is why this section is short:
|
|||
|
|
|
|||
|
|
| | iGPU via DirectML | CPU | ratio |
|
|||
|
|
|---|---:|---:|---|
|
|||
|
|
| fp32 GEMM 2048³ | 1.12 TFLOP/s | 1.54 TFLOP/s | **0.73×** |
|
|||
|
|
| memory bandwidth | 5.0–7.4 GB/s | 20.7 GB/s | **0.30×** |
|
|||
|
|
| host↔device, 64 MB | 39.2 ms (3.3 GB/s) | — | — |
|
|||
|
|
|
|||
|
|
The iGPU is **slower than the CPU it shares memory with, on both axes**: the
|
|||
|
|
D3D12 translation layer costs more than the hardware wins. It is genuinely
|
|||
|
|
usable now, and the honest advice is to route work to it only when the point is
|
|||
|
|
to free the CPU rather than to go faster. A native path — ROCm on bare-metal
|
|||
|
|
Linux, or DirectML inside a Windows process — would not pay the translation and
|
|||
|
|
is the version worth measuring.
|
|||
|
|
|
|||
|
|
Note the pin conflict: `torch-directml` requires `torch==2.4.1`, so it can
|
|||
|
|
never share a virtualenv with the engine (torch 2.13). It gets its own.
|
|||
|
|
|
|||
|
|
### The NPU: working, at 2.6-5.3x the CPU
|
|||
|
|
|
|||
|
|
Not from WSL, and no driver install changes that: it is an **MCDM** device
|
|||
|
|
(`IpuMcdmDriver`, AMD 32.0.203.329), and WSL2's GPU-PV forwards WDDM *display*
|
|||
|
|
adapters only, so it has no device node in Linux. PCI passthrough is not a WSL2
|
|||
|
|
feature. It has to be driven from a Windows-side process.
|
|||
|
|
|
|||
|
|
From Windows it works. Three things have to be right **at once**, and each one
|
|||
|
|
failing on its own produces exactly the same symptom — a session that builds,
|
|||
|
|
runs, returns correct numbers, and executes every node on the CPU:
|
|||
|
|
|
|||
|
|
1. **The right EP.** The appx package ships two.
|
|||
|
|
`onnxruntime_providers_ryzenai.dll` registers cleanly, enumerates the NPU as
|
|||
|
|
`OrtHardwareDeviceType.NPU`, opens a session — and claims **zero nodes** from
|
|||
|
|
everything: fp32, INT8 QDQ symmetric and asymmetric, per-channel,
|
|||
|
|
power-of-two scales, MatMul and Conv, and ahead-of-time `ModelCompiler`
|
|||
|
|
(which emitted zero `EPContext` nodes). It is the *Light* EP. The general one
|
|||
|
|
is `onnxruntime_vitisai_ep.dll`.
|
|||
|
|
2. **The DLL search path.** That one fails to load from a bare
|
|||
|
|
`register_execution_provider_library` with `Error loading … But no
|
|||
|
|
dependency`. It needs its own directory added via `os.add_dll_directory`
|
|||
|
|
first, because it pulls `dyn_dispatch_core.dll` and `vaiml.dll` from beside
|
|||
|
|
it.
|
|||
|
|
3. **A16W8, not A8W8.** The overlay names are embedded in the DLL and say so:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
4x2_psf_model_a8w8_qdq.xclbin
|
|||
|
|
8x4_psu_model_a16w8_qdq.xclbin <- most of them
|
|||
|
|
2x4x4_bmm_model_a16w16.xclbin
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
XDNA2 wants **16-bit activations with 8-bit weights**. A8W8 — the default
|
|||
|
|
shape of `quantize_static` — is rejected as completely as fp32, silently.
|
|||
|
|
|
|||
|
|
Also needed: **onnxruntime ≥ 1.25**, because the EP requests ORT API 25.
|
|||
|
|
`onnxruntime-directml` stops at 1.24.4 and is refused, so the NPU and DirectML
|
|||
|
|
cannot share a process.
|
|||
|
|
|
|||
|
|
With all of it right the partitioner starts talking and names its target,
|
|||
|
|
`AMD_AIE2P_4x8_CMC_Overlay`. Measured against the same INT8 graph on the CPU:
|
|||
|
|
|
|||
|
|
| GEMM | CPU | NPU | speedup | NPU |
|
|||
|
|
|---|---:|---:|---:|---:|
|
|||
|
|
| 256 × 1024 × 1024 | 1.47 ms | 0.56 ms | 2.64× | 0.97 TOPS |
|
|||
|
|
| 512 × 2048 × 2048 | 9.64 ms | 2.14 ms | 4.51× | 2.01 TOPS |
|
|||
|
|
| 1024 × 2048 × 2048 | 19.89 ms | 3.76 ms | **5.29×** | **2.29 TOPS** |
|
|||
|
|
|
|||
|
|
Max relative error against the CPU: **3.2e-05**. It scales the right way — a
|
|||
|
|
small GEMM cannot amortise the dispatch. Dequantize nodes stay on the CPU (the
|
|||
|
|
partitioner says so explicitly), so a graph that is mostly requantization gains
|
|||
|
|
nothing.
|
|||
|
|
|
|||
|
|
The recipe is code in `vllm_omni/edge/npu_ryzenai.py`, including the one check
|
|||
|
|
worth trusting: **count nodes in an ORT profile.** A session reporting the EP in
|
|||
|
|
`get_providers()` may still be running everything on the CPU — and when it does,
|
|||
|
|
its outputs are bit-identical to the CPU, which is precisely what a *correct*
|
|||
|
|
NPU run does not look like. A 0.000e+00 error is the bug signature, not the
|
|||
|
|
success signature.
|
|||
|
|
|
|||
|
|
**What it is worth here.** Not decode: 2.29 TOPS against the dGPU's
|
|||
|
|
48.2 TFLOP/s, over the same 20.7 GB/s system memory the CPU uses, cannot help a
|
|||
|
|
bandwidth-bound step (§3). It earns its place on the compute-bound work off the
|
|||
|
|
per-token path — a vision tower, a draft model for speculative decoding — where
|
|||
|
|
5× the CPU on large GEMMs is real.
|
|||
|
|
|
|||
|
|
### Reaching a phone
|
|||
|
|
|
|||
|
|
The same question for mobile has a cleaner answer. The harness's `AdbDevice`
|
|||
|
|
backend needs only an `adb` binary and a reachable device, and WSL2 is in NAT
|
|||
|
|
mode, so **outbound** connections to the LAN work even though inbound do not.
|
|||
|
|
That makes wireless debugging the path of least resistance:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
sudo apt-get install -y adb # done; adb 1.0.41 (34.0.5)
|
|||
|
|
adb connect <phone-ip>:5555 # Android 11+ wireless debugging
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
No USB passthrough is involved, which matters because WSL2 has none by default
|
|||
|
|
— the USB route needs `usbipd-win` on the Windows side, and is worth the extra
|
|||
|
|
moving part only when the phone cannot be on the same network. `AdbDevice` also
|
|||
|
|
takes an explicit binary path, so `adb="/mnt/c/.../adb.exe"` drives a
|
|||
|
|
USB-attached phone through the Windows adb server with no passthrough at all,
|
|||
|
|
at the cost of translating paths on every `push`/`pull`.
|
|||
|
|
|
|||
|
|
The mobile **export** path is now working as well: the QNN export tests needed
|
|||
|
|
`onnxscript` and `tabulate`, and the ExecuTorch lowering needed `flatc`. All
|
|||
|
|
three are installed, and the one remaining ExecuTorch failure turned out not to
|
|||
|
|
be a failure — pytest run from the repository root lets the bare `executorch/`
|
|||
|
|
source checkout shadow the installed package. Run from `vllm-omni/` the suite
|
|||
|
|
is **124 passed, 1 skipped**.
|
|||
|
|
|
|||
|
|
## 2. The measurement that decides the design
|
|||
|
|
|
|||
|
|
Taken on an **idle** machine (see §7 for why that word is load-bearing):
|
|||
|
|
|
|||
|
|
| path | measured |
|
|||
|
|
|---|---|
|
|||
|
|
| GPU VRAM bandwidth | **472 GB/s** |
|
|||
|
|
| GPU bf16 GEMM 8192³ | **48.2 TFLOP/s** (91 W, 1447 MHz) |
|
|||
|
|
| PCIe H2D, pinned, bulk | **13.9 GB/s** |
|
|||
|
|
| PCIe, as seen by offloaded weight reads | **10.6 GB/s** (§5) |
|
|||
|
|
| CPU RAM bandwidth | **20.7 GB/s** |
|
|||
|
|
| CPU bf16 GEMM 4096³ | **1.54 TFLOP/s** |
|
|||
|
|
|
|||
|
|
Two ratios follow, and everything else in this document is a consequence:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
VRAM : system RAM = 22.8 x
|
|||
|
|
VRAM : PCIe link = 34 x (44x on the effective figure)
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Autoregressive decode reads **every weight once per token** and does two FLOPs
|
|||
|
|
per weight. At batch 1 it is bandwidth-bound with no room for argument, so
|
|||
|
|
decode rate is set by *where the weights are*, and almost nothing else.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 3. Why "use all three devices for decode" is the wrong goal
|
|||
|
|
|
|||
|
|
The intuition -- three processors, so split the model three ways -- inverts on
|
|||
|
|
a bandwidth-bound workload. Splitting weights across pools does not add
|
|||
|
|
bandwidth; the step ends when the *slowest* pool finishes, and the slow pool is
|
|||
|
|
22.8x slower. Putting 10% of a model on the CPU makes the step
|
|||
|
|
`0.9/472 + 0.1/20.7`, which is **3.2x slower** than putting all of it on the
|
|||
|
|
GPU, not 10% slower.
|
|||
|
|
|
|||
|
|
The iGPU and the NPU do not escape this: both read the same LPDDR5X the CPU
|
|||
|
|
does, so all three share one 20.7 GB/s pool — and the iGPU, measured, reaches
|
|||
|
|
only 5-7.4 GB/s of it (§1). Adding them to a decode split adds arithmetic to a
|
|||
|
|
problem that is not short of arithmetic, over a pool that is already the
|
|||
|
|
bottleneck.
|
|||
|
|
|
|||
|
|
So the rule is: **for decode, one device, and it is the one with the memory
|
|||
|
|
bandwidth.** The other devices earn their place on work that is *not* on the
|
|||
|
|
per-token critical path:
|
|||
|
|
|
|||
|
|
| device | job | why it fits there |
|
|||
|
|
|---|---|---|
|
|||
|
|
| dGPU | every decode step, all resident layers | the only 472 GB/s in the machine |
|
|||
|
|
| CPU | `embed_tokens` gather, detokenisation, scheduling, prompt templating | a gather reads one 10 KiB row per token, not the 2.54 GB table |
|
|||
|
|
| NPU | the **vision tower** on image prompts; a **draft model** for speculative decoding | both are compute-bound and off the decode path — *pending the compiler in §1* |
|
|||
|
|
| iGPU | the same class of work, when the goal is to free the CPU | reachable today, but measured at 0.73x the CPU's compute and 0.30x its bandwidth (§1) |
|
|||
|
|
|
|||
|
|
The vision tower is the clean case: 0.92 GB, run **once per image**, never per
|
|||
|
|
token. Moving it off the GPU frees 0.92 GB of VRAM -- which by §5 is worth
|
|||
|
|
86 ms/token -- and costs one prefill-time hop. That is a real heterogeneous win
|
|||
|
|
and it does not touch the decode loop.
|
|||
|
|
|
|||
|
|
Speculative decoding is the other one, and it is the only way a second device
|
|||
|
|
raises *decode* throughput: a draft model on the NPU proposes k tokens, the GPU
|
|||
|
|
verifies them in a single batched pass, and the GPU's per-token weight read is
|
|||
|
|
amortised over k tokens. Qwen3.8-27B **ships an MTP head for exactly this**
|
|||
|
|
(`mtp_num_hidden_layers: 1`, 0.48 GB, in `mtp.safetensors`).
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 4. What the two models actually cost
|
|||
|
|
|
|||
|
|
`vllm_omni/edge/kv_budget.py` now prices both. It could price neither before:
|
|||
|
|
it read `num_hidden_layers` off the top-level config, which `Qwen3_5Config`
|
|||
|
|
does not have, and it charged every layer for a KV cache.
|
|||
|
|
|
|||
|
|
**Qwen3.8-27B-FP8** is a hybrid of a kind the old code had no concept of: of its
|
|||
|
|
64 layers, **48 are gated DeltaNet**, which keep a fixed-size recurrent state
|
|||
|
|
and *no KV cache at any context length*. Only the 16 full-attention layers
|
|||
|
|
cache.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
flat over 64 layers 256 KiB/token <- what a layer-blind budget charges
|
|||
|
|
16 full layers only 64 KiB/token <- the truth, 4x smaller
|
|||
|
|
GDN recurrent state 147 MiB/sequence <- constant, and previously invisible
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The state is constant and the cache is linear, so they cross at ~2350 tokens:
|
|||
|
|
below that, the thing the old budget did not count at all is the larger half.
|
|||
|
|
At a 4096-token context the whole cache plus state is **467 MiB** (147 MiB of
|
|||
|
|
state, 320 MiB of cache), and **787 MiB** at 8192 -- against 30.87 GB of
|
|||
|
|
weights. For this model, **memory is a weights problem and not a
|
|||
|
|
cache problem**, and long context is nearly free. That is the architectural
|
|||
|
|
gift of the 3:1 linear hybrid and it is the strongest reason to want this model
|
|||
|
|
on an edge box.
|
|||
|
|
|
|||
|
|
**Spark-X2.5-4B** is the other hybrid: 27 sliding-window(512) layers to 9 full
|
|||
|
|
ones, 144 KiB/token flat, and a **3.37x** discount if the backend accounts per
|
|||
|
|
layer type. Both sizes now have their published geometry pinned in
|
|||
|
|
`tests/model_executor/models/spark2_5/test_spark2_5.py`, so a change that
|
|||
|
|
assumes the 1.7B's 8 query heads or 28 layers fails in CI rather than on a
|
|||
|
|
checkpoint.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 5. The cost of not fitting, measured
|
|||
|
|
|
|||
|
|
Qwen3.8-27B-FP8 is **30.87 GB** of weights against a 24 GiB card. It only runs
|
|||
|
|
with part of it on the host. Sweeping how much (4096-token context, batch 1,
|
|||
|
|
greedy, `enforce_eager`, idle machine):
|
|||
|
|
|
|||
|
|
| offloaded | tok/s | s/token | marginal |
|
|||
|
|
|---|---:|---:|---|
|
|||
|
|
| 10 GB (the tightest fit; 9 GB fails) | **1.05** | 0.952 | |
|
|||
|
|
| 12 GB | 0.89 | 1.124 | +0.086 s/GB |
|
|||
|
|
| 14 GB | 0.76 | 1.316 | +0.096 s/GB |
|
|||
|
|
| 16 GB | 0.66 | 1.515 | +0.100 s/GB |
|
|||
|
|
|
|||
|
|
Straight line, slope **+94 ms per offloaded GB**, i.e. an effective
|
|||
|
|
**10.6 GB/s** -- the 13.9 GB/s bulk figure less ~20% of overhead. The model
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
s/token = resident_GB / 472 + offloaded_GB / 10.6
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
predicts the 10 GB point at 987 ms against 952 ms measured, and the whole curve
|
|||
|
|
within 5%. At the tightest fit, **the 10 GB on the host is 94% of the step and
|
|||
|
|
the 20.9 GB in VRAM is 4%.**
|
|||
|
|
|
|||
|
|
This is the number to design against: **a gigabyte that does not fit costs
|
|||
|
|
44x what a gigabyte that fits costs** (94 ms against 2.1 ms). Fitting is not
|
|||
|
|
one optimisation among several; it is the only one that matters until it is
|
|||
|
|
solved.
|
|||
|
|
|
|||
|
|
### Which gigabytes to spend
|
|||
|
|
|
|||
|
|
vLLM's `cpu_offload_gb` offloads "non-selectively until the limit is reached",
|
|||
|
|
which is the expensive way. Not every parameter is read once per token, and
|
|||
|
|
`vllm_omni/edge/placement.py` sorts them:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
lm_head 2.543 GB HOT one 248320 x 5120 GEMM per token
|
|||
|
|
embed_tokens 2.543 GB COLD a gather: one row, ~10 KiB per token
|
|||
|
|
visual.* 0.922 GB COLD unless the prompt carries an image
|
|||
|
|
mtp.* 0.478 GB COLD unless speculative decoding is on
|
|||
|
|
---------
|
|||
|
|
cold set 3.94 GB free to evict
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
`embed_tokens` and `lm_head` are the same shape and the same dtype and the
|
|||
|
|
opposite decision. A size-ordered, name-blind policy cannot tell them apart;
|
|||
|
|
`cpu_offload_params`, which matches name segments, can.
|
|||
|
|
|
|||
|
|
For FP8 the cold set is 3.94 GB against a ~10.6 GB deficit, so it helps but
|
|||
|
|
cannot close it -- **selective placement does not rescue this checkpoint**.
|
|||
|
|
|
|||
|
|
And a larger caveat, found after this section was first written: *the cold set
|
|||
|
|
cannot currently be offloaded at all*. `cpu_offload_params` is real and matches
|
|||
|
|
name segments as documented, but the offloader is only ever offered decoder
|
|||
|
|
layers (`make_layers`, `utils.py:812`), so `embed_tokens`, `lm_head`, the vision
|
|||
|
|
tower and the MTP head are out of its reach. The classification below is right
|
|||
|
|
about which bytes are cheap to move; moving them needs an upstream change. See
|
|||
|
|
§6.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 6. The design: make it fit — and the retraction that follows
|
|||
|
|
|
|||
|
|
The previous version of this section said: use a 4-bit checkpoint, it is worth
|
|||
|
|
~20×, everything else is worth less than 2×. That was arithmetic —
|
|||
|
|
`resident_GB / 472 GB/s` — and the arithmetic was **wrong by roughly 30×**. The
|
|||
|
|
4-bit build was then measured, and it is *slower* than the 8-bit one.
|
|||
|
|
|
|||
|
|
| build | weights | offloaded | decode |
|
|||
|
|
|---|---:|---:|---:|
|
|||
|
|
| `Qwen/Qwen3.8-27B-FP8` | 30.87 GB | 10 GB | **1.05 tok/s** |
|
|||
|
|
| `nvidia/Qwen3.8-27B-NVFP4` | 21.95 GB | 3 GB | **0.67 tok/s** |
|
|||
|
|
| the same, torch.compile + CUDA graphs | 21.95 GB | 3 GB | 0.72 tok/s |
|
|||
|
|
| the same, 4 GB offloaded | 21.95 GB | 4 GB | 0.56 tok/s |
|
|||
|
|
|
|||
|
|
Three things went wrong at once, and each is worth stating separately.
|
|||
|
|
|
|||
|
|
**It does not fit, even at 21.95 GB.** The weights load to 19.92 GiB and the
|
|||
|
|
card has 22.58 GiB free (the Windows desktop holds the rest). vLLM's runtime
|
|||
|
|
overhead — non-torch allocations plus the activation peak — is about **2.8 GiB**
|
|||
|
|
on this model, which leaves the KV cache at **−0.31 GiB**. Shrinking
|
|||
|
|
`max_model_len` from 4096 to 1024 moved that to −0.30: the overhead is not the
|
|||
|
|
cache, so context length cannot buy the fit.
|
|||
|
|
|
|||
|
|
**The lever that would have made it fit does not exist.** §5 identified 4.31 GB
|
|||
|
|
of cold parameters and prescribed `cpu_offload_params={embed_tokens, visual,
|
|||
|
|
mtp}`. The flag is real and matches name segments as documented. But the
|
|||
|
|
offloader is only ever *offered decoder layers*: `wrap_modules` is called from
|
|||
|
|
inside `make_layers` (`vllm/model_executor/models/utils.py:812`) and nowhere
|
|||
|
|
else. `embed_tokens`, `lm_head`, the vision tower and the MTP head are built
|
|||
|
|
outside it. Asking for the cold set moved **~0.5 GiB** and the run failed
|
|||
|
|
identically to one with no offload at all. The classification was right about
|
|||
|
|
which bytes are cheap to move; the mechanism to move them is missing, and
|
|||
|
|
closing it is an upstream change.
|
|||
|
|
|
|||
|
|
**And fewer bytes turned out not to mean faster.** This is the part that
|
|||
|
|
invalidates the projection rather than merely complicating it. The traffic model
|
|||
|
|
`resident/472 + offloaded/10.6` fits the FP8 offload curve to within 4.3% across
|
|||
|
|
four points. Against NVFP4 it is **4.7× optimistic**: predicted 3.1 tok/s at
|
|||
|
|
3 GB offloaded, measured 0.67. Its own offload slope is 0.293 s/GB — an
|
|||
|
|
effective **3.4 GB/s**, against FP8's 10.6 — and extrapolating to zero offload
|
|||
|
|
gives ~**1.6 tok/s** even if it fit. Enabling torch.compile and CUDA graphs
|
|||
|
|
changed 0.67 to 0.72, so this is not an eager-mode artifact.
|
|||
|
|
|
|||
|
|
The conclusion is that **NVFP4 decode on this stack is compute-bound, not
|
|||
|
|
bandwidth-bound**, and a model that counts bytes cannot see that. Why it is
|
|||
|
|
compute-bound is not established here: the checkpoint is a `modelopt` mix of
|
|||
|
|
FP8 and NVFP4 with group-16 scales, and this fork already carries a correctness
|
|||
|
|
patch for the W4A4 NVFP4 linear path, which is a hint that the path is not a
|
|||
|
|
well-trodden one on sm_120. Diagnosing it is the obvious next experiment, and
|
|||
|
|
until someone does, the honest statement is the measurement.
|
|||
|
|
|
|||
|
|
### What the design actually is, then
|
|||
|
|
|
|||
|
|
1. **Run the FP8 checkpoint with the smallest offload that fits** — 10 GB, 1.05
|
|||
|
|
tok/s. This is the fastest configuration measured on this machine, and it is
|
|||
|
|
the 8-bit one.
|
|||
|
|
2. **Do not change quantization format on an estimate.** The one estimate this
|
|||
|
|
document made without measuring is the one it got wrong, by 30×.
|
|||
|
|
3. **Size the cache from the model** — 467 MiB at 4096 tokens (§4). Still right,
|
|||
|
|
and still worth little here, because weights dominate.
|
|||
|
|
4. **Speculative decoding via the shipped MTP head** is now the largest
|
|||
|
|
*unexplored* lever, and the only one that attacks the per-token weight read
|
|||
|
|
rather than working within it. It is untested here.
|
|||
|
|
5. **Getting the cold set offloadable upstream** would free 3.94 GB on FP8, worth
|
|||
|
|
~370 ms/token by the traffic model — which, on the FP8 path where that model
|
|||
|
|
demonstrably holds, is the difference between 1.05 and roughly 1.6 tok/s.
|
|||
|
|
|
|||
|
|
The blunt summary: a 27B model on a 24 GB laptop GPU runs at **about one token
|
|||
|
|
per second**, and the things that would change that by an order of magnitude —
|
|||
|
|
a checkpoint that genuinely fits, or speculative decoding — are not yet in hand.
|
|||
|
|
|
|||
|
|
## 7. Where this beats llama.cpp, and where it does not
|
|||
|
|
|
|||
|
|
llama.cpp at the pinned revision **does** support this architecture -- `QWEN35`,
|
|||
|
|
`QWEN35MOE` and `QWEN3NEXT` are in its arch registry and it has gated-DeltaNet
|
|||
|
|
kernels. It is a real competitor here, and on a fitted 4-bit model on one
|
|||
|
|
stream it is a strong one: this repository's own CPU measurements have
|
|||
|
|
llama.cpp's Q4_K_M decode at 90 tok/s against our 74, and `docs/edge/` explains
|
|||
|
|
that the gap is the execution model rather than the kernels.
|
|||
|
|
|
|||
|
|
So the case for vLLM-Omni on this machine is not raw single-stream decode. It is:
|
|||
|
|
|
|||
|
|
| axis | why |
|
|||
|
|
|---|---|
|
|||
|
|
| **Prefill** | measured ~2x llama.cpp on the CPU path (2190 vs 1121 tok/s, pp512). Prompt processing is compute-bound, which is the one regime where the hardware has headroom. |
|
|||
|
|
| **Concurrency** | continuous batching and paged KV. At batch 1 they cost overhead; past ~4 streams they are the whole game, and llama.cpp has no equivalent. |
|
|||
|
|
| **Speculative decoding** | the shipped MTP head is wired in vLLM and is not a GGUF concept. |
|
|||
|
|
| **Multimodal pipelines** | the reason this fork exists: staged vision/audio/codec execution with an OpenAI-compatible server. |
|
|||
|
|
| **Hybrid cache accounting** | 64 KiB/token instead of 256, once §4 is honoured. |
|
|||
|
|
|
|||
|
|
A design that claims to beat llama.cpp at batch-1 decode of a fitted model on
|
|||
|
|
one GPU is claiming something this project has already measured to be false.
|
|||
|
|
The defensible claim is the four rows above, and the way to establish it is
|
|||
|
|
`benchmarks/edge_harness/` against `baselines/`, not assertion.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## 8. What was actually broken
|
|||
|
|
|
|||
|
|
Four faults stood between "the models are supported" and "the models run", none
|
|||
|
|
of them in the model code:
|
|||
|
|
|
|||
|
|
1. **`RuntimeError: UVA is not available`.** vLLM disables pinned memory on WSL
|
|||
|
|
by default; its own gate is kernel ≥ 4.19.121 and this kernel is 6.18. Fixed
|
|||
|
|
by `VLLM_WSL2_ENABLE_PIN_MEMORY=1`. Without it the CUDA runner does not
|
|||
|
|
start **at all**, for any model.
|
|||
|
|
2. **No C compiler**, so Triton could not JIT and the GDN kernels could not
|
|||
|
|
build. `gcc` plus `libc6-dev` (gcc alone did not pull the headers).
|
|||
|
|
3. **FlashInfer's sampler wants `nvcc`**, which a runtime-only CUDA install does
|
|||
|
|
not have. `VLLM_USE_FLASHINFER_SAMPLER=0`. Note this surfaced as a failure in
|
|||
|
|
*warmup*, long after the model had loaded and run a forward pass.
|
|||
|
|
4. **transformers 5 renamed `input_embeds` → `inputs_embeds`**, which broke the
|
|||
|
|
vendored Spark modeling file used as the parity *reference*. Shimmed in
|
|||
|
|
`parity_hf_vs_vllm.py` rather than by editing the cached module, so the
|
|||
|
|
reference stays the code the checkpoint ships.
|
|||
|
|
|
|||
|
|
The required environment, in full:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
export VLLM_WSL2_ENABLE_PIN_MEMORY=1
|
|||
|
|
export VLLM_USE_FLASHINFER_SAMPLER=0
|
|||
|
|
sudo apt-get install -y gcc libc6-dev
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### And one measurement error, which is the real lesson
|
|||
|
|
|
|||
|
|
The first roofline taken on this machine read **3.67 GB/s** H2D, **38 GB/s**
|
|||
|
|
VRAM and **9 TFLOP/s**, with `nvidia-smi` reporting a 33 W power cap and a
|
|||
|
|
220 MHz SM clock. The conclusion drawn from it -- that the dGPU had no
|
|||
|
|
bandwidth advantage over the CPU and the design should lean on the CPU -- was
|
|||
|
|
wrong in every part.
|
|||
|
|
|
|||
|
|
Those numbers were taken **while two model downloads were saturating the disk,
|
|||
|
|
the network and the CPU**. Re-measured on an idle machine the same calls give
|
|||
|
|
13.9 GB/s, 472 GB/s and 48.2 TFLOP/s, at 91 W and 1447 MHz, with the power
|
|||
|
|
limit reading 100.98 W. The GPU was never capped; it was idling because the
|
|||
|
|
benchmark could not feed it.
|
|||
|
|
|
|||
|
|
This repository already has [a methodology
|
|||
|
|
document](methodology.md) that exists because of five retractions of exactly
|
|||
|
|
this shape. This is the sixth, and the rule it re-teaches is the one already
|
|||
|
|
written down: **a number taken on a contended machine is not a number.** The
|
|||
|
|
harness has a contention guard for this; it was not in the path of an
|
|||
|
|
interactive measurement, which is precisely where the mistake got made.
|