anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

[edge-infer] WIP snapshot: uncommitted M0/W1/W4/W5 edge work as of 2026-09-18

Browse Source 
Verbatim copy of the edge/omni-edge working tree (tracked diffs + untracked
edge/local, tests/edge/local, docs/edge, pipeline files). Taken so that
vLLM-Omni v0.29.0rc1 can be merged on a branch without touching the
user's working tree. logs_edge/ and benchmark .npy outputs excluded.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
tzhouamandClaude Fable 5.1 committed 2026-09-18 09:47:29 +08:00
1 parent 3834de3105
commit 4e3e8a1ebc
62 files changed
+11043 -56
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/edge/external-stage-backends.md
+182
View File
@@ -0,0 +1,182 @@
# External stage backends: the Radeon 890M and the AMD NPU
`vllm_omni.edge.local.external` lets the local engine place one stage on a device
vLLM does not drive, prove it actually ran there, and charge its memory to the
same ledger as everything else. Status: the plumbing is done and measured
(M3.0); no model component runs on either device yet (M3.1–M3.4).
## Why these devices need another process
Neither can be opened from the engine's interpreter, and for different reasons.
The **XDNA2 NPU** is an MCDM device. WSL2's GPU-PV forwards WDDM display
adapters only, so `/dev/accel` does not exist inside WSL and no amount of
installing there changes it; VitisAI is a Windows DLL besides.
The **Radeon 890M** *is* reachable from WSL, over DirectML through `/dev/dxg` —
but `torch-directml` pins torch 2.4.1 and Omni runs 2.13.
They cannot even share a Windows venv: `onnxruntime-directml` is pinned at
1.24.4, and the VitisAI EP needs onnxruntime ≥ 1.25.
So: one protocol, four routes, each in its own interpreter.
| route | interpreter | artifact | status on this laptop |
|---|---|---|---|
| `ort-vitisai` | native Windows, plain ORT ≥ 1.25 | `onnx:a16w8` | working |
| `torch-dml` | WSL `.venvs/dml` | `pt:fp16` / `pt:fp32` | working |
| `ort-dml` | native Windows, onnxruntime-directml | `onnx:fp16` / `onnx:fp32` | venv not installed |
| `ort-cpu` | anything with ORT | any ONNX | reference arm |
`python -m vllm_omni.edge.local devices` prints which resolved; a route that did
not resolve says what is missing rather than disappearing.
## Placement is verified, never assumed
A VitisAI session lists the EP in `get_providers()` **whether or not it claimed
a single node**, and a graph it declined runs entirely on the CPU and returns
bit-identical numbers. Identical output cannot confirm NPU execution, and a
small difference cannot confirm it either. The only evidence is ORT's per-node
provider assignment, which exists only after something has run — so `open()`
always does one profiled warm-up run and the planner refuses on the node split.
An *unmeasured* placement (`fraction_on_target is None`) is refused too. It is
not rounded to "fine".
On the torch-directml route there is no partitioner, so the evidence is coarser
and equivalent: the device the output tensors came back from. A module that
fell back to the CPU returns CPU tensors.
## Memory
The 890M and the NPU report `memory_bytes == 0` on purpose. All three devices
allocate out of the CPU's RAM, so repeating the pool size on each row would let
a caller add three copies of the same memory together. Capacity comes from the
host row — the same number the CPU stage is being budgeted against.
The worker is a different process, on Windows a different OS, so the engine's
psutil sampler cannot see it. It reports its own resident set, and that is what
closes the budget-versus-measurement loop.
## Refusal codes
| code | means |
|---|---|
| `no_exported_graph` | the device runs graphs and there is no export for this stage at this precision |
| `execution_provider_declined_graph` | the session ran and left the graph on the CPU |
| `numeric_gate_failed` | it ran on the device and disagreed with the reference |
| `device_unreachable_from_this_process` | the device works; no interpreter here drives it |
| `budget_exceeds_device` | the stage does not fit the shared host pool |
All of them exit 2 from the CLI. A refusal is a result.
## Using it
```bash
python -m vllm_omni.edge.local external \
--graph out/code2wav.onnx --format onnx:fp16 --component code2wav \
--device gpu_integrated:amd --runs 20 --json record.json
```
`--device` requires that device, so "does the NPU run this?" is answered with a
refusal rather than quietly with the iGPU. `--prefer` falls through instead.
`--min-placement` sets how much of the graph the target EP must take.
## First real model component: Qwen3.8-27B's vision tower
`vllm_omni.edge.local.artifacts.vision_tower` exports it at a fixed image size.
It is the component that best fits the transport constraint — 27 blocks run once
per image, 4.8 MB in and 4.0 MB out — and it needs no quantization story for the
iGPU, because the FP8 checkpoint keeps all 333 visual tensors in bf16.
The export wraps `transformers`' `Qwen3_5VisionModel` (whose parameters match the
checkpoint name for name, 333/333) and only fixes its shapes; the wrapper's
output is **bitwise identical** to the model's own `forward`. At a fixed size the
bilinear indices, position ids and `cu_seqlens` are constants, so folding them
into buffers removes every integer `Gather` from the graph.
| | 890M (DirectML, fp32) | AMD NPU (A16W8) |
|---|---|---|
| placement | **85 of 113 nodes** | **0 of 3246** — declined |
| latency | **227.0 ms** device, 237.2 round trip, 11.4 transport | — |
| vs torch CPU (1363 ms) | **6.0×**, or 5.7× including transport | — |
| vs RTX 5090 (95.9 ms) | 2.4× slower | — |
| parity vs torch CPU fp32 | 9.10e-06 normalized, cosine 0.99999994 | — |
Moving the tower off the discrete GPU frees **1862 MiB** of its VRAM, which
matters for a 21–29 GB model on a 23.89 GiB card.
The NPU refusal is about **this artifact**, not the model or the precision: the
same A16W8 scheme applied to the merger subgraph alone *was* accepted (1 of 8
nodes), and cutting requantization from 3058 to 984 Q/DQ nodes did not change the
whole-graph answer. The next step is a bounded bisect of what the partitioner
will take, not a broader claim.
One number worth keeping straight when comparing devices: the 890M's fp32 output
is **closer** to the CPU reference (9.10e-06) than the RTX 5090's is (5.65e-04),
because CUDA runs matmul in TF32 by default. That is a statement about what each
device actually computed, not about which is more accurate.
## Measured, 2026-09-15, this laptop
Transport, WSL ↔ native Windows: **11 µs** half round trip, **330 MB/s**.
| | NPU (A16W8 probe graph) | 890M (100.7 M-param torch.export graph) |
|---|---|---|
| device compute | 0.855 ms | 9.749 ms |
| same graph on CPU | 6.270 ms | 49.661 ms |
| transport | **2.797 ms** (cross-OS) | 0.523 ms (same OS) |
| placement | vitisai 1 / CPU 2 nodes | outputs on `privateuseone` |
| numerics vs CPU | 3.03e-05 normalized | — |
**The transport column is the design constraint.** On the NPU it consumes more
than half of a 5.4 ms compute saving, because that graph moves 1 MiB in and
1 MiB out. An external stage has to be compute-heavy and I/O-light: a vision
tower that runs once per image qualifies, a decode step that runs once per token
does not. This is the measured basis for the proposal's "coarse stages only".
The 890M's 4.8× includes transport, and deserves a caveat: the iGPU hit
1.32 TFLOP/s, matching its recorded 1.23 fp32 rate, while the CPU path reached
only 0.26 TFLOP/s against a 1.54 peak. It is as much a slow CPU path at that
shape as a fast iGPU. Separately measured this session: the 890M does
**2.85 TFLOP/s in fp16**, 2.3× the fp32-only figure the earlier notes carried.
## Exporting a graph these backends will accept
Run `sanitize_onnx` on every dynamo export. It is not a nicety.
torch's dynamo exporter emits `Reshape` with `allowzero=1`. onnxruntime's
DirectML provider rejects that at **session initialisation** with a bare
`E_INVALIDARG` out of `MLOperatorAuthorImpl.cpp` — and onnxruntime then does not
raise. It prints `Falling back to ['CPUExecutionProvider'] and retrying` and
hands back a working CPU session. Measured here: the 27B vision tower placed
**0 of 1171 nodes** on the iGPU and computed correct features entirely on the
CPU. With 58 `allowzero` attributes cleared, the same graph placed **85 of 113**.
`vllm_omni.edge.qnn_export.sanitize_onnx` was written for Qualcomm's converter
and fixes exactly this, along with `IsNaN`/`Where` guard pairs and rank-0 `axes`
initializers — all generic dynamo warts rather than vendor-specific ones. It
preserves external data, so a multi-gigabyte export stays a pair of files.
`vllm_omni.edge.local.artifacts.vision_tower.export_onnx` always calls it.
Two independent guards catch the fallback if a graph slips through: the worker
compares `session.get_providers()` against what was requested and reports the
swap by name, and the node-assignment gate refuses on the placement itself.
## Known limits
- **A16W8 is the only door on XDNA2.** A8W8 — what `quantize_static` produces by
default — and fp32 are declined with no error and every node left on the CPU.
- **Dequantize nodes stay on the CPU** by the partitioner's own account, so a
graph that is mostly requantization will not benefit.
- **TorchScript cannot be placed on DirectML** (torch 2.4.1 / torch-directml
0.2.5): `map_location` raises `Cannot access storage of OpaqueTensorImpl`, and
load-then-`.to(device)` trips `assert isinstance(param, Parameter)` inside
`Module._apply`. Use `torch.export` `.pt2`, which is what
`vllm_omni.edge.decoder_export` already emits — and do not call `.eval()` on
the loaded module, which 2.4.1 does not support.
- Small graphs cannot amortise dispatch; the NPU's advantage grows with GEMM
size (2.6× at 256×1024×1024, 5.3× at 1024×2048×2048).
Raw records: [`analysis/experiments/m3_amd_devices_20260915/`](../../../analysis/experiments/m3_amd_devices_20260915/README.md)
docs/edge/laptop-heterogeneous.md
+466
View File
@@ -0,0 +1,466 @@
# Qwen3.8-27B and Spark-X2.5-4B on a Ryzen AI laptop
Target machine: **AMD Ryzen AI 9 HX 370** (Zen 5, 12 cores / 24 threads,
AVX-512 with VNNI and BF16, **no AMX**), 30.9 GiB LPDDR5X, **RTX 5090 Laptop**
(24463 MiB, sm_120, PCIe 4.0 x8), **Radeon 890M** iGPU and an **XDNA2 NPU**,
running under **WSL2** (kernel 6.18.33.2).
Two models were added and both run:
| model | status | evidence |
|---|---|---|
| `XHToken/Spark-X2.5-4B` | runs; **24/24 greedy tokens identical to the HF reference** | `analysis/experiments/spark_edge/parity_hf_vs_vllm.py` |
| `Qwen/Qwen3.8-27B-FP8` | runs; correct output; 1.05 tok/s at the tightest fit | offload sweep below |
Neither needed a new model class. Spark-4B is the same `Spark2_5ForCausalLM`
the 1.7B uses and the implementation was already config-driven; Qwen3.8 is
`Qwen3_5ForConditionalGeneration`, which vLLM 0.28 already carries. What both
needed was the *sizing* and *placement* code around them, which is what this
document is about, and four environment faults, which are in
[§7](#7-what-was-actually-broken).
---
## 1. Three accelerators, and what each one costs to reach
The request was to use the CPU, the NPU and the GPU together. An earlier
version of this document said the iGPU and the NPU were simply unreachable
under WSL2. That was true of the default setup and false as a statement about
the machine: both have a route, the routes are not equivalent, and finding them
changed one of the two answers.
| device | reachable from WSL? | route | state |
|---|---|---|---|
| RTX 5090 Laptop | yes | `/dev/dxg` → `torch.cuda` | working |
| Radeon 890M iGPU | **yes** | DirectML over `/dev/dxg` (`torch-directml`) | **working, and slower than the CPU** |
| XDNA2 NPU | no | Windows-side ONNX Runtime + **VitisAI** EP | **working** — 2.6–5.3× the CPU on GEMMs |
### The iGPU: reachable, and measured not worth it
There is no `/dev/dri` under WSL2, and there does not need to be. Mesa ships a
D3D12 Gallium driver and Microsoft ships `libd3d12.so` / `libdxcore.so` in
`/usr/lib/wsl/lib`, so any WDDM adapter is addressable through `/dev/dxg`.
`glxinfo` already reports `D3D12 (AMD Radeon(TM) 890M Graphics)`, accelerated,
OpenGL 4.6. For compute, `torch-directml` offers it as a torch device:
```
DirectML device_count: 2
[0] NVIDIA GeForce RTX 5090 Laptop GPU
[1] AMD Radeon(TM) 890M Graphics
```
It computes correctly — a 512×512 matmul agrees with the CPU to **3.6e-07**.
Then it was benchmarked, which is why this section is short:
| | iGPU via DirectML | CPU | ratio |
|---|---:|---:|---|
| fp32 GEMM 2048³ | 1.12 TFLOP/s | 1.54 TFLOP/s | **0.73×** |
| memory bandwidth | 5.0–7.4 GB/s | 20.7 GB/s | **0.30×** |
| host↔device, 64 MB | 39.2 ms (3.3 GB/s) | — | — |
The iGPU is **slower than the CPU it shares memory with, on both axes**: the
D3D12 translation layer costs more than the hardware wins. It is genuinely
usable now, and the honest advice is to route work to it only when the point is
to free the CPU rather than to go faster. A native path — ROCm on bare-metal
Linux, or DirectML inside a Windows process — would not pay the translation and
is the version worth measuring.
Note the pin conflict: `torch-directml` requires `torch==2.4.1`, so it can
never share a virtualenv with the engine (torch 2.13). It gets its own.
### The NPU: working, at 2.6-5.3x the CPU
Not from WSL, and no driver install changes that: it is an **MCDM** device
(`IpuMcdmDriver`, AMD 32.0.203.329), and WSL2's GPU-PV forwards WDDM *display*
adapters only, so it has no device node in Linux. PCI passthrough is not a WSL2
feature. It has to be driven from a Windows-side process.
From Windows it works. Three things have to be right **at once**, and each one
failing on its own produces exactly the same symptom — a session that builds,
runs, returns correct numbers, and executes every node on the CPU:
1. **The right EP.** The appx package ships two.
`onnxruntime_providers_ryzenai.dll` registers cleanly, enumerates the NPU as
`OrtHardwareDeviceType.NPU`, opens a session — and claims **zero nodes** from
everything: fp32, INT8 QDQ symmetric and asymmetric, per-channel,
power-of-two scales, MatMul and Conv, and ahead-of-time `ModelCompiler`
(which emitted zero `EPContext` nodes). It is the *Light* EP. The general one
is `onnxruntime_vitisai_ep.dll`.
2. **The DLL search path.** That one fails to load from a bare
`register_execution_provider_library` with `Error loading … But no
dependency`. It needs its own directory added via `os.add_dll_directory`
first, because it pulls `dyn_dispatch_core.dll` and `vaiml.dll` from beside
it.
3. **A16W8, not A8W8.** The overlay names are embedded in the DLL and say so:
```
4x2_psf_model_a8w8_qdq.xclbin
8x4_psu_model_a16w8_qdq.xclbin <- most of them
2x4x4_bmm_model_a16w16.xclbin
```
XDNA2 wants **16-bit activations with 8-bit weights**. A8W8 — the default
shape of `quantize_static` — is rejected as completely as fp32, silently.
Also needed: **onnxruntime ≥ 1.25**, because the EP requests ORT API 25.
`onnxruntime-directml` stops at 1.24.4 and is refused, so the NPU and DirectML
cannot share a process.
With all of it right the partitioner starts talking and names its target,
`AMD_AIE2P_4x8_CMC_Overlay`. Measured against the same INT8 graph on the CPU:
| GEMM | CPU | NPU | speedup | NPU |
|---|---:|---:|---:|---:|
| 256 × 1024 × 1024 | 1.47 ms | 0.56 ms | 2.64× | 0.97 TOPS |
| 512 × 2048 × 2048 | 9.64 ms | 2.14 ms | 4.51× | 2.01 TOPS |
| 1024 × 2048 × 2048 | 19.89 ms | 3.76 ms | **5.29×** | **2.29 TOPS** |
Max relative error against the CPU: **3.2e-05**. It scales the right way — a
small GEMM cannot amortise the dispatch. Dequantize nodes stay on the CPU (the
partitioner says so explicitly), so a graph that is mostly requantization gains
nothing.
The recipe is code in `vllm_omni/edge/npu_ryzenai.py`, including the one check
worth trusting: **count nodes in an ORT profile.** A session reporting the EP in
`get_providers()` may still be running everything on the CPU — and when it does,
its outputs are bit-identical to the CPU, which is precisely what a *correct*
NPU run does not look like. A 0.000e+00 error is the bug signature, not the
success signature.
**What it is worth here.** Not decode: 2.29 TOPS against the dGPU's
48.2 TFLOP/s, over the same 20.7 GB/s system memory the CPU uses, cannot help a
bandwidth-bound step (§3). It earns its place on the compute-bound work off the
per-token path — a vision tower, a draft model for speculative decoding — where
5× the CPU on large GEMMs is real.
### Reaching a phone
The same question for mobile has a cleaner answer. The harness's `AdbDevice`
backend needs only an `adb` binary and a reachable device, and WSL2 is in NAT
mode, so **outbound** connections to the LAN work even though inbound do not.
That makes wireless debugging the path of least resistance:
```bash
sudo apt-get install -y adb # done; adb 1.0.41 (34.0.5)
adb connect <phone-ip>:5555 # Android 11+ wireless debugging
```
No USB passthrough is involved, which matters because WSL2 has none by default
— the USB route needs `usbipd-win` on the Windows side, and is worth the extra
moving part only when the phone cannot be on the same network. `AdbDevice` also
takes an explicit binary path, so `adb="/mnt/c/.../adb.exe"` drives a
USB-attached phone through the Windows adb server with no passthrough at all,
at the cost of translating paths on every `push`/`pull`.
The mobile **export** path is now working as well: the QNN export tests needed
`onnxscript` and `tabulate`, and the ExecuTorch lowering needed `flatc`. All
three are installed, and the one remaining ExecuTorch failure turned out not to
be a failure — pytest run from the repository root lets the bare `executorch/`
source checkout shadow the installed package. Run from `vllm-omni/` the suite
is **124 passed, 1 skipped**.
## 2. The measurement that decides the design
Taken on an **idle** machine (see §7 for why that word is load-bearing):
| path | measured |
|---|---|
| GPU VRAM bandwidth | **472 GB/s** |
| GPU bf16 GEMM 8192³ | **48.2 TFLOP/s** (91 W, 1447 MHz) |
| PCIe H2D, pinned, bulk | **13.9 GB/s** |
| PCIe, as seen by offloaded weight reads | **10.6 GB/s** (§5) |
| CPU RAM bandwidth | **20.7 GB/s** |
| CPU bf16 GEMM 4096³ | **1.54 TFLOP/s** |
Two ratios follow, and everything else in this document is a consequence:
```
VRAM : system RAM = 22.8 x
VRAM : PCIe link = 34 x (44x on the effective figure)
```
Autoregressive decode reads **every weight once per token** and does two FLOPs
per weight. At batch 1 it is bandwidth-bound with no room for argument, so
decode rate is set by *where the weights are*, and almost nothing else.
---
## 3. Why "use all three devices for decode" is the wrong goal
The intuition -- three processors, so split the model three ways -- inverts on
a bandwidth-bound workload. Splitting weights across pools does not add
bandwidth; the step ends when the *slowest* pool finishes, and the slow pool is
22.8x slower. Putting 10% of a model on the CPU makes the step
`0.9/472 + 0.1/20.7`, which is **3.2x slower** than putting all of it on the
GPU, not 10% slower.
The iGPU and the NPU do not escape this: both read the same LPDDR5X the CPU
does, so all three share one 20.7 GB/s pool — and the iGPU, measured, reaches
only 5-7.4 GB/s of it (§1). Adding them to a decode split adds arithmetic to a
problem that is not short of arithmetic, over a pool that is already the
bottleneck.
So the rule is: **for decode, one device, and it is the one with the memory
bandwidth.** The other devices earn their place on work that is *not* on the
per-token critical path:
| device | job | why it fits there |
|---|---|---|
| dGPU | every decode step, all resident layers | the only 472 GB/s in the machine |
| CPU | `embed_tokens` gather, detokenisation, scheduling, prompt templating | a gather reads one 10 KiB row per token, not the 2.54 GB table |
| NPU | the **vision tower** on image prompts; a **draft model** for speculative decoding | both are compute-bound and off the decode path — *pending the compiler in §1* |
| iGPU | the same class of work, when the goal is to free the CPU | reachable today, but measured at 0.73x the CPU's compute and 0.30x its bandwidth (§1) |
The vision tower is the clean case: 0.92 GB, run **once per image**, never per
token. Moving it off the GPU frees 0.92 GB of VRAM -- which by §5 is worth
86 ms/token -- and costs one prefill-time hop. That is a real heterogeneous win
and it does not touch the decode loop.
Speculative decoding is the other one, and it is the only way a second device
raises *decode* throughput: a draft model on the NPU proposes k tokens, the GPU
verifies them in a single batched pass, and the GPU's per-token weight read is
amortised over k tokens. Qwen3.8-27B **ships an MTP head for exactly this**
(`mtp_num_hidden_layers: 1`, 0.48 GB, in `mtp.safetensors`).
---
## 4. What the two models actually cost
`vllm_omni/edge/kv_budget.py` now prices both. It could price neither before:
it read `num_hidden_layers` off the top-level config, which `Qwen3_5Config`
does not have, and it charged every layer for a KV cache.
**Qwen3.8-27B-FP8** is a hybrid of a kind the old code had no concept of: of its
64 layers, **48 are gated DeltaNet**, which keep a fixed-size recurrent state
and *no KV cache at any context length*. Only the 16 full-attention layers
cache.
```
flat over 64 layers 256 KiB/token <- what a layer-blind budget charges
16 full layers only 64 KiB/token <- the truth, 4x smaller
GDN recurrent state 147 MiB/sequence <- constant, and previously invisible
```
The state is constant and the cache is linear, so they cross at ~2350 tokens:
below that, the thing the old budget did not count at all is the larger half.
At a 4096-token context the whole cache plus state is **467 MiB** (147 MiB of
state, 320 MiB of cache), and **787 MiB** at 8192 -- against 30.87 GB of
weights. For this model, **memory is a weights problem and not a
cache problem**, and long context is nearly free. That is the architectural
gift of the 3:1 linear hybrid and it is the strongest reason to want this model
on an edge box.
**Spark-X2.5-4B** is the other hybrid: 27 sliding-window(512) layers to 9 full
ones, 144 KiB/token flat, and a **3.37x** discount if the backend accounts per
layer type. Both sizes now have their published geometry pinned in
`tests/model_executor/models/spark2_5/test_spark2_5.py`, so a change that
assumes the 1.7B's 8 query heads or 28 layers fails in CI rather than on a
checkpoint.
---
## 5. The cost of not fitting, measured
Qwen3.8-27B-FP8 is **30.87 GB** of weights against a 24 GiB card. It only runs
with part of it on the host. Sweeping how much (4096-token context, batch 1,
greedy, `enforce_eager`, idle machine):
| offloaded | tok/s | s/token | marginal |
|---|---:|---:|---|
| 10 GB (the tightest fit; 9 GB fails) | **1.05** | 0.952 | |
| 12 GB | 0.89 | 1.124 | +0.086 s/GB |
| 14 GB | 0.76 | 1.316 | +0.096 s/GB |
| 16 GB | 0.66 | 1.515 | +0.100 s/GB |
Straight line, slope **+94 ms per offloaded GB**, i.e. an effective
**10.6 GB/s** -- the 13.9 GB/s bulk figure less ~20% of overhead. The model
```
s/token = resident_GB / 472 + offloaded_GB / 10.6
```
predicts the 10 GB point at 987 ms against 952 ms measured, and the whole curve
within 5%. At the tightest fit, **the 10 GB on the host is 94% of the step and
the 20.9 GB in VRAM is 4%.**
This is the number to design against: **a gigabyte that does not fit costs
44x what a gigabyte that fits costs** (94 ms against 2.1 ms). Fitting is not
one optimisation among several; it is the only one that matters until it is
solved.
### Which gigabytes to spend
vLLM's `cpu_offload_gb` offloads "non-selectively until the limit is reached",
which is the expensive way. Not every parameter is read once per token, and
`vllm_omni/edge/placement.py` sorts them:
```
lm_head 2.543 GB HOT one 248320 x 5120 GEMM per token
embed_tokens 2.543 GB COLD a gather: one row, ~10 KiB per token
visual.* 0.922 GB COLD unless the prompt carries an image
mtp.* 0.478 GB COLD unless speculative decoding is on
---------
cold set 3.94 GB free to evict
```
`embed_tokens` and `lm_head` are the same shape and the same dtype and the
opposite decision. A size-ordered, name-blind policy cannot tell them apart;
`cpu_offload_params`, which matches name segments, can.
For FP8 the cold set is 3.94 GB against a ~10.6 GB deficit, so it helps but
cannot close it -- **selective placement does not rescue this checkpoint**.
And a larger caveat, found after this section was first written: *the cold set
cannot currently be offloaded at all*. `cpu_offload_params` is real and matches
name segments as documented, but the offloader is only ever offered decoder
layers (`make_layers`, `utils.py:812`), so `embed_tokens`, `lm_head`, the vision
tower and the MTP head are out of its reach. The classification below is right
about which bytes are cheap to move; moving them needs an upstream change. See
§6.
---
## 6. The design: make it fit — and the retraction that follows
The previous version of this section said: use a 4-bit checkpoint, it is worth
~20×, everything else is worth less than 2×. That was arithmetic —
`resident_GB / 472 GB/s` — and the arithmetic was **wrong by roughly 30×**. The
4-bit build was then measured, and it is *slower* than the 8-bit one.
| build | weights | offloaded | decode |
|---|---:|---:|---:|
| `Qwen/Qwen3.8-27B-FP8` | 30.87 GB | 10 GB | **1.05 tok/s** |
| `nvidia/Qwen3.8-27B-NVFP4` | 21.95 GB | 3 GB | **0.67 tok/s** |
| the same, torch.compile + CUDA graphs | 21.95 GB | 3 GB | 0.72 tok/s |
| the same, 4 GB offloaded | 21.95 GB | 4 GB | 0.56 tok/s |
Three things went wrong at once, and each is worth stating separately.
**It does not fit, even at 21.95 GB.** The weights load to 19.92 GiB and the
card has 22.58 GiB free (the Windows desktop holds the rest). vLLM's runtime
overhead — non-torch allocations plus the activation peak — is about **2.8 GiB**
on this model, which leaves the KV cache at **−0.31 GiB**. Shrinking
`max_model_len` from 4096 to 1024 moved that to −0.30: the overhead is not the
cache, so context length cannot buy the fit.
**The lever that would have made it fit does not exist.** §5 identified 4.31 GB
of cold parameters and prescribed `cpu_offload_params={embed_tokens, visual,
mtp}`. The flag is real and matches name segments as documented. But the
offloader is only ever *offered decoder layers*: `wrap_modules` is called from
inside `make_layers` (`vllm/model_executor/models/utils.py:812`) and nowhere
else. `embed_tokens`, `lm_head`, the vision tower and the MTP head are built
outside it. Asking for the cold set moved **~0.5 GiB** and the run failed
identically to one with no offload at all. The classification was right about
which bytes are cheap to move; the mechanism to move them is missing, and
closing it is an upstream change.
**And fewer bytes turned out not to mean faster.** This is the part that
invalidates the projection rather than merely complicating it. The traffic model
`resident/472 + offloaded/10.6` fits the FP8 offload curve to within 4.3% across
four points. Against NVFP4 it is **4.7× optimistic**: predicted 3.1 tok/s at
3 GB offloaded, measured 0.67. Its own offload slope is 0.293 s/GB — an
effective **3.4 GB/s**, against FP8's 10.6 — and extrapolating to zero offload
gives ~**1.6 tok/s** even if it fit. Enabling torch.compile and CUDA graphs
changed 0.67 to 0.72, so this is not an eager-mode artifact.
The conclusion is that **NVFP4 decode on this stack is compute-bound, not
bandwidth-bound**, and a model that counts bytes cannot see that. Why it is
compute-bound is not established here: the checkpoint is a `modelopt` mix of
FP8 and NVFP4 with group-16 scales, and this fork already carries a correctness
patch for the W4A4 NVFP4 linear path, which is a hint that the path is not a
well-trodden one on sm_120. Diagnosing it is the obvious next experiment, and
until someone does, the honest statement is the measurement.
### What the design actually is, then
1. **Run the FP8 checkpoint with the smallest offload that fits** — 10 GB, 1.05
tok/s. This is the fastest configuration measured on this machine, and it is
the 8-bit one.
2. **Do not change quantization format on an estimate.** The one estimate this
document made without measuring is the one it got wrong, by 30×.
3. **Size the cache from the model** — 467 MiB at 4096 tokens (§4). Still right,
and still worth little here, because weights dominate.
4. **Speculative decoding via the shipped MTP head** is now the largest
*unexplored* lever, and the only one that attacks the per-token weight read
rather than working within it. It is untested here.
5. **Getting the cold set offloadable upstream** would free 3.94 GB on FP8, worth
~370 ms/token by the traffic model — which, on the FP8 path where that model
demonstrably holds, is the difference between 1.05 and roughly 1.6 tok/s.
The blunt summary: a 27B model on a 24 GB laptop GPU runs at **about one token
per second**, and the things that would change that by an order of magnitude —
a checkpoint that genuinely fits, or speculative decoding — are not yet in hand.
## 7. Where this beats llama.cpp, and where it does not
llama.cpp at the pinned revision **does** support this architecture -- `QWEN35`,
`QWEN35MOE` and `QWEN3NEXT` are in its arch registry and it has gated-DeltaNet
kernels. It is a real competitor here, and on a fitted 4-bit model on one
stream it is a strong one: this repository's own CPU measurements have
llama.cpp's Q4_K_M decode at 90 tok/s against our 74, and `docs/edge/` explains
that the gap is the execution model rather than the kernels.
So the case for vLLM-Omni on this machine is not raw single-stream decode. It is:
| axis | why |
|---|---|
| **Prefill** | measured ~2x llama.cpp on the CPU path (2190 vs 1121 tok/s, pp512). Prompt processing is compute-bound, which is the one regime where the hardware has headroom. |
| **Concurrency** | continuous batching and paged KV. At batch 1 they cost overhead; past ~4 streams they are the whole game, and llama.cpp has no equivalent. |
| **Speculative decoding** | the shipped MTP head is wired in vLLM and is not a GGUF concept. |
| **Multimodal pipelines** | the reason this fork exists: staged vision/audio/codec execution with an OpenAI-compatible server. |
| **Hybrid cache accounting** | 64 KiB/token instead of 256, once §4 is honoured. |
A design that claims to beat llama.cpp at batch-1 decode of a fitted model on
one GPU is claiming something this project has already measured to be false.
The defensible claim is the four rows above, and the way to establish it is
`benchmarks/edge_harness/` against `baselines/`, not assertion.
---
## 8. What was actually broken
Four faults stood between "the models are supported" and "the models run", none
of them in the model code:
1. **`RuntimeError: UVA is not available`.** vLLM disables pinned memory on WSL
by default; its own gate is kernel ≥ 4.19.121 and this kernel is 6.18. Fixed
by `VLLM_WSL2_ENABLE_PIN_MEMORY=1`. Without it the CUDA runner does not
start **at all**, for any model.
2. **No C compiler**, so Triton could not JIT and the GDN kernels could not
build. `gcc` plus `libc6-dev` (gcc alone did not pull the headers).
3. **FlashInfer's sampler wants `nvcc`**, which a runtime-only CUDA install does
not have. `VLLM_USE_FLASHINFER_SAMPLER=0`. Note this surfaced as a failure in
*warmup*, long after the model had loaded and run a forward pass.
4. **transformers 5 renamed `input_embeds` → `inputs_embeds`**, which broke the
vendored Spark modeling file used as the parity *reference*. Shimmed in
`parity_hf_vs_vllm.py` rather than by editing the cached module, so the
reference stays the code the checkpoint ships.
The required environment, in full:
```bash
export VLLM_WSL2_ENABLE_PIN_MEMORY=1
export VLLM_USE_FLASHINFER_SAMPLER=0
sudo apt-get install -y gcc libc6-dev
```
### And one measurement error, which is the real lesson
The first roofline taken on this machine read **3.67 GB/s** H2D, **38 GB/s**
VRAM and **9 TFLOP/s**, with `nvidia-smi` reporting a 33 W power cap and a
220 MHz SM clock. The conclusion drawn from it -- that the dGPU had no
bandwidth advantage over the CPU and the design should lean on the CPU -- was
wrong in every part.
Those numbers were taken **while two model downloads were saturating the disk,
the network and the CPU**. Re-measured on an idle machine the same calls give
13.9 GB/s, 472 GB/s and 48.2 TFLOP/s, at 91 W and 1447 MHz, with the power
limit reading 100.98 W. The GPU was never capped; it was idling because the
benchmark could not feed it.
This repository already has [a methodology
document](methodology.md) that exists because of five retractions of exactly
this shape. This is the sixth, and the rule it re-teaches is the one already
written down: **a number taken on a contended machine is not a number.** The
harness has a contention guard for this; it was not in the path of an
interactive measurement, which is precisely where the mistake got made.
docs/edge/local-text-mode.md
+286
View File
@@ -0,0 +1,286 @@
# The local text mode: one stage, one device, and a plan you can argue with
`vllm_omni.edge.local` runs a single autoregressive text stage on whichever
device *this process* can actually drive, and writes down why. It is the M0
milestone of the [single-device engine
proposal](../../../analysis/architecture_local_engine_20260915.md): the smallest
thing that turns "a configuration that happens to run" into "a configuration
that can be explained, reproduced and refused".
It adds no kernels, no scheduler and no graph compiler. vLLM keeps doing all of
that inside the Omni stage. What this adds is the layer above: which device,
which artifact, how many bytes, and what happens when the answer is no.
```bash
python -m vllm_omni.edge.local devices # what this process can drive
python -m vllm_omni.edge.local plan --model <ckpt> # exit 2 == explicit refusal
python -m vllm_omni.edge.local run --model <ckpt> --prompt "..."
python -m vllm_omni.edge.local accept --model <ckpt> --json out.json
```
---
## 1. "It fits" is not "it runs"
`models/Spark-X2.5-1.7B-int8` is 1.84 GiB of compressed-tensors int8 W8A8. The
RTX 5090 Laptop in the reference machine has 23.89 GiB. Every capacity check
passes, the engine starts, the weights load — and the first forward dies:
```
dispatch_scaled_mm, csrc/.../c3x/scaled_mm_helper.hpp:34,
Int8 not supported on SM120. Use FP8 quantization instead, or run on older arch (SM < 100).
```
cutlass' `c3x` has no int8 `scaled_mm` for SM >= 100, and sm_120 is inside that
range. Roughly forty seconds and a stack trace from a subprocess to learn
something that is written in the checkpoint's `quantization_config` and the
device's compute capability.
The planner reads both and refuses in about a second:
```
REFUSED: no device in this process can run this checkpoint as asked.
[weight_format_unsupported] cuda:0: the checkpoint is compressed-tensors:int8 and
cuda:0 (NVIDIA GeForce RTX 5090 Laptop GPU, sm_120) executes only
['awq', 'compressed-tensors:fp8', 'dense', 'gptq', 'modelopt_fp4']
remedy: this device has fp8 tensor cores but no int8 scaled_mm path
(cutlass c3x has none for SM >= 100). Use an fp8 or a dense build here,
or run the int8 build on the CPU backend (.venvs/omni-cpu).
```
The remedy is not decoration. The same checkpoint completes the full acceptance
run on the CPU backend, at 16.3 tok/s median — so this is a property of the
device, not a defect in the artifact, and the refusal says which.
## 2. What a device *is*, here
`hardware_probe` answers "what is in the box". That is the wrong question at
load time, because a vLLM wheel is built for one platform: the CUDA wheel
cannot run a CPU stage however many cores the probe finds, and the CPU wheel
cannot reach the card beside it. So a device is runnable only when the probe
finds it **and** `vllm.platforms.current_platform` is the platform that drives
it. Everything else is enumerated with the reason it is not available:
```
[runnable] cuda:0 NVIDIA GeForce RTX 5090 Laptop GPU (23.9GiB vram) backend=vllm:cuda
[not here] npu:amd NPU Compute Accelerator Device (host_ram) backend=None
An MCDM device; WSL2's GPU-PV forwards WDDM display adapters only...
[not here] gpu_integrated:amd AMD Radeon(TM) 890M Graphics (host_ram) backend=None
No /dev/dri under WSL2, but reachable through DirectML over /dev/dxg...
[not here] cpu AMD Ryzen AI 9 HX 370 (30.9GiB host_ram) backend=None
vllm.platforms.current_platform is 'cuda', not 'cpu'...
```
The iGPU and the NPU are reported rather than dropped, because "the machine has
an NPU" and "a stage can run on it" are different claims and a plan has to be
able to show which one it relied on. They share `host_ram` with the CPU and
carry zero bytes, so nothing can accidentally add three copies of the same RAM
together.
`--mask gpu_discrete` hides the discrete GPU to exercise the no-NVIDIA
deployment class. It is a policy, not amnesia: the device stays in the record
with `masked: true` and its real memory.
## 3. The budget is a sum, and every term is visible
```
ADMITTED on cuda:0 (NVIDIA GeForce RTX 5090 Laptop GPU) via vllm:cuda
artifact 43e867b180f1a3d2 dense 7.66GiB
budget 12.51GiB of 23.89GiB vram (total)
weights 7843 MiB vram session_weights D 5 safetensors shard(s), on-disk size
kv_cache 720 MiB vram session E flat budget at 1x context; 248 MiB if
the backend accounts 27 sliding layers separately
activations 120 MiB vram request E 2048-token prefill chunk x 12 live tensors
backend_workspace 512 MiB vram session E attention scratch, driver/allocator context
load_transient 1569 MiB vram load E freed once the weights are resident
external_reserve 1024 MiB vram session E memory this engine does not own
safety_margin 1024 MiB vram session E fragmentation and allocator slack
```
`D` is read off the artifact; `E` is estimated. The peak is a **max**, not a
sum: the loader's transient is gone before vLLM sizes the KV pool, so adding
both would refuse plans that fit.
The KV line comes from `vllm_omni.edge.kv_budget`, which reads the real
`layer_types`. Spark-4B is 9 full-attention and 27 sliding@512 layers, so a
backend that accounts them separately needs 248 MiB where a flat charge needs
720. The budget takes the flat number (it is the safe one) and reports the
other.
**The estimates are checked, not believed.** After every run the engine reports
the budgeted peak against the sampled one:
| | CUDA / Spark-4B bf16 | CPU / Spark-1.7B int8 |
|---|---:|---:|
| counter | `nvml_device_used` (whole card) | `tree_rss` (process tree) |
| budgeted peak | 12.51 GiB | 6.55 GiB |
| measured attributable peak | 8.76 GiB | 5.48 GiB |
| budget / measured | 1.43x | 1.20x |
A budget above the measurement is the intended direction — admission needs a
provable upper bound, not a prediction — but 1.40x is loose enough to be worth
narrowing, and the number is in the record so somebody can.
### The flag that is not GPU-only
The plan sets `gpu_memory_utilization` on **both** pools, derived from its own
budget. On the CPU backend vLLM reads that same field as the fraction of *host
RAM* to reserve, so leaving it at its default there is not "no opinion" — it is
asking for 92% of the machine. A 6.55 GiB CPU plan failed to start with
`desired CPU memory utilization (0.92, 28.43 GiB)` until this was fixed.
The KV pool is set as an absolute `kv_cache_memory_bytes` rather than left as
whatever remains inside that fraction, so the cache is the size the request
limits imply. Both backends honour it exactly: CUDA logs 754974720 bytes, CPU
logs `Explicitly set (0.14/30.91) GiB for KV cache`.
## 4. Streaming, state and cancellation
Three contracts, which is what the proposal says the first version needs:
* **`ChunkEvent`** carries the *delta*, never the output so far, plus a `seq`
that detects loss and an `epoch` that retires it.
* **`StateHandle`** is an opaque reference to state the backend owns. It is
tied to the artifact that produced it (the same shapes from a different
quantization are not the same state) and is never `migratable` — moving a KV
cache between backends needs a measured layout conversion for the specific
pair.
* **`BoundedEventStream`** bounds chunks *and* bytes and blocks the producer,
which is the backpressure. A single event larger than the whole byte bound
still passes on an empty queue, or it would deadlock.
Cancelling is not "stop calling `next()`". The backend has work in flight and
its tokens arrive after the cancel. `cancel()` retires the epoch first, then
aborts in the backend, then closes the stream — *before* waiting on the
producer, so a backend that hangs while unwinding delays cleanup rather than
the caller. In the recorded runs the fence discarded 3 in-flight events on CUDA
and 2 on CPU, and zero events from the retired epoch reached the consumer.
The device counter does **not** fall on cancel, because vLLM's KV pool is
pre-allocated. What falls is the in-flight count. The report says so, so that
"memory did not drop" is not read as "nothing was released".
## 5. Measuring memory without fooling yourself
Three counters, three different accountings, never summed:
* `nvml_device_used` is the whole card, every process. Under WSL2
`nvmlDeviceGetComputeRunningProcesses` returns an empty list, so GPU bytes
**cannot** be attributed to this engine's PIDs on this host. "Attributable
peak" means peak minus a pre-load baseline (2.79 GiB here), not a private
total.
* `tree_pss` includes a proportional share of file-backed pages, which on an
mmap'd safetensors load is partly page cache.
* `host_available` moves for reasons that are not us.
Sampling is every 0.25 s, so every peak is a lower bound, and the record says
that rather than assuming it away.
## 6. What Spark needed before any of this worked
Two gaps that made "the model is in the registry" and "the model runs under
Omni" different statements:
1. **No `PipelineConfig`.** Spark's model definition had been registered for a
while, but with no pipeline `Omni(model=<Spark checkpoint>)` had nothing to
resolve, so the model was reachable only through plain `vllm.LLM`. Added as
a single `LLM_AR` text stage in
`vllm_omni/model_executor/models/spark2_5/pipeline.py`.
2. **A stage-call mismatch.** Omni's AR model runner hands every stage model
`sampling_metadata`, `logits_index` and `sampler`. Spark's
`ForCausalLM.forward` caught them in `**kwargs` and forwarded them into
`Spark2_5Model.forward`, which takes four arguments — `TypeError` on the
first forward. They now stop at the `ForCausalLM` boundary, where the
decoder stack has no use for them anyway.
One stage is the design, not a placeholder. An autoregressive session stays on
one backend by default: prefill and decode share the KV cache, and a per-token
device boundary buys nothing and costs a synchronization. Multi-stage is M2,
and it needs the tensor/buffer contract that a single text stage — which hands
a buffer to nobody — cannot be used to design.
## 7. Does the Omni path change the answer?
This is the one claim the rest of M0 cannot make for itself. Everything above
is about the engine's own behaviour; this asks whether routing a decode through
the Omni stage changed the model's output. The validation plan makes it a
*hard* initial acceptance condition.
Same checkpoint, same device, same request limits, same environment, greedy
with `ignore_eos` -- the only difference is the path: plain `vllm.LLM` straight
into vLLM against `AsyncOmni` -> StageRuntime -> the stage's EngineCore.
**12 of 12 prompts, 128 greedy tokens each, token-for-token identical -- on
both paths**: CUDA with the bf16 build and CPU with the int8 one. The quantized
build gets its own run rather than inheriting the result, because quantization
is exactly what makes a near-tie likelier to flip.
The comparison is on **token ids, not text**: two different id sequences can
detokenize to the same string, and a text comparison would hide exactly the
drift worth catching. The record keeps the first divergent position rather than
a yes/no, because greedy decoding is not bitwise stable across batch shapes --
a different prefill chunking reorders reductions and can flip an argmax at a
near-tie, so "split at token 120" and "split at token 3" are different
findings. Neither occurred.
The two engines load sequentially in one process, reference first and released
before the second starts; two copies of a 7.66 GB checkpoint would not fit the
budget this milestone is about respecting.
```bash
python -m vllm_omni.edge.local parity --model <ckpt> --json out.json # exit 3 on divergence
```
### The two paths do not need the same environment
The reference cannot start on this host with a default environment, and both
reasons are worth knowing:
* under WSL2, vLLM's own GPU model runner allocates a `UvaBuffer` for its input
batch and raises `RuntimeError: UVA is not available` without pinned memory.
Omni's AR model runner does not take that path and starts either way.
* FlashInfer's sampler JIT-compiles on first use and needs `nvcc`, which is not
installed -- `nvidia-smi` reporting a CUDA version is not a toolkit. This one
kills the engine at the *first token*, not at load.
Both now live in one `RUNTIME_ENV` applied by every entry point that starts an
engine, because a comparison whose reference is configured differently from its
subject is measuring the configuration.
## 8. Limits
* One model, one stage; no multi-stage, no tensor/buffer contract.
* `--mask` is a process-level policy, not a physical absence. A full no-card
verification needs another machine.
* The iGPU and NPU are enumerated only; neither has a stage backend (M3).
* Reference consistency covers **one decode mode**: single sequence, greedy,
128 tokens, on both paths. It does not cover concurrency above 1 (which
changes batch shapes, and greedy is not bitwise stable across those), longer
outputs, or non-greedy sampling -- and it is not a quality evaluation. It
says the Omni path did not change the output, not that the output is good.
* Performance figures are one sample per prompt, not p50/p95, with no fixed
power profile and no cache-cold guarantee.
## 9. Tests
```bash
# hermetic; no GPU, no weights, ~3 s
pytest tests/edge/local --ignore=tests/edge/local/test_e2e_local_text.py
# end to end against a real checkpoint; the venv decides which artifact
pytest tests/edge/local/test_e2e_local_text.py
```
The e2e tests run the acceptance criteria themselves: both startups reach a
decision, twelve fixed prompts each reach 128 tokens, the budget bounds the
measured peak, cancellation leaks nothing, and the Omni path decodes what plain
vLLM decodes. They skip rather than fail when no checkpoint is present.
**78 pass in each venv**; about 4 min on this machine either way.
The twelve prompts are the same twelve
`analysis/experiments/spark_edge/quality_int4.py` measures the quantized builds
against, so these numbers stay comparable with the existing W4A16 / W4A8 /
Q4_K_M fidelity work.
Full run records:
[`analysis/experiments/m0_local_text_20260915/`](../../../analysis/experiments/m0_local_text_20260915/README.md).