vllm-omni/README.md

# vllm-embedding

A fork of [vLLM-Omni](https://github.com/vllm-project/vllm-omni) for running
multimodal models on **edge CPUs** — machines with tens of gigabytes of RAM and
no GPU, rather than servers.

The upstream project targets datacentre inference. Getting the same models onto
a mini-PC or a gateway box turns out to be less about kernels than about three
things nobody profiles: how much memory is resident while the model *loads*,
how much is resident that nothing ever reads, and whether the number you just
measured came from the code path you thought it did.

Upstream's own README is preserved at [docs/README.vllm-omni.md](docs/README.vllm-omni.md).

## What's here

| area | where |
|---|---|
| 4-bit CPU quantization | `vllm_omni/model_executor/layers/quantization/cpu_int4.py` — W4A16 on PyTorch's tinygemm kernel, prefill reading the kernel's own weight layout instead of a duplicate |
| Memory | RoPE tables sized to the servable context; weights repacked in blocks; a KV budget that is actually honoured on CPU |
| CPU platform + vLLM patches | `vllm_omni/platforms/cpu/`, `vllm_omni/patch.py` — vLLM is monkey-patched at import; its *source* is unmodified and no vLLM is redistributed. See [docs/edge/environment.md](docs/edge/environment.md) |
| Device routing | `vllm_omni/edge/weight_path.py` picks the weight format per device class, from measurements rather than assumption |
| Measurement harness | `benchmarks/edge_harness/` — the grid runner, benchmarks, memory ledger, fidelity metric and agent suite every number here came from |
| Raw results | `benchmarks/edge_harness/results/` — 100+ artifacts backing the published figures, each with provenance |
| Baselines | `baselines/` — llama.cpp runs and the comparison scripts |
| Engine analyses | `docs/engines/` — llama.cpp, ExecuTorch, MNN, ncnn, vLLM, vLLM-Omni |
| Study documents | `docs/analysis/` — the full reports these summaries distil |

## Headline results

Spark-X2.5-1.7B, Xeon Platinum 8480C, 24 pinned node-local cores, W4A16 g128.
Every figure is a median of at least three interleaved passes against a control
arm, on a shared host where end-to-end throughput swings ~30% run to run.

**Memory** — each measured as its own A/B with the alternative reachable by a flag:

| lever | saving | cost |
|---|---|---|
| KV budget 1 GiB → 192 MiB | **843 MB** | fewer cached tokens |
| RoPE table capped to `max_model_len` | **618 MB** | none |
| One packed weight layout instead of two | **615 MB** | ~1.6× prefill |
| Blocked weight repack | **489 MB** of load-time peak | none |

These overlap and **must not be added** — see [docs/edge/results.md](docs/edge/results.md).

**Speed**, same machine:

| path | decode | prefill (pp512) |
|---|---|---|
| W4A8 (AMX `int4_scaled_mm_cpu`) | 74.4 tok/s | — |
| W4A16 (tinygemm) | 71.6 tok/s | 2190 tok/s |
| `cpu_gemm_wna16` fallback | 32.3 tok/s | — |
| llama.cpp Q4_K_M (reference) | 90.0 tok/s | 1121 tok/s |

Decode is llama.cpp's; prefill is ours by ~2×. Neither engine is
bandwidth-bound — both sit at 33–39% of the measured 222 GB/s ceiling — so the
decode gap is the execution model, not the kernels. See
[docs/edge/comparisons.md](docs/edge/comparisons.md).

## Why the numbers here are worth reading

This project has retracted five conclusions, every one of them a measurement
error rather than a coding error. The harness exists because of them:

- a "3.1 GB saving" that was `VmRSS` counting the mmap'd checkpoint;
- W4A8 timings taken on a checkpoint whose output head was silently left in bf16;
- a memory benchmark reading `/proc/self/status` while the model ran in a *worker process*;
- "KV sizing saves only 188 MB" — measured through a code path that discarded the setting, so both arms ran the same cache;
- a 1.9× prefill cost quoted from the one pass of a grid whose other passes had crashed.

[docs/edge/methodology.md](docs/edge/methodology.md) is the resulting discipline,
and it is the most transferable thing in this repository.

## Quick start

```bash
# CPU build of vLLM, then this fork
pip install vllm==0.28.0+cpu
VLLM_TARGET_DEVICE=cpu pip install -e .

# a right-sized edge deployment
VLLM_CPU_KVCACHE_SPACE=1 python -c "
from vllm import LLM, SamplingParams
llm = LLM(model='<your-cpu_int4-checkpoint>', quantization='cpu_int4',
          dtype='bfloat16', max_model_len=2048, max_num_seqs=1,
          kv_cache_memory_bytes=192 << 20)   # the single largest memory lever
print(llm.generate(['Hello'], SamplingParams(max_tokens=32))[0].outputs[0].text)
"
```

## Documentation

| | |
|---|---|
| [docs/edge/results.md](docs/edge/results.md) | every measured figure, with spread and baseline |
| [docs/edge/methodology.md](docs/edge/methodology.md) | how to measure on a shared machine without fooling yourself — the five retractions and the rules they left |
| [docs/edge/cpu-4bit.md](docs/edge/cpu-4bit.md) | the two 4-bit paths, the packed layout as measured, building a checkpoint |
| [docs/edge/comparisons.md](docs/edge/comparisons.md) | against llama.cpp and the other engines considered |
| [docs/edge/environment.md](docs/edge/environment.md) | pins, machine, and why there is no vLLM patch series |
| [docs/edge/harness.md](docs/edge/harness.md) | running a comparison |
| [ROADMAP.md](ROADMAP.md) | what is next, with expected gains and how each would be verified — and what was rejected |

## Licence

Apache-2.0, inherited from vLLM-Omni. See [LICENSE](LICENSE) and [NOTICE](NOTICE).