vllm-omni/docs/edge/local-text-mode.md

# The local text mode: one stage, one device, and a plan you can argue with

`vllm_omni.edge.local` runs a single autoregressive text stage on whichever
device *this process* can actually drive, and writes down why. It is the M0
milestone of the [single-device engine
proposal](../../../analysis/architecture_local_engine_20260915.md): the smallest
thing that turns "a configuration that happens to run" into "a configuration
that can be explained, reproduced and refused".

It adds no kernels, no scheduler and no graph compiler. vLLM keeps doing all of
that inside the Omni stage. What this adds is the layer above: which device,
which artifact, how many bytes, and what happens when the answer is no.

```bash
python -m vllm_omni.edge.local devices                       # what this process can drive
python -m vllm_omni.edge.local plan   --model <ckpt>         # exit 2 == explicit refusal
python -m vllm_omni.edge.local run    --model <ckpt> --prompt "..."
python -m vllm_omni.edge.local accept --model <ckpt> --json out.json
```

---

## 1. "It fits" is not "it runs"

`models/Spark-X2.5-1.7B-int8` is 1.84 GiB of compressed-tensors int8 W8A8. The
RTX 5090 Laptop in the reference machine has 23.89 GiB. Every capacity check
passes, the engine starts, the weights load — and the first forward dies:

```
dispatch_scaled_mm, csrc/.../c3x/scaled_mm_helper.hpp:34,
Int8 not supported on SM120. Use FP8 quantization instead, or run on older arch (SM < 100).
```

cutlass' `c3x` has no int8 `scaled_mm` for SM >= 100, and sm_120 is inside that
range. Roughly forty seconds and a stack trace from a subprocess to learn
something that is written in the checkpoint's `quantization_config` and the
device's compute capability.

The planner reads both and refuses in about a second:

```
REFUSED: no device in this process can run this checkpoint as asked.
  [weight_format_unsupported] cuda:0: the checkpoint is compressed-tensors:int8 and
      cuda:0 (NVIDIA GeForce RTX 5090 Laptop GPU, sm_120) executes only
      ['awq', 'compressed-tensors:fp8', 'dense', 'gptq', 'modelopt_fp4']
      remedy: this device has fp8 tensor cores but no int8 scaled_mm path
      (cutlass c3x has none for SM >= 100). Use an fp8 or a dense build here,
      or run the int8 build on the CPU backend (.venvs/omni-cpu).
```

The remedy is not decoration. The same checkpoint completes the full acceptance
run on the CPU backend, at 16.3 tok/s median — so this is a property of the
device, not a defect in the artifact, and the refusal says which.

## 2. What a device *is*, here

`hardware_probe` answers "what is in the box". That is the wrong question at
load time, because a vLLM wheel is built for one platform: the CUDA wheel
cannot run a CPU stage however many cores the probe finds, and the CPU wheel
cannot reach the card beside it. So a device is runnable only when the probe
finds it **and** `vllm.platforms.current_platform` is the platform that drives
it. Everything else is enumerated with the reason it is not available:

```
  [runnable] cuda:0           NVIDIA GeForce RTX 5090 Laptop GPU (23.9GiB vram) backend=vllm:cuda
  [not here] npu:amd          NPU Compute Accelerator Device (host_ram) backend=None
             An MCDM device; WSL2's GPU-PV forwards WDDM display adapters only...
  [not here] gpu_integrated:amd AMD Radeon(TM) 890M Graphics (host_ram) backend=None
             No /dev/dri under WSL2, but reachable through DirectML over /dev/dxg...
  [not here] cpu              AMD Ryzen AI 9 HX 370 (30.9GiB host_ram) backend=None
             vllm.platforms.current_platform is 'cuda', not 'cpu'...
```

The iGPU and the NPU are reported rather than dropped, because "the machine has
an NPU" and "a stage can run on it" are different claims and a plan has to be
able to show which one it relied on. They share `host_ram` with the CPU and
carry zero bytes, so nothing can accidentally add three copies of the same RAM
together.

`--mask gpu_discrete` hides the discrete GPU to exercise the no-NVIDIA
deployment class. It is a policy, not amnesia: the device stays in the record
with `masked: true` and its real memory.

## 3. The budget is a sum, and every term is visible

```
ADMITTED on cuda:0 (NVIDIA GeForce RTX 5090 Laptop GPU) via vllm:cuda
  artifact  43e867b180f1a3d2 dense 7.66GiB
  budget    12.51GiB of 23.89GiB vram (total)
    weights                 7843 MiB  vram  session_weights D  5 safetensors shard(s), on-disk size
    kv_cache                 720 MiB  vram  session         E  flat budget at 1x context; 248 MiB if
                                                               the backend accounts 27 sliding layers separately
    activations              120 MiB  vram  request         E  2048-token prefill chunk x 12 live tensors
    backend_workspace        512 MiB  vram  session         E  attention scratch, driver/allocator context
    load_transient          1569 MiB  vram  load            E  freed once the weights are resident
    external_reserve        1024 MiB  vram  session         E  memory this engine does not own
    safety_margin           1024 MiB  vram  session         E  fragmentation and allocator slack
```

`D` is read off the artifact; `E` is estimated. The peak is a **max**, not a
sum: the loader's transient is gone before vLLM sizes the KV pool, so adding
both would refuse plans that fit.

The KV line comes from `vllm_omni.edge.kv_budget`, which reads the real
`layer_types`. Spark-4B is 9 full-attention and 27 sliding@512 layers, so a
backend that accounts them separately needs 248 MiB where a flat charge needs
720. The budget takes the flat number (it is the safe one) and reports the
other.

**The estimates are checked, not believed.** After every run the engine reports
the budgeted peak against the sampled one:

| | CUDA / Spark-4B bf16 | CPU / Spark-1.7B int8 |
|---|---:|---:|
| counter | `nvml_device_used` (whole card) | `tree_rss` (process tree) |
| budgeted peak | 12.51 GiB | 6.55 GiB |
| measured attributable peak | 8.76 GiB | 5.48 GiB |
| budget / measured | 1.43x | 1.20x |

A budget above the measurement is the intended direction — admission needs a
provable upper bound, not a prediction — but 1.40x is loose enough to be worth
narrowing, and the number is in the record so somebody can.

### The flag that is not GPU-only

The plan sets `gpu_memory_utilization` on **both** pools, derived from its own
budget. On the CPU backend vLLM reads that same field as the fraction of *host
RAM* to reserve, so leaving it at its default there is not "no opinion" — it is
asking for 92% of the machine. A 6.55 GiB CPU plan failed to start with
`desired CPU memory utilization (0.92, 28.43 GiB)` until this was fixed.

The KV pool is set as an absolute `kv_cache_memory_bytes` rather than left as
whatever remains inside that fraction, so the cache is the size the request
limits imply. Both backends honour it exactly: CUDA logs 754974720 bytes, CPU
logs `Explicitly set (0.14/30.91) GiB for KV cache`.

## 4. Streaming, state and cancellation

Three contracts, which is what the proposal says the first version needs:

* **`ChunkEvent`** carries the *delta*, never the output so far, plus a `seq`
  that detects loss and an `epoch` that retires it.
* **`StateHandle`** is an opaque reference to state the backend owns. It is
  tied to the artifact that produced it (the same shapes from a different
  quantization are not the same state) and is never `migratable` — moving a KV
  cache between backends needs a measured layout conversion for the specific
  pair.
* **`BoundedEventStream`** bounds chunks *and* bytes and blocks the producer,
  which is the backpressure. A single event larger than the whole byte bound
  still passes on an empty queue, or it would deadlock.

Cancelling is not "stop calling `next()`". The backend has work in flight and
its tokens arrive after the cancel. `cancel()` retires the epoch first, then
aborts in the backend, then closes the stream — *before* waiting on the
producer, so a backend that hangs while unwinding delays cleanup rather than
the caller. In the recorded runs the fence discarded 3 in-flight events on CUDA
and 2 on CPU, and zero events from the retired epoch reached the consumer.

The device counter does **not** fall on cancel, because vLLM's KV pool is
pre-allocated. What falls is the in-flight count. The report says so, so that
"memory did not drop" is not read as "nothing was released".

## 5. Measuring memory without fooling yourself

Three counters, three different accountings, never summed:

* `nvml_device_used` is the whole card, every process. Under WSL2
  `nvmlDeviceGetComputeRunningProcesses` returns an empty list, so GPU bytes
  **cannot** be attributed to this engine's PIDs on this host. "Attributable
  peak" means peak minus a pre-load baseline (2.79 GiB here), not a private
  total.
* `tree_pss` includes a proportional share of file-backed pages, which on an
  mmap'd safetensors load is partly page cache.
* `host_available` moves for reasons that are not us.

Sampling is every 0.25 s, so every peak is a lower bound, and the record says
that rather than assuming it away.

## 6. What Spark needed before any of this worked

Two gaps that made "the model is in the registry" and "the model runs under
Omni" different statements:

1. **No `PipelineConfig`.** Spark's model definition had been registered for a
   while, but with no pipeline `Omni(model=<Spark checkpoint>)` had nothing to
   resolve, so the model was reachable only through plain `vllm.LLM`. Added as
   a single `LLM_AR` text stage in
   `vllm_omni/model_executor/models/spark2_5/pipeline.py`.
2. **A stage-call mismatch.** Omni's AR model runner hands every stage model
   `sampling_metadata`, `logits_index` and `sampler`. Spark's
   `ForCausalLM.forward` caught them in `**kwargs` and forwarded them into
   `Spark2_5Model.forward`, which takes four arguments — `TypeError` on the
   first forward. They now stop at the `ForCausalLM` boundary, where the
   decoder stack has no use for them anyway.

One stage is the design, not a placeholder. An autoregressive session stays on
one backend by default: prefill and decode share the KV cache, and a per-token
device boundary buys nothing and costs a synchronization. Multi-stage is M2,
and it needs the tensor/buffer contract that a single text stage — which hands
a buffer to nobody — cannot be used to design.

## 7. Does the Omni path change the answer?

This is the one claim the rest of M0 cannot make for itself. Everything above
is about the engine's own behaviour; this asks whether routing a decode through
the Omni stage changed the model's output. The validation plan makes it a
*hard* initial acceptance condition.

Same checkpoint, same device, same request limits, same environment, greedy
with `ignore_eos` -- the only difference is the path: plain `vllm.LLM` straight
into vLLM against `AsyncOmni` -> StageRuntime -> the stage's EngineCore.

**12 of 12 prompts, 128 greedy tokens each, token-for-token identical -- on
both paths**: CUDA with the bf16 build and CPU with the int8 one. The quantized
build gets its own run rather than inheriting the result, because quantization
is exactly what makes a near-tie likelier to flip.

The comparison is on **token ids, not text**: two different id sequences can
detokenize to the same string, and a text comparison would hide exactly the
drift worth catching. The record keeps the first divergent position rather than
a yes/no, because greedy decoding is not bitwise stable across batch shapes --
a different prefill chunking reorders reductions and can flip an argmax at a
near-tie, so "split at token 120" and "split at token 3" are different
findings. Neither occurred.

The two engines load sequentially in one process, reference first and released
before the second starts; two copies of a 7.66 GB checkpoint would not fit the
budget this milestone is about respecting.

```bash
python -m vllm_omni.edge.local parity --model <ckpt> --json out.json   # exit 3 on divergence
```

### The two paths do not need the same environment

The reference cannot start on this host with a default environment, and both
reasons are worth knowing:

* under WSL2, vLLM's own GPU model runner allocates a `UvaBuffer` for its input
  batch and raises `RuntimeError: UVA is not available` without pinned memory.
  Omni's AR model runner does not take that path and starts either way.
* FlashInfer's sampler JIT-compiles on first use and needs `nvcc`, which is not
  installed -- `nvidia-smi` reporting a CUDA version is not a toolkit. This one
  kills the engine at the *first token*, not at load.

Both now live in one `RUNTIME_ENV` applied by every entry point that starts an
engine, because a comparison whose reference is configured differently from its
subject is measuring the configuration.

## 8. Limits

* One model, one stage; no multi-stage, no tensor/buffer contract.
* `--mask` is a process-level policy, not a physical absence. A full no-card
  verification needs another machine.
* The iGPU and NPU are enumerated only; neither has a stage backend (M3).
* Reference consistency covers **one decode mode**: single sequence, greedy,
  128 tokens, on both paths. It does not cover concurrency above 1 (which
  changes batch shapes, and greedy is not bitwise stable across those), longer
  outputs, or non-greedy sampling -- and it is not a quality evaluation. It
  says the Omni path did not change the output, not that the output is good.
* Performance figures are one sample per prompt, not p50/p95, with no fixed
  power profile and no cache-cold guarantee.

## 9. Tests

```bash
# hermetic; no GPU, no weights, ~3 s
pytest tests/edge/local --ignore=tests/edge/local/test_e2e_local_text.py

# end to end against a real checkpoint; the venv decides which artifact
pytest tests/edge/local/test_e2e_local_text.py
```

The e2e tests run the acceptance criteria themselves: both startups reach a
decision, twelve fixed prompts each reach 128 tokens, the budget bounds the
measured peak, cancellation leaks nothing, and the Omni path decodes what plain
vLLM decodes. They skip rather than fail when no checkpoint is present.
**78 pass in each venv**; about 4 min on this machine either way.

The twelve prompts are the same twelve
`analysis/experiments/spark_edge/quality_int4.py` measures the quantized builds
against, so these numbers stay comparable with the existing W4A16 / W4A8 /
Q4_K_M fidelity work.

Full run records:
[`analysis/experiments/m0_local_text_20260915/`](../../../analysis/experiments/m0_local_text_20260915/README.md).