Include the baselines, comparisons and study documents, not just the fork
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
This commit is contained in:
1 parent
b36a3431bd
commit
9264ff81ab
162 files changed
+22640
-9
No files matched your search
@@ -0,0 +1,163 @@
|
||||
# Six-engine comparison
|
||||
|
||||
Sources: the seven engine reports under [engines/](engines/) (each claim there carries a file/symbol citation) and the measurements in [experiments.md](experiments.md). This document compares; it does not re-derive. Labels: [Verified] = confirmed in the engine reports against source at the analyzed commits; [Inferred]; [Proposal]; [Unknown]; "repo-reported" = published by the repository, not reproduced; "measured" = produced in this session on this x86/L20X host.
|
||||
|
||||
## 1. Two kinds of software
|
||||
|
||||
The six repositories split into **serving stacks** (own a request queue, batch across requests, run as long-lived processes) and **execution runtimes** (a library that runs one graph or one decode step when called). This distinction explains most of the comparison below, and it is the first thing the edge plan has to respect: a serving stack can be *re-hosted*, but it cannot be *shrunk* into a runtime without replacing its process model.
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph serving["Serving stacks (request scheduler + process model + HTTP)"]
|
||||
VO["vLLM-Omni<br/>multi-stage orchestrator over vLLM engines"]
|
||||
V["vLLM<br/>V1 engine: scheduler, paged KV, executor"]
|
||||
VO -->|"subclasses EngineCoreProc, Scheduler, model runners; monkeypatches ~10 vLLM symbols"| V
|
||||
end
|
||||
subgraph runtimes["Execution runtimes (library: load graph, run, return)"]
|
||||
L["llama.cpp<br/>ggml graph + libllama + libmtmd<br/>(+ optional llama-server with slot scheduler)"]
|
||||
M["MNN<br/>graph engine + LLM/Omni engine<br/>(+ single-Llm HTTP wrappers)"]
|
||||
N["ncnn<br/>graph engine only"]
|
||||
E["ExecuTorch<br/>AOT compiler + .pte interpreter + LLM runner<br/>(+ experimental batching lib, llm_server example)"]
|
||||
end
|
||||
V -->|"torch, CUDA/ROCm/XPU/CPU kernels"| T["PyTorch 2.13 runtime"]
|
||||
L --> HW["CPU / Metal / Vulkan / OpenCL / CUDA / Hexagon / CANN ..."]
|
||||
M --> HW
|
||||
N --> HW2["CPU / Vulkan"]
|
||||
E --> HW3["delegates chosen at export: XNNPACK / CoreML / MLX / QNN / Vulkan / CUDA ..."]
|
||||
```
|
||||
|
||||
| Engine | Kind | Unit of work the public API accepts | Long-lived process model | Report |
|
||||
|---|---|---|---|---|
|
||||
| vLLM | Serving stack (V1 engine) | A request (prompt + sampling params) into a continuous-batching scheduler | Yes: API server + `EngineCoreProc` + workers, ZMQ between them [Verified] | [engines/vllm.md](engines/vllm.md) |
|
||||
| vLLM-Omni | Serving stack built on vLLM | A request that flows through a DAG of stages (AR / generation / diffusion), each stage a vLLM-style engine process | Yes: `AsyncOmniEngine` + `Orchestrator` + one `StageEngineCoreProc` per AR stage, shared-memory or RDMA connectors between stages [Verified] | [engines/vllm_omni.md](engines/vllm_omni.md), [engines/vllm_omni_models.md](engines/vllm_omni_models.md) |
|
||||
| llama.cpp | Execution runtime (+ optional server tool) | `llama_decode(batch)` over one or more `seq_id`s; `mtmd` chunk encode; audio-gen step API | No in `libllama`; yes only in `tools/server` (slots, single worker queue) [Verified] | [engines/llama_cpp.md](engines/llama_cpp.md) |
|
||||
| ncnn | Execution runtime | `Extractor::extract(blob)` on one `Net` | No [Verified] | [engines/ncnn.md](engines/ncnn.md) |
|
||||
| MNN | Execution runtime (+ LLM/Omni engine) | `Module::onForward`, or `Llm::response/generate` for one conversation | No; `mls_server`/`mnncli` wrap one `Llm` under a mutex [Verified] | [engines/mnn.md](engines/mnn.md) |
|
||||
| ExecuTorch | Execution runtime (AOT + interpreter) | `Method::execute()` on one loaded `.pte`; `IRunner::generate` for LLMs | No in core; experimental `extension/llm/batching` scheduler has no in-tree executor; `examples/llm_server` serializes execution [Verified] | [engines/executorch.md](engines/executorch.md) |
|
||||
|
||||
## 2. Device layer
|
||||
|
||||
| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| In-tree accelerators | CUDA, ROCm, XPU, CPU (x86/aarch64/Apple CPU); TPU via out-of-tree shim; Jetson detected through `/etc/nv_tegra_release` with CUDA arch 8.7/11.0 in the build list [Verified] | CUDA, ROCm, Ascend NPU (incl. 310P), XPU, MUSA; **no CPU platform** (`UnspecifiedOmniPlatform` reports 0 devices) [Verified] | 18 backend dirs: cpu, cuda, hip, musa, metal, vulkan, opencl (Adreno), hexagon (HTP), cann, sycl, openvino, webgpu, blas, rpc, virtgpu, zdnn, zendnn, et (a vendor accelerator platform under `/opt/et`, not ExecuTorch) [Verified] | CPU (ARM/x86/RISC-V/MIPS/LoongArch) + Vulkan only; Apple via MoltenVK [Verified] | 16 backend dirs: cpu (arm82/KleidiAI/SME2/AVX), metal, opencl, vulkan, opengl, cuda (sm60-120), musa, tensorrt, coreml, nnapi, qnn, hiai, hexagon, rknn, neuropilot (stub) [Verified] | Delegates chosen at export: XNNPACK, Vulkan, WebGPU, QNN, MediaTek, Samsung ENN, Ethos-U/VGF, NXP, OpenVINO, CoreML, Metal (exp.), MLX (exp.), CUDA (AOTInductor), Cortex-M, Cadence [Verified] |
|
||||
| Discovery / selection | Exactly one `Platform` activates from builtin detectors or the `vllm.platform_plugins` entry point; out-of-tree wins [Verified] | Same as vLLM plus `vllm_omni.platform_plugins`; devices assigned per stage in deploy YAML [Verified] | `ggml_backend_registry`, dynamic backend `.so` loading, registration order = scheduler priority, CPU last [Verified] | Vulkan device by `rough_score`; CPU ISA tables at load [Verified] | `Schedule::getAppropriateType` AUTO priority HIAI → CoreML/NNAPI → TRT → CUDA → OpenCL → Metal → Vulkan → CPU [Verified] | None at runtime: `BackendDelegate::Init` looks up a string id in a 16-slot static registry and checks `is_available()` [Verified] |
|
||||
| Memory model | Paged KV blocks sized from `gpu_memory_utilization` after a profile run; torch caching allocator; CuMem sleep mode [Verified] | Inherits; adds per-stage `determine_available_memory`, stage co-location admission, diffusion offload, host-weight mmap sharing [Verified] | `ggml_gallocr` worst-case compute buffer reserved at context init; mmap weights; per-backend buffer types [Verified] | Pool allocators, optional mmap weights, Vulkan blob/weight/staging allocators [Verified] | Eager/defer arena allocators, mmap weights and feature maps, KV cache mmap-to-disk option [Verified] | Static planning at export (`non_const_buffer_sizes`), `HierarchicalAllocator` arenas, zero-copy input aliasing and output pointers [Verified] |
|
||||
| Host/device transfers | torch copies; multimodal embeddings scattered on device; CPU offload/KV connectors [Verified] | Default `SharedMemoryConnector` path: **D2H → shared memory (msgpack) → H2D** for every hop; repo docs state connectors are D2H2D, but `MooncakeTransferEngineConnector.put` and the Mori connector pass tensors/`ManagedBuffer` without serialization and support device memory pools, Mori including an intra-node `xgmi` backend [Verified] | Scheduler inserts copies at backend splits; async `tensor_set/get`; RPC backend serializes over TCP [Verified] | Explicit `record_upload/download` per extract; CPU fallback downloads [Verified] | `WrapExecution` copies at per-op fallback boundaries; NPU delegates run whole subgraphs [Verified] | `_h2d_copy/_d2h_copy` inserted by `PropagateDevicePass` at delegate boundaries (CUDA) [Verified] |
|
||||
| Sync / streams | CUDA streams, CUDA graphs, async scheduling overlaps CPU/GPU [Verified] | Adds copy-stream D2H snapshot, async output materialization [Verified] | One stream per backend; events; pipeline parallelism up to 4 copies for multi-GPU [Verified] | One command buffer per extract, `submit_and_wait` [Verified] | Single queue per GPU backend, commit-and-wait; no multi-stream [Verified] | Synchronous interpreter; optional per-method CUDA graph [Verified] |
|
||||
| Portability envelope | Linux servers; aarch64 Linux CPU wheels; macOS CPU experimental; no Android [Verified] | Linux CUDA/ROCm/NPU/XPU/MUSA servers only [Verified] | Linux/macOS/Windows/Android/iOS/WASM; xcframework and Android JNI samples in tree [Verified] | 60 CMake toolchains incl. Android, iOS, HarmonyOS, Jetson, QNX, ESP32, WASM [Verified] | Android/iOS/macOS/Linux/Windows-ARM64/OpenHarmony/WASM/RISC-V; CUDA sm87 present [Verified] | Android AAR, Apple SwiftPM xcframeworks, bare-metal Arm, Zephyr, ESP32, RISC-V, Linux, Windows, WASM [Verified] |
|
||||
|
||||
Take-aways for edge: only the four runtimes reach Android GPU/NPU and Apple GPU/ANE. vLLM reaches ARM CPU and Jetson-class CUDA (untested in CI), and vLLM-Omni does not reach CPU at all. That last point is an **architectural** constraint only in the sense that vLLM-Omni's platform layer requires a device-backed worker; adding a CPU platform is a missing implementation, not a redesign (see §9).
|
||||
|
||||
## 3. Kernel layer
|
||||
|
||||
| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Operator set | torch ops + 132 CUDA custom ops + 73 CPU ops; attention backends by enum registry [Verified] | Inherits; adds diffusion attention registry (11 backends), fused/Triton ops for codecs with torch fallbacks [Verified] | 102 `ggml_op`s; per-backend `supports_op` vtables; `docs/ops.md` support matrix [Verified] | 110 layer types; 64 with Vulkan kernels [Verified] | 186 schema op types; CPU registers 75, Metal 36, OpenCL 39, Vulkan 35/26, QNN 41; geometry decomposition to Raster/Loop reduces what a backend must implement [Verified] | 209 portable + 23 optimized + 19 quantized kernels; LLM custom ops (`sdpa_with_kv_cache`, `update_cache`, ...) [Verified] |
|
||||
| Dispatch | `CustomOp.dispatch_forward` per platform, `vllm.ir` provider priority, `MPLinearKernel.can_implement` chain [Verified] | Adds `forward_npu/musa`, per-role diffusion attention selection [Verified] | `ggml_backend_sched` assigns each node to the highest-priority backend whose `supports_op` is true [Verified] | `Layer_final{layer_cpu, layer_vulkan}` per layer, feature masks per layer [Verified] | Backend creator map; per-op CPU retry on failure [Verified] | Static kernel table resolved at `Method::init`; delegate calls are opaque blobs [Verified] |
|
||||
| Quantization | AWQ/GPTQ/FP8/compressed-tensors/ModelOpt/MXFP4/torchao/...; ARM CPU: W8A8 (oneDNN) and W4A8 (KleidiAI) only [Verified] | Scoped to the AR LM stage; non-AR stages BF16 "not validated" (repo doc) [Verified] | 43 `ggml_type`s (Q4_0..IQ*, TQ, MXFP4); dequant fused into matmul; CPU repack + KleidiAI [Verified] | fp16/bf16, int8 PTQ, LLM block quant W4/W6/W8A8 (CPU only) [Verified] | Export-time W2/W3/W4/W8 block quant (default W4 block-64), runtime int8 dynamic activation; KV int8/TQ3/TQ4; GPU low-bit GEMV [Verified] | torchao 8da4w/4w/8w, SpinQuant, QAT+LoRA; PT2E quantizers per delegate; KV int8 [Verified] |
|
||||
| Compilation / graph capture | `torch.compile` modes + CUDA graph modes, all at warm-up [Verified] | Per-stage compile config; talker-MTP graphs; code2wav stateful graphs per chunk length; diffusion regional compile [Verified] | No JIT; CUDA graph capture; Metal runtime shader compile; Vulkan build-time SPIR-V; OpenCL kernel cache [Verified] | Runtime GLSL → SPIR-V via glslang with pipeline cache [Verified] | No JIT/no CUDA graph; tuning cache files [Verified] | AOT only; optional CUDA graph per method [Verified] |
|
||||
| Fallback | Native torch path per `CustomOp`; Triton C++ fallbacks on CPU [Verified] | Torch fallbacks for Triton codec kernels [Verified] | Unsupported node → CPU with inserted copies [Verified] | Layer without Vulkan impl → CPU with download/upload [Verified] | Per-op CPU retry; NPU subgraph compile is all-or-nothing [Verified] | Non-delegated ops stay portable CPU kernels; missing kernel → load error [Verified] |
|
||||
| Extension | Platform plugin, `CustomOp.register_oot`, `vllm.ir.register_impl`, `register_quantization_config` [Verified] | `register_diffusion_backend`, TTS adapter registry, model/pipeline registries [Verified] | New backend = vtables + `GGML_BACKEND_DL_IMPL`; custom ops `ggml_custom_4d` [Verified] | `Net::register_custom_layer`; custom Vulkan shader op [Verified] | `MNNInsertExtraRuntimeCreator`; plugin ops [Verified] | New delegate = `BackendInterface` + partitioner; custom ops via YAML + kernel registration [Verified] |
|
||||
|
||||
## 4. Model runner
|
||||
|
||||
| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Model format / conversion | HF safetensors directly; no conversion [Verified] | Same; plus per-family Python module code; no export tooling [Verified] | GGUF via `convert_hf_to_gguf.py` (+ separate mmproj GGUF) then `llama-quantize` [Verified] | `.param/.bin` via pnnx (PyTorch/ONNX) or legacy converters; LLM decoder needs seq-len-1 export + param patch [Verified] | `.mnn` FlatBuffer via MNNConvert (ONNX/TF/TorchScript) or `llmexport.py` (HF → MNN with fused attention/RoPE ops); GGUF import [Verified] | `.pte` via `torch.export` → `to_edge_transform_and_lower` → `to_executorch`; LLM CLI `export_llm`; HF via optimum-executorch [Verified] |
|
||||
| Lifecycle | init device → load → profile → KV alloc → compile/warm-up → busy loop [Verified] | Same per stage, sequentially; Orchestrator readiness gate (default 300 s, see experiments) [Verified, measured] | load → init context (reserve) → decode; no warm-up API [Verified] | load_param/model → extractor; no warm-up; first extract compiles Vulkan pipelines [Verified] | `Llm::load` builds tokenizer, modules, clones decode module; optional tuning [Verified] | load program → `Method::init` (kernel resolution, delegate init) → execute [Verified] |
|
||||
| Dynamic shapes | Native; CUDA graphs bucketed by batch size [Verified] | Same; code2wav shapes derived from chunk schedule [Verified] | Native; graph reuse when shapes match [Verified] | Native; shape expressions [Verified] | Bucketed module clones (prefill/decode); `shapeMutable=false` on GPU/NPU + chunked prefill [Verified] | Upper-bound dims at export; XNNPACK reshapes per call; QNN/Ethos-U static [Verified] |
|
||||
| KV / state | Block-paged, prefix caching, hybrid managers, offload connectors, streaming-input sessions [Verified] | Inherits; adds hidden-state prefix cache, inter-stage KV transfer, per-session paged KV in `ARDiffusionEngine` [Verified] | Ring-buffer cells, seq bitsets, no paging, context shift, quantized KV, state save/load [Verified] | Contiguous per-head `Mat`, ×1.5 growth, batch-1 only, host-provided mask/RoPE [Verified] | Contiguous per-head, grows by 64, int8/TQ quant, mmap-to-disk, no paging [Verified] | In-graph mutable buffer `[B,H,max_ctx,D]`, runner tracks position; static-attention I/O mode for NPUs; experimental paged-like cache lib [Verified] |
|
||||
| Multimodal stages | Encoders in worker under an encoder budget, embeddings scattered [Verified] | Encoders, talkers, codec/vocoder, diffusion each as stages or model-owned sub-modules [Verified] | `mtmd` encode chunk → `llama_decode(embd)`; audio-gen step API [Verified] | Separate `Net`s wired by the app (Piper: 5 nets) [Verified] | `Omni` class: vision/audio encoders, talker, DiT, vocoder threads [Verified] | `MultimodalRunner` method conventions (`vision_encoder`, `text_decoder`, ...); Voxtral TTS runner [Verified] |
|
||||
|
||||
## 5. Scheduler
|
||||
|
||||
Request/token/pipeline scheduling is separated from graph/operator/thread scheduling, as the task requires.
|
||||
|
||||
| Level | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Request / token | Token-budget continuous batching, chunked prefill, preemption by recompute, async scheduling, priority, admission backpressure, abort [Verified] | Per-stage vLLM schedulers (`OmniARScheduler`, async variant) + `OmniGenerationScheduler`; **pipeline-level**: Orchestrator routes, pre-warms downstream requests parked in `WAITING_FOR_CHUNK`, propagates abort [Verified] | `libllama`: none (caller batches `seq_id`s). `llama-server`: slots, one worker queue, continuous batching, cancel, speculative [Verified] | N/A [Verified] | N/A; batch 1 baked into export; cancel only through wav callback or per-token stepping [Verified] | N/A in core; experimental `DecodeFirstScheduler` without executor; `llm_server` serializes; cooperative `stop()` [Verified] |
|
||||
| Pipeline (multi-stage) | N/A (single model) | Yes: stage DAG, connectors, async chunk hand-off (Talker → Code2Wav in 25-frame chunks after a 1-frame first chunk) [Verified, measured] | App-level only (`mtmd` helper loops) [Verified] | App-level (Piper example) [Verified] | Built-in for Omni: thinker/talker interleave and codec-LM → DiT → vocoder worker threads with `WavChunk` queues [Verified] | App-level (Voxtral TTS runner chunk loop) [Verified] |
|
||||
| Graph / operator / thread | CUDA graph dispatch, DBO micro-batching; OpenMP binding on CPU; no op scheduler [Verified] | Inherits | `ggml_backend_sched` 5-pass split, `ggml_threadpool` with cpumask/polling, separate batch/decode thread counts [Verified] | Lazy DAG pull, OpenMP intra-op, big/little affinity [Verified] | Linear command walk, own thread pool (max 2 tasks), big/little pinning [Verified] | Sequential instruction chain; pthreadpool sized by performant cores; delegate-internal threading [Verified] |
|
||||
| Concurrency | Many requests per step | Many per stage, stage-parallel | Server slots; library caller-managed | One Extractor per thread | One `Llm` per thread | One `Method` at a time |
|
||||
| Cancellation | Immediate at next step | Propagated across stages | Server cancel task; library abort callback (CPU/Metal) | None | Weak (callback return / stepping) | Token-boundary stop |
|
||||
|
||||
## 6. I/O layer
|
||||
|
||||
| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Tokenizer | HF `tokenizers`, Mistral, incremental detok [Verified] | Inherits; per-model speech tokenizers (12/25 Hz Qwen3-TTS, s3tokenizer, DAC, XCodec) [Verified] | In-tree SPM/BPE/WPM/UGM/RWKV [Verified] | None [Verified] | In-house SentencePiece/tiktoken/BERT/HF-BPE + vendored Jinja [Verified] | Submodule (uninitialized here): HF json, Tiktoken, SentencePiece, Llama2c, Tekken [Verified/Unknown internals] |
|
||||
| Image / audio preprocessing | HF processors in API process; PyAV/soxr audio [Verified] | Adds Silero VAD, forced aligner, resamplers, soundfile encoders [Verified] | stb_image + slicing; miniaudio + mel/FFT [Verified] | `Mat::from_pixels`, Spectrogram layers; no audio I/O [Verified] | ImageProcess, per-family vision preprocess, fbank → `audio.mnn` [Verified] | `ImageProcessor`, exportable mel spectrogram, ASR runners [Verified] |
|
||||
| Streaming | Token SSE, WS realtime audio input [Verified] | Token + audio chunk streaming, `/v1/audio/speech/stream`, duplex sessions [Verified] | Server SSE; audio-gen step API but CLI writes one WAV [Verified] | None [Verified] | `setWavformCallback` chunked audio (Qwen2.5-Omni); Qwen3-TTS single callback [Verified] | Token callback; Voxtral TTS `AudioChunkCallback` [Verified] |
|
||||
| Application boundary | Python, OpenAI/Anthropic HTTP, gRPC, CLI; no C API/mobile [Verified] | Python `Omni/AsyncOmni`, HTTP + WS incl. OpenPI robot endpoint; no C API/mobile [Verified] | C API, CLI, server, Android JNI, Swift; no Python runtime [Verified] | C++/C API, Python, Android/iOS packaging [Verified] | C++, Python (`MNN.llm`), Android/iOS/HarmonyOS apps, HTTP wrappers [Verified] | C++ `Module`, Python, Android AAR, Swift packages, WASM [Verified] |
|
||||
|
||||
## 7. Deployment complexity, conversion, extensibility
|
||||
|
||||
| Dimension | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Minimum deployable unit | Python env with torch 2.13 + CUDA/CPU wheel (≤500 MB wheel cap) [Verified] | vLLM env + vLLM-Omni + transformers/diffusers/soundfile/onnxruntime + per-family extras; 2 engine processes for a 2-stage TTS [Verified] | Single static binary; libs of ~0.9 MB (`ggml-base`) + 1.5 MB (`ggml-cpu`) + `libllama` measured here on x86 [measured] | `libncnn` (size deltas only in CI) [Unknown absolute] | Android core `.so` ≈ 800 KB, iOS static ≈ 12 MB repo-reported [repo-reported] | Core ≈ 50 kB (CI thresholds 45–52 kB) + delegates + kernels [repo-reported] |
|
||||
| Cold start | Seconds to minutes (compile + graph capture) | **157–188 s observed** for Qwen3-TTS 2-stage on L20X with default CUDA graphs and `torch.compile` (sequential stage init; `enforce_eager`, per-stage `compilation_config`, and parallel init exist but were not measured) [measured, not a lower bound] | Sub-second to seconds (mmap) [Inferred from design; not measured here except mtmd run ≈ 8 s incl. 1 GB load] | Sub-second (first Vulkan pipeline compile aside) [Inferred] | Seconds (module clones) [Inferred] | Sub-second (static planning) [Inferred] |
|
||||
| Conversion effort for a new HF model | None (Python model class) | Python model class + pipeline config + deploy YAML | Converter class in `conversion/` + graph class in `src/models/` + optional mmproj | pnnx export + possibly custom layers + param patching | `llmexport.py` mapper entry or ONNX path; fused ops emitted by exporter | `torch.export` of each sub-module with static/upper-bound shapes; partitioner per target; quantizer recipe |
|
||||
| Adding a backend | Platform plugin (out-of-tree possible) | Same + omni platform overlay | ggml backend vtables (in-tree or dynamic) | Third slot in `Layer_final` (in-tree) | `MNNInsertExtraRuntimeCreator` (in-tree) | `BackendInterface` + partitioner (in- or out-of-tree) |
|
||||
| Adding an op | `CustomOp` / `vllm.ir` | Same | `ggml_custom_4d` or new op enum across backends | `register_custom_layer` | Plugin op | YAML + kernel or delegate-internal |
|
||||
|
||||
## 8. What vLLM-Omni inherits from vLLM and what it adds
|
||||
|
||||
Inherited (used as-is or subclassed) [Verified in engines/vllm_omni.md]:
|
||||
|
||||
- Engine processes and IPC: `EngineCoreProc` (subclassed as `StageEngineCoreProc`), `AsyncMPClient` over ZMQ, msgspec serialization.
|
||||
- Scheduler: `Scheduler`/`AsyncScheduler` subclassed into `OmniARScheduler`/`OmniARAsyncScheduler`; token budgets, chunked prefill, preemption, prefix caching.
|
||||
- KV cache manager and block pool; multimodal processor and caches; tokenizers; quantization configs; `torch.compile`/CUDA-graph machinery; platform plugin mechanism; OpenAI server scaffolding.
|
||||
- Model layers for talkers (`Qwen2Model`/`Qwen3Model`, `ParallelLMHead`, paged attention).
|
||||
|
||||
Added [Verified]:
|
||||
|
||||
- Stage abstraction: `PipelineConfig` (LLM_AR / LLM_GENERATION / DIFFUSION) + `DeployConfig` YAML; `AsyncOmniEngine`, `Orchestrator`, `StageRuntime/StagePool`, headless remote replicas.
|
||||
- `LLM_GENERATION` runner (`GPUGenerationModelRunner`: no logits/sampling, tensor outputs) and `OmniGenerationScheduler`.
|
||||
- Diffusion runtime: `DiffusionEngine` with request/step schedulers, workers, 11-backend attention registry, offload, parallelism (TP/SP/CFG/HSDP/VAE).
|
||||
- Connectors (`OmniConnector`: shared memory, Mooncake, Mori, Yuanrong) and the async-chunk protocol (`WAITING_FOR_CHUNK`, `OmniChunkTransferAdapter`).
|
||||
- Model families: 15 TTS, 12 speech-to-speech/omni, 5 image, 4 video/world, 2 VLA pipelines; TTS adapter registry; audio encoders/resamplers; OpenPI robot WebSocket; duplex sessions.
|
||||
- Monkeypatches of vLLM internals (`patch.py`) and injection of omni model classes into vLLM workers via the `vllm.general_plugins` entry point.
|
||||
|
||||
Consequence: vLLM-Omni is not separable from vLLM's process model. It imports 284 `vllm.*` modules and patches request/output classes; "vLLM-Omni without vLLM" is not a meaningful target. What *is* separable is the **model code of individual stages** (see [engines/vllm_omni_models.md](engines/vllm_omni_models.md) §5: decoders, codecs, action heads are plain torch) and the **pipeline semantics** (stage graph, chunk sizes, abort rules), which is what the edge design re-implements.
|
||||
|
||||
## 9. Architectural constraints versus missing implementations
|
||||
|
||||
| Engine | Architectural constraint (would need a redesign) | Missing implementation (could be added without changing the architecture) |
|
||||
|---|---|---|
|
||||
| vLLM | Multi-process engine with ZMQ IPC and a Python control loop; server-class dependency set (torch, transformers) | Jetson wheels/CI; Android/Vulkan/QNN (would be new platform plugins, but the torch dependency remains) |
|
||||
| vLLM-Omni | Every AR/generation stage is a vLLM engine process and diffusion stages are a worker process or inline; the default inter-stage transport is host-staged shared memory (the Mooncake/Mori transfer-engine connectors have device-memory fast paths, and Mori's `xgmi` backend covers intra-node AMD GPU-to-GPU per `deploy/qwen3_omni_moe_mori_intranode.yaml`; none is validated for an edge target); start-up = per-stage load + optional compile/capture | CPU platform; a device-direct connector for single-GPU/edge co-location (repo docs list D2D as roadmap); action keys in `OmniPayload`; CPU/edge deploy YAML overlays; a codec-token streaming wire contract (the speech stream endpoint consumes PCM today) |
|
||||
| llama.cpp | Single-graph-per-arch C++ model definitions (each new architecture is C++ work); no request scheduler in the library | Server TTS endpoint; batched audio generation; abort callback on GPU backends; VLA action head (proposal in report) |
|
||||
| ncnn | No LLM runtime, no tokenizer, batch-1 KV cache, host-provided masks/RoPE; app owns all pipeline logic | NPU delegates; more Vulkan ops (Embed, MatMul, GLU, RNN); LLM export tooling in-tree |
|
||||
| MNN | Batch 1 baked at export; static shapes on GPU/NPU; one `Llm` per conversation; no request queue | Streaming callback for Qwen3-TTS (single callback today); Python TTS API; more TTS families; VLA (nothing yet) |
|
||||
| ExecuTorch | Everything is AOT: no runtime op fallback across devices, static kernel table, upper-bound shapes, one `Method` at a time | In-tree batching executor; mobile TTS example; VLA models; generic TTS runner |
|
||||
|
||||
## 10. TTS and VLA capability matrix
|
||||
|
||||
| Capability | vLLM | vLLM-Omni | llama.cpp | ncnn | MNN | ExecuTorch |
|
||||
|---|---|---|---|---|---|---|
|
||||
| TTS model families in tree | none | 15 (Qwen3-TTS, CosyVoice3, Fish S2 Pro, MOSS, IndexTTS-2, VoxCPM2, GLM-TTS, Higgs, OmniVoice, Ming, Voxtral TTS, Gepard, dots, Audex, ...) | Qwen3-TTS, Pocket-TTS (`MTMD_GEN_AUDIO_TYPE_*`), legacy WavTokenizer arch | Piper/VITS example | Qwen3-TTS (`Llm::generateTTS`), Qwen2.5-Omni talker, `mnn_tts` SDK (Bert-VITS2, Supertonic, Piper), sherpa-mnn (Kokoro, Matcha, VITS) | Voxtral-4B-TTS example (streaming), Supertonic (MLX only) |
|
||||
| Streaming audio out | N/A | Yes, 1-frame first chunk then 25-frame chunks; measured TTFA ≈ 28 ms on L20X | Step API (app-level streaming possible), CLI writes one WAV | App-level | Qwen2.5-Omni chunked callback; Qwen3-TTS single callback | Voxtral chunk callback (25 frames, initial 5) |
|
||||
| Codec/vocoder state across chunks | N/A | Explicit (conv ctx, 72-frame window, ref-prefix cache) for Qwen3-TTS; MOSS ring KV; CosyVoice3 re-decodes cumulative history | Qwen3-TTS code2wav 72-frame windows | App-owned | App/engine-owned worker queues | Runner-owned (left context 25) |
|
||||
| VLA models in tree | OpenVLA (image-only, discrete action bins) | GR00T N1.7, π0, DreamZero, InternVLA-A1 (offline), Cosmos3 action modes; OpenPI WebSocket serving | none (vision encoders + `embd` input only) | none | none (vision exporters + DiT runtime) | none (VLM runners, DiT on QNN) |
|
||||
| Structured (proprioceptive) inputs | none (`EmbedsPrompt` only) | Via diffusion sampling-param `extra_args["robot_obs"]`, not through `OmniPayload` | `llama_batch.embd` | Any `Mat` input | Any `VARP` input | Any tensor input |
|
||||
| Action head type | discrete tokens | Flow/diffusion heads (4–16 Euler/UniPC steps), stateless per step, batch 1, eager | Would be a sidecar graph (proposal) | Host denoise loop per extract | Host denoise loop; DiT runtime exists | Loop inside `.pte` (`while_loop` supported) or runner loop |
|
||||
|
||||
## 11. Measured data points that anchor the comparison
|
||||
|
||||
From [experiments.md](experiments.md), this host only (x86 Xeon 8480C; L20X):
|
||||
|
||||
| Measurement | Value | What it tells the edge plan |
|
||||
|---|---|---|
|
||||
| vLLM-Omni Qwen3-TTS 1.7B, 1 GPU, batch 1: TTFA p50 / p95 | 27.7 / 30.7 ms | First-audio is nearly free once graphs are warm, but the default 1→25 schedule leaves an ≈80 ms playback gap (80 ms of audio at 30 ms, next chunk at 190 ms); the deploy YAML's `codec_chunk_ramp`/`codec_chunk_adaptive` options exist to remove that cliff. An edge port must keep the chunk semantics configurable and gate on uninterrupted playback, not on first-chunk arrival. |
|
||||
| Same: steady cadence per 25-frame (2 s) chunk | ≈160 ms | About 6.4 ms per codec frame for talker + code predictor + vocoder; the edge budget is 80 ms per frame. |
|
||||
| Same: RTF | 0.08–0.10 | ~10× real-time headroom on a data-center GPU. No device extrapolation is made here: predictor and decoder scaling on edge hardware is unmeasured. |
|
||||
| Same: engine initialization (default config) | 157–188 s | Observed with CUDA graphs and `torch.compile`, cold and warm caches; the minimum with `enforce_eager`/parallel init is unmeasured. Either way the edge design precompiles/exports ahead of time. |
|
||||
| llama.cpp CPU, 409 M-param LLM, Q4_0, 4 threads: decode | 135.6 tok/s | Backbone-only microbenchmark. A Qwen3-TTS frame also needs 15 code-predictor sub-steps (`qwen3_code_predictor.py`) and the decoder, none of which `llama-bench` measures; it bounds only the talker backbone's share of the 80 ms frame budget. |
|
||||
| llama.cpp CPU, SigLIP-class vision encoder, one 512 px tile, 8 threads | ≈0.6–1.2 s | Vision encoding, not the LLM, dominates per-frame VLA cost on CPU; tile count must be pinned to 1 per camera. |
|
||||
|
||||
## 12. Recommendation summary (detailed in the design documents)
|
||||
|
||||
- Treat vLLM-Omni as the **reference implementation and cloud/edge-server tier**, not as code to ship on a phone. What transfers to edge is its stage graph, chunk schedule, state ownership rules, abort semantics, and API shapes.
|
||||
- For **TTS on edge**, llama.cpp and MNN already run the same Qwen3-TTS family; ExecuTorch has a streaming TTS runner pattern (Voxtral). Choose per target: llama.cpp (ARM CPU, Apple Metal, Adreno OpenCL, Hexagon), MNN (Android CPU/OpenCL/QNN, iOS Metal/CoreML), ExecuTorch (CoreML/MLX, QNN, XNNPACK with the strongest AOT story), ncnn (only as a vocoder/VITS-class runtime).
|
||||
- For **VLA on edge**, no runtime has a policy model in tree; the closest building blocks are ExecuTorch (static memory, zero-copy I/O, `while_loop` for the denoise loop, QNN/CoreML delegates) and MNN (DiT runtime, vision exporters, NPU delegation). llama.cpp is viable for the VLM backbone but not for the action head today; ncnn fits only the vision encoder or a small MLP head.
|
||||
- See [vllm_omni_edge_feasibility.md](vllm_omni_edge_feasibility.md) for the strategy comparison, [tts_edge_design.md](tts_edge_design.md) and [vla_edge_design.md](vla_edge_design.md) for the designs, and [implementation_roadmap.md](implementation_roadmap.md) for phases and acceptance criteria.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Simulating edge devices on this host, and why the CPU path is slow
|
||||
|
||||
Companion to [edge_engine_comparison.md](edge_engine_comparison.md). Measured 2026-09-09; raw data under [experiments/edge_engine_compare/results/gpu_emu](experiments/edge_engine_compare/results/gpu_emu/) and [results/cpu_attrib](experiments/edge_engine_compare/results/cpu_attrib/), driver scripts `gpu_emu_chain2.sh` (MPS-capped runs) and the branch's `hw_emulation.py`.
|
||||
|
||||
## 1. What this GPU is, and what can be emulated without root
|
||||
|
||||
`nvidia-smi` reports the eight cards as "NVIDIA L20X, Ada Lovelace, 143,771 MiB", but the measured dense bf16 matmul throughput is **804 TFLOPS** (8192³, 20 iterations) with 132 SMs at 1980 MHz and 140 GiB, which is H200-class silicon, not an Ada L20 (≈120 TFLOPS). The comparison and the branch's results should therefore be read as **H200-class** numbers; the label in earlier documents is the driver's.
|
||||
|
||||
| Edge property | Emulable here? | How | Not covered |
|
||||
|---|---|---|---|
|
||||
| Fewer SMs (Orin AGX 16 SMs, Orin NX 8, Thor 20 vs 132) | **yes** | user-level MPS daemon + `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` (verified: 12 % cap → 100.6 TFLOPS, ratio 0.125). Two traps: the Unix-socket path of `CUDA_MPS_PIPE_DIRECTORY` must be < 108 bytes, and clients inside the MPS namespace must use `CUDA_VISIBLE_DEVICES=0` even when `gpu run` assigned another index | lower SM clocks (Orin 1.3 GHz vs 1.98 GHz) |
|
||||
| Small VRAM / unified memory budgets | yes | branch `deploy_profile="auto"` with `VLLM_OMNI_HW_PROFILE` overrides: absolute KV budgets, eager mode, small batches (`hw_emulation.py` jetson_orin_8g / 32g / cuda_discrete_8g) | unified-memory bandwidth sharing with the CPU |
|
||||
| Weak host CPU (Orin: 8–12 Cortex-A78AE at 2.2 GHz) | partly | `taskset` to 4–6 Xeon cores for the whole engine; x86 cores are still ≈2× faster per core | ARM ISA, big.LITTLE |
|
||||
| Memory bandwidth (HBM 4.8 TB/s vs LPDDR5 102–273 GB/s) | **no** | analytic projection only (§4) | – |
|
||||
| Clock / power caps, MIG partitions | no | `nvidia-smi -lgc/-pl` and MIG need root; MIG is disabled on these cards | – |
|
||||
| Thermal throttling, CPU/GPU memory contention | no | – | – |
|
||||
|
||||
## 2. GPU: sensitivity to SM count (MPS caps, H200, 0.6B CustomVoice, edge profile)
|
||||
|
||||
Same 12 prompts, one utterance at a time, `deploy_profile="edge"`, 0.6B CustomVoice; MPS `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` = share of the 132 SMs available to every process of the deployment (both stages). llama.cpp rows: `llama-tts` CUDA build, all layers offloaded, same cap. The "Jetson-like" row combines a 12 % SM cap (16 SMs ≈ Orin AGX), 6 host cores for the whole engine, and the branch's `jetson_orin_8g` hardware profile (eager mode, absolute 8 GB budget).
|
||||
|
||||
| SM share (≈ SMs) | vLLM-Omni talker step ms/frame | vLLM-Omni TTFA p50 / p90 | vLLM-Omni RTF | llama.cpp talker ms/frame | llama.cpp vocoder ms/frame | llama.cpp RTF |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 100 % (132) | 6.0 | 41–46 / 49 ms | 0.10 | 17.4 | 3.8 | 0.28 |
|
||||
| 25 % (33) | 6.5–7.0 | 43 / 49–53 ms | 0.08–0.09 | 23.1 | 2.1 | 0.33 |
|
||||
| 12 % (16, Orin AGX class) | 9.1 | 52 / 58 ms | 0.12 | 33.7 | 4.7 | 0.50 |
|
||||
| 6 % (8, Orin NX class) | 9.3 | 58 / 66 ms | 0.13 | 48.0 | 4.1 | 0.68 |
|
||||
| Jetson-like combined: 12 % + 6 host cores + 8 GB profile (eager) | ≈13 (from RTF) | 160 / – ms | 0.50 | – | – | – |
|
||||
| (reference) 8 GB profile alone, uncapped, from §8.5 | – | 181 ms | 0.57 | – | – | – |
|
||||
|
||||
Reading: with CUDA graphs and one utterance in flight, the talker step is latency-bound: cutting the SMs to 6 % of an H200 costs vLLM-Omni only +55 % per frame (6.0 → 9.3 ms) and +15 ms of TTFA, and RTF stays ≈0.13. llama.cpp's kernels are launch-bound per layer and lose 2.8× at the same cap (17 → 48 ms per frame). The combined Jetson-like case shows where the real cost sits: forcing eager mode (the 8 GB profile disables CUDA graphs so the small budget is not consumed by graph pools) raises the frame time to ≈13 ms and TTFA to 160 ms regardless of the SM cap, i.e. **on a small-memory Jetson the branch should keep graph capture for the talker/predictor and spend the memory elsewhere**, which is the next tuning item for `deploy/edge/hardware/jetson.yaml`. What SM caps cannot show is the memory-bandwidth term, projected in §4.
|
||||
|
||||
## 3. Where a GPU frame goes (synchronized attribution, uncapped)
|
||||
|
||||
`VLLM_OMNI_STEP_STATS_SYNC=1` synchronizes the device around the model-level timers (adds ≈3 ms per step of sync overhead, so `core.step` reads 9.6 ms here instead of the 6.0 ms measured without sync):
|
||||
|
||||
| per talker step (= one 80 ms frame) | ms |
|
||||
|---|---|
|
||||
| model forward (28-layer talker, CUDA graph) incl. sampling hook | 1.8 |
|
||||
| sampler | 0.6 |
|
||||
| talker-MTP graph (15 code-predictor sub-steps + embeddings, one CUDA graph) | inside the forward's 1.8 ms (graph replay; the separate `model.mtp_graph_ms` timer measures 0.04–0.43 ms of launch time without sync) |
|
||||
| scheduler (`schedule` + `update_from_output`) | 0.2 |
|
||||
| remainder: input preparation, D2H of codes/hidden, Python bookkeeping | ≈3–4 |
|
||||
|
||||
The GPU compute for a frame is a couple of milliseconds; more than half of the 6 ms step is host-side runner work, which is what a slow ARM host CPU would stretch first.
|
||||
|
||||
## 4. Analytic projection to Jetson-class devices
|
||||
|
||||
The talker step is memory-bound at batch 1: every frame streams the talker weights once (0.6B bf16: 1.2 GB) and the code-predictor weights fifteen times (5 layers × 1024 hidden ≈ 160 MB each pass, 2.4 GB per frame). Using the measured H200 split and datasheet bandwidths (weights from the HF config; bandwidths: H200 4.8 TB/s, Orin AGX 204.8 GB/s, Orin NX 102.4 GB/s, Thor 273 GB/s):
|
||||
|
||||
| device | weight streaming per frame (talker + 15 predictor passes) | host-side runner work (≈4 ms on Xeon, scaled by single-core speed) | projected ms/frame | projected RTF (80 ms frame) |
|
||||
|---|---|---|---|---|
|
||||
| H200 (measured) | 0.75 ms | ≈4 ms | 6.0 (measured) | 0.10 (measured) |
|
||||
| Jetson AGX Orin 64 GB | 17.6 ms | ≈8 ms | ≈26 | ≈0.33 |
|
||||
| Jetson Orin NX 16 GB | 35 ms | ≈8 ms | ≈43 | ≈0.55 |
|
||||
| Jetson Thor | 13 ms | ≈6 ms | ≈19 | ≈0.24 |
|
||||
|
||||
These are floors from bandwidth plus measured host overhead, not measurements: kernel launch latency at small SM counts, unified-memory contention with the code2wav stage on the same memory bus, and thermal limits push them up. The MPS 12 % run (§2) is the compute-side check of the Orin AGX case (16 of 132 SMs); a 4-bit talker (llama.cpp Q4_0 runs at 3.1 ms/token on 16 Xeon threads for the same backbone) would cut the streaming term by ≈3× on all rows. **Conclusion:** the branch's talker stays real-time on Orin AGX / Thor with 2–4× headroom by this model, is marginal on Orin NX, and its 15-pass code predictor is the term to attack (see §5).
|
||||
|
||||
## 5. Why the CPU path is slow
|
||||
|
||||
### 5.1 Measured attribution (vLLM-Omni CPU platform, 0.6B CustomVoice, 12+4 threads on one socket, bf16 with AMX)
|
||||
|
||||
| per frame (one talker step) | ms | share of the 80 ms frame |
|
||||
|---|---|---|
|
||||
| **code predictor loop** (15 sub-steps × 6.4 ms, 5-layer 1024-hidden model, batch 1) | **97.1** | 121 % |
|
||||
| talker forward (28 layers, 0.45 B params) | 25.4 | 32 % |
|
||||
| sampler (top-k/top-p on a 3072-way vocabulary, penalties, CPU) | 9.9 | 12 % |
|
||||
| scheduler + bookkeeping | ≈2 | 2 % |
|
||||
| **engine-core step** | **133.9** | **167 %** |
|
||||
|
||||
(`core.step` 134 ms here vs 147 ms in the earlier sweep: the attribution run had the host to itself.) The code2wav stage runs concurrently in its own process and does not add to this cadence.
|
||||
|
||||
### 5.2 The same split in llama.cpp (native C++, same host, 16 threads)
|
||||
|
||||
`llama-bench` on the converted talker GGUF gives the backbone alone at **126.7 tok/s f16 (7.9 ms/token)** and **325.8 tok/s Q4_0 (3.1 ms/token)**, yet `llama-tts` needs **85–97 ms per frame**: ≈80 ms of every frame is the fifteen code-predictor passes (≈5.3 ms each) plus sampling. llama.cpp's predictor pass is only ≈20 % cheaper than vLLM-Omni's 6.4 ms even though it is hand-written C++ with no Python — so the cost is not "Python overhead"; it is the structure of the work.
|
||||
|
||||
### 5.3 Root causes, in order
|
||||
|
||||
1. **Sixteen sequential forward passes per 80 ms frame.** One talker pass plus fifteen residual-codebook passes, each depending on the previous sample. Real time needs the whole chain under 80 ms, i.e. **< 5 ms per pass including sampling**. Both engines sit at 5.3–6.4 ms per predictor pass on 16 Xeon threads.
|
||||
2. **Each pass is latency-bound, not bandwidth-bound.** A predictor pass reads ≈160 MB of bf16 weights, which is ≈1.6 ms at the ≈100 GB/s one socket-half delivers; the measured 6.4 ms is 4× that. The rest is per-operator cost: ≈100 small kernels per pass (5 layers × QKV/attention/RoPE/norms/MLP/gate) each paying OpenMP fork-join across 12 threads, cache-cold tiny GEMVs, and (in PyTorch) operator dispatch. The talker's 28-layer pass is at 2× its bandwidth floor (25 ms vs 12 ms) for the same reason.
|
||||
3. **Sampling costs 10 ms on CPU.** vLLM's sampler applies penalties, top-k and top-p over the padded vocabulary per step with several full-vocabulary passes; on GPU this is 0.6 ms. The fifteen residual codebooks are also sampled (Gumbel top-k) inside the predictor loop.
|
||||
4. **Threads stop helping at 16.** 16 → 32 threads changed the frame time by 8 % (147 → 136 ms) because the work is latency-bound (item 2) and OpenMP synchronization grows with thread count; the two stages also compete for the same memory channels.
|
||||
5. **bf16/AMX is the only CPU precision on the branch.** There is no int8/int4 talker path in vLLM's CPU backend for this model; llama.cpp's Q4_0 backbone shows the bandwidth term can shrink 2.5×, but that only helps the part that is bandwidth-bound (talker pass, ≈25 → ≈12 ms), not the predictor's overhead floor.
|
||||
6. **Vocoder**: not on the vLLM-Omni critical path (concurrent stage), but in llama.cpp the BigVGAN decoder costs a further 55 ms per frame serially, which is why its end-to-end RTF matches vLLM-Omni's despite the faster talker.
|
||||
|
||||
### 5.4 What would make CPU real-time, with expected effect
|
||||
|
||||
| change | where | expected per-frame effect (0.6B, 16 threads) | evidence |
|
||||
|---|---|---|---|
|
||||
| Fuse each predictor pass into one kernel / one OpenMP region (C++ or a compiled graph), drop per-op dispatch | predictor | 6.4 → ≈2–3 ms per pass, i.e. −50…−65 ms per frame | bandwidth floor 1.6 ms; llama.cpp's eager C++ is already at 5.3 ms, so fusion (not language) is the lever |
|
||||
| Greedy or cheap top-k for the 15 residual codebooks; skip penalties for codec ids | sampler + predictor | −8 ms (sampler) and part of each pass | GPU sampler is 0.6 ms for the same work |
|
||||
| Int8/int4 talker weights (AMX int8 or a GGUF-style kernel) | talker pass | 25 → ≈10–12 ms | llama-bench Q4_0 3.1 ms vs f16 7.9 ms per token |
|
||||
| `torch.compile` (inductor) on the CPU runner | whole step | −6 % measured (`cpu16_inductor` in §8.4) | measured |
|
||||
| Architectural: fewer residual codebooks per pass (predict 2–4 codebooks per pass) or a parallel/non-autoregressive predictor | model | 15 → 4–8 passes per frame | requires retraining or a different checkpoint; out of scope for the branch |
|
||||
|
||||
Adding the first three rows gives ≈134 → ≈45–60 ms per frame, i.e. real time with little margin on 16 Xeon threads; on a 4–8-core ARM box (≈2–3× slower per core, see the emulated `x86_cpu_avx2` case at RTF 12) the architectural row is the only one that reaches real time. That matches the study's original conclusion: CPU-only real-time TTS with this model family needs a lighter talker/predictor, not a faster runtime.
|
||||
@@ -0,0 +1,121 @@
|
||||
# Edge engine comparison: Qwen3-TTS on vLLM-Omni (edge branch) vs native edge runtimes
|
||||
|
||||
Measured on 2026-09-09 on the same host as the rest of this study (Xeon 8480C, one NVIDIA L20X through `gpu run --gpus 1`), same 12 English prompts (`vllm-omni/benchmarks/tts/prompts_12.txt`), same model family (Qwen3-TTS-12Hz, 0.6B and 1.7B), same reference speaker wav, one utterance at a time. Raw results, driver scripts and logs live in [experiments/edge_engine_compare/](experiments/edge_engine_compare/); the merged table is [results_summary.md](experiments/edge_engine_compare/results_summary.md) (regenerate with `python summarize_compare.py --out results_summary.md`).
|
||||
|
||||
## 1. What was compared, and what could not be
|
||||
|
||||
| Engine (checkout) | Qwen3-TTS path | Status here |
|
||||
|---|---|---|
|
||||
| vLLM-Omni `edge/omni-edge` (this branch) | two-stage talker → code2wav pipeline, streaming chunks, edge profile; CPU platform from WP3 | measured on GPU and CPU |
|
||||
| llama.cpp `e71b80510` | `llama-tts` + `libmtmd` (talker GGUF + mmproj holding the speaker encoder and the codec decoder; `conversion/qwen3tts.py`) | built out of tree (CPU and CUDA), models converted locally from the cached HF checkpoints (f16, Q8_0, Q4_0), measured on GPU and CPU |
|
||||
| MNN `bef71b97` | `transformers/llm/engine/demo/qwen3_tts_demo` + `llmexport.py` | built (`MNNConvert`, `qwen3_tts_demo`), but the exporter imports the external `qwen_tts` package (the QwenLM/Qwen3-TTS GitHub repo), which is not on this host; cloning it is a download and was not done without permission. **Not measured.** |
|
||||
| ExecuTorch `cfc6ecd516` | no Qwen3-TTS runner (the streaming TTS runner is Voxtral); the branch's decoder export targets it | needs `pip install executorch` (permission gate). **Not measured.** |
|
||||
| ncnn `3b7bdba7` | no TTS model | not applicable |
|
||||
|
||||
Model/task parity: llama.cpp and MNN only run the **Base** checkpoints with a reference speaker (x-vector cloning); the branch's headline numbers are for **CustomVoice** (preset speakers). So this comparison runs vLLM-Omni on the *same* Base checkpoints with the *same* reference wav in `x_vector_only_mode`, and quotes the CustomVoice numbers separately where the reference-audio cost matters. The reference wav is a 4.5 s utterance synthesized earlier by the two-stage server; it drives timing only, not quality.
|
||||
|
||||
## 2. Metric definitions
|
||||
|
||||
- **TTFA**: vLLM-Omni: measured time from request submission to the first streamed PCM chunk (p50 over 12 prompts × repeats). llama.cpp: the tool is not streaming (all frames, then one vocoder pass, then the WAV), so the column is an **estimate**: prompt eval + 2 talker frames + 2 frames' share of the vocoder time (the edge profile's first chunk is 2 frames). llama.cpp's speaker encoding happens at load time and is excluded; vLLM-Omni's per-request x-vector extraction is included in its Base TTFA.
|
||||
- **Talker ms/frame**: llama.cpp: generation time / frames from its own timing line. vLLM-Omni: the stage-0 engine-core step from the WP8 counters (one step = one frame, includes the 15-codebook predictor).
|
||||
- **Vocoder ms/frame**: llama.cpp's vocoder time / frames. vLLM-Omni's code2wav runs concurrently in its own stage, so its cost does not add to the frame period; it is not tabulated.
|
||||
- **RTF**: total processing wall time / seconds of audio (llama.cpp: its "total" line; vLLM-Omni: request wall time / audio).
|
||||
- **Startup**: vLLM-Omni: engine init to ready (`init_s`, both stages). llama.cpp: process start + weight load + speaker encoding (`wall − total`, per invocation; the tool reloads everything for every utterance).
|
||||
- **Memory**: peak RSS of the process tree (both), peak per-process GPU memory (vLLM-Omni, `nvidia-smi` samples; llama.cpp GPU memory was not sampled).
|
||||
|
||||
## 2b. Mobile (real devices via Qualcomm AI Hub, 2026-09-10)
|
||||
|
||||
Measured on Samsung Galaxy S24 (Snapdragon 8 Gen 3) and Galaxy S25 (Snapdragon 8 Elite): QNN context binaries on the Hexagon NPU, TFLite on the Adreno GPU and the phone CPU, AI Hub on-device medians. Details, job ids and fidelity in [mobile_htp_simulation.md §5](mobile_htp_simulation.md); export code in `vllm_omni/edge/qnn_export.py` and `decoder_export.py`.
|
||||
|
||||
| component (0.6B, real weights), per 80 ms frame | Hexagon NPU | Adreno GPU | phone CPU |
|
||||
|---|---|---|---|
|
||||
| talker decode step (v2 export), fp16 | **15.8 ms** | 29.9 ms | 29.8 ms |
|
||||
| code predictor, 15 cached steps, fp16 | **33.9 ms** | 65.0 ms | 72.3 ms |
|
||||
| Code2Wav vocoder, fp16 | 182 ms | **21.7 ms** | 109 ms |
|
||||
| Code2Wav vocoder, fp32 (numerically exact) | – | **45.8 ms** | 109 ms |
|
||||
|
||||
Best placement: talker + predictor on the NPU, vocoder on the GPU. The two are separate stages exchanging codec tokens, so they pipeline: **49.7 ms per 80 ms frame, RTF 0.62** (96 ms, RTF 1.19 if run serially with the exact vocoder). Fidelity: NPU fp16 39.2 dB (vocoder) / 48.3 dB (predictor); GPU fp32 108.4 dB; GPU fp16 27.5 dB. Every automatic int8 build is 5–17× faster and numerically broken (vocoder −4.3 dB, predictor 3.1 dB), so int8 is not counted.
|
||||
|
||||
Runtime comparison on the same graphs and phone (per frame): the talker step costs 15.8 ms on a QNN context binary, 16.3 ms on TensorFlow Lite and 16.6 ms on ONNX Runtime — all NPU paths converge because they all end in QNN, so the export layout matters and the runtime does not. They separate elsewhere: the vocoder is 21.7 ms on TFLite/GPU, 122.5 ms on ONNX Runtime/GPU, 109–124 ms on the phone CPU, and cannot run on the NPU under TFLite or ONNX Runtime at all (fails after compiling; out of device memory).
|
||||
|
||||
Reading: with the heterogeneous split the branch's 0.6B pipeline is **real time on a 2024–2025 flagship phone, in faithful precision, without quantization**. llama.cpp and MNN have no Hexagon NPU path for Qwen3-TTS at all; llama.cpp's CPU-only pipeline extrapolates above RTF 2 on phone cores. The mistake to avoid is sending everything to the NPU: that is 3.7× slower than the split.
|
||||
|
||||
## 2c. Mobile inference engines on the same graphs (x86 CPU, 2026-09-10)
|
||||
|
||||
MNN, ncnn and ExecuTorch were the three engines this study could not reach earlier. All three are now installed
|
||||
(`.venvs/edge-engines`) and fed the *same* exported graphs as the QNN/TFLite/ONNX-Runtime runs: the v2 talker decode
|
||||
step (256-token cache), the code-predictor step (16-token cache) and the Code2Wav chunk (25 frames of audio from a
|
||||
97-frame window). Median of 15 runs after 3 warm-ups, pinned to 8 or 16 physical cores of the Xeon 8480C, fp32.
|
||||
Accuracy is signal-to-noise against the PyTorch output for the same input; the harness is
|
||||
`experiments/edge_engine_compare/qnn/engines/bench_engines.py`.
|
||||
|
||||
| graph | PyTorch 2.14 | ONNX Runtime 1.29 | MNN 3.x | ExecuTorch (XNNPACK) | ncnn |
|
||||
|---|---|---|---|---|---|
|
||||
| code predictor step, 8 / 16 threads | 4.47 / 3.30 ms | 4.27 / **2.51** ms | **3.69** / 4.13 ms | 6.89 / 5.01 ms | segfault |
|
||||
| talker decode step, 8 / 16 threads | 35.96 / 20.32 ms | **26.40 / 17.04** ms | 37.46 / 31.39 ms | 73.04 / 61.87 ms | segfault |
|
||||
| Code2Wav chunk (2 s audio), 8 / 16 threads | **957 / 590** ms | 1562 / 1248 ms | 1621 / 1059 ms | 1059 / 633 ms | 3538 / 2331 ms, silent |
|
||||
| accuracy vs PyTorch (16 threads) | reference | 86–114 dB | 85–109 dB | 84–107 dB | NaN inside the graph |
|
||||
|
||||
Reading:
|
||||
|
||||
- **ncnn cannot run this model.** pnnx converts all three graphs at its default optimisation level, but the two
|
||||
transformer steps segfault inside `Extractor::extract` and the vocoder fills with NaN partway through, which the
|
||||
final `clamp(-1, 1)` turns into constant silence — the timing column above is therefore meaningless for it. At
|
||||
`optlevel=0` the graph will not even load (`layer F.linear not exists or registered`). This is a converter-quality
|
||||
problem in the ncnn toolchain for these graphs, not a missing runtime feature.
|
||||
- **MNN and ExecuTorch both work and are numerically faithful** (85–109 dB, i.e. fp32 arithmetic differences only).
|
||||
MNN is the fastest engine on the small predictor step at 8 threads; ExecuTorch is consistently the slowest on the
|
||||
transformer steps (2–3× ONNX Runtime) but mid-pack on the vocoder.
|
||||
- **ONNX Runtime is the best CPU engine for the transformer steps** (1.4–1.8× MNN and ExecuTorch); **PyTorch is the
|
||||
best for the vocoder**, where its fused convolution/activation kernels beat every exported runtime.
|
||||
- Caveat that matters: this is x86. MNN, ncnn and ExecuTorch are tuned primarily for ARM (NEON, dotprod, i8mm), so
|
||||
this ranks the engines *on this host*, not on a phone. The phone-side ranking is in §2b, where the same graphs run
|
||||
on the actual accelerators.
|
||||
- MNN's Interpreter (C-style) API silently mis-feeds tensors of these ranks and produced −4 dB output; the numbers
|
||||
above come from its Module/expr API. Anyone benchmarking MNN on multi-input graphs should check parity first.
|
||||
|
||||
## 3. GPU (one L20X)
|
||||
|
||||
| engine | model | TTFA | talker ms/frame | vocoder ms/frame | RTF | startup | peak RSS |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| llama.cpp CUDA, all layers | 0.6B Base f16 | 109 ms (est.) | 17.7 | 3.6 | 0.31 | 4.5 s per invocation | not sampled |
|
||||
| llama.cpp CUDA, all layers | 0.6B Base Q4_0 | 125 ms (est.) | 16.8 | 2.4 | 0.28 | 3.4 s | not sampled |
|
||||
| llama.cpp CUDA, all layers | 1.7B Base f16 | 146 ms (est.) | 17.9 | 2.1 | 0.28 | 6.0 s | not sampled |
|
||||
| vLLM-Omni edge profile | 0.6B Base, x-vector | 561 ms p50 (incl. per-request x-vector: chunks then 27–43 ms apart) | 6.0 | – | 0.20 | 148 s | 9.0 GiB + 36.6 GB GPU |
|
||||
| vLLM-Omni edge profile | 1.7B Base, x-vector | 374 ms p50 (same) | 6.4 | – | 0.20 | 149 s | 8.8 GiB + 37.0 GB GPU |
|
||||
| vLLM-Omni edge profile | 0.6B CustomVoice (no ref audio) | 41–54 ms p50 | 6.0 | – | 0.10 | 87 s | 8.7 GiB + 37 GB GPU |
|
||||
| vLLM-Omni edge profile | 1.7B CustomVoice (no ref audio) | 44–46 ms p50 | 6.4 | – | 0.10 | 88 s | 7.8 GiB + 37 GB GPU |
|
||||
|
||||
Reading:
|
||||
- **Steady-state generation** is where vLLM-Omni wins on the GPU: 6 ms per codec frame (CUDA graphs, talker-MTP graph, compiled code predictor) against 17–18 ms in llama.cpp, i.e. 3× faster per frame, and its code2wav runs in parallel in a second stage, so RTF is 0.10 (CustomVoice) / 0.20 (Base) against 0.28–0.31. Both are far below real time; the difference only matters for concurrency headroom or for smaller GPUs.
|
||||
- **First audio** favours llama.cpp once reference-audio handling is in the picture: vLLM-Omni spends ≈0.3–0.5 s per request on x-vector extraction (the chunk cadence after the first chunk is normal), while llama.cpp encodes the speaker once at load. Without reference audio (CustomVoice) vLLM-Omni's measured TTFA of 41–54 ms is below llama.cpp's estimated 109–146 ms. Caching the x-vector per speaker on the server would bring the Base path to the same range; that is the obvious next fix on the branch.
|
||||
- **Startup and footprint** favour llama.cpp by more than an order of magnitude: 3–6 s to load and be ready (per invocation, speaker encoding included) versus 87–150 s (and 355 s with cold compile caches) for vLLM-Omni, whose two stages each carry a vLLM engine process with CUDA-graph capture and a `torch.compile` warm-up, and 36–37 GB of reserved GPU memory (memory fractions, not need) versus a few GB of weights.
|
||||
|
||||
## 4. CPU only (NUMA node 1 of the Xeon 8480C, pinned)
|
||||
|
||||
Full grid in [results_summary.md](experiments/edge_engine_compare/results_summary.md); the WP3 CustomVoice sweep is in [experiments.md §8.4](experiments.md#84-cpu-platform-wp3-and-cpu-thread-sweep). Both engines pinned to the same cores of NUMA node 1 (stage 0 gets 3/4 of the threads in vLLM-Omni).
|
||||
|
||||
| engine | model | threads | TTFA | playback-start | talker ms/frame | vocoder ms/frame | RTF | startup | peak RSS |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| llama.cpp | 0.6B Base f16 | 8 / 16 / 32 | 443 / 413 / 407 ms (est.) | – | 94 / 97 / 93 | 55 | 1.9 / 2.0 / 2.0 | 3–4 s | not sampled (Q8_0 run: 6.8 GiB) |
|
||||
| llama.cpp | 0.6B Base Q8_0 | 16 | 368 ms (est.) | – | 94 | 56 | 1.9 | 2.9 s | 6.8 GiB |
|
||||
| llama.cpp | 0.6B Base Q4_0 | 8 / 16 | 342 / 332 ms (est.) | – | 85 / 87 | 54 | 1.9 / 1.9 | 2.2 s | 6.3 GiB |
|
||||
| llama.cpp | 1.7B Base f16 | 16 | 541 ms (est.) | – | 114 | 56 | 2.2 | 5.1 s | 12.2 GiB |
|
||||
| llama.cpp | 1.7B Base Q4_0 | 16 | 407 ms (est.) | – | 91 | 53 | 2.0 | 3.0 s | 8.0 GiB |
|
||||
| vLLM-Omni CPU platform | 0.6B Base, x-vector | 8 / 16 / 32 | 855 / 581 / 707 ms | 8.4 / 5.1 / 4.6 s | ≈250 / 147 / 136 (from the CustomVoice sweep) | concurrent stage | 2.6 / 1.8 / 1.7 | 102–104 s | 13.2–13.4 GiB |
|
||||
| **vLLM-Omni CPU platform after the 2026-09-10 CPU changes** ([mobile_htp_simulation.md §3](mobile_htp_simulation.md); faithful fp32 predictor path, the branch default) | 0.6B Base, x-vector | 16 / 32 | 686 / 486 ms (x-vector included) | 3.0 / 2.2 s | 109 / 91 mean (talker 25 / 21 + predictor 69 / 56 + sampler <1) | concurrent stage | **1.27 / 0.99** | ≈100 s | ≈13 GiB |
|
||||
| same, CustomVoice (no reference audio) | 0.6B | 8 / 16 / 32 | 388 / 224 / 195 ms | 5.7 / 2.6 / 1.9 s | 148 / 93 / 77 | concurrent stage | 1.93 / **1.15 / 0.89** | ≈100 s | ≈13 GiB |
|
||||
| same with the int8-dynamic predictor (measured, then **rejected**: 22 % top-1 agreement with the fp32 predictor, see §3 of the HTP document) | 0.6B Base x-vector / CustomVoice | 16 / 32 | 696 / 175 ms | 2.6 / 1.7 s | 80 / 59 | concurrent stage | 1.12 / 0.80 | ≈100 s | ≈13 GiB |
|
||||
| vLLM-Omni CPU platform | 0.6B CustomVoice (WP3 sweep) | 8 / 16 / 32 | 426 / 292 / 252 ms | 9.4 / 5.3 / 5.0 s | 250 / 147 / 136 | concurrent stage | 2.8 / 1.8 / 1.7 | 110–156 s | 13.3 GiB |
|
||||
|
||||
Reading:
|
||||
- Update 2026-09-10: after the KV-cached code predictor with a compiled shape-static step, the compiled post-step (lm head + top-k sampling + embedding) and the model-side sampler ([mobile_htp_simulation.md](mobile_htp_simulation.md) §3), the branch generates a frame in 77–109 ms and runs at RTF 0.89–1.27 on 16–32 threads (real time at 32 threads), i.e. 1.6–2.2× faster end-to-end than llama.cpp, whose RTF stays 1.9–2.0 because its talker frame (85–97 ms) is followed by a serial ≈55 ms vocoder pass per frame. At 8 threads the two are level (RTF 1.93 vs 1.9), where llama.cpp's Q4_0 talker frame (85 ms) is still shorter than the branch's 148 ms; an int8 predictor would close that gap but the only int8 path available in this torch build (dynamic per-tensor int8) was rejected for fidelity. The paragraphs below describe the state before that change.
|
||||
- Neither engine reaches real time on this CPU class with the 0.6B model. llama.cpp's talker is faster per frame (85–97 ms vs 136–250 ms; vLLM-Omni's step includes the 15-codebook predictor eagerly on CPU) but llama.cpp then pays ≈55 ms per frame in its non-streamed BigVGAN vocoder, so the end-to-end RTF is the same ≈1.9–2.0 for llama.cpp and 1.7–2.6 for vLLM-Omni (whose decoder runs concurrently in the second stage).
|
||||
- llama.cpp needs half the RAM (6–8 GiB vs 13 GiB) and starts in 2–5 s instead of ≈100 s. Quantization trims its talker (Q4_0: −10 %) and its footprint but not the f16 vocoder; vLLM-Omni's CPU platform has no quantized talker path on this branch (bf16/AMX), the first thing to add if the CPU tier matters.
|
||||
- Playback-start latency is the honest streaming metric on CPU: vLLM-Omni streams but underruns (first audio at 0.6–0.9 s, then the player would stall for 4–8 s over the utterance); llama.cpp does not stream at all (the whole utterance is ready after ≈2× its duration). Threads beyond 16 help neither engine.
|
||||
|
||||
## 5. Bottom line for the edge plan
|
||||
|
||||
- On a Jetson/discrete-GPU class device, the branch's vLLM-Omni is the faster steady-state engine (3× per frame) and the better streaming server (measured 41–54 ms first audio, ramped chunks, concurrent decoder stage), but it costs 100 s+ of startup, a ~9 GiB host process and a Python/vLLM stack; llama.cpp starts in seconds, needs one process and a few GB, and is within 3× on generation speed. For an always-on edge server the branch is the right tier; for a device that must start on demand, or has < 8 GB, llama.cpp is.
|
||||
- On CPU-only devices neither runs Qwen3-TTS in real time; the study's conclusion stands (real-time CPU TTS needs a smaller/quantized talker and a streamed, cheaper vocoder), and llama.cpp's quantized GGUF path is the better starting point there.
|
||||
- MNN and ExecuTorch numbers are still missing for the reasons above; the exact unblock is `git clone https://github.com/QwenLM/Qwen3-TTS` (then re-run `llmexport.py` with `PYTHONPATH=<clone>`) and `pip install executorch` respectively.
|
||||
@@ -0,0 +1,724 @@
|
||||
# ExecuTorch (pytorch/executorch) — Engine Analysis
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Local path | `/data/zhoutaichang/embedding_infer/executorch` |
|
||||
| Remote | `https://github.com/pytorch/executorch.git` (origin) |
|
||||
| Branch | `main` |
|
||||
| HEAD | `cfc6ecd5166bec70c5fde2bad836fed26018fe52` — "Qualcomm AI Engine Direct - Enable QNN Windows ARM64 build in CI (#22438)", 2026-09-07 23:47 +0800 |
|
||||
| Declared version | `1.5.0` ([version.txt](../../executorch/version.txt)) |
|
||||
| Dirty status | clean (`git status --short` empty) |
|
||||
| Submodules | **all 23 uninitialized** (`git submodule status` shows `-` for every entry), including `extension/llm/tokenizers`, `backends/xnnpack/third-party/{XNNPACK,pthreadpool,cpuinfo,FP16,FXdiv}`, `backends/mlx/third-party/mlx`, `backends/vulkan/third-party/*`, `third-party/{ao,flatbuffers,flatcc,gflags,googletest,json,pocketfft,pybind11,ios-cmake,prelude}`, `kernels/optimized/third-party/eigen`, `shim`, `backends/cadence/utils/FACTO` |
|
||||
| Analysis date | 2026-09-08 (read-only; no builds, installs or GPU use) |
|
||||
| Repo guidance followed | [CLAUDE.md](../../executorch/CLAUDE.md) (AGENTS.md is a symlink to it), `.claude/*.md` reference docs, `.wiki/index.md` |
|
||||
|
||||
Evidence labels: **[Verified]** read in code/docs at this HEAD; **[Inferred]** reasoned from structure; **[Proposal]** design suggestion; **[Unknown]** not determinable from the tree. Repo-published numbers are labelled "repo-reported, not reproduced".
|
||||
|
||||
---
|
||||
|
||||
## 0. Summary
|
||||
|
||||
- **What it is** [Verified]: ExecuTorch is an *ahead-of-time (AOT) compiler + minimal C++ execution runtime* for PyTorch models. Python side (`exir/`) lowers `torch.export` programs through Edge IR, partitions subgraphs to hardware "delegates" (`to_backend`), plans memory statically, and serializes a FlatBuffer `.pte` ([schema/program.fbs](../../executorch/schema/program.fbs)). The C++ runtime ([runtime/executor/method.cpp](../../executorch/runtime/executor/method.cpp)) walks a linear instruction list per method.
|
||||
- **Execution runtime, not a serving stack** [Verified]: the core is a single-`Method`, single-thread, sequential interpreter (`Method::execute` loops chains → instructions; comment says "Chains are executed sequentially today"). There is no request queue, batcher or server in `runtime/`.
|
||||
- **Serving-adjacent layers exist above the core** [Verified]: `extension/llm/runner/` (TextLLMRunner / MultimodalRunner with token callbacks), an experimental `extension/llm/batching/` (`DecodeFirstScheduler`, `Runner`, `Session`, `GenerationHandle` with cancel) and `examples/llm_server/` (OpenAI-compatible Python control plane over JSONL worker binaries). No in-tree class implements `batching::Executor` outside its tests.
|
||||
- **Backends in tree** [Verified]: XNNPACK, Vulkan, WebGPU, Qualcomm QNN (HTP/GPU/LPAI), MediaTek Neuron, Samsung Exynos (ENN), Arm (Ethos-U via TOSA/Vela, VGF, Cortex-M CMSIS-NN), Cadence DSP, NXP Neutron, OpenVINO, Apple CoreML, Apple Metal (experimental), Apple "Core AI" (under construction), MLX (experimental), CUDA (AOTInductor + Triton; experimental ROCm), a "native" portable delegate, and the AOTI common library. **No `backends/apple/mps` directory exists at this HEAD.**
|
||||
- **Device model** [Verified]: backend selection happens entirely at export (partitioner). At runtime `BackendDelegate::Init` looks up the delegate by string id in a 16-slot static registry (`kMaxRegisteredBackends = 16`, [runtime/backend/interface.cpp](../../executorch/runtime/backend/interface.cpp#L20)) and calls `is_available()`. There is no runtime device discovery/selection.
|
||||
- **Memory** [Verified]: static memory planning in Python (`greedy` default, optional banked planning) produces `non_const_buffer_sizes`; the runtime `MemoryManager` = method allocator + `HierarchicalAllocator` (planned arenas, ≤16 spans) + temp allocator. A new `Device{CPU,CUDA}` concept and `DeviceAllocator` registry let planned buffers live in CUDA memory with explicit `_h2d_copy/_d2h_copy` ops inserted at delegate boundaries.
|
||||
- **Kernels** [Verified]: portable (209 ops in `functions.yaml`), optimized (23), quantized (19), ATen wrapper (225) plus LLM custom ops (`custom_sdpa`, `update_cache`, `quantized_moe_ffn`, `gated_delta_rule`, ...). Kernel table is a static array (default 250 ops × 8 kernels), selective build generates `selected_max_kernel_num.h`.
|
||||
- **Quantization** [Verified]: torchao source transforms (`8da4w`, `int8`, `4w`, `torchao:8da*`, `torchao:fpa*`) and PT2E quantizers per backend (XNNPACK, QNN, CoreML, Vulkan, OpenVINO, TOSA/Ethos-U/VGF). CUDA backend has int4/5/6/8 packed GEMM shims; MLX has int2/4/8 weight quant.
|
||||
- **LLM decode loop** [Verified]: `TextPrefiller::prefill` (chunked) → `TextTokenGenerator::generate` → per-token `TextDecoderRunner::step` → `Module::execute("forward"|"decode")` → `Sampler` → tokenizer `decode` → `token_callback`. KV cache is a mutable buffer inside the `.pte` (static, `[B,H,max_context,D]` layout), updated by `update_cache`/`custom_sdpa`.
|
||||
- **TTS in tree** [Verified]: `examples/models/voxtral_tts` (LM + flow head + codec decoder, 5 methods, CPU/XNNPACK and CUDA, streaming codec chunks with `AudioChunkCallback`) and `examples/models/supertonic` (MLX, 4 methods incl. `vocoder`, dynamic FP16). Mimi codec on QNN under `examples/qualcomm/oss_scripts/moshi`. Audio front end: `extension/audio/mel_spectrogram.py` (exportable Whisper mel), `extension/asr/runner` (seq2seq + transducer), `extension/llm/runner/audio.h`.
|
||||
- **VLA in tree** [Verified]: none. No OpenVLA/pi0/LeRobot/diffusion-policy/proprioceptive references anywhere. Vision encoders exist for VLMs (LLaVA, Llama 3.2 Vision, Gemma3, SmolVLM, InternVL3) via `MultimodalRunner` with `vision_encoder`/`token_embedding`/`text_decoder` method naming.
|
||||
- **Edge profile** [Verified]: core runtime CI thresholds ~45–52 KB stripped (Linux gcc/clang size_test), ~118 KB bare-metal Cortex-M / ~136 KB Zephyr all-in; CMake presets for android-arm64-v8a, ios, macos, linux, windows, arm-baremetal, arm-ethosu-linux, esp-baremetal, riscv64-linux, zephyr, llm-release-{cuda,rocm,metal}, mlx-release. Android AAR (Maven), iOS xcframeworks (SwiftPM), Python wheels incl. aarch64 CUDA wheels.
|
||||
- **Jetson** [Unknown]: no explicit Jetson/Orin documentation; an aarch64-Linux CUDA wheel workflow exists ([.github/workflows/build-wheels-cuda-aarch64-linux.yml](../../executorch/.github/workflows/build-wheels-cuda-aarch64-linux.yml)), so aarch64+CUDA is at least built.
|
||||
- **Tokenizers** [Verified]: C++ tokenizers (HF JSON, Tiktoken, SentencePiece, Llama2c, Tekken) live in the `extension/llm/tokenizers` submodule, which is **not checked out here**; only the `.claude/tokenizers.md` summary and call sites could be inspected.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph AOT["AOT (Python, exir/)"]
|
||||
TE["torch.export → ExportedProgram"] --> TOEDGE["to_edge / to_edge_transform_and_lower<br/>EdgeProgramManager"]
|
||||
TOEDGE --> PART["Partitioner(s) → to_backend()<br/>BackendDetails.preprocess → delegate blob"]
|
||||
PART --> PASSES["edge_to_executorch_passes<br/>ConstraintBasedSymShapeEvalPass, reinplace, PropagateDevicePass"]
|
||||
PASSES --> MP["MemoryPlanningPass (greedy / banked)"]
|
||||
MP --> EMIT["_Emitter → ExecutionPlan(FlatBuffer)"]
|
||||
EMIT --> PTE[".pte (+ .ptd via flat_tensor)"]
|
||||
end
|
||||
subgraph RT["Runtime (C++, runtime/)"]
|
||||
PTE --> DL["DataLoader (file/mmap/buffer)"]
|
||||
DL --> PROG["Program::load → Program::load_method"]
|
||||
PROG --> METH["Method::init<br/>parse_values / resolve delegates / resolve_operator"]
|
||||
METH --> EXEC["Method::execute → execute_instruction<br/>KernelCall | DelegateCall | JumpFalse | Move | Free"]
|
||||
EXEC --> KREG["Kernel registry<br/>portable / optimized / quantized / custom"]
|
||||
EXEC --> BREG["Backend registry (≤16)<br/>BackendInterface::init/execute"]
|
||||
MM["MemoryManager<br/>method alloc + HierarchicalAllocator + temp"] --> METH
|
||||
DEV["DeviceAllocatorRegistry<br/>CPU / CUDA"] --> MM
|
||||
PAL["Platform PAL (log, clock, alloc, abort)"] --> EXEC
|
||||
end
|
||||
subgraph EXT["Extensions"]
|
||||
MOD["extension/module::Module"] --> PROG
|
||||
LLM["extension/llm/runner<br/>TextLLMRunner / MultimodalRunner"] --> MOD
|
||||
BATCH["extension/llm/batching<br/>DecodeFirstScheduler / Runner"] -.-> LLM
|
||||
ASR["extension/asr/runner"] --> MOD
|
||||
THR["extension/threadpool (pthreadpool+cpuinfo)"] --> KREG
|
||||
THR --> BREG
|
||||
BIND["android JNI / apple Swift / pybindings / wasm"] --> MOD
|
||||
SRV["examples/llm_server (OpenAI HTTP)"] -.-> LLM
|
||||
end
|
||||
subgraph BE["Delegates (backends/)"]
|
||||
BREG --> XNN["XNNPACK"] & VK["Vulkan"] & QNN["Qualcomm QNN"] & CML["CoreML"] & MLX["MLX"] & CUDA["CUDA (AOTI)"] & ARM["Arm Ethos-U / VGF"] & OTH["MediaTek / Samsung / NXP / OpenVINO / Cadence / WebGPU / Metal / native"]
|
||||
end
|
||||
```
|
||||
|
||||
**Key modules**
|
||||
|
||||
| Module | Path | Responsibility | Key symbols |
|
||||
|---|---|---|---|
|
||||
| Export IR | [exir/program/_program.py](../../executorch/exir/program/_program.py) | `to_edge`, `to_edge_transform_and_lower`, `EdgeProgramManager.to_backend/to_executorch`, `ExecutorchProgramManager.buffer/save` | `to_edge` (L1414), `EdgeProgramManager` (L1496), `ExecutorchProgramManager` (L1823) |
|
||||
| Backend API | [exir/backend/backend_api.py](../../executorch/exir/backend/backend_api.py), [exir/backend/partitioner.py](../../executorch/exir/backend/partitioner.py), [exir/backend/backend_details.py](../../executorch/exir/backend/backend_details.py) | Partition + preprocess into delegate blobs | `to_backend` (singledispatch), `Partitioner.partition`, `BackendDetails.preprocess/preprocess_multimethod`, `PreprocessResult` |
|
||||
| Memory planning | [exir/memory_planning.py](../../executorch/exir/memory_planning.py), [exir/banked_memory_planning.py](../../executorch/exir/banked_memory_planning.py) | Lifetime analysis, arena assignment | `greedy`, `naive`, `MemoryPlanningAlgorithmSuite`, `_partition_specs_by_device`, `TargetMemoryMap` |
|
||||
| Config | [exir/capture/_config.py](../../executorch/exir/capture/_config.py) | Export/backend knobs | `EdgeCompileConfig`, `ExecutorchBackendConfig` |
|
||||
| Emitter | [exir/emit/_emitter.py](../../executorch/exir/emit/_emitter.py) | FX → FlatBuffer instructions | `_Emitter`, `_TopLevelEmitter` |
|
||||
| Schema | [schema/program.fbs](../../executorch/schema/program.fbs) | `.pte` file format | `Program`, `ExecutionPlan`, `Chain`, `KernelCall`, `DelegateCall`, `NonConstBufferDevice` |
|
||||
| Program/Method | [runtime/executor/program.h](../../executorch/runtime/executor/program.h), [runtime/executor/method.cpp](../../executorch/runtime/executor/method.cpp) | Load, init, execute | `Program::load/load_method`, `Method::init/execute/step/set_input/get_outputs`, `BackendDelegate` |
|
||||
| Memory | [runtime/executor/memory_manager.h](../../executorch/runtime/executor/memory_manager.h), [runtime/core/hierarchical_allocator.h](../../executorch/runtime/core/hierarchical_allocator.h), [runtime/core/memory_allocator.h](../../executorch/runtime/core/memory_allocator.h) | Arena allocators | `MemoryManager`, `HierarchicalAllocator::get_offset_address`, `MemoryAllocator` |
|
||||
| Device memory | [runtime/core/device_allocator.h](../../executorch/runtime/core/device_allocator.h), [runtime/core/device_memory_buffer.h](../../executorch/runtime/core/device_memory_buffer.h), [runtime/core/portable_type/device.h](../../executorch/runtime/core/portable_type/device.h) | Non-CPU planned buffers | `DeviceAllocator`, `DeviceAllocatorRegistry`, `DeviceMemoryBuffer`, `Device{CPU,CUDA}` |
|
||||
| Backend registry | [runtime/backend/interface.h](../../executorch/runtime/backend/interface.h) | Delegate ABI | `BackendInterface::{is_available,init,execute,set_option,get_option,destroy}`, `register_backend` |
|
||||
| Kernel registry | [runtime/kernel/operator_registry.h](../../executorch/runtime/kernel/operator_registry.h) | Op → kernel resolution | `Kernel`, `KernelKey`, `register_kernels`, `get_op_function_from_registry` |
|
||||
| PAL | [runtime/platform/platform.h](../../executorch/runtime/platform/platform.h) | OS abstraction | `et_pal_init/abort/current_ticks/emit_log_message/allocate/free`, `PalImpl` |
|
||||
| Kernels | [kernels/portable/functions.yaml](../../executorch/kernels/portable/functions.yaml), [kernels/optimized/optimized.yaml](../../executorch/kernels/optimized/optimized.yaml), [kernels/quantized/quantized.yaml](../../executorch/kernels/quantized/quantized.yaml) | CPU op libraries | YAML op defs + `kernels/*/cpu/op_*.cpp` |
|
||||
| Selective build | [tools/cmake/Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake), [codegen/gen.py](../../executorch/codegen/gen.py) | Generate registration for selected ops | `gen_selected_ops`, `EXECUTORCH_SELECT_OPS_*` |
|
||||
| Module API | [extension/module/module.h](../../executorch/extension/module/module.h) | Convenience wrapper | `Module::{load_method,execute,forward,set_input,set_output,method_meta}`, `LoadMode` |
|
||||
| LLM runner | [extension/llm/runner/](../../executorch/extension/llm/runner/README.md) | Prefill/decode loop | `IRunner`, `TextLLMRunner`, `TextPrefiller`, `TextDecoderRunner`, `TextTokenGenerator`, `IOManager`, `MultimodalRunner`, `MultimodalPrefiller`, `LLMSession`, `LLMEngine` |
|
||||
| LLM batching | [extension/llm/batching/](../../executorch/extension/llm/batching/runner.h) | Experimental multi-session scheduler | `Scheduler`, `DecodeFirstScheduler`, `Executor`, `Runner`, `Session`, `GenerationHandle` |
|
||||
| LLM cache | [extension/llm/cache/](../../executorch/extension/llm/cache/cache.h) | Off-graph KV cache abstractions | `CacheBase`, `SequenceCache`, `CellCache`, `RingPolicy`, `kvcache::update_and_attend` |
|
||||
| Custom ops | [extension/llm/custom_ops/custom_ops.py](../../executorch/extension/llm/custom_ops/custom_ops.py) | LLM kernels | `sdpa_with_kv_cache`, `custom_sdpa`, `update_cache`, `quantized_moe_ffn`, `gated_delta_rule` |
|
||||
| Threadpool | [extension/threadpool/threadpool.h](../../executorch/extension/threadpool/threadpool.h), [runtime/kernel/thread_parallel_interface.h](../../executorch/runtime/kernel/thread_parallel_interface.h) | CPU parallelism | `ThreadPool`, `get_threadpool`, `get_pthreadpool`, `parallel_for` |
|
||||
| Devtools | [devtools/etdump/](../../executorch/devtools/etdump/etdump_flatcc.h), [devtools/inspector/_inspector.py](../../executorch/devtools/inspector/_inspector.py), [devtools/bundled_program/](../../executorch/devtools/bundled_program/bundled_program.h) | Profiling / debugging | ETDump, `Inspector`, bundled program |
|
||||
|
||||
---
|
||||
|
||||
## 2. Device layer
|
||||
|
||||
### 2.1 Backends actually in tree [Verified]
|
||||
|
||||
Every backend registers itself with `register_backend(...)` into a static 16-entry table ([runtime/backend/interface.cpp](../../executorch/runtime/backend/interface.cpp#L20), `kMaxRegisteredBackends = 16`, with a TODO noting global statics prevent multiple executors). Registration sites:
|
||||
|
||||
| Backend id / dir | Registration | Hardware / OS | Status note |
|
||||
|---|---|---|---|
|
||||
| XNNPACK — [backends/xnnpack](../../executorch/backends/xnnpack/README.md) | [XNNPACKBackend.cpp#L272](../../executorch/backends/xnnpack/runtime/XNNPACKBackend.cpp#L272) | ARM64/ARMv7/x86 CPU on Android, iOS, macOS, Linux, Windows ([xnnpack-overview.md](../../executorch/docs/source/backends/xnnpack/xnnpack-overview.md)); KleidiAI via `EXECUTORCH_XNNPACK_ENABLE_KLEIDI` ([CMakeLists.txt#L34](../../executorch/backends/xnnpack/CMakeLists.txt#L34)) | default CPU delegate |
|
||||
| Vulkan — [backends/vulkan](../../executorch/backends/vulkan/README.md) | [VulkanBackend.cpp#L931](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp#L931) | Android GPUs (Adreno/Mali/PowerVR noted in wiki), desktop experimental | mature on Android |
|
||||
| WebGPU — [backends/webgpu](../../executorch/backends/webgpu/README.md) | [WebGPUBackend.cpp#L263](../../executorch/backends/webgpu/runtime/WebGPUBackend.cpp#L263) | Dawn/Tint WGSL; browser via Emscripten, native Metal/Vulkan | under active development |
|
||||
| Qualcomm QNN — [backends/qualcomm](../../executorch/backends/qualcomm/README.md) | [QnnExecuTorchBackend.cpp#L394](../../executorch/backends/qualcomm/runtime/QnnExecuTorchBackend.cpp#L394) | HTP / GPU / LPAI; 38 chipset enum entries in [qc_schema.py](../../executorch/backends/qualcomm/serialization/qc_schema.py) (`QcomChipset`) | no BC guarantees stated |
|
||||
| MediaTek Neuron — [backends/mediatek](../../executorch/backends/mediatek/README.md) | [NeuronBackend.cpp#L261](../../executorch/backends/mediatek/runtime/NeuronBackend.cpp#L261) | Dimensity 9300/9400 NPU | |
|
||||
| Samsung Exynos — [backends/samsung](../../executorch/backends/samsung/README.md) | [enn_backend.cpp#L96](../../executorch/backends/samsung/runtime/enn_backend.cpp#L96) | Exynos 2500 NPU/DSP (`EXECUTORCH_BUILD_ENN`) | |
|
||||
| Arm Ethos-U — [backends/arm](../../executorch/backends/arm/README.md) | [EthosUBackend.cpp#L451](../../executorch/backends/arm/runtime/EthosUBackend.cpp#L451) | Ethos-U55/65/85 via TOSA → Vela | bare-metal |
|
||||
| Arm VGF — same dir | [VGFBackend.cpp#L803](../../executorch/backends/arm/runtime/VGFBackend.cpp#L803) | Vulkan ML extensions (SPIR-V ML) | |
|
||||
| Arm Cortex-M — [backends/cortex_m](../../executorch/backends/cortex_m/README.md) | not a delegate: an operator dialect (`cortex_m::*`) on CMSIS-NN with portable fallback | Cortex-M MCUs | Beta |
|
||||
| Cadence — [backends/cadence](../../executorch/backends/cadence/README.md) | no delegation; custom ops for HiFi4/FusionG3/Vision DSP (see wiki "No-delegation architecture") | Xtensa DSP | |
|
||||
| NXP Neutron — [backends/nxp](../../executorch/backends/nxp/README.md) | [NeutronBackend.cpp#L676](../../executorch/backends/nxp/runtime/NeutronBackend.cpp#L676) | eIQ Neutron NPU (i.MX) | |
|
||||
| OpenVINO — [backends/openvino](../../executorch/backends/openvino/README.md) | [OpenvinoBackend.cpp#L419](../../executorch/backends/openvino/runtime/OpenvinoBackend.cpp#L419) | Intel CPU / iGPU / dGPU / NPU | |
|
||||
| Apple CoreML — [backends/apple/coreml](../../executorch/backends/apple/coreml/README.md) | [coreml_backend_delegate.mm#L438](../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm#L438) | iOS/macOS ANE/GPU/CPU (compute units via compile spec) | |
|
||||
| Apple Metal — [backends/apple/metal](../../executorch/backends/apple/metal/README.md) | [metal_backend.cpp#L696](../../executorch/backends/apple/metal/runtime/metal_backend.cpp#L696) | AOTI-driven Metal | "EXPERIMENTAL" |
|
||||
| Apple Core AI — [backends/apple/coreai](../../executorch/backends/apple/coreai/README.md) | Python only at this HEAD | | "Under construction — not for use" |
|
||||
| MLX — [backends/mlx](../../executorch/backends/mlx/README.md) | [MLXBackend.cpp#L570](../../executorch/backends/mlx/runtime/MLXBackend.cpp#L570) | Apple Silicon GPU, macOS ≥ 14, iOS ≥ 17 experimental ([mlx-overview.md](../../executorch/docs/source/backends/mlx/mlx-overview.md)) | experimental |
|
||||
| CUDA — [backends/cuda](../../executorch/backends/cuda/cuda_backend.py) | [cuda_backend.cpp#L1429](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L1429) | NVIDIA Linux/Windows via AOTInductor + Triton; ROCm experimental ([rocm.md](../../executorch/backends/cuda/rocm.md)) | |
|
||||
| AOTI common — [backends/aoti](../../executorch/backends/aoti/README.md) | library, not a backend | shared shims for CUDA/Metal | |
|
||||
| native — [backends/native](../../executorch/backends/native/preprocess.py) | `NativeBackend` serializes an fx subgraph to a generic flatbuffer executed by [backends/native/runtime/Program.h](../../executorch/backends/native/runtime/Program.h) | portable | new |
|
||||
| example/test — [backends/example](../../executorch/backends/example/README.md), [backends/test](../../executorch/backends/test/README.md) | reference delegate + test harness | | |
|
||||
|
||||
Reference table also in [.claude/backends.md](../../executorch/.claude/backends.md) and [docs/source/backends-overview.md](../../executorch/docs/source/backends-overview.md).
|
||||
|
||||
### 2.2 Device discovery / selection [Verified]
|
||||
|
||||
- **AOT**: the user chooses the target by passing a `Partitioner` to `to_backend()`/`to_edge_transform_and_lower()`. Multiple partitioners are applied in priority order; unmatched ops stay as `KernelCall`s for the CPU kernel library ([.claude/backends.md](../../executorch/.claude/backends.md)).
|
||||
- **Runtime**: `BackendDelegate::Init` ([method.cpp#L61](../../executorch/runtime/executor/method.cpp#L61)) does `get_backend_class(backend_id)` then `backend->is_available()` and fails with `NotFound` if either check fails. There is no fallback to another backend at runtime and no enumeration of physical devices. Backend-specific selection (e.g., CoreML compute units, QNN SoC/HTP arch, CUDA graph enable) is carried in `CompileSpec`s embedded in the `.pte` and parsed in `init` (e.g., [coreml_backend_delegate.mm#L278](../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm#L278), `kEnableCudaGraphForMethod` in [cuda_backend.cpp#L85](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L85)).
|
||||
- Runtime knobs can be pushed to a backend by name with `set_option(backend_name, Span<BackendOption>)` ([interface.h](../../executorch/runtime/backend/interface.h)); a `BackendOptionsMap` can be passed at `load_method` time.
|
||||
|
||||
### 2.3 Memory allocation / pools [Verified]
|
||||
|
||||
- `MemoryManager(method_allocator, planned_memory, temp_allocator)` ([memory_manager.h](../../executorch/runtime/executor/memory_manager.h)); method and temp allocator must differ. `planned_memory` is a `HierarchicalAllocator` over `Span<Span<uint8_t>>` — one span per `mem_id`, hard cap `kSpanArraySize = 16` for the deprecated constructor; `get_offset_address(memory_id, offset, size)` with overflow checks ([hierarchical_allocator.h](../../executorch/runtime/core/hierarchical_allocator.h)).
|
||||
- Sizes come from `ExecutionPlan.non_const_buffer_sizes` ([program.fbs#L368](../../executorch/schema/program.fbs#L368)); the app must allocate those arenas. `Method::resolve_operator` uses temp allocator (falls back to method allocator) for `TensorMeta` scratch ([method.cpp#L731](../../executorch/runtime/executor/method.cpp#L731)).
|
||||
- Temp allocator is reset after each instruction and at the start of `execute()` ([method.cpp#L1697](../../executorch/runtime/executor/method.cpp#L1697)) — kernels get per-op scratch.
|
||||
- Planning algorithms: `greedy` (default, allows overlapping lifetimes), `naive`, `MemoryPlanningAlgorithmSuite` picks the minimum total; `banked_memory_planning.TargetMemoryMap` adds capacity-aware placement across memory banks with spill ([banked_memory_planning.py](../../executorch/exir/banked_memory_planning.py)). Mutable buffers (KV caches) are placeholders that are "never freed" ([memory_planning.py#L492](../../executorch/exir/memory_planning.py#L492)) and are planned only if `alloc_mutable_buffers`.
|
||||
- Inputs/outputs: if not memory-planned, `Method::set_input` aliases user memory (`share_tensor_data`) instead of copying; otherwise `copy_tensor_data` after `resize_tensor` ([method.cpp#L1146](../../executorch/runtime/executor/method.cpp#L1146)); `set_output_data_ptr` lets the caller supply output buffers.
|
||||
|
||||
### 2.4 Host ↔ device transfers [Verified]
|
||||
|
||||
- `Device{type: CPU|CUDA, index}` ([device.h](../../executorch/runtime/core/portable_type/device.h)); `DeviceAllocator` has `allocate/deallocate/copy_host_to_device/copy_device_to_host`; `DeviceAllocatorRegistry` holds one allocator per device type; `DeviceMemoryBuffer::create(size, type)` RAII ([device_allocator.h](../../executorch/runtime/core/device_allocator.h), [device_memory_buffer.h](../../executorch/runtime/core/device_memory_buffer.h)).
|
||||
- `ExecutionPlan.non_const_buffer_device: [NonConstBufferDevice]` marks planned arenas on CUDA ([program.fbs#L417](../../executorch/schema/program.fbs#L417)); `MemoryManager::has_device_memory()` exposes this.
|
||||
- At export, `ExecutorchBackendConfig.enable_non_cpu_memory_planning=True` (default) partitions specs by device (`_partition_specs_by_device`) and `PropagateDevicePass` inserts `_h2d_copy`/`_d2h_copy` ops at delegate boundaries ([_config.py#L126](../../executorch/exir/capture/_config.py#L126), [propagate_device_pass.py](../../executorch/exir/passes/propagate_device_pass.py), [_device_copy_ops_registry.py#L27](../../executorch/exir/passes/_device_copy_ops_registry.py#L27)).
|
||||
- CUDA backend auto-registers `CudaAllocator` ([cuda_backend.cpp#L1431](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L1431), [cuda_allocator.h](../../executorch/backends/cuda/runtime/cuda_allocator.h)) and uses `cudaMemcpyAsync` device-to-device for CUDA-graph static I/O staging ([cuda_backend.cpp#L574](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L574)).
|
||||
- Other delegates copy internally (XNNPACK: `xnn_setup_runtime_v2` on external tensors; Vulkan: `maybe_resize_input`/staging in [VulkanBackend.cpp#L529](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp#L529); QNN: `RegisterMem`/`PreRegisterMem` shared buffers and rpcmem in [QnnManager.h#L89](../../executorch/backends/qualcomm/runtime/QnnManager.h#L89), [rpc_mem.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h)).
|
||||
|
||||
### 2.5 Synchronization / streams [Verified]
|
||||
|
||||
- Core runtime is synchronous: every `KernelCall`/`DelegateCall` returns before the next instruction. No stream abstraction in `runtime/`.
|
||||
- CUDA backend: optional CUDA-graph capture per method (`CudaGraphPhase::{Warmup,Replay,Disabled}`, `kCudaGraphWarmupSteps`, `cudaGraphLaunch` on a stream — [cuda_backend.cpp#L434](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L434)); enable with compile-spec `enable_cuda_graph_for_method`. Unavailable on ROCm ([rocm.md](../../executorch/backends/cuda/rocm.md)).
|
||||
- Vulkan: `ComputeGraph::submit_current_cmd[_and_wait]`, `submit_deferred_cmds_and_wait`, `prepack`, `optional_warmup_execute`, `propagate_resize` ([ComputeGraph.h#L1103](../../executorch/backends/vulkan/runtime/graph/ComputeGraph.h#L1103)).
|
||||
- MLX: `async_eval` with keep-alive of input data ([MLXBackend.cpp#L94](../../executorch/backends/mlx/runtime/MLXBackend.cpp#L94)).
|
||||
|
||||
### 2.6 Portability [Verified]
|
||||
|
||||
- PAL functions are weak symbols (`ET_INTERNAL_PLATFORM_WEAKNESS`) or a `PalImpl` table ([platform.h](../../executorch/runtime/platform/platform.h)); defaults in `runtime/platform/default`. Examples of custom PALs: Arduino ([examples/arduino/arduino_pal.cpp](../../executorch/examples/arduino/arduino_pal.cpp)), Zephyr ([zephyr/](../../executorch/zephyr)), CUDA ([backends/cuda/runtime/platform/platform.cpp](../../executorch/backends/cuda/runtime/platform/platform.cpp)).
|
||||
- Build presets ([CMakePresets.json](../../executorch/CMakePresets.json)): `android-arm64-v8a`, `android-x86_64`, `macos`, `ios`, `ios-simulator`, `linux`, `windows`, `pybind`, `zephyr`, `arm-baremetal`, `arm-ethosu-linux`, `esp-baremetal`, `riscv64-linux`, `llm`, `llm-release[-cuda|-rocm|-metal]`, `llm-debug[-cuda|-metal|-vulkan]`, `mlx[-release|-debug]`, `profiling`.
|
||||
- Host OS for building: Linux x86_64, macOS x86_64/ARM64, Windows 10+ (VS2022 + Clang-CL) or WSL ([using-executorch-building-from-source.md](../../executorch/docs/source/using-executorch-building-from-source.md)).
|
||||
|
||||
---
|
||||
|
||||
## 3. Kernel layer
|
||||
|
||||
### 3.1 Operator set & registration [Verified]
|
||||
|
||||
- Op definitions are YAML (`- op:`/`- func:` entries): portable 209 ([functions.yaml](../../executorch/kernels/portable/functions.yaml)) + 2 custom ([custom_ops.yaml](../../executorch/kernels/portable/custom_ops.yaml)); optimized 23 ([optimized.yaml](../../executorch/kernels/optimized/optimized.yaml) — `op_add/bmm/div/elu/exp/fft_*/gelu/grid_sampler_2d/le/linear/log_softmax/mm/mul/native_layer_norm/sub/sum/where`); quantized 19 ([quantized.yaml](../../executorch/kernels/quantized/quantized.yaml)); ATen wrapper 225 ([kernels/aten/functions.yaml](../../executorch/kernels/aten/functions.yaml)); prim ops (`et_copy_index`, `et_view`, sym ops) in [kernels/prim_ops/register_prim_ops.cpp](../../executorch/kernels/prim_ops/register_prim_ops.cpp).
|
||||
- Codegen ([codegen/gen.py](../../executorch/codegen/gen.py), [codegen/templates](../../executorch/codegen/templates)) emits `Kernel{name, KernelKey, OpFunction}` arrays registered via `register_kernels(Span<const Kernel>)` ([operator_registry.h#L179](../../executorch/runtime/kernel/operator_registry.h#L179)). `KernelKey` encodes dtype/dim-order per tensor arg (`kKernelKeyBufSize = 659`); a null key is the fallback kernel.
|
||||
- Table capacity: `MAX_KERNEL_NUM` → `EXECUTORCH_SELECTED_MAX_KERNEL_NUM` (generated by selective build) → default `250 ops × 8 kernels` ([operator_registry.cpp#L30](../../executorch/runtime/kernel/operator_registry.cpp#L30)). Duplicate registration aborts (see [.claude/faq.md](../../executorch/.claude/faq.md)).
|
||||
- Portable kernels must use ExecuTorch types (`executorch::aten::Tensor`, no `at::Tensor`) — [kernels/README.md](../../executorch/kernels/README.md).
|
||||
|
||||
### 3.2 Dispatch [Verified]
|
||||
|
||||
- At `Method::init`, each `KernelCall` instruction is resolved once: `resolve_operator` builds `TensorMeta` for the args and calls `get_op_function_from_registry(name, meta_list)`; the resulting `OpFunction` pointer is stored in `chain.kernels_[instr]` ([method.cpp#L731](../../executorch/runtime/executor/method.cpp#L731), [#L1040](../../executorch/runtime/executor/method.cpp#L1040)).
|
||||
- At `execute_instruction`, `KernelCall` invokes `chain.kernels_[idx](context, args)` with a `KernelRuntimeContext(event_tracer, temp_allocator)`; `DelegateCall` invokes `delegates_[idx].Execute(BackendExecutionContext, args)`; `JumpFalseCall`, `MoveCall`, `FreeCall` implement control flow, aliasing and explicit frees ([method.cpp#L1450](../../executorch/runtime/executor/method.cpp#L1450)).
|
||||
- Routing to a backend is therefore decided *entirely at export*: an op either became part of a delegate blob (`executorch_call_delegate` → `DelegateCall`) or stays a CPU `KernelCall`. There is no runtime per-op dispatcher across backends.
|
||||
- When the `Method` is loaded with an explicit `kernel_registry` span, resolution uses that list instead of the global table ([method.h](../../executorch/runtime/executor/method.h) `Span<const Kernel> kernel_registry_`).
|
||||
|
||||
### 3.3 Quantization formats & where dequant happens [Verified]
|
||||
|
||||
- **LLM source transforms** ([quantize.py](../../executorch/examples/models/llama/source_transformation/quantize.py)): `int8` (weight-only), `8da4w`/`8da8w` (int8 dynamic act + int4/int8 weight, torchao `Int8DynamicActivationIntxWeightConfig`), `4w`, `torchao:8da*` and `torchao:fpa*` low-bit kernels for Arm, embedding quant `4,32` etc. Config in [llm_config.py](../../executorch/extension/llm/export/config/llm_config.py) (`QuantizationConfig.qmode`, `pt2e_quantize`, `use_qat`, `use_spin_quant`, `preq_mode` values `8da4w` and `8da4w_output_8da8w`). SpinQuant R1/R2 rotations: [apply_spin_quant_r1_r2.py](../../executorch/examples/models/llama/source_transformation/apply_spin_quant_r1_r2.py).
|
||||
- **KV-cache quant**: `QuantizedKVCache` (int8 per-token, `QuantizedCacheType`), `StaticQuantizedKVCache` with calibration, `QuantizedRingKVCache`, `CustomKVCacheWithAttentionSink` ([custom_kv_cache.py](../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py)); `QuantizedSDPA` consumes int8 caches ([sdpa.py#L104](../../executorch/examples/models/llama/source_transformation/sdpa.py#L104)). Dequant happens inside `custom_quantized_sdpa`/XNNPACK kernels at runtime.
|
||||
- **PT2E quantizers** ([quantizer_lib.py](../../executorch/extension/llm/export/quantizer_lib.py)): XNNPACK ([xnnpack_quantizer.py](../../executorch/backends/xnnpack/quantizer/xnnpack_quantizer.py)), QNN (`get_qnn_quantizer`, 8a8w/16a4w/16a8w — see wiki [recipes.md](../../executorch/.wiki/quantization/recipes.md)), CoreML, Vulkan, OpenVINO, TOSA/Ethos-U/VGF, Cortex-M `CortexMQuantizer`. Q/DQ nodes are fused into the delegate blob by the backend's preprocess; ops that stay on CPU run `quantized_decomposed::*` kernels from [kernels/quantized](../../executorch/kernels/quantized/quantized.yaml) (see `quant_fusion_pass.py` in [exir/passes](../../executorch/exir/passes/quant_fusion_pass.py)).
|
||||
- **GPU**: CUDA int4 tile-packed (`tile_packed_to_4d`, `_weight_int4pack_mm`) plus int5/int6/int8 planar GEMM shims ([backends/cuda/runtime/shims](../../executorch/backends/cuda/runtime/shims/int4mm.cuh), [quantize_op_dispatch](../../executorch/backends/cuda/quantize_op_dispatch/int4_dispatch.py)); MLX INT2/4/8 weight quant via torchao ([mlx-overview.md](../../executorch/docs/source/backends/mlx/mlx-overview.md)); WebGPU 4-bit weight-only, 8da4w, q8ta ([webgpu README](../../executorch/backends/webgpu/README.md)). Also `extension/llm/export/{int4,mx,nvfp4,gguf}.py` for GGUF loading, MX and NVFP4 formats.
|
||||
|
||||
### 3.4 Compilation / JIT / graph capture [Verified]
|
||||
|
||||
- Everything is AOT: `torch.export` → `to_edge` (ATen→Edge dialect, decompositions, dim-order) → partition/preprocess → `to_executorch` (passes incl. `ConstraintBasedSymShapeEvalPass`, `reinplace`, `MemoryPlanningPass`, `ToOutVarPass`) → emit. No JIT at runtime.
|
||||
- Delegate compilers: XNNPACK builds an `xnn_subgraph` at `init` (`xnn_create_runtime_v4` with pthreadpool, weights cache, workspace — [XNNCompiler.cpp#L2296](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp#L2296)); Vulkan builds `ComputeGraph` from a flatbuffer, prepacks and optionally warms up ([VulkanBackend.cpp#L676](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp#L676)); QNN loads precompiled context binaries (or online-prepare) via `QnnManager::Compile` ([QnnManager.h#L85](../../executorch/backends/qualcomm/runtime/QnnManager.h#L85)); CoreML compiles/caches `.mlmodelc` at first load; CUDA `dlopen`s the AOTInductor `.so` from the `.ptd` and calls `AOTInductorModelContainerCreateWithDevice/Run` ([cuda_backend.cpp#L164](../../executorch/backends/cuda/runtime/cuda_backend.cpp#L164)); MLX interprets its own flatbuffer bytecode ([backends/mlx/serialization](../../executorch/backends/mlx/serialization)).
|
||||
|
||||
### 3.5 Fallback path [Verified]
|
||||
|
||||
- Unsupported ops simply remain `KernelCall`s to the portable/optimized CPU kernels ([backends-overview.md](../../executorch/docs/source/backends-overview.md): "Operators not supported by the delegate are executed using the portable CPU fallback"). If the CPU kernel is also missing (selective build), `Method::init` fails with `OperatorMissing` (0x14) ([.claude/faq.md](../../executorch/.claude/faq.md)). Cortex-M's dialect explicitly relies on portable-ops fallback ([cortex_m/README.md](../../executorch/backends/cortex_m/README.md)).
|
||||
- `extension/llm/custom_ops/op_fallback.cpp` provides an AOT-side fallback op ([op_fallback.py](../../executorch/extension/llm/custom_ops/op_fallback.py)).
|
||||
|
||||
### 3.6 Extension points [Verified]
|
||||
|
||||
- **Custom op**: define via `torch.library` (see [custom_ops.py](../../executorch/extension/llm/custom_ops/custom_ops.py)), implement an `out`-variant C++ kernel using ExecuTorch tensors, register with `EXECUTORCH_LIBRARY` / YAML + codegen (`gen_selected_ops`, [Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake)); docs [kernel-library-custom-aten-kernel.md](../../executorch/docs/source/kernel-library-custom-aten-kernel.md), [examples/portable/custom_ops](../../executorch/examples/portable/custom_ops).
|
||||
- **Custom backend**: Python `BackendDetails.preprocess(edge_program, compile_specs) -> PreprocessResult` + a `Partitioner` returning `PartitionResult` ([backend_details.py#L53](../../executorch/exir/backend/backend_details.py#L53), [partitioner.py#L37](../../executorch/exir/backend/partitioner.py#L37)); C++ `BackendInterface` subclass + `static auto _ = register_backend({"Id", &instance})`; template in [docs/source/backends/template](../../executorch/docs/source/backends/template) and the [backends/example](../../executorch/backends/example/README.md) reference delegate. Multi-method delegates via `preprocess_multimethod` and `MethodProgramsPartitionerSpec` ([backend_api.py#L664](../../executorch/exir/backend/backend_api.py#L664)).
|
||||
- **Custom memory planning**: pass your own `MemoryPlanningPass`/algorithm list or custom `mem_id` pools ([compiler-memory-planning.md](../../executorch/docs/source/compiler-memory-planning.md)).
|
||||
- **Custom passes**: `EdgeProgramManager.transform(passes)` and `ExecutorchBackendConfig.passes` ([compiler-custom-compiler-passes.md](../../executorch/docs/source/compiler-custom-compiler-passes.md)).
|
||||
|
||||
---
|
||||
|
||||
## 4. Model runner
|
||||
|
||||
### 4.1 Loading / export / conversion pipeline [Verified]
|
||||
|
||||
- Formats: `.pte` (FlatBuffer program with inline or segmented constants and delegate blobs; extended header in [schema/extended_header.h](../../executorch/schema/extended_header.h); [pte-file-format.md](../../executorch/docs/source/pte-file-format.md)); `.ptd` external tensor files (`extension/flat_tensor`, `FlatTensorDataMap`; [ptd-file-format.md](../../executorch/docs/source/ptd-file-format.md)) used for external constants (`ExecutorchBackendConfig.external_constants`), CUDA AOTI blobs (`aoti_cuda_blob.ptd`), Whisper weights, LoRA adapters.
|
||||
- Export API: `to_edge_transform_and_lower(ep, partitioner=[...], transform_passes=..., compile_config=EdgeCompileConfig)` then `.to_executorch(ExecutorchBackendConfig)` and `.buffer`/`.save()` ([_program.py#L1280](../../executorch/exir/program/_program.py#L1280)). Also `executorch.export` recipe API ([export/](../../executorch/export)).
|
||||
- LLM export: `python -m executorch.extension.llm.export.export_llm` (Hydra/YAML `LlmConfig`: `base`, `model`, `export`, `quantization`, `backend`, `debug`) → [export_llm.py](../../executorch/extension/llm/export/export_llm.py) → [export_llama_lib.py](../../executorch/examples/models/llama/export_llama_lib.py) → `LLMEdgeManager` in [builder.py](../../executorch/extension/llm/export/builder.py). Model classes in `ModelType` (stories110m, llama2/3/3_1/3_2, llama3_2_vision, static_llama, qwen2_5*, qwen3*, qwen3_5*, phi_4_mini, smollm2, lfm2*). HF models via *optimum-executorch* (out of tree; [export-llm-optimum.md](../../executorch/docs/source/llm/export-llm-optimum.md)). QNN LLMs use a separate path [examples/qualcomm/oss_scripts/llama/llama.py](../../executorch/examples/qualcomm/oss_scripts/llama/llama.py) (wiki warns against `export_llama --qnn`, [model-specific.md](../../executorch/.wiki/export/model-specific.md)).
|
||||
- Loading: `DataLoader` variants (`FileDataLoader`, `MmapDataLoader` with mlock/madvise, `BufferDataLoader`, `FileDescriptorDataLoader` — [extension/data_loader](../../executorch/extension/data_loader/mmap_data_loader.h)); `Module::LoadMode {File, Mmap, MmapUseMlock, MmapUseMlockIgnoreErrors, MmapUseMadvise}` ([module.h#L50](../../executorch/extension/module/module.h#L50)); LLM runner default `MmapUseMlockIgnoreErrors` ([llm_runner_helper.h#L106](../../executorch/extension/llm/runner/llm_runner_helper.h#L106)).
|
||||
|
||||
### 4.2 Execution lifecycle [Verified]
|
||||
|
||||
1. `Program::load(loader, Verification)` → header check, optional flatbuffer verification ([program.h#L97](../../executorch/runtime/executor/program.h#L97)).
|
||||
2. `Program::load_method(name, MemoryManager*, EventTracer*, NamedDataMap*)` → `Method::init`: `parse_values` (materialize `values_` table, tensors pointing into planned arenas / constant segments / external `.ptd` data), allocate and `BackendDelegate::Init` each delegate (calls `backend->init` → `DelegateHandle*`), `resolve_operator` for every `KernelCall` ([method.cpp#L883](../../executorch/runtime/executor/method.cpp#L883)).
|
||||
3. `Method::set_input(s)` (resize + copy or alias) → `Method::execute()` (checks all inputs set, resets temp allocator, sequentially runs all chains/instructions, guards against >`ET_MAX_INSTRUCTIONS`=10 M, then `reset_execution`) → `Method::get_outputs` (shallow copies) ([method.cpp#L1697](../../executorch/runtime/executor/method.cpp#L1697)).
|
||||
4. Alternative: `Method::step()` executes one instruction (experimental; used by devtools/debuggers), `reset_execution()` after `EndOfMethod`.
|
||||
5. **Warmup**: no core warmup phase; `TextLLMRunner::warmup(prompt, max_new_tokens)` runs a throwaway generation ([text_llm_runner.h#L146](../../executorch/extension/llm/runner/text_llm_runner.h#L146)); Vulkan `optional_warmup_execute`; CUDA-graph warmup steps.
|
||||
6. Multiple `Method`s per program (e.g., `encoder`, `text_decoder`, `prefill`, `decode`, `vision_encoder`, `token_embedding`, TTS `duration_predictor/text_encoder/vector_estimator/vocoder`) are loaded independently, each with its own planned memory; a global `MergedDataMap` lets them share `.ptd` constants.
|
||||
|
||||
### 4.3 Dynamic shapes [Verified]
|
||||
|
||||
- Export with `torch.export.Dim` dynamic shapes; `ConstraintBasedSymShapeEvalPass` (default in `ExecutorchBackendConfig.sym_shape_eval_pass`) converts symbolic sizes to upper bounds for memory planning (`DynamicMemoryPlanningMode.UPPER_BOUND` default; `SYMBOLIC` exists) ([sym_shape_eval_pass.py#L257](../../executorch/exir/passes/sym_shape_eval_pass.py#L257), [dynamic_shape.py](../../executorch/exir/dynamic_shape.py), [_config.py#L56](../../executorch/exir/capture/_config.py#L56)).
|
||||
- Runtime tensors carry `TensorShapeDynamism {STATIC, DYNAMIC_BOUND, DYNAMIC_UNBOUND}` ([tensor_shape_dynamism.h](../../executorch/runtime/core/tensor_shape_dynamism.h)); `set_input` calls `resize_tensor` then copies; `DYNAMIC_UNBOUND` is not memory-planned and must be provided by the caller.
|
||||
- Delegates handle dynamic shapes themselves: XNNPACK `xnn_reshape_external_value` + `xnn_reshape_runtime` + `resize_outputs` per call ([XNNExecutor.cpp#L150](../../executorch/backends/xnnpack/runtime/XNNExecutor.cpp#L150)); Vulkan `maybe_resize_input`/`propagate_resize`; MLX advertises dynamic shape support; QNN graphs are static (wiki known-issues). LLM examples default to static (`enable_dynamic_shape=False` in [model_args.py#L102](../../executorch/examples/models/llama/model_args.py#L102)); the LLM runner reads `enable_dynamic_shape` metadata ([constants.h](../../executorch/extension/llm/runner/constants.h)) and otherwise pads prefill to fixed length.
|
||||
|
||||
### 4.4 State & KV cache [Verified]
|
||||
|
||||
- **In-graph mutable buffers**: `KVCache` registers `k_cache/v_cache` buffers of shape `[max_batch, n_heads, max_context_len, head_dim]` and updates with `index_copy_`/slice writes ([attention.py#L99](../../executorch/examples/models/llama/attention.py#L99)); `CustomKVCache` uses `llama::update_cache` + `llama::custom_sdpa` ([custom_kv_cache.py#L579](../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py#L579), [op_update_cache.cpp](../../executorch/extension/llm/custom_ops/op_update_cache.cpp), [op_sdpa.cpp](../../executorch/extension/llm/custom_ops/op_sdpa.cpp)). Ring buffer (`CustomRingKVCache`, `QuantizedRingKVCache`) and attention-sink variants exist; `ReplaceKVCache...` transforms are selected by `ModelConfig` flags (`use_kv_cache`, `use_sdpa_with_kv_cache`, `quantize_kv_cache`, `use_attention_sink`, `use_ring_kv_cache`).
|
||||
- The cache lives in the method's planned non-const memory (mutable buffer, never freed) and persists across `execute()` calls; the runner tracks `pos_` and passes `start_pos` as an input tensor. `reset()` zeroes position; there is **no paging, eviction or cross-request sharing** in the core runner — one `Method` = one sequence. `max_context_len` vs `max_seq_len` (prefill chunk) are metadata methods (`get_max_context_len`, `get_max_seq_len`).
|
||||
- **Static attention (NPU-friendly)**: `StaticKVCache`/`StaticAttention`/`StaticAttentionIOManager` in Python ([static_attention.py](../../executorch/examples/models/llama/static_attention.py)) and C++ ([static_attention_io_manager.h](../../executorch/examples/models/llama/runner/static_attention_io_manager.h)) keep caches *outside* the graph as method inputs/outputs; update styles `SHIFT_POINTER` and `SMART_MASK` (fixed I/O pointers enabling persistent AP↔NPU mapping; supports per-layer cache lengths for sliding-window layers, batch 1, lookahead n-gram cache). Used by the QNN runner ([kv_manager.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/kv_manager.h), [rpc_mem.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h)).
|
||||
- **Off-graph cache library** (new, experimental): `extension/llm/cache` defines `CacheBase`, `SequenceControl::rewind`, `SequencePlanner::plan/commit`, `LayoutPolicy` (`FlatPolicy`, `RingPolicy`), `SequenceCache`, and a paged-like `CellCache` (cells claimed per position with owner bitsets, `kMaxSeqs = 64`, `free_cells()`), reached from the graph through the functional custom op `kvcache::update_and_attend` keyed by `layer_id` ([cache.h](../../executorch/extension/llm/cache/cache.h), [cell_cache.h](../../executorch/extension/llm/cache/cell_cache.h), [sequence_cache.h](../../executorch/extension/llm/cache/sequence_cache.h), [update_and_attend.py](../../executorch/extension/llm/cache/update_and_attend.py)). MLX supports per-session mutable state (`mlx_mutable_state.h`, `skip_mutable_buffer_init` — [MLXBackend.cpp#L298](../../executorch/backends/mlx/runtime/MLXBackend.cpp#L298)); CUDA has `cuda_mutable_state.h` ([backends/cuda/runtime/cuda_mutable_state.h](../../executorch/backends/cuda/runtime/cuda_mutable_state.h)). **[Inferred]** these are the substrate for the multi-session `LLMEngine` implementations (Muse Glimmer, Qwen3.5-MoE, Gemma4-31B engines).
|
||||
|
||||
### 4.5 Multimodal stages [Verified]
|
||||
|
||||
- `MultimodalRunner` (`IRunner`) takes `std::vector<MultimodalInput>` (text / tokens / `Image` / `Audio` (mel) / `RawAudio`); `MultimodalPrefiller` runs `vision_encoder` or `audio_encoder`, `token_embedding`, then `text_decoder` for prefill; decode continues with `MultimodalDecoderRunner`/`TextTokenGenerator` ([multimodal_runner.h](../../executorch/extension/llm/runner/multimodal_runner.h), [multimodal_prefiller.cpp#L112](../../executorch/extension/llm/runner/multimodal_prefiller.cpp#L112), [multimodal_input.h](../../executorch/extension/llm/runner/multimodal_input.h)).
|
||||
- Encoders/preprocessors as separate methods or `.pte`s: LLaVA (`export_image_encoder`, CLIP preprocessing in Python; [export_llava.py#L138](../../executorch/examples/models/llava/export_llava.py#L138)); Llama 3.2 Vision (exportable preprocess + C++ preprocess in [llama3_2_vision/preprocess](../../executorch/examples/models/llama3_2_vision/preprocess/preprocess.h)); Whisper mel `preprocessor.pte` ([extension/audio/mel_spectrogram.py](../../executorch/extension/audio/mel_spectrogram.py)); Voxtral Realtime `preprocessor.pte` ([voxtral_realtime README](../../executorch/examples/models/voxtral_realtime/README.md)).
|
||||
- Decoders/vocoders: Voxtral TTS `codec_decoder.pte` (Conv1d/ConvTranspose1d + 8 transformer layers → 24 kHz) driven by `voxtral_tts_runner` with a 7-step Euler flow head; Supertonic `vocoder` method; Mimi decoder on QNN (`qnn_mimi_decoder_runner.cpp`).
|
||||
- Diffusion: `examples/models/stable_diffusion` and `stable_diffusion_3_5_large` are model definitions/export scripts only (no runner); DiT on QNN ([dit.py](../../executorch/examples/qualcomm/oss_scripts/dit.py)).
|
||||
|
||||
---
|
||||
|
||||
## 5. Scheduler
|
||||
|
||||
### 5(a) Request / token / pipeline-level scheduling
|
||||
|
||||
- **Core runtime**: **N/A** [Verified]. One `Method` executes one request synchronously; `execute()` refuses re-entry while `in_progress()`; there is no queue, no batching across requests, no admission control. Batch size is baked into the exported shapes (LLM examples: batch 1).
|
||||
- **LLM runner (`extension/llm/runner`)** [Verified]: sequential prefill → decode per call; `GenerationConfig` (`max_new_tokens`, `seq_len`, `temperature`, `echo`, `warming`, `grammar`) ([irunner.h](../../executorch/extension/llm/runner/irunner.h)); chunked prefill in `TextPrefiller::prefill` when prompt > `max_seq_len` ([text_llm_runner.cpp#L143](../../executorch/extension/llm/runner/text_llm_runner.cpp#L143)); optional two-method PTE (`prefill` + `decode`) ([constants.h#L28](../../executorch/extension/llm/runner/constants.h#L28)). Cancellation: `IRunner::stop()` sets an atomic checked per token in `TextTokenGenerator::generate` ([text_token_generator.h#L131](../../executorch/extension/llm/runner/text_token_generator.h#L131)); `LLMSession::stop()` is documented as a token-boundary cooperative stop ([llm_session.h#L104](../../executorch/extension/llm/runner/llm_session.h#L104)). No backpressure concept (callbacks are synchronous on the calling thread).
|
||||
- **`extension/llm/batching` (experimental)** [Verified]: `Scheduler` interface (`submit/has_work/get_work/cancel(sid)/clear/max_prefill_chunk_size`), `DecodeFirstScheduler::create(max_batch_tokens=544, max_decode_sequences=32, max_prefill_chunk_size=256)` takes all pending decodes first then fills the token budget with round-robin prefill chunks (continuous batching with chunked prefill; admission rejects over-budget tasks) ([decode_first_scheduler.h](../../executorch/extension/llm/batching/decode_first_scheduler.h)); `Executor` (`open_session/close_session/set_sampling/execute(BatchInput, BatchOutput)`); `Runner(executor, scheduler)` with `open_session_async`, `Session::generate_async(delta, GenConfig, callback)` returning `GenerationHandle{cancel, done, wait, finish_reason}` and `FinishReason {StopToken, NewTokenLimit, Cancelled, Failed}` ([runner.h](../../executorch/extension/llm/batching/runner.h)). **Gap [Verified]**: no in-tree `Executor` implementation outside `extension/llm/batching/test`; no batched `.pte` example wires into it yet.
|
||||
- **`examples/llm_server`** [Verified]: OpenAI-compatible `/v1/chat/completions` (streaming + non-streaming), one worker process with *serialized execution*, multiple isolated sessions on one weight load when the engine reports `LLMServingCapacity > 1`, bounded request cancellation, structured 400s for unsupported params (`n>1`, penalties, logprobs...) ([examples/llm_server/README.md](../../executorch/examples/llm_server/README.md), [worker_loop.h](../../executorch/examples/llm_server/cpp/worker_loop.h), [worker_prefill_plan.h](../../executorch/examples/llm_server/cpp/worker_prefill_plan.h)). `LLMEngine` implementations: [muse_glimmer_engine.h](../../executorch/examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.h) (with DFlash speculative decoding), [qwen35_moe_engine.h](../../executorch/examples/models/qwen3_5_moe/qwen35_moe_engine.h), [gemma4_31b_engine.h](../../executorch/examples/models/gemma4_31b/gemma4_31b_engine.h).
|
||||
- **Multi-stage pipelines** (TTS/ASR): hand-written in each example runner (e.g., `VoxtralTtsRunner::synthesize_streaming` LM → flow head → codec chunks with `streaming_chunk_frames_=25`, `streaming_initial_chunk_=5`, `streaming_left_context_=25` — [voxtral_tts_runner.h#L121](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h#L121)); QNN has a Python `genai_pipeline` with model-preparation / quantization / compilation / inference stages ([genai_pipeline.py](../../executorch/backends/qualcomm/genai_pipeline/genai_pipeline.py)).
|
||||
|
||||
### 5(b) Graph / operator / thread-level scheduling
|
||||
|
||||
- **Op ordering** [Verified]: fixed at export (topological order emitted into `Chain.instructions`); executed strictly sequentially by `Method::execute` — "Chains are executed sequentially today, but future async designs may branch and run many in parallel" ([method.cpp#L1722](../../executorch/runtime/executor/method.cpp#L1722)). No inter-op parallelism, no stream assignment in the core.
|
||||
- **Intra-op threading** [Verified]: `extension/threadpool` wraps `pthreadpool` and sizes the pool from `cpuinfo` performant cores (`get_num_performant_cores`, [cpuinfo_utils.cpp](../../executorch/extension/threadpool/cpuinfo_utils.cpp)); `ThreadPool::run(fn, range)` is blocking `pthreadpool_parallelize_1d`; `_unsafe_reset_threadpool(n)` changes size ([threadpool.cpp#L65](../../executorch/extension/threadpool/threadpool.cpp#L65)); kernels use `parallel_for(begin,end,grain,fn)` which degrades to a serial loop without `ET_USE_THREADPOOL` ([thread_parallel_interface.h#L53](../../executorch/runtime/kernel/thread_parallel_interface.h#L53)). XNNPACK shares the same `get_pthreadpool()` ([XNNCompiler.cpp#L2300](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp#L2300)). `NoThreadPoolGuard` prevents nested parallelism.
|
||||
- **Delegate-internal scheduling** [Verified]: Vulkan builds command buffers per graph and submits/waits (`submit_current_cmd_and_wait`); CUDA optionally replays a captured CUDA graph; QNN executes a named graph on HTP with optional multi-graph context; XNNPACK runs the subgraph on the pthreadpool.
|
||||
- **Batching** [Verified]: static batch dim only; `HierarchicalAllocator` sizes are per-method so batch changes require re-export.
|
||||
|
||||
---
|
||||
|
||||
## 6. I/O layer
|
||||
|
||||
- **Text tokenization** [Verified via docs, submodule absent]: `extension/llm/tokenizers` (pytorch-labs tokenizers, submodule commit `1d7ca636`) provides C++ `HFTokenizer` (tokenizer.json), `Tiktoken`, `SentencePiece`, `Llama2c`, `Tekken` plus Python `pytorch_tokenizers.get_tokenizer` ([.claude/tokenizers.md](../../executorch/.claude/tokenizers.md)); `llm::load_tokenizer(path, special_tokens)` auto-detects ([llm_runner_helper.h#L47](../../executorch/extension/llm/runner/llm_runner_helper.h#L47)); llama runner tries Tiktoken → SentencePiece → Llama2c ([runner.cpp](../../executorch/examples/models/llama/runner/runner.cpp)). WASM tokenizers bindings in [extension/wasm/tokenizers](../../executorch/extension/wasm/tokenizers).
|
||||
- **Image preprocessing** [Verified]: C++ `ImageProcessor::process/process_yuv/process_into` (resize + normalize, NV12/NV21, SIMD and Apple GPU variants) in [extension/image/image_processor.h](../../executorch/extension/image/image_processor.h); Swift `ExecuTorch+ImageProcessor.swift`; JNI `jni_layer_image.cpp`; examples use `stb_image` ([gemma3/e2e_runner.cpp](../../executorch/examples/models/gemma3/e2e_runner.cpp)) and `llm::Image` ([image.h](../../executorch/extension/llm/runner/image.h)); custom op `tile_crop` ([op_tile_crop.cpp](../../executorch/extension/llm/custom_ops/op_tile_crop.cpp)).
|
||||
- **Audio** [Verified]: `WhisperAudioProcessor` (torch.stft + mel filterbank, exportable to `whisper_preprocess.pte`, 16 kHz/80 bins/30 s defaults) in [extension/audio/mel_spectrogram.py](../../executorch/extension/audio/mel_spectrogram.py); `RawAudio{data, batch, n_channels, n_samples}` and `Audio` (mel, float/bf16) types ([audio.h](../../executorch/extension/llm/runner/audio.h)); `wav_loader.h`; ASR runners `Seq2SeqRunner` (Whisper-style `encoder` + `text_decoder`) and `TransducerRunner` (Parakeet TDT) in [extension/asr/runner](../../executorch/extension/asr/runner/seq2seq_runner.h); Silero VAD streaming runner ([stream_main.cpp](../../executorch/examples/models/silero_vad/stream_main.cpp)).
|
||||
- **Streaming** [Verified]: token-level `token_callback(const std::string&)` per generated token and `stats_callback(const Stats&)` (`first_token_ms`, `prompt_eval_end_ms`, `num_generated_tokens`, `gpu_total_bytes` — [stats.h](../../executorch/extension/llm/runner/stats.h)); audio-chunk `AudioChunkCallback` in Voxtral TTS; Voxtral Realtime processes 80 ms chunks live; Supertonic persistent mode emits JSONL results per request. HTTP SSE streaming in `examples/llm_server`.
|
||||
- **Buffering** [Verified]: planned arenas allocated by the app; `Module` allocates them internally; `set_output_data_ptr` for zero-copy outputs; XNNPACK weights cache for `.ptd` weights ([XNNCompiler.cpp#L2283](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp#L2283)).
|
||||
- **Postprocessing** [Verified]: `Sampler` (temperature, top-k `sample_topk`, argmax) + `LogitProcessor` chain (grammar hooks) ([sampler.h](../../executorch/extension/llm/sampler/sampler.h), [logit_processor.h](../../executorch/extension/llm/sampler/logit_processor.h)); tokenizer `decode(prev, cur)`; WAV writers in TTS examples ([wav_writer.h](../../executorch/examples/models/voxtral_tts/wav_writer.h)).
|
||||
- **Application boundaries** [Verified]:
|
||||
- C++: `runtime/` core API, `extension/module::Module`, `extension/tensor` (`TensorPtr`), `extension/runner_util`, LLM/ASR runners, `executor_runner` CLI ([examples/portable/executor_runner/executor_runner.cpp](../../executorch/examples/portable/executor_runner/executor_runner.cpp)).
|
||||
- Python: `executorch.runtime.Runtime/Program/Method` and `portable_lib._load_for_executorch` ([.claude/runtime-api.md](../../executorch/.claude/runtime-api.md), [extension/pybindings/pybindings.cpp](../../executorch/extension/pybindings/pybindings.cpp)); LLM runner pybindings ([extension/llm/runner/pybindings.cpp](../../executorch/extension/llm/runner/pybindings.cpp)).
|
||||
- Android: Kotlin `Module`, `Tensor`, `EValue`, `ExecuTorchRuntime`, `BackendOptionsMap` + JNI layers for runtime/llama/asr/image/training ([extension/android](../../executorch/extension/android/README.md), [jni_layer_llama.cpp](../../executorch/extension/android/jni/jni_layer_llama.cpp)); shipped as AAR on Maven Central ([using-executorch-android.md](../../executorch/docs/source/using-executorch-android.md)).
|
||||
- Apple: Objective-C/Swift `ExecuTorchModule/Tensor/Value/ImageProcessor/BackendOptions` ([extension/apple/ExecuTorch/Exported](../../executorch/extension/apple/ExecuTorch/Exported/ExecuTorchModule.h)) and `ExecuTorchLLM` ([extension/llm/apple](../../executorch/extension/llm/apple)); xcframeworks via SwiftPM ([using-executorch-ios.md](../../executorch/docs/source/using-executorch-ios.md)).
|
||||
- WASM: [extension/wasm/wasm_bindings.cpp](../../executorch/extension/wasm/wasm_bindings.cpp).
|
||||
- HTTP: `examples/llm_server` (Python control plane; C++ single-binary server listed as future).
|
||||
- Demo apps in tree: only [examples/demo-apps/react-native/rnllama](../../executorch/examples/demo-apps/react-native) (native Android/iOS demo apps have moved out of this tree; docs still reference them).
|
||||
|
||||
---
|
||||
|
||||
## 7. Execution flow
|
||||
|
||||
### 7(a) Text LLM decode (TextLLMRunner over XNNPACK-delegated `.pte`) [Verified]
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App
|
||||
participant R as TextLLMRunner
|
||||
participant Tok as Tokenizer
|
||||
participant P as TextPrefiller
|
||||
participant G as TextTokenGenerator
|
||||
participant D as TextDecoderRunner
|
||||
participant M as Module/Method
|
||||
participant K as Kernels / XNNPACK delegate
|
||||
participant S as Sampler
|
||||
App->>R: create_text_llm_runner(pte, tokenizer, data_files)
|
||||
App->>R: load()
|
||||
R->>M: load_method("forward") → Method::init (parse_values, BackendDelegate::Init → XNNPACKBackend::init builds xnn runtime, resolve_operator)
|
||||
App->>R: generate(prompt, GenerationConfig, token_cb, stats_cb)
|
||||
R->>Tok: encode(prompt, bos, eos)
|
||||
R->>P: prefill(tokens, pos_) (chunked ≤ max_seq_len)
|
||||
P->>D: step(tokens_tensor, start_pos)
|
||||
D->>M: IOManager::prepare_prefill → execute("forward", {tokens, start_pos})
|
||||
M->>K: Method::execute: KernelCall/DelegateCall sequence (update_cache, custom_sdpa, xnn subgraph)
|
||||
K-->>M: logits
|
||||
M-->>D: logits tensor
|
||||
D->>S: logits_to_token(temperature)
|
||||
S-->>P: first token
|
||||
P-->>R: first token, pos_ += n
|
||||
R->>G: generate(first token, start_pos, max_new_tokens, temperature, token_cb)
|
||||
loop until EOS / max_new_tokens / stop()
|
||||
G->>D: step({cur_token}, pos)
|
||||
D->>M: execute("forward"|"decode")
|
||||
M->>K: instructions (KV cache mutable buffer updated in planned memory)
|
||||
K-->>D: logits
|
||||
D->>S: sample → cur_token
|
||||
G->>Tok: decode(prev, cur)
|
||||
G-->>App: token_cb(text)
|
||||
end
|
||||
G-->>R: n_generated
|
||||
R-->>App: stats_cb(Stats: TTFT, tok/s ...)
|
||||
```
|
||||
|
||||
Source: [text_llm_runner.cpp](../../executorch/extension/llm/runner/text_llm_runner.cpp), [text_prefiller.h](../../executorch/extension/llm/runner/text_prefiller.h), [text_token_generator.h#L83](../../executorch/extension/llm/runner/text_token_generator.h#L83), [text_decoder_runner.cpp#L40](../../executorch/extension/llm/runner/text_decoder_runner.cpp#L40), [io_manager.h](../../executorch/extension/llm/runner/io_manager/io_manager.h), [method.cpp#L1697](../../executorch/runtime/executor/method.cpp#L1697).
|
||||
|
||||
### 7(b) TTS: Voxtral TTS streaming synthesis (CPU or CUDA) [Verified structure; per-method details Inferred]
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App
|
||||
participant TR as VoxtralTtsRunner
|
||||
participant Tok as Tekken tokenizer
|
||||
participant LM as model.pte (5 methods: LM + flow head)
|
||||
participant CD as codec_decoder.pte
|
||||
participant BE as XNNPACK / CUDA(AOTI) delegate
|
||||
App->>TR: ctor(model_path, codec_path, tokenizer, voice embedding, [aoti .ptd])
|
||||
TR->>LM: Module::load_method(...) for LM + flow-head methods
|
||||
TR->>CD: Module::load_method("forward")
|
||||
App->>TR: synthesize_streaming(text, AudioChunkCallback)
|
||||
TR->>Tok: encode(text)
|
||||
TR->>LM: prefill(text tokens + voice embedding)
|
||||
loop per audio frame (autoregressive)
|
||||
TR->>LM: decode step → hidden state
|
||||
TR->>LM: flow head: 7-step Euler ODE (x0 from xorshift64 rng) → 37 codebook tokens
|
||||
LM->>BE: DelegateCall (AOTI container run / xnn runtime)
|
||||
end
|
||||
Note over TR: after streaming_initial_chunk_ (5 frames) then every streaming_chunk_frames_ (25) with left context 25
|
||||
TR->>CD: execute(codebook tokens window) → PCM 24 kHz
|
||||
CD->>BE: DelegateCall
|
||||
TR-->>App: AudioChunkCallback(float PCM chunk)
|
||||
TR-->>App: final WAV via wav_writer
|
||||
```
|
||||
|
||||
Source: [voxtral_tts_runner.h](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h), [voxtral_tts_runner.cpp](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.cpp), [README](../../executorch/examples/models/voxtral_tts/README.md). Exact method names inside `model.pte` were not enumerated here [Unknown].
|
||||
|
||||
Multimodal VLM path (for VLA-style vision+text) is the `MultimodalRunner` sequence: `vision_encoder(image)` → `token_embedding(tokens)` → `text_decoder(embeds, start_pos)` prefill → token loop ([multimodal_prefiller.cpp](../../executorch/extension/llm/runner/multimodal_prefiller.cpp)).
|
||||
|
||||
---
|
||||
|
||||
## 8. Tests and examples
|
||||
|
||||
- **Python tests** [Verified]: `pytest -n auto` from repo root ([CLAUDE.md](../../executorch/CLAUDE.md)); [pytest.ini](../../executorch/pytest.ini) ignores `backends/arm/**` (needs `examples/arm/setup.sh`), some XNNPACK op tests, `backends/test` (WIP harness), LLaVA tests, backend-specific Muse Glimmer pipelines. Tests live next to code: `exir/tests`, `exir/backend/test`, `backends/*/test`, `extension/llm/export/test`, `examples/models/llama/tests` (incl. `test_static_attention.py`), `runtime/test/test_runtime.py`.
|
||||
- **C++ tests** [Verified]: gtest via CMake `EXECUTORCH_BUILD_TESTS=ON` then `ctest --output-on-failure`; [Test.cmake](../../executorch/Test.cmake) enumerates `runtime/*/test`, `kernels/test`, `extension/*/test`; [test/run_oss_cpp_tests.sh](../../executorch/test/run_oss_cpp_tests.sh); size tests [test/build_size_test.sh](../../executorch/test/build_size_test.sh) and [test/size_test.cpp](../../executorch/test/size_test.cpp); end-to-end [test/end2end](../../executorch/test/end2end).
|
||||
- **Backend op-coverage harness**: [backends/test](../../executorch/backends/test/README.md) (`suite`, `harness`, FACTO), `multi_method_delegate_test.cpp`.
|
||||
- **CI** [Verified]: ~60 workflows in [.github/workflows](../../executorch/.github/workflows) — `pull.yml`, `trunk`/`periodic.yml`, `nightly.yml`, `_unittest.yml`, `_android.yml`, `apple.yml`, `cuda.yml`, `cuda-windows.yml`, `cuda-perf.yml`, `rocm.yml`, `metal.yml`, `mlx.yml`, `riscv64.yml`, `qnn-windows-msvc.yml`, `test-backend-{arm,coreml,cortex-m,nxp,openvino}.yml`, `_test_cadence.yml`, `_xtensa_*.yml`, `_llm_server.yml`, `_test_arduino_library.yml`, wheel builds for linux/macos/windows/aarch64/cuda.
|
||||
- **Example apps / runners** [Verified]: LLM: [examples/models/llama](../../executorch/examples/models/llama/README.md) (`main.cpp`, `runner/`, `export_llama.py`, eval, Android perf table), `examples/models/{qwen3,phi-3-mini,gemma3,gemma4,smollm2,lfm2,deepseek-r1-distill-llama-8B,...}`; VLM: llava, llama3_2_vision, gemma3, smolvlm, internvl3, muse_glimmer; audio: whisper, voxtral, voxtral_realtime, parakeet, silero_vad, sortformer, granite_speech, emformer_rnnt, wav2letter; TTS: voxtral_tts, supertonic; vision: mobilenet_v2/v3, resnet, deit, dinov2, efficient_sam, yolo12/26, torchvision_vit, edsr; diffusion: stable_diffusion, stable_diffusion_3_5_large. Platform examples: [examples/qualcomm](../../executorch/examples/qualcomm/README.md) (40+ oss_scripts incl. llama, whisper, t5, gemma4, moshi/mimi, DiT, EfficientSAM), [examples/apple/coreml](../../executorch/examples/apple/coreml/README.md), [examples/arm](../../executorch/examples/arm/README.md) (Ethos-U/VGF/Cortex-M notebooks: smollm2, silero_vad, tinystories, mobilesam), [examples/mediatek](../../executorch/examples/mediatek/README.md), [examples/samsung](../../executorch/examples/samsung), [examples/nxp](../../executorch/examples/nxp), [examples/openvino](../../executorch/examples/openvino), [examples/vulkan](../../executorch/examples/vulkan), [examples/xnnpack](../../executorch/examples/xnnpack/README.md), [examples/cuda](../../executorch/examples/cuda/README.md) (ROCm pointwise), [examples/arduino](../../executorch/examples/arduino/README.md), [examples/espressif](../../executorch/examples/espressif/README.md), [examples/raspberry_pi/pico2](../../executorch/examples/raspberry_pi/pico2), [examples/riscv](../../executorch/examples/riscv/README.md), [examples/zephyr](../../executorch/examples/zephyr), [examples/wasm](../../executorch/examples/wasm), [examples/llm_pte_finetuning](../../executorch/examples/llm_pte_finetuning), [examples/demo-apps/react-native](../../executorch/examples/demo-apps/react-native).
|
||||
- **Make targets** [Verified]: [Makefile](../../executorch/Makefile) + per-example `CMakePresets.json` (`make whisper-cpu|whisper-cuda|whisper-metal`, `make silero-vad-cpu`, `make sortformer-cpu`, `make supertonic-mlx`).
|
||||
|
||||
---
|
||||
|
||||
## 9. TTS relevance
|
||||
|
||||
**Exists** [Verified]:
|
||||
- **Voxtral-4B-TTS** end-to-end: export ([export_voxtral_tts.py](../../executorch/examples/models/voxtral_tts/export_voxtral_tts.py)) producing `model.pte` (Mistral 4B decoder + 3-layer flow-matching head, 5 methods) and `codec_decoder.pte`; backends `portable`, `xnnpack`, `cuda`, `cuda-windows`; quant `--qlinear 4w|8w|8da4w|8da8w`, `--qlinear-codec`, `--qembedding`; `--streaming` chunk metadata; C++ runner with `synthesize_streaming` + `AudioChunkCallback`, xorshift RNG for flow noise, WAV writer. Repo-reported (not reproduced): CUDA 4w RTF 0.31× on RTX 5080 with ~2.6 s time-to-first-audio, 3.4 GB `.ptd`; FP32 CUDA RTF 51× (codec on portable CPU) ([README](../../executorch/examples/models/voxtral_tts/README.md)).
|
||||
- **Supertonic 3 on MLX**: single dynamic FP16 `.pte` with methods `duration_predictor`, `text_encoder`, `vector_estimator` (5 flow steps), `vocoder`; one-shot and persistent JSONL server mode; repo-reported RTF ≈ 0.028 on Apple Silicon ([README](../../executorch/examples/models/supertonic/README.md), [supertonic_runner.cpp](../../executorch/examples/models/supertonic/runtime/supertonic_runner.cpp), [model/vocoder.py](../../executorch/examples/models/supertonic/model/vocoder.py)). macOS/Apple-Silicon only.
|
||||
- **Neural audio codecs**: Mimi (Moshi) tests ([examples/models/moshi/mimi/test_mimi.py](../../executorch/examples/models/moshi/mimi/test_mimi.py)) and a QNN Mimi decoder runner ([qnn_mimi_decoder_runner.cpp](../../executorch/examples/qualcomm/oss_scripts/moshi/qnn_mimi_decoder_runner.cpp)); Voxtral codec decoder.
|
||||
- **Audio ops**: portable/optimized FFT (`op_fft_r2c/c2r` in [kernels/optimized/cpu](../../executorch/kernels/optimized/cpu/op_fft_r2c.cpp), pocketfft third-party), Conv1d→Conv2d transform ([backends/transforms/convert_conv1d_to_conv2d_pass.py](../../executorch/backends/transforms/convert_conv1d_to_conv2d_pass.py)) for XNNPACK; `ConvTranspose1d` handled in Voxtral codec on CPU/CUDA; mel front end exportable.
|
||||
- **Streaming audio output**: chunked codec decoding with left context (Voxtral TTS); real-time 80 ms chunk ASR (Voxtral Realtime) shows the runtime can sustain streaming loops; VAD streaming runner.
|
||||
- **Text front end**: Supertonic `text_processor.cpp`; no general G2P/phonemizer library.
|
||||
- **NXP** README explicitly lists TTS as a Neutron NPU use-case ([backends/nxp/README.md](../../executorch/backends/nxp/README.md)) [Verified statement, no TTS example for NXP found].
|
||||
|
||||
**Missing / gaps** [Verified unless noted]:
|
||||
- No generic TTS runner abstraction (unlike `extension/llm/runner` or `extension/asr/runner`); each TTS model has a bespoke C++ runner.
|
||||
- No mobile (Android/iOS/QNN/CoreML) TTS example; Supertonic is MLX-only, Voxtral TTS is CPU/CUDA. [Inferred] the XNNPACK path is fp32 (`--dtype fp32` default) and 4B params — heavy for phones.
|
||||
- No vocoder zoo (HiFi-GAN, BigVGAN, WaveRNN, etc.) or codec encoders (EnCodec/SNAC/DAC) in tree; `stable_diffusion` has no audio variant.
|
||||
- Flow-matching ODE loops are unrolled by the runner (7 Euler steps as repeated method calls), not expressed as a graph; no CFG batching helper.
|
||||
- No audio-specific streaming buffer/jitter management; callbacks are synchronous.
|
||||
- Sampling for audio tokens (multi-codebook, 37 codebooks per frame) is handled inside the example, not by `Sampler`.
|
||||
|
||||
---
|
||||
|
||||
## 10. VLA relevance
|
||||
|
||||
**Exists** [Verified]:
|
||||
- **Vision encoders**: CLIP (LLaVA), Llama 3.2 Vision encoder + cross-attention decoder, SigLIP-style Gemma3/SmolVLM/InternVL3 via optimum-executorch converters ([examples/models/smolvlm/convert_weights.py](../../executorch/examples/models/smolvlm/convert_weights.py)), DINOv2, ViT, DeiT, EfficientSAM, YOLO, MobileNet; `MultimodalRunner` handles image → embeddings → decoder; C++ `ImageProcessor` for resize/normalize/YUV.
|
||||
- **Structured inputs**: methods accept arbitrary tensor tuples (`Method::set_input`), so proprioceptive vectors are trivially additional inputs [Inferred: no example does this].
|
||||
- **Low-latency loops**: static memory planning, no allocation in steady state, `set_output_data_ptr` zero-copy outputs, CUDA-graph replay for fixed-shape methods, Vulkan/QNN prepacked graphs, `Method::execute` overhead is a flat instruction walk — good fit for fixed-rate control loops. Static attention IO manager demonstrates fixed-pointer I/O for NPUs.
|
||||
- **Diffusion/flow machinery**: SD/SD3.5 model definitions, DiT on QNN, Voxtral/Supertonic flow-matching heads driven by runner-side Euler loops; `MoveCall`/`JumpFalseCall` allow `torch.cond`/`while` control flow in a single method (`get_while_nodes`, `get_scan_nodes` in memory planning) — [Inferred] a fixed-step denoising loop could be exported as `torch.ops.higher_order.while_loop` but no example does so.
|
||||
- **Action heads**: nothing VLA-specific; an MLP/transformer action head is just another method.
|
||||
|
||||
**Missing** [Verified]:
|
||||
- No VLA models (OpenVLA, π0, ACT, Diffusion Policy, SmolVLA, Octo), no LeRobot integration, no "robot" mentions in docs/examples/wiki.
|
||||
- No multi-method orchestration primitive for encoder + policy + action-decoder pipelines beyond writing a C++ runner; no shared-memory/DMA-BUF camera input path (image bytes must be copied into an ExecuTorch tensor).
|
||||
- No batched multi-camera encoder execution beyond exporting a batched shape.
|
||||
- Jetson/embedded-CUDA deployment is undocumented (see §11).
|
||||
|
||||
**[Proposal]** For an edge VLA on ExecuTorch: export `vision_encoder` (per camera, batched), `proprio_proj`, `text/token_embedding`, `policy_decoder` (static KV, `SMART_MASK` style) and `action_head` (flow/diffusion step as a method called N times) into one `.pte` with shared `.ptd` weights; drive with a small C++ runner modelled on `MultimodalPrefiller` + `VoxtralTtsRunner`; pin planned arenas once and reuse `Method` objects across control ticks.
|
||||
|
||||
---
|
||||
|
||||
## 11. Edge deployment profile
|
||||
|
||||
- **Binary size** [Verified, repo-reported]: docs state the core library is "around 50 kB with no operators/kernels or delegates" ([kernel-library-selective-build.md](../../executorch/docs/source/kernel-library-selective-build.md)); CI thresholds in [pull.yml](../../executorch/.github/workflows/pull.yml#L555): stripped `size_test` ≤ 52,168 B (gcc9) / ≤ 45,000 B (clang); bare-metal Arm `size_test` ≤ 118,000 B (GCC 15.2 baseline 115,868 B) and Zephyr preset ≤ 136,020 B. Size tooling: `EXECUTORCH_OPTIMIZE_SIZE` (`-Os -fno-exceptions -fno-rtti`), `ET_LOG_ENABLED=0`, bloaty ([.claude/skills/binary-size/SKILL.md](../../executorch/.claude/skills/binary-size/SKILL.md), [devtools/size_analysis_tool](../../executorch/devtools/size_analysis_tool)).
|
||||
- **Selective build** [Verified]: `EXECUTORCH_SELECT_OPS_YAML|LIST|MODEL`, `EXECUTORCH_ENABLE_DTYPE_SELECTIVE_BUILD`, `gen_selected_ops(...)` macro; generates `selected_max_kernel_num.h` ([tools/cmake/Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake), [examples/selective_build](../../executorch/examples/selective_build)).
|
||||
- **Dependencies** [Verified]: core needs only C++17 and flatbuffers headers (`third-party/flatbuffers`); optional pthreadpool + cpuinfo (`EXECUTORCH_BUILD_PTHREADPOOL`, `EXECUTORCH_BUILD_CPUINFO`), XNNPACK, Eigen (optimized BLAS), gflags for runners, flatcc for ETDump, torchao (`third-party/ao`) for quant kernels, tokenizers submodule for LLM; C++ `c10` headers vendored (`check-c10-sync.yml`). Python side pins PyTorch via [torch_pin.py](../../executorch/torch_pin.py).
|
||||
- **Build system** [Verified]: CMake (`CMakeLists.txt`, presets in [tools/cmake/preset](../../executorch/tools/cmake/preset/default.cmake) with `EXECUTORCH_BUILD_{XNNPACK,VULKAN,QNN,COREML,METAL,MLX,CUDA,ROCM,NEURON,ENN,NXP_NEUTRON,OPENVINO,CADENCE,CORTEX_M,VGF,ARM_BAREMETAL,ARM_ETHOSU_LINUX,WEBGPU,WASM,...}`, `EXECUTORCH_BUILD_EXTENSION_{MODULE,TENSOR,DATA_LOADER,FLAT_TENSOR,LLM,LLM_RUNNER,ASR_RUNNER,IMAGE,TRAINING,APPLE,...}`, `EXECUTORCH_BUILD_KERNELS_{OPTIMIZED,QUANTIZED,LLM}`); Buck2 (`BUCK`/`targets.bzl`) for Meta-internal; Gradle for AAR; `Package.swift` for SwiftPM; Arduino library builder; `pip install executorch` wheels for Linux/macOS/Windows (+ CUDA and aarch64-CUDA wheels).
|
||||
- **Platform matrix** [Verified from backend docs/dirs]:
|
||||
- **Embedded NVIDIA / Jetson**: [Unknown] — CUDA backend documented only for "Linux or Windows" x86-class hosts; aarch64 Linux CUDA wheels are built in CI, which [Inferred] would cover Jetson/GB10-class aarch64 if the AOTI `.so` is compiled for the target SM, but no doc, example or CI run targets Jetson.
|
||||
- **ARM CPU**: XNNPACK (ARM64/ARMv7, KleidiAI i8mm/dotprod kernels by default on ARM), torchao low-bit kernels (`torchao:8da4w`), `riscv64-linux` preset also exists; Raspberry Pi tutorial ([raspberry_pi_llama_tutorial.md](../../executorch/docs/source/raspberry_pi_llama_tutorial.md)).
|
||||
- **Cortex-M / MCU**: `arm-baremetal`, `zephyr`, `esp-baremetal` presets; Cortex-M CMSIS-NN dialect; Ethos-U55/65/85 delegate; Arduino library; Pico2 example; Alif/Zephyr tutorial ([zephyr_alif_tutorial.md](../../executorch/docs/source/zephyr_alif_tutorial.md)).
|
||||
- **Apple**: CoreML (ANE/GPU/CPU, iOS/macOS), MLX (Apple-Silicon GPU, macOS ≥ 14, iOS ≥ 17 experimental), Metal (experimental AOTI), XNNPACK; distributed as xcframeworks `executorch, executorch_llm, backend_coreml, backend_mlx, backend_xnnpack, kernels_llm/optimized/quantized/torchao`.
|
||||
- **Android**: XNNPACK, Vulkan (Adreno/Mali; Maven artifact `executorch-android-vulkan`), QNN HTP (Snapdragon; 38 chipset ids incl. automotive SA8295/SA8797), MediaTek Neuron (D9300/D9400), Samsung ENN (Exynos 2500), Arm VGF (Vulkan ML). NNAPI: not present in tree. OpenCL: not present (Vulkan only).
|
||||
- **Intel**: OpenVINO CPU/iGPU/dGPU/NPU.
|
||||
- **Browser**: WebGPU + WASM bindings.
|
||||
- **Model conversion constraints** [Verified]: must be `torch.export`-able (no data-dependent Python control flow except `torch.cond`/`while_loop`; static or bounded dynamic shapes); one `.pte` per backend; QNN/Ethos-U/CoreML prefer static shapes and quantized graphs; Ethos-U requires Vela-compilable int8 TOSA; dim-order/channels-last pitfalls documented in [.wiki/export/common-pitfalls.md](../../executorch/.wiki/export/common-pitfalls.md); CUDA export needs `EdgeCompileConfig(_check_ir_validity=False, _skip_dim_order=True)` ([cuda-overview.md](../../executorch/docs/source/backends/cuda/cuda-overview.md)); PTE BC guarantee is ≥ one following minor release ([runtime/COMPATIBILITY.md](../../executorch/runtime/COMPATIBILITY.md)).
|
||||
- **Repo-reported perf (not reproduced)**: Llama 3.2 1B SpinQuant on OnePlus 12 (XNNPACK+KleidiAI): 50.2 tok/s decode, 0.3 s TTFT, 1,083 MiB PTE, 1,921 MiB RSS; 3B SpinQuant 19.7 tok/s ([llama/README.md#L70](../../executorch/examples/models/llama/README.md#L70)); WebGPU Llama 3.2 1B 4-bit on M4 Pro 188 tok/s decode @128 ctx ([webgpu README](../../executorch/backends/webgpu/README.md)).
|
||||
|
||||
---
|
||||
|
||||
## 12. Limitations and unknowns
|
||||
|
||||
- [Verified] Single-method, synchronous, sequential execution; no request-level batching in the core; `extension/llm/batching` has no production `Executor` implementation in tree.
|
||||
- [Verified] Global static registries (backends ≤ 16, kernels ≤ 2000 by default) with an explicit TODO that multiple `Executor` instances are not supported in the long term.
|
||||
- [Verified] Backend availability is checked only at delegate init; no runtime fallback if a delegate is unavailable — a `.pte` is tied to its backend.
|
||||
- [Verified] KV cache is a static, pre-sized mutable buffer per method; no paging/eviction/prefix sharing in the core (experimental `CellCache`/`SequenceCache` exist but are wired only through custom ops and engine examples).
|
||||
- [Verified] Dynamic shapes are upper-bound planned; `DYNAMIC_UNBOUND` outputs need caller-provided memory; several NPU delegates (QNN, Ethos-U) are static-shape only per wiki.
|
||||
- [Verified] No VLA/policy models, no robotics integration, no general TTS runner, no vocoder/codec library; TTS examples are CUDA/MLX-centric.
|
||||
- [Verified] Tokenizers, XNNPACK, pthreadpool, cpuinfo, MLX, Vulkan headers, flatbuffers, torchao are uninitialized submodules in this checkout — their internals (e.g., tokenizer implementations, XNNPACK version, KleidiAI integration details) could not be inspected.
|
||||
- [Unknown] Jetson/embedded CUDA support status; whether AOTI-compiled `.so` blobs are portable across SM versions and aarch64.
|
||||
- [Unknown] Exact method set and shapes inside Voxtral TTS `model.pte`; exact QNN LLM multi-graph memory layout; Metal and Core AI backend maturity.
|
||||
- [Unknown] `examples/llm_server` C++ single-binary server ("future"); Android/iOS native demo apps are referenced by docs but not present in tree (only react-native).
|
||||
- [Inferred] Cross-method sharing of a KV cache (e.g., encoder→decoder cross-attn cache, `update_cross_attn_cache` op) is supported by custom ops but memory planning treats each method independently, so shared state must be threaded as inputs/outputs or held off-graph.
|
||||
- [Inferred] Thread-count control is global (`_unsafe_reset_threadpool`), so co-scheduling two models (e.g., VLM + TTS) on CPU has no per-model thread budget.
|
||||
|
||||
---
|
||||
|
||||
## 13. Reference index
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| [../../executorch/CLAUDE.md](../../executorch/CLAUDE.md) | Repo agent guidance, quick build/test commands |
|
||||
| [../../executorch/.claude/backends.md](../../executorch/.claude/backends.md) | Backend table + partitioner imports |
|
||||
| [../../executorch/.claude/runtime-api.md](../../executorch/.claude/runtime-api.md) | Python runtime API summary |
|
||||
| [../../executorch/.claude/quantization.md](../../executorch/.claude/quantization.md) | Quantizer list |
|
||||
| [../../executorch/.claude/llm-export.md](../../executorch/.claude/llm-export.md) | LLM export config summary |
|
||||
| [../../executorch/.claude/tokenizers.md](../../executorch/.claude/tokenizers.md) | Tokenizer classes (submodule absent) |
|
||||
| [../../executorch/.claude/faq.md](../../executorch/.claude/faq.md) | Error codes, threadpool reset |
|
||||
| [../../executorch/.claude/skills/binary-size/SKILL.md](../../executorch/.claude/skills/binary-size/SKILL.md) | Size measurement procedure |
|
||||
| [../../executorch/.wiki/index.md](../../executorch/.wiki/index.md) | Tribal knowledge index |
|
||||
| [../../executorch/.wiki/export/model-specific.md](../../executorch/.wiki/export/model-specific.md) | LLM export codepaths, dynamic-shape limits |
|
||||
| [../../executorch/.wiki/export/common-pitfalls.md](../../executorch/.wiki/export/common-pitfalls.md) | Export pitfalls |
|
||||
| [../../executorch/.wiki/quantization/recipes.md](../../executorch/.wiki/quantization/recipes.md) | Quant recipe selection |
|
||||
| [../../executorch/version.txt](../../executorch/version.txt) | Version 1.5.0 |
|
||||
| [../../executorch/CMakePresets.json](../../executorch/CMakePresets.json) | Build presets |
|
||||
| [../../executorch/tools/cmake/preset/default.cmake](../../executorch/tools/cmake/preset/default.cmake) | `EXECUTORCH_BUILD_*` options |
|
||||
| [../../executorch/tools/cmake/Codegen.cmake](../../executorch/tools/cmake/Codegen.cmake) | Selective build macros |
|
||||
| [../../executorch/codegen/gen.py](../../executorch/codegen/gen.py) | Kernel registration codegen |
|
||||
| [../../executorch/Makefile](../../executorch/Makefile) | Runner make targets |
|
||||
| [../../executorch/Test.cmake](../../executorch/Test.cmake) | C++ test enumeration |
|
||||
| [../../executorch/pytest.ini](../../executorch/pytest.ini) | Python test config |
|
||||
| [../../executorch/test/build_size_test.sh](../../executorch/test/build_size_test.sh) | Size test build |
|
||||
| [../../executorch/test/size_test.cpp](../../executorch/test/size_test.cpp) | Minimal runtime binary |
|
||||
| [../../executorch/.github/workflows/pull.yml](../../executorch/.github/workflows/pull.yml) | CI incl. size thresholds |
|
||||
| [../../executorch/.github/workflows/build-wheels-cuda-aarch64-linux.yml](../../executorch/.github/workflows/build-wheels-cuda-aarch64-linux.yml) | aarch64 CUDA wheels |
|
||||
| [../../executorch/schema/program.fbs](../../executorch/schema/program.fbs) | PTE schema |
|
||||
| [../../executorch/schema/extended_header.h](../../executorch/schema/extended_header.h) | PTE extended header |
|
||||
| [../../executorch/runtime/COMPATIBILITY.md](../../executorch/runtime/COMPATIBILITY.md) | PTE/runtime BC policy |
|
||||
| [../../executorch/runtime/executor/program.h](../../executorch/runtime/executor/program.h) | `Program` API |
|
||||
| [../../executorch/runtime/executor/method.h](../../executorch/runtime/executor/method.h) | `Method` API |
|
||||
| [../../executorch/runtime/executor/method.cpp](../../executorch/runtime/executor/method.cpp) | `BackendDelegate`, init/execute/step |
|
||||
| [../../executorch/runtime/executor/memory_manager.h](../../executorch/runtime/executor/memory_manager.h) | `MemoryManager` |
|
||||
| [../../executorch/runtime/core/hierarchical_allocator.h](../../executorch/runtime/core/hierarchical_allocator.h) | Planned arenas |
|
||||
| [../../executorch/runtime/core/memory_allocator.h](../../executorch/runtime/core/memory_allocator.h) | Bump allocator |
|
||||
| [../../executorch/runtime/core/device_allocator.h](../../executorch/runtime/core/device_allocator.h) | `DeviceAllocator` registry |
|
||||
| [../../executorch/runtime/core/device_memory_buffer.h](../../executorch/runtime/core/device_memory_buffer.h) | RAII device buffer |
|
||||
| [../../executorch/runtime/core/portable_type/device.h](../../executorch/runtime/core/portable_type/device.h) | `Device{CPU,CUDA}` |
|
||||
| [../../executorch/runtime/core/tensor_shape_dynamism.h](../../executorch/runtime/core/tensor_shape_dynamism.h) | Shape dynamism enum |
|
||||
| [../../executorch/runtime/backend/interface.h](../../executorch/runtime/backend/interface.h) | `BackendInterface` |
|
||||
| [../../executorch/runtime/backend/interface.cpp](../../executorch/runtime/backend/interface.cpp) | Backend registry (16) |
|
||||
| [../../executorch/runtime/kernel/operator_registry.h](../../executorch/runtime/kernel/operator_registry.h) | Kernel registry API |
|
||||
| [../../executorch/runtime/kernel/operator_registry.cpp](../../executorch/runtime/kernel/operator_registry.cpp) | Kernel table limits |
|
||||
| [../../executorch/runtime/kernel/thread_parallel_interface.h](../../executorch/runtime/kernel/thread_parallel_interface.h) | `parallel_for` |
|
||||
| [../../executorch/runtime/platform/platform.h](../../executorch/runtime/platform/platform.h) | PAL |
|
||||
| [../../executorch/exir/program/_program.py](../../executorch/exir/program/_program.py) | `to_edge`, managers |
|
||||
| [../../executorch/exir/capture/_config.py](../../executorch/exir/capture/_config.py) | Compile/backend configs |
|
||||
| [../../executorch/exir/memory_planning.py](../../executorch/exir/memory_planning.py) | Planning algorithms |
|
||||
| [../../executorch/exir/banked_memory_planning.py](../../executorch/exir/banked_memory_planning.py) | Capacity-aware planning |
|
||||
| [../../executorch/exir/dynamic_shape.py](../../executorch/exir/dynamic_shape.py) | `DynamicMemoryPlanningMode` |
|
||||
| [../../executorch/exir/passes/sym_shape_eval_pass.py](../../executorch/exir/passes/sym_shape_eval_pass.py) | Upper-bound shape eval |
|
||||
| [../../executorch/exir/passes/propagate_device_pass.py](../../executorch/exir/passes/propagate_device_pass.py) | h2d/d2h insertion |
|
||||
| [../../executorch/exir/passes/_device_copy_ops_registry.py](../../executorch/exir/passes/_device_copy_ops_registry.py) | `_h2d_copy/_d2h_copy` ops |
|
||||
| [../../executorch/exir/passes/quant_fusion_pass.py](../../executorch/exir/passes/quant_fusion_pass.py) | Q/DQ fusion |
|
||||
| [../../executorch/exir/emit/_emitter.py](../../executorch/exir/emit/_emitter.py) | Emitter |
|
||||
| [../../executorch/exir/backend/backend_api.py](../../executorch/exir/backend/backend_api.py) | `to_backend` |
|
||||
| [../../executorch/exir/backend/partitioner.py](../../executorch/exir/backend/partitioner.py) | `Partitioner` |
|
||||
| [../../executorch/exir/backend/backend_details.py](../../executorch/exir/backend/backend_details.py) | `BackendDetails` |
|
||||
| [../../executorch/export](../../executorch/export) | Recipe export API |
|
||||
| [../../executorch/kernels/README.md](../../executorch/kernels/README.md) | Kernel lib layout |
|
||||
| [../../executorch/kernels/portable/functions.yaml](../../executorch/kernels/portable/functions.yaml) | Portable ops (209) |
|
||||
| [../../executorch/kernels/portable/custom_ops.yaml](../../executorch/kernels/portable/custom_ops.yaml) | Portable custom ops |
|
||||
| [../../executorch/kernels/optimized/optimized.yaml](../../executorch/kernels/optimized/optimized.yaml) | Optimized ops (23) |
|
||||
| [../../executorch/kernels/optimized/cpu/op_fft_r2c.cpp](../../executorch/kernels/optimized/cpu/op_fft_r2c.cpp) | FFT kernel |
|
||||
| [../../executorch/kernels/quantized/quantized.yaml](../../executorch/kernels/quantized/quantized.yaml) | Quantized ops (19) |
|
||||
| [../../executorch/kernels/aten/functions.yaml](../../executorch/kernels/aten/functions.yaml) | ATen mode ops |
|
||||
| [../../executorch/kernels/prim_ops/register_prim_ops.cpp](../../executorch/kernels/prim_ops/register_prim_ops.cpp) | Prim ops |
|
||||
| [../../executorch/backends/xnnpack/README.md](../../executorch/backends/xnnpack/README.md) | XNNPACK backend |
|
||||
| [../../executorch/backends/xnnpack/CMakeLists.txt](../../executorch/backends/xnnpack/CMakeLists.txt) | KleidiAI flag |
|
||||
| [../../executorch/backends/xnnpack/runtime/XNNPACKBackend.cpp](../../executorch/backends/xnnpack/runtime/XNNPACKBackend.cpp) | Registration |
|
||||
| [../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp](../../executorch/backends/xnnpack/runtime/XNNCompiler.cpp) | Subgraph → runtime |
|
||||
| [../../executorch/backends/xnnpack/runtime/XNNExecutor.cpp](../../executorch/backends/xnnpack/runtime/XNNExecutor.cpp) | Dynamic reshape/forward |
|
||||
| [../../executorch/backends/xnnpack/quantizer/xnnpack_quantizer.py](../../executorch/backends/xnnpack/quantizer/xnnpack_quantizer.py) | XNNPACK quantizer |
|
||||
| [../../executorch/backends/vulkan/README.md](../../executorch/backends/vulkan/README.md) | Vulkan backend |
|
||||
| [../../executorch/backends/vulkan/runtime/VulkanBackend.cpp](../../executorch/backends/vulkan/runtime/VulkanBackend.cpp) | Vulkan delegate |
|
||||
| [../../executorch/backends/vulkan/runtime/graph/ComputeGraph.h](../../executorch/backends/vulkan/runtime/graph/ComputeGraph.h) | Vulkan graph exec |
|
||||
| [../../executorch/backends/webgpu/README.md](../../executorch/backends/webgpu/README.md) | WebGPU backend |
|
||||
| [../../executorch/backends/webgpu/runtime/WebGPUBackend.cpp](../../executorch/backends/webgpu/runtime/WebGPUBackend.cpp) | Registration |
|
||||
| [../../executorch/backends/qualcomm/README.md](../../executorch/backends/qualcomm/README.md) | QNN backend |
|
||||
| [../../executorch/backends/qualcomm/runtime/QnnExecuTorchBackend.cpp](../../executorch/backends/qualcomm/runtime/QnnExecuTorchBackend.cpp) | Registration |
|
||||
| [../../executorch/backends/qualcomm/runtime/QnnManager.h](../../executorch/backends/qualcomm/runtime/QnnManager.h) | Graph exec, shared mem |
|
||||
| [../../executorch/backends/qualcomm/serialization/qc_schema.py](../../executorch/backends/qualcomm/serialization/qc_schema.py) | Chipset enum |
|
||||
| [../../executorch/backends/qualcomm/genai_pipeline/genai_pipeline.py](../../executorch/backends/qualcomm/genai_pipeline/genai_pipeline.py) | QNN GenAI pipeline |
|
||||
| [../../executorch/backends/mediatek/README.md](../../executorch/backends/mediatek/README.md) | MediaTek backend |
|
||||
| [../../executorch/backends/mediatek/runtime/NeuronBackend.cpp](../../executorch/backends/mediatek/runtime/NeuronBackend.cpp) | Registration |
|
||||
| [../../executorch/backends/samsung/README.md](../../executorch/backends/samsung/README.md) | Samsung backend |
|
||||
| [../../executorch/backends/samsung/runtime/enn_backend.cpp](../../executorch/backends/samsung/runtime/enn_backend.cpp) | Registration |
|
||||
| [../../executorch/backends/arm/README.md](../../executorch/backends/arm/README.md) | Arm TOSA/Ethos-U/VGF |
|
||||
| [../../executorch/backends/arm/runtime/EthosUBackend.cpp](../../executorch/backends/arm/runtime/EthosUBackend.cpp) | Ethos-U delegate |
|
||||
| [../../executorch/backends/arm/runtime/VGFBackend.cpp](../../executorch/backends/arm/runtime/VGFBackend.cpp) | VGF delegate |
|
||||
| [../../executorch/backends/cortex_m/README.md](../../executorch/backends/cortex_m/README.md) | Cortex-M dialect |
|
||||
| [../../executorch/backends/cadence/README.md](../../executorch/backends/cadence/README.md) | Cadence DSP |
|
||||
| [../../executorch/backends/nxp/README.md](../../executorch/backends/nxp/README.md) | NXP Neutron |
|
||||
| [../../executorch/backends/nxp/runtime/NeutronBackend.cpp](../../executorch/backends/nxp/runtime/NeutronBackend.cpp) | Registration |
|
||||
| [../../executorch/backends/openvino/README.md](../../executorch/backends/openvino/README.md) | OpenVINO |
|
||||
| [../../executorch/backends/openvino/runtime/OpenvinoBackend.cpp](../../executorch/backends/openvino/runtime/OpenvinoBackend.cpp) | Registration |
|
||||
| [../../executorch/backends/apple/coreml/README.md](../../executorch/backends/apple/coreml/README.md) | CoreML backend |
|
||||
| [../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm](../../executorch/backends/apple/coreml/runtime/delegate/coreml_backend_delegate.mm) | CoreML delegate |
|
||||
| [../../executorch/backends/apple/metal/README.md](../../executorch/backends/apple/metal/README.md) | Metal (experimental) |
|
||||
| [../../executorch/backends/apple/metal/runtime/metal_backend.cpp](../../executorch/backends/apple/metal/runtime/metal_backend.cpp) | Registration |
|
||||
| [../../executorch/backends/apple/coreai/README.md](../../executorch/backends/apple/coreai/README.md) | Core AI (WIP) |
|
||||
| [../../executorch/backends/mlx/README.md](../../executorch/backends/mlx/README.md) | MLX backend |
|
||||
| [../../executorch/backends/mlx/runtime/MLXBackend.cpp](../../executorch/backends/mlx/runtime/MLXBackend.cpp) | MLX runtime |
|
||||
| [../../executorch/backends/mlx/serialization](../../executorch/backends/mlx/serialization) | MLX bytecode |
|
||||
| [../../executorch/backends/cuda/cuda_backend.py](../../executorch/backends/cuda/cuda_backend.py) | CUDA AOTI export |
|
||||
| [../../executorch/backends/cuda/runtime/cuda_backend.cpp](../../executorch/backends/cuda/runtime/cuda_backend.cpp) | CUDA delegate, CUDA graphs |
|
||||
| [../../executorch/backends/cuda/runtime/cuda_allocator.h](../../executorch/backends/cuda/runtime/cuda_allocator.h) | CUDA `DeviceAllocator` |
|
||||
| [../../executorch/backends/cuda/runtime/cuda_mutable_state.h](../../executorch/backends/cuda/runtime/cuda_mutable_state.h) | Per-session state |
|
||||
| [../../executorch/backends/cuda/runtime/shims/int4mm.cuh](../../executorch/backends/cuda/runtime/shims/int4mm.cuh) | int4 GEMM |
|
||||
| [../../executorch/backends/cuda/quantize_op_dispatch/int4_dispatch.py](../../executorch/backends/cuda/quantize_op_dispatch/int4_dispatch.py) | Quant dispatch |
|
||||
| [../../executorch/backends/cuda/rocm.md](../../executorch/backends/cuda/rocm.md) | ROCm status |
|
||||
| [../../executorch/backends/aoti/README.md](../../executorch/backends/aoti/README.md) | AOTI common lib |
|
||||
| [../../executorch/backends/native/preprocess.py](../../executorch/backends/native/preprocess.py) | Native delegate |
|
||||
| [../../executorch/backends/native/runtime/Program.h](../../executorch/backends/native/runtime/Program.h) | Native runtime |
|
||||
| [../../executorch/backends/example/README.md](../../executorch/backends/example/README.md) | Example delegate |
|
||||
| [../../executorch/backends/test/README.md](../../executorch/backends/test/README.md) | Backend test harness |
|
||||
| [../../executorch/backends/transforms/convert_conv1d_to_conv2d_pass.py](../../executorch/backends/transforms/convert_conv1d_to_conv2d_pass.py) | Conv1d transform |
|
||||
| [../../executorch/extension/module/module.h](../../executorch/extension/module/module.h) | `Module` |
|
||||
| [../../executorch/extension/data_loader/mmap_data_loader.h](../../executorch/extension/data_loader/mmap_data_loader.h) | mmap loader |
|
||||
| [../../executorch/extension/flat_tensor/flat_tensor_data_map.h](../../executorch/extension/flat_tensor/flat_tensor_data_map.h) | `.ptd` loader |
|
||||
| [../../executorch/extension/threadpool/threadpool.h](../../executorch/extension/threadpool/threadpool.h) | Threadpool |
|
||||
| [../../executorch/extension/threadpool/threadpool.cpp](../../executorch/extension/threadpool/threadpool.cpp) | Impl |
|
||||
| [../../executorch/extension/threadpool/cpuinfo_utils.cpp](../../executorch/extension/threadpool/cpuinfo_utils.cpp) | Performant cores |
|
||||
| [../../executorch/extension/llm/runner/README.md](../../executorch/extension/llm/runner/README.md) | Runner framework |
|
||||
| [../../executorch/extension/llm/runner/irunner.h](../../executorch/extension/llm/runner/irunner.h) | `IRunner`, `GenerationConfig` |
|
||||
| [../../executorch/extension/llm/runner/text_llm_runner.h](../../executorch/extension/llm/runner/text_llm_runner.h) | `TextLLMRunner` |
|
||||
| [../../executorch/extension/llm/runner/text_llm_runner.cpp](../../executorch/extension/llm/runner/text_llm_runner.cpp) | generate() |
|
||||
| [../../executorch/extension/llm/runner/text_prefiller.h](../../executorch/extension/llm/runner/text_prefiller.h) | Prefill |
|
||||
| [../../executorch/extension/llm/runner/text_token_generator.h](../../executorch/extension/llm/runner/text_token_generator.h) | Decode loop |
|
||||
| [../../executorch/extension/llm/runner/text_decoder_runner.h](../../executorch/extension/llm/runner/text_decoder_runner.h) | Decoder step |
|
||||
| [../../executorch/extension/llm/runner/text_decoder_runner.cpp](../../executorch/extension/llm/runner/text_decoder_runner.cpp) | step impl |
|
||||
| [../../executorch/extension/llm/runner/io_manager/io_manager.h](../../executorch/extension/llm/runner/io_manager/io_manager.h) | `IOManager` |
|
||||
| [../../executorch/extension/llm/runner/multimodal_runner.h](../../executorch/extension/llm/runner/multimodal_runner.h) | `MultimodalRunner` |
|
||||
| [../../executorch/extension/llm/runner/multimodal_prefiller.cpp](../../executorch/extension/llm/runner/multimodal_prefiller.cpp) | Encoder invocation |
|
||||
| [../../executorch/extension/llm/runner/multimodal_input.h](../../executorch/extension/llm/runner/multimodal_input.h) | Input variant |
|
||||
| [../../executorch/extension/llm/runner/image.h](../../executorch/extension/llm/runner/image.h) | `Image` |
|
||||
| [../../executorch/extension/llm/runner/audio.h](../../executorch/extension/llm/runner/audio.h) | `Audio`, `RawAudio` |
|
||||
| [../../executorch/extension/llm/runner/constants.h](../../executorch/extension/llm/runner/constants.h) | Method name conventions |
|
||||
| [../../executorch/extension/llm/runner/llm_session.h](../../executorch/extension/llm/runner/llm_session.h) | `LLMSession`, `LLMEngine` |
|
||||
| [../../executorch/extension/llm/runner/llm_runner_helper.h](../../executorch/extension/llm/runner/llm_runner_helper.h) | Factories, tokenizer loading |
|
||||
| [../../executorch/extension/llm/runner/stats.h](../../executorch/extension/llm/runner/stats.h) | `Stats` |
|
||||
| [../../executorch/extension/llm/runner/pybindings.cpp](../../executorch/extension/llm/runner/pybindings.cpp) | Runner pybindings |
|
||||
| [../../executorch/extension/llm/sampler/sampler.h](../../executorch/extension/llm/sampler/sampler.h) | `Sampler` |
|
||||
| [../../executorch/extension/llm/sampler/logit_processor.h](../../executorch/extension/llm/sampler/logit_processor.h) | `LogitProcessor` |
|
||||
| [../../executorch/extension/llm/batching/runner.h](../../executorch/extension/llm/batching/runner.h) | Batching runner/session |
|
||||
| [../../executorch/extension/llm/batching/decode_first_scheduler.h](../../executorch/extension/llm/batching/decode_first_scheduler.h) | Scheduler impl |
|
||||
| [../../executorch/extension/llm/cache/cache.h](../../executorch/extension/llm/cache/cache.h) | Cache interfaces |
|
||||
| [../../executorch/extension/llm/cache/cell_cache.h](../../executorch/extension/llm/cache/cell_cache.h) | Cell (paged-like) cache |
|
||||
| [../../executorch/extension/llm/cache/sequence_cache.h](../../executorch/extension/llm/cache/sequence_cache.h) | Flat/Ring policies |
|
||||
| [../../executorch/extension/llm/cache/update_and_attend.py](../../executorch/extension/llm/cache/update_and_attend.py) | `kvcache::update_and_attend` |
|
||||
| [../../executorch/extension/llm/custom_ops/custom_ops.py](../../executorch/extension/llm/custom_ops/custom_ops.py) | Custom op defs |
|
||||
| [../../executorch/extension/llm/custom_ops/op_sdpa.cpp](../../executorch/extension/llm/custom_ops/op_sdpa.cpp) | SDPA kernel |
|
||||
| [../../executorch/extension/llm/custom_ops/op_update_cache.cpp](../../executorch/extension/llm/custom_ops/op_update_cache.cpp) | KV update kernel |
|
||||
| [../../executorch/extension/llm/custom_ops/op_tile_crop.cpp](../../executorch/extension/llm/custom_ops/op_tile_crop.cpp) | Image tile crop |
|
||||
| [../../executorch/extension/llm/custom_ops/op_fallback.py](../../executorch/extension/llm/custom_ops/op_fallback.py) | Fallback op |
|
||||
| [../../executorch/extension/llm/export/export_llm.py](../../executorch/extension/llm/export/export_llm.py) | LLM export CLI |
|
||||
| [../../executorch/extension/llm/export/builder.py](../../executorch/extension/llm/export/builder.py) | `LLMEdgeManager` |
|
||||
| [../../executorch/extension/llm/export/config/llm_config.py](../../executorch/extension/llm/export/config/llm_config.py) | `LlmConfig`, `ModelType` |
|
||||
| [../../executorch/extension/llm/export/partitioner_lib.py](../../executorch/extension/llm/export/partitioner_lib.py) | Backend partitioners for LLM |
|
||||
| [../../executorch/extension/llm/export/quantizer_lib.py](../../executorch/extension/llm/export/quantizer_lib.py) | PT2E quantizers |
|
||||
| [../../executorch/extension/llm/apple](../../executorch/extension/llm/apple) | `ExecuTorchLLM` Swift |
|
||||
| [../../executorch/extension/asr/runner/seq2seq_runner.h](../../executorch/extension/asr/runner/seq2seq_runner.h) | ASR seq2seq |
|
||||
| [../../executorch/extension/audio/mel_spectrogram.py](../../executorch/extension/audio/mel_spectrogram.py) | Mel front end |
|
||||
| [../../executorch/extension/image/image_processor.h](../../executorch/extension/image/image_processor.h) | Image preprocessing |
|
||||
| [../../executorch/extension/android/README.md](../../executorch/extension/android/README.md) | Android bindings |
|
||||
| [../../executorch/extension/android/jni/jni_layer_llama.cpp](../../executorch/extension/android/jni/jni_layer_llama.cpp) | LLM JNI |
|
||||
| [../../executorch/extension/apple/ExecuTorch/Exported/ExecuTorchModule.h](../../executorch/extension/apple/ExecuTorch/Exported/ExecuTorchModule.h) | Apple bindings |
|
||||
| [../../executorch/extension/pybindings/pybindings.cpp](../../executorch/extension/pybindings/pybindings.cpp) | Python bindings |
|
||||
| [../../executorch/extension/wasm/wasm_bindings.cpp](../../executorch/extension/wasm/wasm_bindings.cpp) | WASM bindings |
|
||||
| [../../executorch/extension/wasm/tokenizers](../../executorch/extension/wasm/tokenizers) | WASM tokenizers |
|
||||
| [../../executorch/devtools/etdump/etdump_flatcc.h](../../executorch/devtools/etdump/etdump_flatcc.h) | ETDump |
|
||||
| [../../executorch/devtools/inspector/_inspector.py](../../executorch/devtools/inspector/_inspector.py) | Inspector |
|
||||
| [../../executorch/devtools/bundled_program/bundled_program.h](../../executorch/devtools/bundled_program/bundled_program.h) | Bundled program |
|
||||
| [../../executorch/devtools/size_analysis_tool](../../executorch/devtools/size_analysis_tool) | Size analysis |
|
||||
| [../../executorch/examples/models/llama/README.md](../../executorch/examples/models/llama/README.md) | Llama example + perf |
|
||||
| [../../executorch/examples/models/llama/export_llama_lib.py](../../executorch/examples/models/llama/export_llama_lib.py) | Export pipeline |
|
||||
| [../../executorch/examples/models/llama/model_args.py](../../executorch/examples/models/llama/model_args.py) | `ModelArgs` |
|
||||
| [../../executorch/examples/models/llama/attention.py](../../executorch/examples/models/llama/attention.py) | `KVCache`, attention |
|
||||
| [../../executorch/examples/models/llama/static_attention.py](../../executorch/examples/models/llama/static_attention.py) | Static attention (Python) |
|
||||
| [../../executorch/examples/models/llama/runner/static_attention_io_manager.h](../../executorch/examples/models/llama/runner/static_attention_io_manager.h) | Static attention (C++) |
|
||||
| [../../executorch/examples/models/llama/runner/runner.cpp](../../executorch/examples/models/llama/runner/runner.cpp) | Llama runner factory |
|
||||
| [../../executorch/examples/models/llama/source_transformation/quantize.py](../../executorch/examples/models/llama/source_transformation/quantize.py) | Quant modes |
|
||||
| [../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py](../../executorch/examples/models/llama/source_transformation/custom_kv_cache.py) | KV cache variants |
|
||||
| [../../executorch/examples/models/llama/source_transformation/sdpa.py](../../executorch/examples/models/llama/source_transformation/sdpa.py) | SDPA variants |
|
||||
| [../../executorch/examples/models/llama/source_transformation/apply_spin_quant_r1_r2.py](../../executorch/examples/models/llama/source_transformation/apply_spin_quant_r1_r2.py) | SpinQuant |
|
||||
| [../../executorch/examples/models/llava/README.md](../../executorch/examples/models/llava/README.md) | LLaVA |
|
||||
| [../../executorch/examples/models/llava/export_llava.py](../../executorch/examples/models/llava/export_llava.py) | LLaVA export |
|
||||
| [../../executorch/examples/models/llama3_2_vision/preprocess/preprocess.h](../../executorch/examples/models/llama3_2_vision/preprocess/preprocess.h) | Vision preprocess C++ |
|
||||
| [../../executorch/examples/models/smolvlm/convert_weights.py](../../executorch/examples/models/smolvlm/convert_weights.py) | SmolVLM |
|
||||
| [../../executorch/examples/models/gemma3/e2e_runner.cpp](../../executorch/examples/models/gemma3/e2e_runner.cpp) | Gemma3 VLM runner |
|
||||
| [../../executorch/examples/models/voxtral_tts/README.md](../../executorch/examples/models/voxtral_tts/README.md) | Voxtral TTS |
|
||||
| [../../executorch/examples/models/voxtral_tts/export_voxtral_tts.py](../../executorch/examples/models/voxtral_tts/export_voxtral_tts.py) | TTS export |
|
||||
| [../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.h) | TTS runner API |
|
||||
| [../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.cpp](../../executorch/examples/models/voxtral_tts/voxtral_tts_runner.cpp) | TTS runner impl |
|
||||
| [../../executorch/examples/models/voxtral_tts/wav_writer.h](../../executorch/examples/models/voxtral_tts/wav_writer.h) | WAV output |
|
||||
| [../../executorch/examples/models/supertonic/README.md](../../executorch/examples/models/supertonic/README.md) | Supertonic TTS (MLX) |
|
||||
| [../../executorch/examples/models/supertonic/runtime/supertonic_runner.cpp](../../executorch/examples/models/supertonic/runtime/supertonic_runner.cpp) | Supertonic runner |
|
||||
| [../../executorch/examples/models/supertonic/model/vocoder.py](../../executorch/examples/models/supertonic/model/vocoder.py) | Vocoder |
|
||||
| [../../executorch/examples/models/voxtral_realtime/README.md](../../executorch/examples/models/voxtral_realtime/README.md) | Streaming ASR |
|
||||
| [../../executorch/examples/models/voxtral/README.md](../../executorch/examples/models/voxtral/README.md) | Audio LLM via optimum |
|
||||
| [../../executorch/examples/models/whisper/README.md](../../executorch/examples/models/whisper/README.md) | Whisper runner |
|
||||
| [../../executorch/examples/models/parakeet/README.md](../../executorch/examples/models/parakeet/README.md) | Parakeet TDT |
|
||||
| [../../executorch/examples/models/silero_vad/README.md](../../executorch/examples/models/silero_vad/README.md) | VAD |
|
||||
| [../../executorch/examples/models/silero_vad/stream_main.cpp](../../executorch/examples/models/silero_vad/stream_main.cpp) | Streaming VAD |
|
||||
| [../../executorch/examples/models/sortformer/README.md](../../executorch/examples/models/sortformer/README.md) | Diarization |
|
||||
| [../../executorch/examples/models/moshi/mimi/test_mimi.py](../../executorch/examples/models/moshi/mimi/test_mimi.py) | Mimi codec test |
|
||||
| [../../executorch/examples/models/muse_glimmer/README.md](../../executorch/examples/models/muse_glimmer/README.md) | 30B VLM + serving |
|
||||
| [../../executorch/examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.h](../../executorch/examples/models/muse-glimmer/runtime/engine/muse_glimmer_engine.h) | `LLMEngine` impl |
|
||||
| [../../executorch/examples/models/qwen3_5_moe/qwen35_moe_engine.h](../../executorch/examples/models/qwen3_5_moe/qwen35_moe_engine.h) | `LLMEngine` impl |
|
||||
| [../../executorch/examples/models/gemma4_31b/gemma4_31b_engine.h](../../executorch/examples/models/gemma4_31b/gemma4_31b_engine.h) | `LLMEngine` impl |
|
||||
| [../../executorch/examples/models/stable_diffusion/model.py](../../executorch/examples/models/stable_diffusion/model.py) | SD model def |
|
||||
| [../../executorch/examples/models/stable_diffusion_3_5_large/export_sd3_5_large.py](../../executorch/examples/models/stable_diffusion_3_5_large/export_sd3_5_large.py) | SD3.5 export |
|
||||
| [../../executorch/examples/llm_server/README.md](../../executorch/examples/llm_server/README.md) | OpenAI server |
|
||||
| [../../executorch/examples/llm_server/cpp/worker_loop.h](../../executorch/examples/llm_server/cpp/worker_loop.h) | Worker loop |
|
||||
| [../../executorch/examples/llm_server/cpp/worker_prefill_plan.h](../../executorch/examples/llm_server/cpp/worker_prefill_plan.h) | Prefill planning |
|
||||
| [../../executorch/examples/qualcomm/README.md](../../executorch/examples/qualcomm/README.md) | QNN examples |
|
||||
| [../../executorch/examples/qualcomm/oss_scripts/llama/llama.py](../../executorch/examples/qualcomm/oss_scripts/llama/llama.py) | QNN LLM export |
|
||||
| [../../executorch/examples/qualcomm/oss_scripts/llama/runner/kv_manager.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/kv_manager.h) | QNN KV manager |
|
||||
| [../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h](../../executorch/examples/qualcomm/oss_scripts/llama/runner/rpc_mem.h) | rpcmem shared buffers |
|
||||
| [../../executorch/examples/qualcomm/oss_scripts/moshi/qnn_mimi_decoder_runner.cpp](../../executorch/examples/qualcomm/oss_scripts/moshi/qnn_mimi_decoder_runner.cpp) | Mimi on QNN |
|
||||
| [../../executorch/examples/qualcomm/oss_scripts/dit.py](../../executorch/examples/qualcomm/oss_scripts/dit.py) | DiT on QNN |
|
||||
| [../../executorch/examples/apple/coreml/README.md](../../executorch/examples/apple/coreml/README.md) | CoreML examples |
|
||||
| [../../executorch/examples/arm/README.md](../../executorch/examples/arm/README.md) | Arm examples |
|
||||
| [../../executorch/examples/mediatek/README.md](../../executorch/examples/mediatek/README.md) | MediaTek examples |
|
||||
| [../../executorch/examples/samsung](../../executorch/examples/samsung) | Samsung examples |
|
||||
| [../../executorch/examples/nxp](../../executorch/examples/nxp) | NXP examples |
|
||||
| [../../executorch/examples/openvino](../../executorch/examples/openvino) | OpenVINO examples |
|
||||
| [../../executorch/examples/vulkan](../../executorch/examples/vulkan) | Vulkan examples |
|
||||
| [../../executorch/examples/xnnpack/README.md](../../executorch/examples/xnnpack/README.md) | XNNPACK examples |
|
||||
| [../../executorch/examples/cuda/README.md](../../executorch/examples/cuda/README.md) | ROCm/AOTI example |
|
||||
| [../../executorch/examples/portable/executor_runner/executor_runner.cpp](../../executorch/examples/portable/executor_runner/executor_runner.cpp) | Generic runner |
|
||||
| [../../executorch/examples/portable/custom_ops](../../executorch/examples/portable/custom_ops) | Custom op example |
|
||||
| [../../executorch/examples/selective_build](../../executorch/examples/selective_build) | Selective build example |
|
||||
| [../../executorch/examples/arduino/README.md](../../executorch/examples/arduino/README.md) | Arduino |
|
||||
| [../../executorch/examples/arduino/arduino_pal.cpp](../../executorch/examples/arduino/arduino_pal.cpp) | Arduino PAL |
|
||||
| [../../executorch/examples/espressif/README.md](../../executorch/examples/espressif/README.md) | ESP32 |
|
||||
| [../../executorch/examples/raspberry_pi/pico2](../../executorch/examples/raspberry_pi/pico2) | Pico 2 |
|
||||
| [../../executorch/examples/riscv/README.md](../../executorch/examples/riscv/README.md) | RISC-V |
|
||||
| [../../executorch/examples/zephyr](../../executorch/examples/zephyr) | Zephyr |
|
||||
| [../../executorch/examples/wasm](../../executorch/examples/wasm) | WASM |
|
||||
| [../../executorch/examples/llm_pte_finetuning](../../executorch/examples/llm_pte_finetuning) | On-device finetuning |
|
||||
| [../../executorch/examples/demo-apps/react-native](../../executorch/examples/demo-apps/react-native) | RN demo |
|
||||
| [../../executorch/zephyr](../../executorch/zephyr) | Zephyr module |
|
||||
| [../../executorch/torch_pin.py](../../executorch/torch_pin.py) | PyTorch pin |
|
||||
| [../../executorch/docs/source/backends-overview.md](../../executorch/docs/source/backends-overview.md) | Backend matrix |
|
||||
| [../../executorch/docs/source/backends/xnnpack/xnnpack-overview.md](../../executorch/docs/source/backends/xnnpack/xnnpack-overview.md) | XNNPACK platforms |
|
||||
| [../../executorch/docs/source/backends/cuda/cuda-overview.md](../../executorch/docs/source/backends/cuda/cuda-overview.md) | CUDA docs |
|
||||
| [../../executorch/docs/source/backends/mlx/mlx-overview.md](../../executorch/docs/source/backends/mlx/mlx-overview.md) | MLX docs |
|
||||
| [../../executorch/docs/source/backends/template](../../executorch/docs/source/backends/template) | Backend doc template |
|
||||
| [../../executorch/docs/source/kernel-library-selective-build.md](../../executorch/docs/source/kernel-library-selective-build.md) | Selective build |
|
||||
| [../../executorch/docs/source/kernel-library-custom-aten-kernel.md](../../executorch/docs/source/kernel-library-custom-aten-kernel.md) | Custom kernels |
|
||||
| [../../executorch/docs/source/compiler-memory-planning.md](../../executorch/docs/source/compiler-memory-planning.md) | Memory planning |
|
||||
| [../../executorch/docs/source/compiler-custom-compiler-passes.md](../../executorch/docs/source/compiler-custom-compiler-passes.md) | Custom passes |
|
||||
| [../../executorch/docs/source/pte-file-format.md](../../executorch/docs/source/pte-file-format.md) | PTE format |
|
||||
| [../../executorch/docs/source/ptd-file-format.md](../../executorch/docs/source/ptd-file-format.md) | PTD format |
|
||||
| [../../executorch/docs/source/runtime-overview.md](../../executorch/docs/source/runtime-overview.md) | Runtime overview |
|
||||
| [../../executorch/docs/source/getting-started-architecture.md](../../executorch/docs/source/getting-started-architecture.md) | Architecture |
|
||||
| [../../executorch/docs/source/using-executorch-building-from-source.md](../../executorch/docs/source/using-executorch-building-from-source.md) | Build options/presets |
|
||||
| [../../executorch/docs/source/using-executorch-android.md](../../executorch/docs/source/using-executorch-android.md) | Android AAR |
|
||||
| [../../executorch/docs/source/using-executorch-ios.md](../../executorch/docs/source/using-executorch-ios.md) | iOS xcframeworks |
|
||||
| [../../executorch/docs/source/llm/export-llm.md](../../executorch/docs/source/llm/export-llm.md) | LLM export doc |
|
||||
| [../../executorch/docs/source/llm/export-llm-optimum.md](../../executorch/docs/source/llm/export-llm-optimum.md) | optimum-executorch |
|
||||
| [../../executorch/docs/source/etdump.md](../../executorch/docs/source/etdump.md) | ETDump |
|
||||
| [../../executorch/docs/source/raspberry_pi_llama_tutorial.md](../../executorch/docs/source/raspberry_pi_llama_tutorial.md) | RPi tutorial |
|
||||
| [../../executorch/docs/source/zephyr_alif_tutorial.md](../../executorch/docs/source/zephyr_alif_tutorial.md) | Zephyr/Alif tutorial |
|
||||
@@ -0,0 +1,558 @@
|
||||
# llama.cpp (ggml-org/llama.cpp) — Engine Analysis
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Local path | `/data/zhoutaichang/embedding_infer/llama.cpp` |
|
||||
| Remote | `https://github.com/ggml-org/llama.cpp.git` (origin) |
|
||||
| Branch | `master` |
|
||||
| HEAD | `e71b80510c848c00175924ecf3c40333ccae8eb5` (2026-09-07, "Revert CUDA: size routed MoE MMQ N-tiles ... (#28551)") |
|
||||
| Dirty status | clean (`git status --porcelain` empty) |
|
||||
| Submodules | none declared (`git submodule status` prints nothing); third-party code is vendored under `vendor/` (cpp-httplib, miniaudio, nlohmann, stb, sheredom, hash) |
|
||||
| Analysis date | 2026-09-08 |
|
||||
| Method | read-only source/doc inspection; no builds, no runs; evidence labels: [Verified] / [Inferred] / [Proposal] / [Unknown] |
|
||||
|
||||
Note: this checkout is far ahead of common public knowledge about llama.cpp. Several features described here (mtmd audio generation, Hexagon/HTP backend, backend samplers, `llama_model_init_from_user`, DSv4/MSA KV caches, unified `llama` app binary) may not exist in older releases. Everything below is grounded in this tree.
|
||||
|
||||
## 0. Summary
|
||||
|
||||
- [Verified] llama.cpp is a C/C++ inference engine built on the `ggml` tensor library. Two layers: `ggml` (tensor/graph/backends/scheduler, `../../llama.cpp/ggml/`) and `libllama` (model loading, KV/memory, graph building, sampling, tokenization, `../../llama.cpp/src/`), plus `libmtmd` (multimodal encoders + audio generation, `../../llama.cpp/tools/mtmd/`) and `common/` (CLI args, chat templates, speculative decoding).
|
||||
- [Verified] It is primarily an **execution runtime + library**; the "serving stack" is a separate tool, `llama-server` (`../../llama.cpp/tools/server/`), which implements slot-based continuous batching, an OpenAI/Anthropic-compatible HTTP API, SSE streaming, a router mode that spawns child server processes, and prompt caching.
|
||||
- [Verified] 18 ggml backend directories exist in-tree: `blas, cann, cpu, cuda, et, hexagon, hip, metal, musa, opencl, openvino, rpc, sycl, virtgpu, vulkan, webgpu, zdnn, zendnn` (`../../llama.cpp/ggml/src/`). All are registered via `ggml_backend_registry` (`../../llama.cpp/ggml/src/ggml-backend-reg.cpp`) and can be built as dynamically loaded plugins (`GGML_BACKEND_DL`).
|
||||
- [Verified] Heterogeneous execution: `ggml_backend_sched` (`../../llama.cpp/ggml/src/ggml-backend.cpp`) assigns each graph node to a backend (weights location, `supports_op`, `offload_op`), splits the graph, inserts inter-backend copies, and supports pipeline parallelism with up to `GGML_SCHED_MAX_COPIES=4` in-flight copies + events. Unsupported ops fall back to the CPU backend (last-priority backend).
|
||||
- [Verified] Quantization: 43 `ggml_type` ids incl. Q4_0/Q4_1/Q5_x/Q8_0, K-quants (Q2_K..Q6_K), IQ1..IQ4 variants, TQ1_0/TQ2_0 ternary, MXFP4, NVFP4, Q1_0/Q2_0 (`../../llama.cpp/ggml/include/ggml.h#L390`). Dequantization is done on the fly inside matmul kernels (CPU `vec_dot` over quantized blocks after quantizing activations to `vec_dot_type`; CUDA `mmq`/`mmvq`).
|
||||
- [Verified] Edge CPU acceleration: runtime weight repacking (`GGML_CPU_REPACK`, on by default) into interleaved Q4_0/Q4_K/Q2_K/Q5_K/Q6_K/Q8_0/IQ4_NL/MXFP4 layouts (`../../llama.cpp/ggml/src/ggml-cpu/repack.cpp`), optional Arm KleidiAI microkernels (`GGML_CPU_KLEIDIAI`, `../../llama.cpp/ggml/src/ggml-cpu/kleidiai/`), AMX on x86, per-arch SIMD in `ggml-cpu/arch/{arm,x86,riscv,loongarch,powerpc,s390,wasm}`.
|
||||
- [Verified] Mobile/edge GPU/NPU backends present: Metal (Apple, default ON on APPLE), Vulkan, OpenCL (Qualcomm Adreno-optimized, prebuilt Adreno binary kernel library optional), Hexagon HTP (Snapdragon NPU, `../../llama.cpp/ggml/src/ggml-hexagon/`), CANN (Ascend), OpenVINO (Intel NPU, "in progress"), WebGPU, ET (RISC-V manycore). No CoreML, no NNAPI, no QNN backend in tree.
|
||||
- [Verified] Model runner: GGUF format only (converted by `convert_hf_to_gguf.py` + `conversion/*.py` + `gguf-py`). `llama_model_load_from_file` -> `llama_init_from_model` -> `llama_decode`/`llama_encode`. Graph is rebuilt per ubatch unless `llm_graph_result::can_reuse()` matches (graph reuse), and compute buffers are reserved up front (`llama_context::sched_reserve`).
|
||||
- [Verified] KV/memory abstraction `llama_memory_i` has many implementations: unified/per-stream KV cache, iSWA (two caches), DSA/DSA-iSWA, MSA, DSv4, recurrent (Mamba/RWKV, with rollback snapshots), hybrid, hybrid-iSWA, hybrid-idx (`../../llama.cpp/src/llama-memory.h`, `../../llama.cpp/src/llama-model.cpp#L2232`). KV cells are a ring buffer with per-cell sequence bitsets; no paged attention.
|
||||
- [Verified] TTS is first-class at this commit: `tools/tts` (`llama-tts`) drives a 3-stage pipeline (backbone -> code predictor / flow net -> code2wav / Mimi decoder) through `mtmd` experimental audio-generation API. Supported pipelines: **Qwen3-TTS** and **Pocket-TTS (Kyutai)**; legacy **WavTokenizer decoder** arch remains in `libllama` (OuteTTS-era). Output is WAV to file, non-streaming in the CLI, but the API is frame-stepped (`step_gen`) so streaming is feasible.
|
||||
- [Verified] Audio input (ASR/audio understanding) is supported through mtmd audio encoders (Whisper-style, Conformer/Parakeet, Ultravox, Qwen2/3-Audio, Voxtral, Gemma-4 audio, etc.). Server exposes `/v1/audio/transcriptions`. Mel preprocessing is a whisper.cpp-derived FFT in `mtmd-audio.cpp`.
|
||||
- [Verified] Vision: `clip.cpp` in mtmd implements ~60 projector types (SigLIP/CLIP ViTs, Qwen-VL mergers, Gemma3/4, Pixtral, InternVL, Kimi-VL, MobileNetV5 for Gemma3n, etc.), with image preprocessing (Pillow-compatible resize, LLaVA-UHD slicing) and video (ffmpeg subprocess).
|
||||
- [Verified] No VLA support: no action heads, no proprioceptive input path, no diffusion/flow policy heads for robotics. The only generic "numeric input" path is `llama_batch.embd` (pre-computed input embeddings) and `mtmd` encoder outputs. Diffusion **language** models (Dream/LLaDA) exist (`../../llama.cpp/examples/diffusion/`), and Pocket-TTS contains a flow-matching net in mtmd, which are the closest existing building blocks.
|
||||
- [Verified] Application boundaries: C API (`include/llama.h`, `tools/mtmd/mtmd.h`), unified `llama` binary (`app/`), `llama-cli`, `llama-server`, `llama-mtmd-cli`, `llama-tts`, Android JNI/Kotlin sample (`examples/llama.android`), iOS SwiftUI + XCFramework (`examples/llama.swiftui`, `build-xcframework.sh`), RPC server for distributed backends.
|
||||
- [Unknown] The repo publishes no binary-size numbers; footprint must be measured.
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Apps["Applications / boundaries"]
|
||||
CLI[llama-cli / llama app<br/>tools/cli, app/]
|
||||
SRV[llama-server<br/>tools/server]
|
||||
TTS[llama-tts<br/>tools/tts]
|
||||
MTMDCLI[llama-mtmd-cli]
|
||||
MOB[Android JNI / iOS Swift<br/>examples/llama.android, llama.swiftui]
|
||||
end
|
||||
subgraph Common["common/ (C++ helpers)"]
|
||||
ARG[arg.cpp params]
|
||||
CHAT[chat.cpp + jinja engine]
|
||||
SPEC[speculative.cpp]
|
||||
SAMP[sampling.cpp]
|
||||
end
|
||||
subgraph MTMD["libmtmd (tools/mtmd)"]
|
||||
MTMDAPI[mtmd.h / mtmd-helper.h]
|
||||
CLIP[clip.cpp encoders + projectors]
|
||||
AUD[mtmd-audio.cpp mel/FFT]
|
||||
IMG[mtmd-image.cpp preprocessing]
|
||||
GEN[mtmd-helper-gen.cpp<br/>TTS pipelines]
|
||||
end
|
||||
subgraph LLAMA["libllama (src/)"]
|
||||
LOAD[llama-model-loader.cpp<br/>GGUF + mmap]
|
||||
MODEL[llama-model.cpp + src/models/*<br/>tensors + graph builders]
|
||||
CTX[llama-context.cpp<br/>decode/encode, sched, outputs]
|
||||
BATCH[llama-batch.cpp<br/>ubatch splitting]
|
||||
MEM[llama-memory-*.cpp / llama-kv-cache*.cpp]
|
||||
GRAPH[llama-graph.cpp<br/>llm_graph_context]
|
||||
VOCAB[llama-vocab.cpp tokenizers]
|
||||
SMPL[llama-sampler.cpp]
|
||||
end
|
||||
subgraph GGML["ggml (ggml/)"]
|
||||
GG[ggml.c tensors/ops/graphs]
|
||||
SCHED[ggml-backend.cpp<br/>ggml_backend_sched]
|
||||
ALLOC[ggml-alloc.c gallocr]
|
||||
REG[ggml-backend-reg.cpp registry + DL]
|
||||
BE[backends: cpu cuda metal vulkan opencl hexagon sycl hip cann rpc webgpu ...]
|
||||
end
|
||||
CLI --> Common --> LLAMA
|
||||
SRV --> Common
|
||||
SRV --> MTMD
|
||||
TTS --> MTMD
|
||||
MTMDCLI --> MTMD
|
||||
MOB --> LLAMA
|
||||
MTMD --> LLAMA
|
||||
MTMD --> GGML
|
||||
LLAMA --> GGML
|
||||
SCHED --> ALLOC
|
||||
SCHED --> BE
|
||||
REG --> BE
|
||||
```
|
||||
|
||||
Key modules:
|
||||
|
||||
| Module | Path | Responsibility | Key symbols |
|
||||
|---|---|---|---|
|
||||
| ggml core | `../../llama.cpp/ggml/src/ggml.c`, `../../llama.cpp/ggml/include/ggml.h` | tensor struct, op enum (102 ops), graph build | `ggml_tensor`, `enum ggml_op`, `ggml_build_forward_expand`, `ggml_custom_4d` |
|
||||
| Backend API | `../../llama.cpp/ggml/include/ggml-backend.h`, `../../llama.cpp/ggml/src/ggml-backend-impl.h` | device/buffer/backend vtables | `ggml_backend_device_i`, `ggml_backend_i`, `ggml_backend_reg_i`, `ggml_backend_dev_props` |
|
||||
| Backend registry | `../../llama.cpp/ggml/src/ggml-backend-reg.cpp` | compile-time registration, dynamic loading | `ggml_backend_registry`, `ggml_backend_load_all`, `ggml_backend_load_best` |
|
||||
| Scheduler | `../../llama.cpp/ggml/src/ggml-backend.cpp` | node->backend assignment, splits, copies, pipeline parallel | `ggml_backend_sched`, `ggml_backend_sched_split_graph`, `ggml_backend_sched_compute_splits` |
|
||||
| Graph allocator | `../../llama.cpp/ggml/src/ggml-alloc.c` | compute buffer planning (in-place reuse, chunked vbuffers) | `ggml_gallocr`, `ggml_dyn_tallocr`, `ggml_backend_alloc_ctx_tensors` |
|
||||
| CPU backend | `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c`, `ggml-cpu.cpp`, `ops.cpp` | threadpool, op kernels, extra buffer types | `ggml_threadpool`, `ggml_graph_compute`, `type_traits_cpu`, `ggml_backend_cpu_device_supports_op` |
|
||||
| GPU/NPU backends | `../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu`, `../../llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp`, `../../llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp`, `../../llama.cpp/ggml/src/ggml-opencl/ggml-opencl.cpp`, `../../llama.cpp/ggml/src/ggml-hexagon/ggml-hexagon.cpp` | per-backend kernels | `ggml_backend_*_device_supports_op`, `*_reg()` |
|
||||
| RPC backend | `../../llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp`, `../../llama.cpp/tools/rpc/rpc-server.cpp` | remote device over sockets | `enum rpc_cmd` |
|
||||
| Model loader | `../../llama.cpp/src/llama-model-loader.cpp`, `../../llama.cpp/src/llama-mmap.cpp` | GGUF parse, splits, mmap/mlock, tensor data upload | `llama_model_loader`, `load_all_data`, `init_mappings` |
|
||||
| Model | `../../llama.cpp/src/llama-model.cpp`, `../../llama.cpp/src/models/models.h` (+ one `.cpp` per arch in `../../llama.cpp/src/models/`) | hparams/tensors per arch, buffer placement, `create_memory`, `build_graph` | `llama_model::load_tensors`, `make_cpu_buft_list`, `make_gpu_buft_list`, `llama_model::create_memory` |
|
||||
| Context | `../../llama.cpp/src/llama-context.cpp` | backends/sched creation, reserve, decode/encode loop, outputs, state save/load | `llama_context::decode`, `process_ubatch`, `sched_reserve`, `graph_compute` |
|
||||
| Batch | `../../llama.cpp/src/llama-batch.cpp` | validate batch, split into ubatches | `llama_batch_allocr::split_simple/split_equal/split_seq` |
|
||||
| Memory/KV | `../../llama.cpp/src/llama-memory.h`, `../../llama.cpp/src/llama-kv-cache.cpp`, `llama-kv-cells.h`, `llama-memory-recurrent.cpp` | KV storage, slot finding, masks, shifts, state IO | `llama_memory_i`, `llama_kv_cache::find_slot`, `apply_ubatch`, `llama_kv_cells` |
|
||||
| Graph builder | `../../llama.cpp/src/llama-graph.cpp` | shared building blocks, graph inputs, reuse checks | `llm_graph_context::build_attn/build_ffn/build_moe_ffn/build_rs`, `llm_graph_result::can_reuse` |
|
||||
| Vocab | `../../llama.cpp/src/llama-vocab.cpp` | tokenizers SPM/BPE/WPM/UGM/RWKV/PLaMo2, detokenize | `llm_tokenizer_*_session`, `llama_vocab::impl` |
|
||||
| Sampling | `../../llama.cpp/src/llama-sampler.cpp` | CPU sampler chain + backend (in-graph) samplers | `llama_sampler_chain_*`, `llama_set_sampler` |
|
||||
| Multimodal | `../../llama.cpp/tools/mtmd/mtmd.cpp`, `clip.cpp`, `mtmd-audio.cpp`, `mtmd-image.cpp`, `mtmd-helper.cpp` | encoders, preprocessing, chunked prompt eval | `mtmd_tokenize`, `mtmd_encode_chunk`, `mtmd_helper_eval_chunks`, `clip_image_batch_encode` |
|
||||
| Audio generation | `../../llama.cpp/tools/mtmd/mtmd-helper-gen.cpp`, `../../llama.cpp/tools/mtmd/models/qwen3tts-gen.cpp`, `pockettts-gen.cpp` | TTS stage 2/3 graphs and stateful driver | `mtmd_gen_audio_process`, `qwen3tts_gen_audio_pipeline`, `pockettts_gen_audio_pipeline` |
|
||||
| Server | `../../llama.cpp/tools/server/server-context.cpp`, `server-queue.cpp`, `server-http.cpp` | slots, continuous batching, HTTP/SSE | `server_slot`, `update_slots`, `server_queue`, `set_chunked_content_provider` |
|
||||
| Conversion | `../../llama.cpp/convert_hf_to_gguf.py`, `../../llama.cpp/conversion/base.py`, `../../llama.cpp/gguf-py/gguf/` | HF -> GGUF | `ModelBase`, `TextModel`, `MmprojModel`, `GGUFWriter` |
|
||||
|
||||
## 2. Device layer
|
||||
|
||||
**Backend abstraction.** [Verified] Three-level object model in `../../llama.cpp/ggml/include/ggml-backend.h`: `ggml_backend_reg_t` (a backend library, enumerates devices) -> `ggml_backend_dev_t` (a physical device with `ggml_backend_dev_props{name, description, memory_free/total, type, device_id, caps}`) -> `ggml_backend_t` (a "stream"/execution context created by `ggml_backend_dev_init`). Buffer types (`ggml_backend_buffer_type_t`) allocate `ggml_backend_buffer_t`. Device types: `CPU`, `GPU`, `IGPU` (integrated, host memory), `ACCEL` (BLAS/AMX used together with CPU), `META` (tensor-parallel wrapper) (`../../llama.cpp/ggml/include/ggml-backend.h#L134`).
|
||||
|
||||
**Device capability flags.** [Verified] `ggml_backend_dev_caps{async, host_buffer, buffer_from_host_ptr, events, mmap_support}` (`../../llama.cpp/ggml/include/ggml-backend.h#L148`). `llama_context` enables pipeline parallelism only if all non-CPU devices report `async` and `events` (`../../llama.cpp/src/llama-context.cpp#L426`).
|
||||
|
||||
**Backends in tree and registration.** [Verified] `../../llama.cpp/ggml/src/ggml-backend-reg.cpp#L115` `struct ggml_backend_registry` registers at static-init time (guarded by `GGML_USE_*`): CUDA, Metal, SYCL, Vulkan, WebGPU, zDNN, VirtGPU frontend, OpenCL, ZenDNN, Hexagon, CANN, BLAS, RPC, OpenVINO, ET, and CPU last. HIP and MUSA compile the CUDA source tree (`ggml-hip/`, `ggml-musa/` are thin CMake wrappers; `ggml-hip` has 0 source lines). `ggml_backend_load_all_from_path` (`#L578`) dlopens `ggml-{blas,zendnn,cann,cuda,hip,metal,rpc,sycl,vulkan,virtgpu,opencl,hexagon,musa,openvino,cpu}` plugins when `GGML_BACKEND_DL` is used; `ggml_backend_load_best` picks the best CPU variant (`GGML_CPU_ALL_VARIANTS`). Backend order in the registry defines scheduler priority (GPU first, CPU last).
|
||||
|
||||
**Device discovery/selection in llama.** [Verified] `../../llama.cpp/src/llama.cpp#L139-L303`: `llama_model_load_from_file_impl` enumerates `ggml_backend_dev_count()`; if `params.devices` is NULL it collects GPU devices, then IGPU devices only if no GPU, RPC servers first; `split_mode == LLAMA_SPLIT_MODE_NONE` keeps only `main_gpu`. `llama_model_params{devices, tensor_buft_overrides, n_gpu_layers, split_mode, load_mode, lazy_mode, main_gpu, tensor_split, use_extra_bufts, no_host, no_alloc}` (`../../llama.cpp/include/llama.h`). CLI `--device`, `--list-devices` (`../../llama.cpp/docs/build.md#L838`).
|
||||
|
||||
**Weight placement.** [Verified] `../../llama.cpp/src/llama-model.cpp#L1031` `make_cpu_buft_list` builds an ordered candidate list: ACCEL device bufts, GPU host (pinned) bufts (unless `no_host`), CPU extra bufts (AMX/KleidiAI/repack, when `use_extra_bufts`), then plain CPU. `make_gpu_buft_list` (`#L1093`) adds split-row bufts for `LLAMA_SPLIT_MODE_ROW`, the device buft, and GPU extra bufts. Layers `[n_layer+1-n_gpu_layers, n_layer]` map to GPU devices proportionally to `tensor_split` (`#L1479-L1504`); input embeddings always on CPU (`dev_input`). Per-tensor overrides via regex `tensor_buft_overrides` (`--override-tensor`). Each weight is placed in the first buft in its layer list whose device `supports_op` for a probe op (`llama_model_loader` `select_weight_buft`; [Inferred] from function names at `#L1031-L1138` and loader code).
|
||||
|
||||
**Memory allocation & pools.** [Verified] Weights: `ggml_backend_alloc_ctx_tensors_from_buft` (`../../llama.cpp/ggml/src/ggml-alloc.c#L1239`), or zero-copy `ggml_backend_dev_buffer_from_host_ptr` from the mmap'd file when the device supports it and the buft is the default one (`../../llama.cpp/src/llama-model.cpp#L1734`). `LLAMA_LOAD_MODE_{AUTO,MMAP,MLOCK,MMAP_MLOCK,NONE}` control mmap/mlock (`../../llama.cpp/src/llama-model-loader.cpp#L559`). Compute buffers: `ggml_gallocr` in `ggml-alloc.c` plans in-place reuse (`ggml_op_can_inplace`, `#L22`), splits large allocations into chunks (`struct vbuffer`, `#L398`), and reallocates only when `ggml_gallocr_needs_realloc` (`#L1009`). KV cache: one `ggml_context` per buffer type, K/V tensors allocated per layer (`../../llama.cpp/src/llama-kv-cache.cpp#L104-L152`); `offload_kqv` places the cache on the layer's device. Output logits/embeddings: pinned host buffer `buf_output` (`../../llama.cpp/src/llama-context.cpp#L380`). Backend-internal pools (e.g. CUDA VMM pool) are inside each backend ([Verified] `GGML_CUDA_NO_VMM` option in `../../llama.cpp/ggml/CMakeLists.txt#L204`).
|
||||
|
||||
**Host <-> device transfers.** [Verified] `ggml_backend_tensor_set/get` (sync), `_async` variants, strided 2D variants `ggml_backend_tensor_set_2d_async` (`../../llama.cpp/ggml/include/ggml-backend.h#L86-L96`), `ggml_backend_tensor_copy_async` between backends with automatic sync fallback (`#L116`). The scheduler copies split inputs: user inputs (`GGML_TENSOR_FLAG_INPUT`) are copied synchronously after waiting on the copy's event; other inputs use `cpy_tensor_async` if the backend supports it, else sync copy (`../../llama.cpp/ggml/src/ggml-backend.cpp#L1670-L1760`). MoE weights that live on host are copied expert-by-expert only for the experts used (`#L1690`). Model loading uploads via `load_all_data` with optional async upload backend ([Inferred] from `upload_backend`/`ggml_backend_event` usage grep in `llama-model-loader.cpp`).
|
||||
|
||||
**Synchronization / streams.** [Verified] One `ggml_backend_t` per device = one stream. `ggml_backend_synchronize`, `ggml_backend_event_{new,record,wait,synchronize}` (`ggml-backend.h#L121-L128`). Scheduler keeps `events[GGML_SCHED_MAX_BACKENDS=16][GGML_SCHED_MAX_COPIES=4]` (`ggml-backend.cpp#L763-L772, #L819`) and rotates `cur_copy` per graph for pipeline parallelism; `n_copies = parallel ? 4 : 1` (`#L1865`). `llama_context::graph_compute` calls `ggml_backend_sched_graph_compute_async` and results are fetched with async `ggml_backend_tensor_get_async` then `llama_synchronize` (`../../llama.cpp/src/llama-context.cpp#L2492`, `#L714`).
|
||||
|
||||
**Tensor parallelism / multi-device.** [Verified] `GGML_BACKEND_DEVICE_TYPE_META` + `ggml_backend_meta_device` (`../../llama.cpp/ggml/src/ggml-backend-meta.cpp`) implement experimental tensor parallelism (`LLAMA_SPLIT_MODE_TENSOR`, `../../llama.cpp/docs/multi-gpu.md#L80`); NCCL optional (`GGML_CUDA_NCCL`). Default multi-GPU is layer split with pipeline parallelism.
|
||||
|
||||
**RPC backend.** [Verified] `../../llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp#L61` defines `rpc_cmd` {ALLOC_BUFFER, GET_ALIGNMENT, GET_MAX_SIZE, BUFFER_GET_BASE, FREE_BUFFER, BUFFER_CLEAR, SET_TENSOR, SET_TENSOR_HASH, GET_TENSOR, COPY_TENSOR, GRAPH_COMPUTE, GET_DEVICE_MEMORY, INIT_TENSOR, GET_ALLOC_SIZE, HELLO, DEVICE_COUNT, GRAPH_RECOMPUTE, MEMSET_TENSOR}; graphs are serialized and computed remotely; `SET_TENSOR_HASH` dedups uploads > 10 MiB. Server binary: `../../llama.cpp/tools/rpc/rpc-server.cpp`. Apple transport variant `transport-apple.cpp`.
|
||||
|
||||
**Portability (OS/arch actually in tree).** [Verified]
|
||||
- CPU arches: `ggml/src/ggml-cpu/arch/{arm,x86,riscv,loongarch,powerpc,s390,wasm}`; build options for SSE4.2/AVX/AVX2/AVX512(+VNNI/BF16)/AMX, RVV/Zfh/Zvfh, LASX/LSX, VXE (`../../llama.cpp/ggml/CMakeLists.txt#L154-L181`). Android NDK arm64-v8a documented (`../../llama.cpp/docs/android.md`), Termux build, WASM/Emscripten (`LLAMA_WASM_*`).
|
||||
- GPU/NPU: CUDA (arch list incl. `86-real`, `89-real`, `120a/121a-real`; [Inferred] Jetson Orin sm_87 would rely on `native` or user-provided `CMAKE_CUDA_ARCHITECTURES`, `../../llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt#L28-L54`), HIP, MUSA, Metal (macOS/iOS/visionOS/tvOS via `build-xcframework.sh`), Vulkan (Windows/Linux/Android/macOS via MoltenVK or KosmicKrisp, `../../llama.cpp/docs/build.md#L402-L557`), OpenCL for Adreno (Android, Windows Arm64; Adreno 750/810/830/840/X1-85/X2-90 listed, `../../llama.cpp/docs/backend/OPENCL.md#L47-L54`), Hexagon HTP v73+ via Hexagon SDK containers (`../../llama.cpp/docs/backend/snapdragon/README.md`), CANN, SYCL (Intel GPU), OpenVINO, zDNN (IBM Z), WebGPU (Dawn; browser via Emscripten), VirtGPU (VM API remoting), ET (RISC-V ET-SOC).
|
||||
|
||||
## 3. Kernel layer
|
||||
|
||||
**Operator set.** [Verified] `enum ggml_op` in `../../llama.cpp/ggml/include/ggml.h#L482-L593` has 102 entries incl. `MUL_MAT`, `MUL_MAT_ID` (MoE), `FLASH_ATTN_EXT`, `ROPE`, `RMS_NORM`, `GROUP_NORM`, `L2_NORM`, `SSM_CONV/SSM_SCAN`, `RWKV_WKV6/7`, `GATED_LINEAR_ATTN`, `GATED_DELTA_NET`, `LIGHTNING_INDEXER`, `DSV4_HC_*`, `SOLVE_TRI`, `TRI`, conv family (`CONV_2D`, `CONV_3D`, `CONV_2D_DW`, `CONV_TRANSPOSE_1D/2D`, `IM2COL`, `IM2COL_3D`, `COL2IM_1D`), `POOL_1D/2D`, `UPSCALE`, `PAD`, `PAD_REFLECT_1D`, `ROLL`, `ARANGE`, `TIMESTEP_EMBEDDING`, `ARGSORT`, `TOP_K`, `CUMSUM`, `SET_ROWS`, `GET_ROWS`, optimizer ops (`OPT_STEP_ADAMW/SGD`, `CROSS_ENTROPY_LOSS`), `MAP_CUSTOM1..3`, `CUSTOM`, `UNARY`, `GLU`. Unary and GLU sub-ops are separate enums (`#L596`, `#L623`). Per-backend support matrix is generated into `../../llama.cpp/docs/ops.md` from `docs/ops/*.csv` (columns: BLAS, CANN, CPU, CUDA, ET, HTP, MTL, OpenCL, SYCL, Vulkan, WebGPU, ZenDNN, zDNN) via `test-backend-ops support --output csv` + `../../llama.cpp/scripts/create_ops_docs.py`.
|
||||
|
||||
**Dispatch mechanism.** [Verified] There is no per-op registry table across backends. Each backend implements `ggml_backend_device_i::supports_op(dev, op)` and `offload_op(dev, op)` (`../../llama.cpp/ggml/src/ggml-backend-impl.h#L205-L212`) plus a `graph_compute(backend, cgraph)` entry (`#L146`) that walks the nodes and switches on `op->op`. The scheduler queries `supports_op`/`supports_buft` to assign nodes (Section 5b). Examples: `ggml_backend_cpu_device_supports_op` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp#L424`, which also delegates to extra buffer types' `supports_op`), `ggml_backend_cuda_device_supports_op` (`../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu#L5060`), `ggml_backend_metal_device_supports_op` (`../../llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp#L741`), `ggml_backend_vk_device_supports_op` (`../../llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp#L19037`), `ggml_opencl_supports_op` (`../../llama.cpp/ggml/src/ggml-opencl/ggml-opencl.cpp#L8393`), `ggml_backend_hexagon_device_supports_op` (`../../llama.cpp/ggml/src/ggml-hexagon/ggml-hexagon.cpp#L5831`; HTP op list at `#L4962-L4992`: FA, MUL_MAT(_ID), binary ops, CPY, GET/SET_ROWS, ARGSORT, norms, SOFT_MAX, SSM_CONV, GATED_DELTA_NET, ROPE, ...). `offload_op` lets a GPU claim ops on host-resident weights when the batch is large enough (CUDA: `GGML_OP_OFFLOAD_MIN_BATCH`, default 32, `ggml-cuda.cu#L5714`).
|
||||
|
||||
**CPU kernel selection.** [Verified] `type_traits_cpu[GGML_TYPE_COUNT]` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c#L215`) maps each quant type to `{from_float, vec_dot, vec_dot_type, nrows}`. In `ggml_compute_forward_mul_mat` (`../../llama.cpp/ggml/src/ggml-cpu/ops.cpp#L5240-L5290`) the activation matrix is quantized to `vec_dot_type` (e.g. Q8_0/Q8_K) into `params->wdata`, then `vec_dot` runs over quantized weight blocks; there is no full dequantization of weights. Optional tinyBLAS `llamafile_sgemm` (`../../llama.cpp/ggml/src/ggml-cpu/llamafile/sgemm.cpp`, `GGML_LLAMAFILE`) and BLAS/Accelerate for large F32/F16 matmuls.
|
||||
|
||||
**CPU extra buffer types (repack / KleidiAI / AMX).** [Verified] `ggml_backend_cpu_get_extra_buffer_types` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp#L42`) registers AMX, SpacemiT RISC-V, KleidiAI, and repack buffer types. An extra buffer type implements `extra_buffer_type::supports_op` and `get_tensor_traits` (`../../llama.cpp/ggml/src/ggml-cpu/traits.h#L29-L31`); `tensor_traits::compute_forward` overrides the kernel for ops whose weight lives in that buffer. `repack.cpp` converts Q4_0 -> Q4_0x4/x8/x16, Q4_K -> Q4_Kx8/x16, Q2_K/Q5_K/Q6_K, Q8_0, IQ4_NL, MXFP4 into interleaved blocks at load time (`make_block_*`, `repack_*_bl`, `ggml_repack_get_optimal_repack_type` at `#L4528`). KleidiAI (`../../llama.cpp/ggml/src/ggml-cpu/kleidiai/kleidiai.cpp`) selects Arm microkernels by dotprod/i8mm/SVE/SME2 at runtime (`../../llama.cpp/docs/build.md#L616-L712`).
|
||||
|
||||
**Quantization formats.** [Verified] `enum ggml_type` (`../../llama.cpp/ggml/include/ggml.h#L390-L433`): F32, F16, BF16, F64, I8/I16/I32/I64, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1, Q2_K..Q6_K, Q8_K, IQ2_XXS/XS/S, IQ3_XXS/S, IQ1_S/M, IQ4_NL/XS, TQ1_0/TQ2_0, MXFP4, NVFP4, Q1_0, Q2_0. Quantize/dequantize reference in `../../llama.cpp/ggml/src/ggml-quants.c`; model-level `llama_ftype` (42 `LLAMA_FTYPE_MOSTLY_*` entries) and `llama_model_quantize` in `../../llama.cpp/src/llama-quant.cpp` (imatrix-aware, per-tensor type overrides, layer pruning). GPU: CUDA `mmq.cuh` (int8 tensor-core MMQ on quantized weights), `mmvq.cu` (matvec), `dequantize.cuh`/`convert.cu`; Metal kernels in `../../llama.cpp/ggml/src/ggml-metal/kernels/` (23 files incl. `dequantize.h`); Vulkan 180 shader sources in `../../llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/`; OpenCL 178 `.cl` kernels (`../../llama.cpp/ggml/src/ggml-opencl/kernels/`), Adreno path supports Q4_0/Q4_1/Q4_K/Q6_K/MXFP4 ([Verified] `../../llama.cpp/docs/backend/OPENCL.md#L59-L83`). KV cache may be quantized (`type_k/type_v`, e.g. q8_0), Flash-attention kernels handle quantized KV on CPU/CUDA/Metal/Vulkan ([Inferred] from `GGML_CUDA_FA_ALL_QUANTS` and `k_vec_dot_type` in `ops.cpp#L8688`).
|
||||
|
||||
**Compilation / JIT / graph capture.** [Verified]
|
||||
- Metal: shaders compiled from embedded source at runtime with `newLibraryWithSource` per op-source, or loaded from a prebuilt `.metallib` (`GGML_METAL_EMBED_LIBRARY`, `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m#L300-L395`); pipelines cached (`ggml_metal_pipelines_t`). Metal 4 tensor API gated by `has_tensor`.
|
||||
- Vulkan: GLSL compiled to SPIR-V at build time by the host tool `vulkan-shaders-gen` (`../../llama.cpp/ggml/src/ggml-vulkan/CMakeLists.txt#L174-L199`, `../../llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp`); pipelines created lazily at runtime; cooperative-matrix variants selected by device features (`coopmat_support`, `ggml-vulkan.cpp#L890`).
|
||||
- OpenCL: kernels embedded (`GGML_OPENCL_EMBED_KERNELS`) and built at init with an on-disk program binary cache (`../../llama.cpp/ggml/src/ggml-opencl/cl-program-cache.cpp`); optional Adreno prebuilt binary kernel library (`GGML_OPENCL_USE_ADRENO_BIN_KERNELS`).
|
||||
- CUDA: CUDA graph capture for decode graphs (`GGML_CUDA_GRAPHS`, `ggml_cuda_graph_evaluate_and_capture` at `../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu#L4181`; keyed per `graph_key`); HIP graphs, SYCL graphs, MUSA graphs (experimental) also have options (`../../llama.cpp/ggml/CMakeLists.txt#L207-L250`).
|
||||
- Hexagon: HTP kernels compiled with the Hexagon SDK into `libggml-htp-v73.so`-style DSP libraries invoked over FastRPC (`../../llama.cpp/ggml/src/ggml-hexagon/htp/`, `htp_iface.idl`; [Verified] build log excerpt in `../../llama.cpp/docs/backend/snapdragon/README.md`).
|
||||
- Backend-side graph rewriting hook: `ggml_backend_i::graph_optimize` (`../../llama.cpp/ggml/src/ggml-backend-impl.h#L155`) is invoked per split (`ggml-backend.cpp#L1462`). llama also probes fused ops per backend (`llm_fused_op_probe`, `llama_context::resolve_fused_ops`, `../../llama.cpp/src/llama-context.cpp#L35`, `#L505`).
|
||||
|
||||
**Fallback when an op is unsupported.** [Verified] In `ggml_backend_sched_backend_id_from_cur` (`../../llama.cpp/ggml/src/ggml-backend.cpp#L921`): a node whose weight buffer's backend does not `supports_op` searches other backends that support both the buffer type and op; if none, the weight is copied ("the weight will need to be copied"). Graph inputs default to the last backend (CPU). Pass 2/3 expand assignments to neighbours and "upgrade" nodes to higher-priority backends only when buffer types are compatible; remaining unsupported nodes end up on CPU, producing extra splits and copies (`#L1124-L1297`). `docs/ops.md` marks partial support; the ET backend doc explicitly states "some/most operations will likely fallback to CPU backend" (`../../llama.cpp/docs/backend/ET.md`).
|
||||
|
||||
**Extension points (custom op / backend).** [Verified] Custom ops: `ggml_map_custom1/2/3` and `ggml_custom_4d(ctx, type, ne0..3, args, n_args, fun, n_tasks, userdata)` with `ggml_custom_op_t` executed by the CPU backend (`../../llama.cpp/ggml/include/ggml.h#L2679-L2747`). New backend: implement `ggml_backend_reg_i`, `ggml_backend_device_i`, `ggml_backend_i`, `ggml_backend_buffer_type_i`, `ggml_backend_buffer_i` vtables (`ggml-backend-impl.h`), export with `GGML_BACKEND_DL_IMPL(reg_fn)` (`#L258-L285`), add to `ggml-backend-reg.cpp` and CMake. Optional proc-address extensions: `ggml_backend_set_n_threads`, `ggml_backend_dev_get_extra_bufts`, `ggml_backend_set_abort_callback`, `ggml_backend_get_features` (`ggml-backend.h#L205-L225`). New model arch: `../../llama.cpp/docs/development/HOWTO-add-model.md` (conversion class in `conversion/`, arch enum in `llama-arch.cpp`, `llama_model_<arch>` class in `src/models/models.h` with `load_arch_hparams/load_arch_tensors/build_graph`). Tests for new ops go in `../../llama.cpp/tests/test-backend-ops.cpp`.
|
||||
|
||||
## 4. Model runner
|
||||
|
||||
**Formats and conversion.** [Verified] Runtime format is GGUF only (`../../llama.cpp/ggml/src/gguf.cpp`, `../../llama.cpp/gguf-py/gguf/gguf_writer.py`, `gguf_reader.py`, `constants.py`, `tensor_mapping.py`). `../../llama.cpp/convert_hf_to_gguf.py` (312 lines) dispatches to registered classes in `../../llama.cpp/conversion/` (`ModelBase`, `TextModel`, `MmprojModel` in `../../llama.cpp/conversion/base.py`; 100+ per-family files incl. `qwen3tts.py`, `pockettts.py`, `wavtokenizer.py`, `talkie.py`, `ultravox.py`, `qwen3vl.py`). Multimodal encoders are converted into a separate `mmproj` GGUF (`--mmproj`). Also `convert_lora_to_gguf.py`, `convert_llama_ggml_to_gguf.py`, `llama-quantize` (`../../llama.cpp/tools/quantize/`), `llama-imatrix`, `gguf-split`. Model sharding/splits supported (`llama_model_load_from_splits`, `llama_get_list_splits` in `llama-model-loader.cpp#L82`). Models can also be constructed programmatically without a file: `llama_model_init_from_user` (`../../llama.cpp/src/llama.cpp#L446`).
|
||||
|
||||
**Load pipeline.** [Verified] `llama_model_load_from_file_impl` (`../../llama.cpp/src/llama.cpp#L380`) -> device list -> `llama_model_load` (`#L316`) -> `llama_model_loader` parses GGUF metadata (+ splits), `model.load_arch/hparams/vocab/stats`, `model.load_tensors` (buffer placement, mmap zero-copy or `load_all_data` upload; `../../llama.cpp/src/llama-model.cpp#L1403-L1834`), optional `check_tensors`, progress callback that can abort loading. `lazy_mode` allows on-demand reading of arch-marked tensors. 152 `LLM_ARCH_*` enums (`../../llama.cpp/src/llama-arch.h`).
|
||||
|
||||
**Execution lifecycle.** [Verified]
|
||||
1. `llama_backend_init()`; optionally `ggml_backend_load_all()` for DL builds (`../../llama.cpp/examples/simple/simple.cpp#L82`).
|
||||
2. `llama_init_from_model(model, params)` -> `llama_context` ctor (`../../llama.cpp/src/llama-context.cpp#L83`): computes `n_ctx` padded to 256 and `n_ctx_seq = n_ctx / n_seq_max` unless `kv_unified` (`#L288-L301`), `create_memory`, inits one `ggml_backend_t` per model device + ACCEL devices + CPU (`#L331-L357`), collects `set_n_threads` hooks, allocates output buffer, decides pipeline parallelism, then `sched_reserve()` (`#L582`).
|
||||
3. `sched_reserve`: builds `ggml_backend_sched_new(backends, bufts, n, max_nodes, pipeline_parallel, op_offload)` and reserves compute buffers by building worst-case graphs for prompt-processing (`n_ubatch` tokens) and token-generation (1 token) shapes via `graph_reserve` (`#L2416`), logging split counts `n_splits_pp/tg` and buffer sizes. If allocation fails with pipeline parallelism it retries without (`#L637`).
|
||||
4. Warmup: `llama_set_warmup(ctx, true)` and a decode of BOS/EOS is done by `common` init (deprecated-marked in API but present).
|
||||
5. Step: `llama_decode(ctx, batch)` (`#L1644`) -> `balloc->init` validates the `llama_batch` (tokens or embeddings, positions, `seq_id`, `logits` output flags) -> `memory_update(false)` applies pending shifts/copies -> `memory->init_batch(balloc, n_ubatch, output_all)` splits into ubatches and finds KV slots (retrying once after `memory_update(true)` optimization) -> `output_reserve` -> for each ubatch `process_ubatch` (`#L1334`): `mctx->apply()`, graph reuse check, else `ggml_backend_sched_reset` + `model.build_graph(gparams)` + `ggml_backend_sched_alloc_graph`; `res->set_inputs(&ubatch)`; `graph_compute` (async) -> logits/embeddings copied with `ggml_backend_tensor_get_async` into host buffers, reordered by `output_reorder`. `llama_encode` handles encoder-only / encoder half of enc-dec (T5) and non-causal models (`#L1406`).
|
||||
6. `llama_get_logits_ith`, `llama_get_embeddings_{ith,seq}`, backend-sampler results `llama_get_sampled_token_ith` (`../../llama.cpp/include/llama.h#L1035-L1092`).
|
||||
|
||||
**Dynamic shapes / graph reuse.** [Verified] Graphs are built per ubatch as `ggml_cgraph` (no static shape compilation). `llm_graph_params` records `ubatch` shape, `n_outputs`, memory context, and each `llm_graph_input_*::can_reuse` compares against the previous graph (`../../llama.cpp/src/llama-graph.h#L100-L760`, `#L771-L892`); when identical topology, `process_ubatch` skips rebuilding and reallocating (`n_reused++`), which is what makes CUDA graph capture effective. Env `LLAMA_GRAPH_REUSE_DISABLE` disables it (`../../llama.cpp/src/llama-context.cpp#L280`). Because `ggml_gallocr` reserved worst-case buffers, varying `n_tokens <= n_ubatch` needs no reallocation (`ggml_gallocr_needs_realloc`).
|
||||
|
||||
**Batch splitting.** [Verified] `llama_batch_allocr::split_simple(n_ubatch)` (sequential chunk), `split_equal(n_ubatch, sequential, n_keep_tail)` (equal-length per-sequence groups, used by non-unified KV and recurrent memories), `split_seq(n_ubatch)` (one sequence per ubatch) (`../../llama.cpp/src/llama-batch.h#L103-L111`). KV cache picks `split_simple` when `n_stream == 1`, else `split_equal` (`../../llama.cpp/src/llama-kv-cache.cpp#L713`). `llama_ubatch` holds `token/embd/pos/n_seq_id/seq_id/seq_id_unq/output` (`llama-batch.h#L15`).
|
||||
|
||||
**State and KV cache.** [Verified]
|
||||
- Interface: `llama_memory_i` {`init_batch`, `init_full`, `init_update`, `clear`, `seq_rm/seq_cp/seq_keep/seq_add/seq_div`, `seq_pos_min/max`, `state_write/read`, `memory_breakdown`} and `llama_memory_context_i` {`next`, `apply`, `get_ubatch`, `get_status`} (`../../llama.cpp/src/llama-memory.h#L51-L126`). Public C API `llama_memory_*` (`../../llama.cpp/include/llama.h#L739-L805`).
|
||||
- Factory: `llama_model::create_memory` (`../../llama.cpp/src/llama-model.cpp#L2232-L2700`) returns `nullptr` for encoder-only/diffusion/WavTokenizer archs (no cache), `llama_kv_cache_msa` (MiniMax-M3), `llama_kv_cache` (standard), `llama_kv_cache_dsa` / `dsa_iswa` (DeepSeek sparse attention), `llama_kv_cache_iswa` (two caches: full + sliding-window, `../../llama.cpp/src/llama-kv-cache-iswa.h`), `llama_kv_cache_dsv4`, `llama_memory_recurrent` (Mamba/RWKV: per-seq `r_l`/`s_l` state tensors, `n_rs_seq` rollback snapshots, `../../llama.cpp/src/llama-memory-recurrent.h#L70-L113`), `llama_memory_hybrid`, `hybrid_iswa`, `hybrid_idx`.
|
||||
- Layout: `llama_kv_cache` stores per-layer `k_l/v_l` tensors of `kv_size` cells (K contiguous per head, V optionally transposed `v_trans` for non-FA path), one **stream** per sequence when `unified=false` (`n_stream = n_seq_max`) or a single shared stream when unified (`../../llama.cpp/src/llama-kv-cache.cpp#L84`, `#L135-L152`). Cells are tracked by `llama_kv_cells` with `pos`, `shift`, `seq` bitset (`std::bitset<LLAMA_MAX_SEQ=256>`) and a `seq_pos` index (`../../llama.cpp/src/llama-kv-cells.h#L489-L524`). `find_slot(ubatch, cont)` (`llama-kv-cache.cpp#L898`) scans the ring buffer for free cells (contiguous or not; non-contiguous slots are expressed via `k_idxs/v_idxs` and `GGML_OP_SET_ROWS`), honoring SWA masking to free expired cells. There is **no paging** and no defragmentation step any more (`defrag_thold` marked deprecated in `llama_context_params`); eviction is explicit via `llama_memory_seq_rm`/context-shift (`seq_add` with K-shift graph, `update(do_shift)` at `#L817`).
|
||||
- Sharing across requests: `seq_cp` copies cells between sequences (full-buffer copy only for non-unified streams, `#L496-L506`); server prompt cache saves idle slot KV to RAM and restores by common prefix; `llama_state_seq_{get,set}_data(_ext)` with flags `SWA_ONLY`, `PARTIAL_ONLY`, `ON_DEVICE` (`../../llama.cpp/include/llama.h#L906-L916`), file variants `llama_state_seq_save_file/load_file`, whole-context `llama_state_get_data/set_data`.
|
||||
- KV dtypes: `type_k/type_v` (F16 default, quantized allowed), `offload_kqv`, `flash_attn_type`, `swa_full`, `kv_unified` in `llama_context_params`.
|
||||
|
||||
**Multimodal stages.** [Verified] Encoders are in `libmtmd` (`../../llama.cpp/tools/mtmd/`), not in `libllama`: `mtmd_init_from_file(mmproj, model, params)` loads a `clip_ctx` per modality (vision and/or audio, with own `ggml_backend_sched` and a chosen device `mtmd_context_params.device`, `../../llama.cpp/tools/mtmd/clip.cpp#L153-L202`). Pipeline (`../../llama.cpp/tools/mtmd/README-dev.md#L18`): bitmap (RGB or PCM) -> `mtmd_tokenize` (splits prompt into text/image/audio chunks, runs preprocessor: `mtmd_image_preprocessor_*` slicing/resizing in `mtmd-image.cpp`, mel spectrogram in `mtmd-audio.cpp`) -> `mtmd_encode_chunk`/`mtmd_batch_encode` (runs `clip_image_batch_encode`, graph built by `clip_get_graph_builder` per projector type at `clip.cpp#L930`, computed at `#L4437-L5749`) -> `mtmd_get_output_embd` -> `mtmd_helper_decode_image_chunk` feeds embeddings to `llama_decode` via `llama_batch.embd` (with M-RoPE 2D positions via `mtmd_decoder_pos`, non-causal attention toggled by `llama_set_causal_attn` for some models) (`../../llama.cpp/tools/mtmd/mtmd-helper.cpp#L100-L293`). Decoders/vocoders for TTS live in mtmd as "gen" graphs (Section 9). Diffusion LMs are driven by `../../llama.cpp/examples/diffusion/diffusion.cpp` on top of `llama_decode` with `llama_model_is_diffusion`.
|
||||
|
||||
## 5. Scheduler
|
||||
|
||||
### 5a. Request / token / pipeline-level scheduling
|
||||
|
||||
**Library level (libllama).** [Verified] `libllama` has no request scheduler: the caller composes a `llama_batch` with tokens from any number of sequences (`seq_id` per token, up to `n_seq_max`, `LLAMA_MAX_SEQ=256`), and `llama_decode` processes it as ubatches of at most `n_ubatch` tokens, sharing weights across sequences (this is the batching primitive). `examples/batched` and `examples/parallel` demonstrate multi-sequence decoding with `llama_memory_seq_cp` for shared prefixes (`../../llama.cpp/examples/parallel/parallel.cpp#L263-L307`). Cancellation inside a decode: `llama_set_abort_callback` (CPU backend `ggml_cplan.abort_callback` checked per node, `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c#L3150`; Metal also supports an abort callback, `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-context.m#L620`); other backends: [Unknown]/not observed. Backpressure: `llama_decode` returns `1` when no KV slot is found (`../../llama.cpp/src/llama-context.cpp#L1779`), leaving the caller to shrink or retry.
|
||||
|
||||
**Server level (tools/server).** [Verified]
|
||||
- Slots: `n_parallel` `server_slot`s, each bound to a `seq_id`, with `n_ctx_slot = n_ctx_seq` (or capped with `kv_unified_per_slot`) (`../../llama.cpp/tools/server/server-context.cpp#L239`, `#L1250-L1295`). States: `IDLE, WAIT_OTHER, STARTED, PROCESSING_PROMPT, DONE_PROMPT, GENERATING` (`#L100`).
|
||||
- Queue: `server_queue` (`../../llama.cpp/tools/server/server-queue.cpp`) is a mutex/condvar-protected deque; `post()`, `defer()` (task waits for a free slot), `pop_deferred_task(id_slot)`, `start_loop` runs `process_single_task` then `update_slots` on one worker thread; `yield_to_queue` lets `update_slots` interleave new tasks (`#L223`).
|
||||
- Continuous batching: `update_slots()` (`#L2777`) adds one sampled token per generating slot (plus speculative draft tokens) to a shared `llama_batch`, then fills remaining capacity up to `n_batch` with pending prompt tokens (`#L3087-L3129`), reusing common prefixes (`get_common_prefix`, `n_cache_reuse` KV shifting `#L3202-L3268`, context checkpoints), decodes in chunks of `n_batch` (`#L2853-L2873`; on KV-slot failure it halves `n_batch` and retries), and dispatches results per slot. Slots that cannot batch together (`can_batch_with`, e.g. different LoRA) are processed in separate iterations (`#L2973`, `#L3108`).
|
||||
- Prefill vs decode: no separate phases/engines; prompt tokens and generation tokens share ubatches (chunked prefill is natural since prompts are consumed `n_batch` at a time). Context shift when full (`n_discard`, `seq_rm` + `seq_add`, `#L2898-L2948`), disabled for multimodal.
|
||||
- Cancellation: `SERVER_TASK_TYPE_CANCEL` releases the slot (`#L2438`); `post()` drops pending tasks with the same id (`server-queue.cpp#L31`). HTTP disconnect during streaming ends the chunked provider.
|
||||
- Multi-stage: speculative decoding (draft model, EAGLE-3, DFlash, DSpark, n-gram variants; `../../llama.cpp/docs/speculative.md`, `../../llama.cpp/common/speculative.h`) runs draft + verify inside a slot; parent/child slots exist for `n > 1` completions (`launch_slots_with_parent_task`, `#L2264`). Router mode (`--models-dir`, `server-models.cpp`) spawns/unloads child `llama-server` processes per model and proxies requests (`../../llama.cpp/tools/server/README.md#L230-L233`, `#L1800-L2040`).
|
||||
- Backpressure: `n_parallel` bounds concurrency; excess tasks are deferred; `/slots` and `/metrics` expose state; `--cache-ram` bounds prompt-cache RAM.
|
||||
|
||||
**TTS pipeline scheduling.** [Verified] `mtmd_helper_gen_audio` (`../../llama.cpp/tools/mtmd/mtmd-helper-gen.cpp`) is a single-sequence stateful driver: `set_input` -> `step_prompt(n_batch)` loops -> per-frame `step_gen(sampled, h_state)` running GEN_CODE then buffered GEN_WAV flushes (`flush_gen_wav`, window of 72 frames for Qwen3-TTS `window_frames`, `#L400-L437`) -> `get_output`. No batching across TTS requests; the server does not expose it.
|
||||
|
||||
### 5b. Graph / operator / thread-level scheduling
|
||||
|
||||
**Graph split scheduling (`ggml_backend_sched`).** [Verified] `ggml_backend_sched_split_graph` (`../../llama.cpp/ggml/src/ggml-backend.cpp#L1066`) runs five passes: (1) assign nodes/leafs with pre-allocated tensors (weights -> their backend; graph inputs -> CPU; `op_offload` may move host-weight ops to a GPU if `offload_op` says so, `#L921-L995`); (2) expand assignments to adjacent nodes, GPU backends up/down but never expanding CPU (`#L1124`); (3) upgrade nodes to higher-priority backends when buffer types are compatible (`#L1204`); (4) assign remaining sources from destination/view_src (`#L1265`); (5) cut the node list into contiguous same-backend splits, record split inputs that need copies (`#L1297-L1423`), call each backend's `graph_optimize` (`#L1462`), and build a copy graph with copy nodes inserted (`#L1475-L1513`). `ggml_backend_sched_alloc_splits` allocates via `ggml_gallocr` with per-node buffer ids, reallocating only if needed. `ggml_backend_sched_compute_splits` (`#L1643`) executes splits in order, one async `graph_compute` per split, with event-based cross-backend synchronization and `cur_copy` rotation; `callback_eval` can observe intermediate tensors and abort. Op ordering within a split is the topological order of `ggml_build_forward_expand`; there is no reordering/fusion in the scheduler itself (fusion is backend-internal, e.g. CUDA/Metal fusing RMS_NORM+MUL, probed via `llm_fused_op_probe`).
|
||||
|
||||
**Threads (CPU backend).** [Verified] `ggml_threadpool` (`../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c#L481`) with `n_threads`, per-thread `cpumask`, priority (`GGML_SCHED_PRIO_{LOW,NORMAL,MEDIUM,HIGH,REALTIME}` mapped to SCHED_FIFO/OTHER on Linux/macOS, thread priority classes on Windows, `#L2557-L2700`), polling level, pause/resume (`ggml_threadpool_pause/resume`, `../../llama.cpp/ggml/include/ggml-cpu.h#L58-L62`), OpenMP mode when `GGML_OPENMP` (default ON; Android docs recommend OFF). Each op is split across threads by row ranges inside `ggml_compute_forward_*`, with `ggml_barrier` between ops (`#L576`). `llama_context` keeps separate `threadpool` and `threadpool_batch` (`n_threads` vs `n_threads_batch`) and sets them per compute (`../../llama.cpp/src/llama-context.cpp#L2492-L2510`); CLI exposes `--threads`, `--cpu-mask`, `--cpu-range`, `--prio`, `--poll` (`../../llama.cpp/tools/cli/README.md`). GPU backends use a single stream per device; Hexagon HTP has its own DSP worker pool (`htp/worker-pool.c` referenced in the Snapdragon doc build log).
|
||||
|
||||
**Stream assignment.** [Verified] One stream per backend; pipeline parallelism overlaps up to 4 ubatches across devices via copies/events. Backends may additionally use internal multi-streams ([Unknown] not audited per backend).
|
||||
|
||||
## 6. I/O layer
|
||||
|
||||
**Text tokenization.** [Verified] Built-in, no external library: `llama_vocab_type` {NONE, SPM, BPE, WPM, UGM, RWKV, PLAMO2} (`../../llama.cpp/include/llama.h#L73-L79`), implementations `llm_tokenizer_spm_session`, `llm_tokenizer_bpe_session` (with `unicode_regex_split` pre-tokenizer regexes per `LLAMA_VOCAB_PRE_TYPE_*`), `llm_tokenizer_wpm_session`, `llm_tokenizer_ugm_session` (naive trie + Viterbi-style), `llm_tokenizer_rwkv_session`, `llm_tokenizer_plamo2_session`, plus `hybriddna` and `whitespace` variants (`../../llama.cpp/src/llama-vocab.cpp#L110-L1723`). Unicode tables in `../../llama.cpp/src/unicode-data.cpp`. API: `llama_tokenize`, `llama_token_to_piece`, `llama_detokenize` (`../../llama.cpp/include/llama.h#L1172-L1200`), special tokens (BOS/EOS/EOT/SEP/PAD/MASK/FIM), `llama_vocab_is_eog`. Chat templates: in-tree Jinja engine (`../../llama.cpp/common/jinja/README.md`, not minja) used by `common/chat.cpp` with tool-call grammar constraining and PEG-based output parsers (`../../llama.cpp/docs/autoparser.md`, `../../llama.cpp/common/peg-parser.cpp`); `llama_chat_apply_template` C API with built-in templates. Grammar-constrained sampling: GBNF (`../../llama.cpp/grammars/README.md`, `../../llama.cpp/src/llama-grammar.cpp`), JSON-schema -> grammar, optional LLGuidance.
|
||||
|
||||
**Image preprocessing.** [Verified] Decoding via vendored `stb_image` (JPEG/PNG/...), WebP and video via `ffmpeg`/`ffprobe` subprocess (`MTMD_VIDEO` compile option, `../../llama.cpp/tools/mtmd/mtmd-helper.cpp#L36`, `#L419`, `../../llama.cpp/tools/mtmd/mtmd-helper.h#L23-L53`). Resizing with Pillow-compatible bilinear/bicubic (`img_tool::resize_pillow`, `../../llama.cpp/tools/mtmd/mtmd-image.cpp#L204`), smart-resize to patch multiples, padding, LLaVA-UHD/MiniCPM-V slicing (`mtmd_image_preprocessor_llava_uhd`, `#L485`), normalization with `image_mean/std` from the mmproj GGUF.
|
||||
|
||||
**Audio preprocessing.** [Verified] Decoding via vendored `miniaudio` (WAV/MP3/FLAC, resampled to model sample rate, `mtmd-helper.cpp#L23-L33`). Mel spectrogram from a whisper.cpp-derived implementation: Hann window, Cooley-Tukey FFT, mel filterbank matrix (`mtmd_audio_cache::fill_mel_filterbank_matrix`, `fft_impl`, `../../llama.cpp/tools/mtmd/mtmd-audio.cpp#L13-L268`). `mtmd_get_audio_sample_rate` reports e.g. 16000 for Whisper-type encoders. Audio encoders in mtmd: `whisper-enc.cpp`, `conformer.cpp`, `parakeet.cpp`, `granite-speech.cpp`, `qwen3a.cpp`, `mimo-audio.cpp`, `gemma4a.cpp/gemma4ua.cpp` (`../../llama.cpp/tools/mtmd/models/`).
|
||||
|
||||
**Streaming.** [Verified] Token streaming: server SSE via `httplib` chunked content provider (`../../llama.cpp/tools/server/server-http.cpp#L546-L576`), `server-stream.cpp` assembles OpenAI/Anthropic-style deltas; CLI prints pieces as sampled. Audio streaming output: `llama-tts` accumulates PCM and writes one WAV (`write_wav16`, `../../llama.cpp/tools/tts/tts.cpp#L53`); the helper API is frame-stepped (`step_gen` returns after each frame; GEN_WAV state carried via `state_data` so windows can be decoded incrementally, `mtmd.h#L426-L433`), so chunked audio streaming is achievable at the application layer [Inferred]. No audio-streaming HTTP endpoint exists.
|
||||
|
||||
**Buffering/postprocessing.** [Verified] Logits are returned only for tokens flagged in `llama_batch.logits`; output rows are reordered to batch order (`output_reorder`). Sampling: CPU sampler chain (greedy, dist, top-k/p, min-p, typical, temp, XTC, top-n-sigma, mirostat, penalties, DRY, adaptive-p, logit-bias, grammar, infill; `../../llama.cpp/include/llama.h#L1367-L1524`) or experimental backend samplers executed in-graph (`llama_set_sampler`, `llama_get_sampled_token_ith`; `llm_graph_input_sampling` in `llama-graph.h#L744`). Detokenization is incremental via `llama_token_to_piece`; server handles partial UTF-8. Vocoder/codec postprocessing for TTS is inside mtmd gen graphs (Section 9).
|
||||
|
||||
**Application boundaries.** [Verified]
|
||||
- C API: `../../llama.cpp/include/llama.h` (`llama_*`), `../../llama.cpp/tools/mtmd/mtmd.h` + `mtmd-helper.h` (`mtmd_*`), ggml headers.
|
||||
- C++ helpers: `../../llama.cpp/common/` (`common_params`, `common_sampler`, `common_chat_*`, `common_speculative_*`); `llama-cpp.h`/`ggml-cpp.h` RAII wrappers.
|
||||
- CLI: unified `llama` binary (`../../llama.cpp/app/llama.cpp`: `server`, `cli`, hidden `completion/bench/batched-bench/fit-params/quantize/perplexity/download`), `llama-cli` (`../../llama.cpp/tools/cli/`, which embeds a server context), `llama-completion`, `llama-mtmd-cli`, `llama-tts`, `llama-bench`, `llama-diffusion-cli`.
|
||||
- HTTP: `llama-server` endpoints `/completion`, `/tokenize`, `/detokenize`, `/apply-template`, `/embedding(s)`, `/reranking`, `/infill`, `/props`, `/slots`, `/metrics`, `/lora-adapters`, `/v1/models`, `/v1/completions`, `/v1/chat/completions` (+`/control`, `/input_tokens`), `/v1/responses`, `/v1/embeddings`, `/v1/messages` (Anthropic), `/v1/audio/transcriptions` (audio-input models only), `/models` (router mode load/unload/download/SSE), MCP tool servers, Web UI (`../../llama.cpp/tools/server/README.md#L473-L2040`, `../../llama.cpp/tools/server/server.cpp#L261`).
|
||||
- Mobile bindings: Android JNI + Kotlin `InferenceEngine`/`AiChat` with Kotlin `Flow` token stream and GGUF metadata reader (`../../llama.cpp/examples/llama.android/lib/src/main/cpp/ai_chat.cpp`, `../../llama.cpp/examples/llama.android/lib/src/main/java/com/arm/aichat/InferenceEngine.kt`, CMake enabling KleidiAI+OpenMP on arm64-v8a `../../llama.cpp/examples/llama.android/lib/src/main/cpp/CMakeLists.txt`); iOS/macOS/visionOS/tvOS `llama.xcframework` via `../../llama.cpp/build-xcframework.sh` and Swift sample `../../llama.cpp/examples/llama.swiftui/llama.cpp.swift/LibLlama.swift` (direct C API from Swift: `llama_batch_init`, `llama_decode`, `llama_sampler_chain_*`).
|
||||
- Python: only conversion/tooling (`gguf-py`); no runtime Python binding in tree (third-party `llama-cpp-python` is not vendored). [Verified] `../../llama.cpp/README.md` (no bindings list at this commit).
|
||||
- Distributed: `rpc-server` (`../../llama.cpp/tools/rpc/README.md`).
|
||||
|
||||
## 7. Execution flow
|
||||
|
||||
### (a) Text LLM decode (single sequence, e.g. `examples/simple`)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App as App (simple.cpp)
|
||||
participant L as libllama (llama-context.cpp)
|
||||
participant B as llama_batch_allocr
|
||||
participant M as llama_memory_i (KV cache)
|
||||
participant G as llm_graph_context / model.build_graph
|
||||
participant S as ggml_backend_sched
|
||||
participant BE as Backends (GPU..., CPU)
|
||||
App->>L: llama_backend_init(); ggml_backend_load_all()
|
||||
App->>L: llama_model_load_from_file(gguf, mparams)
|
||||
L->>BE: enumerate devices, place weights (mmap / upload)
|
||||
App->>L: llama_init_from_model(model, cparams)
|
||||
L->>M: model.create_memory()
|
||||
L->>S: ggml_backend_sched_new(); sched_reserve() (pp + tg worst-case graphs)
|
||||
App->>L: llama_tokenize(prompt)
|
||||
loop each llama_decode(batch)
|
||||
App->>L: llama_decode(ctx, batch)
|
||||
L->>B: balloc->init(batch) validate seq_id/pos/outputs
|
||||
L->>M: memory_update(false); init_batch(balloc, n_ubatch)
|
||||
M-->>L: memory context (ubatches + slot_info per ubatch)
|
||||
loop each ubatch
|
||||
L->>M: mctx->apply() (apply_ubatch: write cells)
|
||||
alt graph reusable (can_reuse)
|
||||
L->>L: reuse gf_res_prev
|
||||
else
|
||||
L->>G: build_graph(gparams)
|
||||
G-->>L: ggml_cgraph
|
||||
L->>S: sched_reset(); sched_alloc_graph(gf)
|
||||
S->>S: split_graph (assign backends, splits, copies)
|
||||
end
|
||||
L->>L: res->set_inputs(ubatch) (tokens, pos, kq_mask, k/v idxs)
|
||||
L->>S: ggml_backend_sched_graph_compute_async(gf)
|
||||
S->>BE: per split: copy inputs, graph_compute_async, record event
|
||||
L->>BE: ggml_backend_tensor_get_async(logits -> host)
|
||||
end
|
||||
App->>L: llama_sampler_sample(smpl, ctx, -1)
|
||||
L->>L: llama_synchronize(); apply sampler chain on logits
|
||||
App->>L: llama_token_to_piece(); next llama_batch_get_one(token)
|
||||
end
|
||||
```
|
||||
|
||||
### (b) TTS path (`llama-tts` with Qwen3-TTS or Pocket-TTS)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant T as tools/tts/tts.cpp
|
||||
participant H as mtmd_helper gen_audio pipeline (mtmd-helper-gen.cpp)
|
||||
participant L as libllama (backbone talker)
|
||||
participant MT as libmtmd (mtmd.cpp / clip.cpp)
|
||||
participant SPK as clip graph: speaker encoder
|
||||
participant GC as clip graph: GEN_CODE (code_predictor / flow net)
|
||||
participant GW as clip graph: GEN_WAV (code2wav / Mimi decoder)
|
||||
T->>L: load backbone GGUF (LLM_ARCH_QWEN3TTS / POCKETTTS)
|
||||
T->>MT: mtmd_init_from_file(mmproj) -> gen_audio_get_info(type, sample_rate)
|
||||
T->>MT: mtmd_helper_bitmap_init_from_file(speaker.wav/mp3) (miniaudio decode)
|
||||
T->>H: set_input(prompt, lang, speaker_ref, top_k/top_p/seed)
|
||||
H->>MT: mtmd_tokenize + encode speaker chunk
|
||||
MT->>SPK: ECAPA-TDNN (qwen3tts-spkenc) / Mimi encoder (pockettts-spkenc)
|
||||
SPK-->>H: speaker embedding row(s)
|
||||
loop step_prompt(n_batch) until 0
|
||||
H->>L: llama_decode(prompt tokens + speaker embd via llama_batch.embd)
|
||||
end
|
||||
T->>L: sampled = common_sampler_sample(); h_state = llama_get_embeddings_ith(-1)
|
||||
loop step_gen(sampled, h_state) until EOS / max frames
|
||||
H->>MT: mtmd_gen_audio_process(GEN_CODE, code0=sampled, embd=h_state)
|
||||
MT->>GC: code_predictor: 15 more codebooks (on-device top-k/top-p sampling) | flow-matching latent + EOS score
|
||||
GC-->>H: codes / feats, embd (next backbone input), is_eos
|
||||
H->>H: buffer codes; every window_frames (72) -> flush_gen_wav
|
||||
H->>MT: mtmd_gen_audio_process(GEN_WAV, codes, state_data)
|
||||
MT->>GW: code2wav (Qwen3-TTS) | Mimi/SEANet decoder (Pocket-TTS), state carried across calls
|
||||
GW-->>H: PCM samples + new state
|
||||
H->>L: llama_decode(next backbone input embedding)
|
||||
T->>L: sample next codec_0 token
|
||||
end
|
||||
T->>H: get_output() -> WAV bytes (write_wav16)
|
||||
```
|
||||
|
||||
## 8. Tests and examples
|
||||
|
||||
- [Verified] Unit tests: `../../llama.cpp/tests/` (71 `llama_build_and_test`/`llama_test` registrations in `../../llama.cpp/tests/CMakeLists.txt`), run with `ctest` after `cmake -B build -DLLAMA_BUILD_TESTS=ON && cmake --build build`; labels `main` (`ctest -L main`), e.g. `test-backend-ops.cpp` (per-op cross-backend correctness/`perf`/`grad`/`support` modes, CSV output used for `docs/ops.md`), `test-tokenizer-0/1-bpe/1-spm`, `test-quantize-fns/perf`, `test-alloc`, `test-batch-alloc`, `test-thread-safety`, `test-save-load-state`, `test-state-restore-fragmented`, `test-recurrent-state-rollback`, `test-mtmd-c-api.c`, `test-mtmd-impl.cpp`, `test-backend-sampler`, `test-rpc-multi-server`, `test-chat*`, `test-jinja`, `test-grammar*`, `test-llama-archs`. Debugging guide: `../../llama.cpp/docs/development/debugging-tests.md` (`scripts/debug-test.sh`). CONTRIBUTING requires running `test-backend-ops` after ggml changes (`../../llama.cpp/CONTRIBUTING.md#L48`).
|
||||
- [Verified] Server tests: pytest scenarios in `../../llama.cpp/tools/server/tests/` (`tests.sh`, `../../llama.cpp/tools/server/tests/README.md`).
|
||||
- [Verified] CI: `.github/workflows` includes `build-android.yml`, `build-apple.yml` (builds xcframework, `xcodebuild` of `llama.swiftui`, runs `ctest -L main`), `build-and-test-snapdragon.yml` (Hexagon/OpenCL presets), `build-opencl.yml` (Windows Arm64 Adreno), `build-vulkan.yml`, `build-webgpu.yml`, `build-cann.yml`, `build-openvino.yml`, `build-riscv.yml`, `build-wasm.yml`, `server.yml`, `update-ops-docs.yml`; self-hosted `ggml-ci` via `../../llama.cpp/ci/run.sh` (`../../llama.cpp/ci/README.md`).
|
||||
- [Verified] Examples relevant to edge/TTS/vision/audio: `../../llama.cpp/tools/tts/tts.cpp` (TTS), `../../llama.cpp/tools/mtmd/mtmd-cli.cpp` (vision/audio chat), `../../llama.cpp/examples/llama.android/` (Kotlin app + JNI lib), `../../llama.cpp/examples/llama.swiftui/` (iOS), `../../llama.cpp/examples/simple/simple.cpp`, `../../llama.cpp/examples/simple-chat/`, `../../llama.cpp/examples/batched/batched.cpp`, `../../llama.cpp/examples/parallel/parallel.cpp`, `../../llama.cpp/examples/embedding/`, `../../llama.cpp/examples/retrieval/`, `../../llama.cpp/examples/diffusion/diffusion-cli.cpp`, `../../llama.cpp/examples/speculative-simple/`, `../../llama.cpp/examples/eval-callback/` (tensor observation via `cb_eval`), `../../llama.cpp/tools/llama-bench/`, `../../llama.cpp/tools/batched-bench/`, `../../llama.cpp/tools/rpc/`. Multimodal model docs: `../../llama.cpp/docs/multimodal.md`, `../../llama.cpp/docs/multimodal/`.
|
||||
- [Verified] Snapdragon build helper: `../../llama.cpp/scripts/snapdragon/build.py` (Docker toolchain, `--push` to adb).
|
||||
|
||||
## 9. TTS relevance
|
||||
|
||||
**What exists.** [Verified]
|
||||
- Tool: `llama-tts` (`../../llama.cpp/tools/tts/tts.cpp`, README `../../llama.cpp/tools/tts/README.md`): `llama-tts -hf ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF -p "Hello world" --output out.wav`, options `--tts-lang` (zh/en/de/it/pt/es/ja/ko/fr/ru), `--tts-speaker-file` (wav/mp3 reference voice), `-n` frames, standard sampling and `-ngl/-b/-ub`. README states the tool "used to serve as a demo for OuteTTS, but it was converted to a more model-agnostic tool".
|
||||
- Supported pipelines (`enum mtmd_gen_audio_type`, `../../llama.cpp/tools/mtmd/mtmd.h#L370-L374`): `QWEN3TTS` (24 kHz; backbone `LLM_ARCH_QWEN3TTS` reusing the Qwen3-VL graph, `../../llama.cpp/src/models/qwen3tts.cpp`; speaker encoder ECAPA-TDNN `../../llama.cpp/tools/mtmd/models/qwen3tts-spkenc.cpp`; `code_predictor` producing 15 additional codebooks with on-graph top-k/top-p sampling `../../llama.cpp/tools/mtmd/models/qwen3tts-gen.cpp`; `code2wav` vocoder with 72-frame windows and carried state) and `POCKETTTS` (Kyutai Pocket-TTS: CALM backbone without lm_head `../../llama.cpp/src/models/pockettts.cpp`, flow-matching latent generator + EOS head `../../llama.cpp/tools/mtmd/models/pockettts-gen.cpp`, Mimi/SEANet codec encoder+decoder `pockettts-seanet.cpp`, `pockettts-spkenc.cpp`; requires a speaker file). Conversion classes: `../../llama.cpp/conversion/qwen3tts.py` (`Qwen3TTSTalkerModel`, `Qwen3TTSSpeakerEncoderModel`), `../../llama.cpp/conversion/pockettts.py`.
|
||||
- Legacy path: `LLM_ARCH_WAVTOKENIZER_DEC` (`../../llama.cpp/src/models/wavtokenizer-dec.cpp`: posnet/convnext decoder, group norm, conv1d) and `conversion/wavtokenizer.py` remain in libllama (OuteTTS-era vocoder), but no OuteTTS driver exists in `tools/tts` anymore. `LLM_ARCH_TALKIE` (`../../llama.cpp/src/models/talkie.cpp`) is a 13B text model class ([Unknown] its exact purpose; name suggests a speech LLM but nothing audio-specific is in its graph file header).
|
||||
- API: experimental, stateless core `mtmd_gen_audio_process(ctx, mtmd_gen_inp{type=GEN_CODE|GEN_WAV, code0, embd, top_k, top_p, seed, temp, codes/feats, state_data}, mtmd_gen_out{codes/feats, embd, is_eos, audio, n_samples, state_data})` (`../../llama.cpp/tools/mtmd/mtmd.h#L385-L438`), and stateful helper `mtmd_helper_gen_audio_{init,set_input,step_prompt,step_gen,get_output}` with a C++ `mtmd_helper::gen_audio` wrapper (`../../llama.cpp/tools/mtmd/mtmd-helper.h#L189-L291`). Design rules for porting new TTS models are documented in `../../llama.cpp/tools/mtmd/README-dev.md#L38-L90` (3-stage pipeline, sidecar models in mmproj, `a.gen.*` tensor prefixes, most logic in `mtmd-helper-gen.cpp`).
|
||||
- Audio ops available in ggml relevant to vocoders: `CONV_TRANSPOSE_1D`, `IM2COL`, `COL2IM_1D`, `PAD_REFLECT_1D`, `GROUP_NORM`, `CUMSUM`, `TOP_K`, `ARGSORT`, `SSM_CONV`, `TIMESTEP_EMBEDDING` (used by Pocket-TTS flow net time embedding). Backend coverage for these varies (see `../../llama.cpp/docs/ops.md`: e.g. `COL2IM_1D` only SYCL/Vulkan; `CONV_TRANSPOSE_1D` CPU/CANN/CUDA/Metal/SYCL/Vulkan; not OpenCL/HTP), so on Adreno/Hexagon those ops fall back to CPU.
|
||||
- Audio input for ASR/audio-LLMs: Whisper-style encoders, Qwen2/3-Audio, Ultravox, Voxtral, Granite-Speech, Parakeet, Gemma-4 audio, Qwen2.5/3-Omni (audio+vision input) (`../../llama.cpp/docs/multimodal.md#L108-L138`, projector enum `../../llama.cpp/tools/mtmd/clip-impl.h#L445-L507`).
|
||||
- Timing instrumentation: `llama-tts` reports prompt / generation / vocoder seconds and frames (`tts.cpp#L188-L201`).
|
||||
|
||||
**What is missing / limitations.**
|
||||
- [Verified] No streaming audio output in the CLI (single WAV at end) and no server endpoint for speech synthesis (`/v1/audio/speech` absent; only `/v1/audio/transcriptions`).
|
||||
- [Verified] Only two generation pipelines; OuteTTS driver removed; no Kokoro/CosyVoice/Orpheus/SNAC/Dia/Chatterbox etc. in tree.
|
||||
- [Verified] `mtmd_gen_audio` API is marked experimental / subject to breaking changes; hparams for code2wav are hard-coded in `clip.cpp` (`conversion/qwen3tts.py#L257`).
|
||||
- [Verified] Single-sequence TTS driver; no batching of multiple utterances, no cross-request scheduling; state (`c2w_state`) is a byte blob managed by the caller.
|
||||
- [Inferred] Latency-critical stage 2/3 graphs run through `clip_ctx`'s own `ggml_backend_sched` with per-call graph rebuild (`clip_image_batch_encode` resets and reallocates each call), so no graph reuse/CUDA-graph benefit on the vocoder path.
|
||||
- [Unknown] No published RTF/latency numbers for TTS in docs.
|
||||
|
||||
## 10. VLA relevance
|
||||
|
||||
**What exists.** [Verified]
|
||||
- Vision encoders: ~60 projector/encoder types in `clip.cpp` (`../../llama.cpp/tools/mtmd/clip-impl.h#L445-L507`), including SigLIP (`tools/mtmd/models/siglip.cpp`), Qwen2/2.5/3-VL mergers with M-RoPE, Gemma3/3n/4 (MobileNetV5 `mobilenetv5.cpp`), Pixtral, InternVL, Kimi-VL, LLaVA/LLaVA-UHD, MiniCPM-V, Idefics3/SmolVLM, LFM2-VL, PaddleOCR/DeepSeek-OCR, Llama-4, Janus-Pro, CogVLM, Step3-VL, Hunyuan-VL, Nemotron-V2-VL, YouTu-VL, MiMo-VL, Granite-Vision. Encoder-only use is possible via `mtmd_encode`/`clip_image_batch_encode` + `mtmd_get_output_embd` without decoding (embedding extraction), and `clip_init` can be called standalone.
|
||||
- Numeric / continuous inputs: `llama_batch.embd` lets an application inject arbitrary pre-computed input embeddings (rows of `n_embd_inp`) instead of tokens (`../../llama.cpp/include/llama.h#L262-L270`); `mtmd_bitmap_init_from_audio(n_samples, float*)` accepts raw float sequences (audio); no generic "sensor" tensor input.
|
||||
- Encoder-decoder and non-causal graphs: `llama_encode`, `llama_set_causal_attn`, T5 (`src/models/t5.cpp`), BERT-family embeddings, classification heads (`llama_model_n_cls_out`, `llama_model_cls_label`), pooling types.
|
||||
- Diffusion / iterative denoising in text: Dream / LLaDA / LLaDA-MoE / RND1 masked-diffusion LMs driven by `examples/diffusion` (`../../llama.cpp/examples/diffusion/README.md`; algorithms origin/entropy/margin/random/confidence; timestep or block scheduling). Flow matching in mtmd: Pocket-TTS `clip_graph_pockettts_gen` (`../../llama.cpp/tools/mtmd/models/pockettts-gen.cpp`, `time_embed`, `modulate` = adaLN-style conditioning) shows a working pattern for a small flow-matching head on top of a backbone hidden state, generating continuous 32-d latents per step.
|
||||
- Low-latency loop primitives: graph reuse + CUDA graphs for 1-token steps; backend samplers to avoid host round-trips; `llama_decode` on embeddings; state snapshot/restore (`llama_state_seq_*`); recurrent-state rollback (`n_rs_seq`); abort callback; threadpool pinning/priorities.
|
||||
- Training hooks: `llama_opt_init/llama_opt_epoch` and `ggml-opt.cpp` (AdamW/SGD) for fine-tuning in ggml (`../../llama.cpp/include/llama.h#L1609-L1625`).
|
||||
|
||||
**What is missing.** [Verified unless noted]
|
||||
- No VLA model architecture (no OpenVLA, pi0/pi0.5, SmolVLA, GR00T, RT-2 style heads): grep for `proprio|openvla|smolvla|pi0|action head|diffusion policy` finds nothing in `src/`, `tools/`, `conversion/`, `docs/`.
|
||||
- No action head abstraction: `libllama` outputs are logits (`n_vocab`), embeddings (`n_embd`/pooled), classification (`n_cls_out`), or backend-sampled tokens; a continuous action regression head would need a new arch class in `src/models/` or a sidecar graph in mtmd (as Pocket-TTS does).
|
||||
- No proprioception/state tokenizer or projector; `llama_batch.embd` requires the caller to compute embeddings externally.
|
||||
- No diffusion/flow policy sampler loop at library level (Pocket-TTS's is model-specific inside `mtmd-helper-gen.cpp`).
|
||||
- No structured multi-stage scheduler for a control loop (perception encoder -> backbone -> action decoder) beyond the sequential helper used for TTS; no fixed-rate/real-time scheduling.
|
||||
- [Inferred] Vision encoder latency on mobile GPUs depends on backend op coverage (`CONV_2D`, `IM2COL`, `UPSCALE`, `PAD`, 2D-RoPE); OpenCL/HTP have gaps in `docs/ops.md`, so parts of encoders may fall back to CPU.
|
||||
- [Proposal] A VLA port would: (1) convert the VLM backbone via `conversion/` + mmproj; (2) add a `PROJECTOR_TYPE_*` for the vision tower if new; (3) add a sidecar "action expert" graph in `tools/mtmd/models/` driven by a new `mtmd_gen_*`-style stateless process API and a helper driver mirroring `mtmd-helper-gen.cpp`; (4) feed proprio via `llama_batch.embd` rows after a small projector computed in the sidecar graph.
|
||||
|
||||
## 11. Edge deployment profile
|
||||
|
||||
- **Build system.** [Verified] CMake (`../../llama.cpp/CMakeLists.txt`, `../../llama.cpp/ggml/CMakeLists.txt`), presets `CMakePresets.json`, Nix flake, Makefile shim. Static builds via `-DBUILD_SHARED_LIBS=OFF` (`../../llama.cpp/docs/build.md#L63`, `#L335`), `GGML_STATIC`, `GGML_LTO`. Optional components can be disabled: `LLAMA_BUILD_{COMMON,TESTS,TOOLS,EXAMPLES,SERVER,APP,UI}`, `LLAMA_OPENSSL=OFF`, `LLAMA_SUBPROCESS`, `LLAMA_LLGUIDANCE`, `GGML_OPENMP`, `GGML_LLAMAFILE`, `GGML_CPU_REPACK`, `GGML_BACKEND_DL`. `LLAMA_BUILD_MTMD` builds libmtmd standalone (`../../llama.cpp/CMakeLists.txt#L256`).
|
||||
- **Dependencies.** [Verified] Core (`ggml` + `libllama`) has no external runtime dependencies beyond libc/libstdc++ and optional pthreads/OpenMP. Vendored: cpp-httplib, nlohmann/json, miniaudio, stb_image, sheredom (`../../llama.cpp/vendor/`). Optional: OpenSSL/BoringSSL (HTTPS downloads), CURL (not required; `download.cpp` uses httplib), ffmpeg binaries for video/WebP (runtime, subprocess), backend SDKs (CUDA, ROCm, Vulkan SDK + glslc at build time, OpenCL headers/ICD, Hexagon SDK, CANN, OpenVINO, Dawn for WebGPU, KleidiAI fetched when enabled).
|
||||
- **Binary size.** [Unknown] No size numbers in docs. [Inferred] CPU-only footprint is dominated by per-arch kernel variants (`GGML_CPU_ALL_VARIANTS` multiplies) and `unicode-data.cpp` (7 k lines of tables); Metal embeds shader source; Vulkan embeds SPIR-V for 180 shaders (times variants); OpenCL embeds 178 kernels; CUDA fat binaries for multiple archs are large (limit `CMAKE_CUDA_ARCHITECTURES`).
|
||||
- **Platforms.**
|
||||
- [Verified] Apple Silicon: Metal on by default (`GGML_METAL_DEFAULT ON` when `APPLE`, `../../llama.cpp/ggml/CMakeLists.txt#L95`), Accelerate/BLAS; iOS 16.4+, macOS 13.3+, visionOS 1.0+, tvOS 16.4+ xcframework (`../../llama.cpp/build-xcframework.sh#L7-L10`), `GGML_METAL_EMBED_LIBRARY` for app bundles. No CoreML/ANE backend.
|
||||
- [Verified] Android: NDK arm64-v8a cross-compile with `GGML_NATIVE=OFF GGML_OPENMP=OFF GGML_LLAMAFILE=OFF LLAMA_OPENSSL=OFF` (`../../llama.cpp/docs/android.md`), KleidiAI for SME2/i8mm/dotprod, OpenCL for Adreno (`../../llama.cpp/docs/build.md#L712-L760`), Hexagon HTP NPU via Snapdragon toolchain containers with `GGML_HEXAGON=ON GGML_OPENCL=ON` preset (`../../llama.cpp/docs/backend/snapdragon/README.md`), Vulkan (generic). No NNAPI, no QNN; Adreno prebuilt kernel library is a manual download.
|
||||
- [Verified] Embedded NVIDIA (Jetson): CUDA backend; the CMake default arch list does not include sm_87 explicitly (`../../llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt#L28-L54`), so a `native` build on-device or explicit `-DCMAKE_CUDA_ARCHITECTURES=87` is needed ([Inferred]). `GGML_CUDA_GRAPHS` benefits small-batch decode. Unified memory documented (`../../llama.cpp/docs/build.md#L285`).
|
||||
- [Verified] Generic ARM Linux CPU: NEON/dotprod/i8mm/SVE/SME runtime dispatch, repack, KleidiAI; RISC-V RVV incl. SpacemiT (`../../llama.cpp/docs/build-riscv64-spacemit.md`); s390x; LoongArch.
|
||||
- [Verified] Windows on Arm (Snapdragon X): OpenCL Adreno and Hexagon docs (`../../llama.cpp/docs/backend/snapdragon/windows.md`).
|
||||
- [Verified] Browser/WASM: WebGPU via Emscripten (`../../llama.cpp/docs/build.md#L806-L821`), `build-wasm.yml`.
|
||||
- **Model conversion path and constraints.** [Verified] HF safetensors -> `convert_hf_to_gguf.py` (per-arch class must exist in `conversion/`; unsupported archs fail) -> optional `llama-quantize` (with `llama-imatrix` for IQ/low-bit types) -> optional `gguf-split`. Multimodal encoders need a separate `--mmproj` conversion; TTS sidecars (code2wav, Mimi) must live in the mmproj GGUF (`README-dev.md#L61-L80`). Pre-quantized GGUFs downloadable with `-hf` (HF cache in `common/hf-cache.cpp`). Constraints: no ONNX/TorchScript import; tokenizer must be one of the supported vocab types with a recognized pre-tokenizer (`convert_hf_to_gguf_update.py` maintains hashes); quantized weight repacking happens at load (`use_extra_bufts`), not offline, and is CPU-only.
|
||||
- **Memory fit tooling.** [Verified] `llama-fit-params` / `common/fit.cpp` and `no_alloc` metadata-only loads (`llama_model_params.no_alloc`) estimate memory before allocating; `llama_get_memory_breakdown` (`../../llama.cpp/src/llama-ext.h#L91`).
|
||||
- **Runtime knobs for edge.** [Verified] `-ngl`, `--device`, `-t/-tb`, `--cpu-mask/--prio/--poll`, `-c` (KV size), `-ctk/-ctv` (KV quant), `-fa`, `-b/-ub`, `--load-mode` (mmap/mlock/none), `--override-tensor`, `--no-repack`/`use_extra_bufts`, `--kv-unified`.
|
||||
|
||||
## 12. Limitations and unknowns
|
||||
|
||||
1. [Verified] No paged/blocked KV cache; sequences occupy fixed slots in a ring buffer (`n_ctx_seq` each unless unified); long-context multi-tenant serving relies on prompt cache save/restore and context shift.
|
||||
2. [Verified] No dedicated prefill/decode disaggregation or priority scheduling in `llama-server`; single worker thread, FIFO with deferral.
|
||||
3. [Verified] Cancellation within a running `llama_decode` only works on CPU (and Metal via abort callback); GPU backends run the whole split.
|
||||
4. [Verified] Graph is rebuilt per ubatch unless reusable; `clip.cpp` encoder/generator graphs are rebuilt and reallocated on every call.
|
||||
5. [Verified] TTS API is experimental; only Qwen3-TTS and Pocket-TTS; CLI-only; no streaming HTTP; code2wav hparams hard-coded in `clip.cpp`.
|
||||
6. [Verified] No VLA/action-head/proprioception support; continuous inputs only via `llama_batch.embd`.
|
||||
7. [Verified] Op coverage on mobile accelerators is partial (OpenCL, HTP columns in `docs/ops.md`); vocoder/conv ops frequently fall back to CPU on those backends.
|
||||
8. [Verified] No NNAPI/QNN/CoreML/TensorRT backends; Hexagon requires the proprietary Hexagon SDK and signed DSP libraries; Adreno binary kernel lib is a manual download.
|
||||
9. [Verified] Python is conversion-only; runtime bindings are out of tree.
|
||||
10. [Unknown] Binary sizes, per-platform performance, TTS real-time factor, Hexagon vs OpenCL relative speed: no numbers in this tree (benchmarks in `../../llama.cpp/benches/` were not audited).
|
||||
11. [Unknown] Whether pipeline-parallel copies (`n_copies=4`) apply on single-device mobile builds: [Inferred] no, since `pipeline_parallel` requires more than one non-CPU device (`llama-context.cpp#L428`).
|
||||
12. [Inferred] The `talkie` and `muse-glimmer` archs are recent additions whose exact application is not documented in-tree.
|
||||
13. [Unknown] ET backend (`ggml-et`) maturity and availability of hardware.
|
||||
|
||||
## 13. Reference index
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `../../llama.cpp/AGENTS.md` | Contributor/agent policy (read-first) |
|
||||
| `../../llama.cpp/README.md` | Backend table, docs index |
|
||||
| `../../llama.cpp/CMakeLists.txt` | Top-level build options (`LLAMA_BUILD_*`, `LLAMA_BUILD_MTMD`) |
|
||||
| `../../llama.cpp/ggml/CMakeLists.txt` | ggml backend/SIMD options (`GGML_*`) |
|
||||
| `../../llama.cpp/ggml/include/ggml.h` | tensor types, `ggml_op` enum, custom ops |
|
||||
| `../../llama.cpp/ggml/include/ggml-backend.h` | backend/device/buffer/sched public API |
|
||||
| `../../llama.cpp/ggml/include/ggml-cpu.h` | CPU backend API, threadpool, feature queries |
|
||||
| `../../llama.cpp/ggml/src/ggml.c` | tensor/graph core |
|
||||
| `../../llama.cpp/ggml/src/ggml-backend.cpp` | `ggml_backend_sched` (split/assign/compute) |
|
||||
| `../../llama.cpp/ggml/src/ggml-backend-impl.h` | backend vtables, `GGML_BACKEND_DL_IMPL` |
|
||||
| `../../llama.cpp/ggml/src/ggml-backend-reg.cpp` | registry, dynamic loading |
|
||||
| `../../llama.cpp/ggml/src/ggml-backend-meta.cpp` | meta device (tensor parallel) |
|
||||
| `../../llama.cpp/ggml/src/ggml-backend-dl.h` | dlopen helpers |
|
||||
| `../../llama.cpp/ggml/src/ggml-alloc.c` | graph/tensor allocators |
|
||||
| `../../llama.cpp/ggml/src/ggml-quants.c` | reference quant/dequant |
|
||||
| `../../llama.cpp/ggml/src/gguf.cpp` | GGUF reader/writer |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c` | threadpool, `type_traits_cpu`, graph compute |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/ggml-cpu.cpp` | CPU device/reg, extra buffer types, `supports_op` |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/ops.cpp` | CPU op kernels incl. mul_mat quantized path |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/repack.cpp` | runtime weight repacking (Q4_0x4/x8, Q4_Kx8, ...) |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/traits.h` | `extra_buffer_type`/`tensor_traits` interfaces |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/kleidiai/kleidiai.cpp` | Arm KleidiAI microkernel integration |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/amx/amx.cpp` | Intel AMX buffer type |
|
||||
| `../../llama.cpp/ggml/src/ggml-cpu/llamafile/sgemm.cpp` | tinyBLAS sgemm |
|
||||
| `../../llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu` | CUDA backend, `supports_op`, CUDA graphs |
|
||||
| `../../llama.cpp/ggml/src/ggml-cuda/CMakeLists.txt` | CUDA arch selection |
|
||||
| `../../llama.cpp/ggml/src/ggml-cuda/mmq.cuh` | quantized matmul (int8 MMA) |
|
||||
| `../../llama.cpp/ggml/src/ggml-cuda/mmvq.cu` | quantized matvec |
|
||||
| `../../llama.cpp/ggml/src/ggml-metal/ggml-metal.cpp` | Metal device/backend registration |
|
||||
| `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-device.m` | Metal library compile / metallib load |
|
||||
| `../../llama.cpp/ggml/src/ggml-metal/ggml-metal-context.m` | Metal command encoding, abort callback |
|
||||
| `../../llama.cpp/ggml/src/ggml-metal/kernels/` | Metal shader sources |
|
||||
| `../../llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp` | Vulkan backend |
|
||||
| `../../llama.cpp/ggml/src/ggml-vulkan/CMakeLists.txt` | shader generation at build |
|
||||
| `../../llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp` | GLSL->SPIR-V generator |
|
||||
| `../../llama.cpp/ggml/src/ggml-opencl/ggml-opencl.cpp` | OpenCL/Adreno backend |
|
||||
| `../../llama.cpp/ggml/src/ggml-opencl/cl-program-cache.cpp` | OpenCL program binary cache |
|
||||
| `../../llama.cpp/ggml/src/ggml-opencl/kernels/` | OpenCL kernels |
|
||||
| `../../llama.cpp/ggml/src/ggml-hexagon/ggml-hexagon.cpp` | Hexagon HTP host-side backend |
|
||||
| `../../llama.cpp/ggml/src/ggml-hexagon/htp/` | DSP-side HVX/HMX kernels |
|
||||
| `../../llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp` | RPC backend protocol |
|
||||
| `../../llama.cpp/ggml/src/ggml-opt.cpp` | optimizer/training |
|
||||
| `../../llama.cpp/include/llama.h` | public C API |
|
||||
| `../../llama.cpp/src/llama.cpp` | model load entry, device selection |
|
||||
| `../../llama.cpp/src/llama-ext.h` | extended (non-stable) API |
|
||||
| `../../llama.cpp/src/llama-model-loader.cpp` | GGUF loading, splits, mmap |
|
||||
| `../../llama.cpp/src/llama-mmap.cpp` | mmap/mlock |
|
||||
| `../../llama.cpp/src/llama-model.cpp` | tensors, buffer placement, `create_memory`, `build_graph` |
|
||||
| `../../llama.cpp/src/models/models.h` | per-arch model classes |
|
||||
| `../../llama.cpp/src/models/qwen3tts.cpp` | Qwen3-TTS backbone |
|
||||
| `../../llama.cpp/src/models/pockettts.cpp` | Pocket-TTS backbone |
|
||||
| `../../llama.cpp/src/models/wavtokenizer-dec.cpp` | WavTokenizer decoder (legacy vocoder) |
|
||||
| `../../llama.cpp/src/models/talkie.cpp` | Talkie arch |
|
||||
| `../../llama.cpp/src/models/t5.cpp` | encoder-decoder example |
|
||||
| `../../llama.cpp/src/llama-arch.h` | arch enum |
|
||||
| `../../llama.cpp/src/llama-context.cpp` | context, decode/encode, sched, state |
|
||||
| `../../llama.cpp/src/llama-batch.h` | ubatch splitting |
|
||||
| `../../llama.cpp/src/llama-batch.cpp` | batch allocator |
|
||||
| `../../llama.cpp/src/llama-memory.h` | memory interface |
|
||||
| `../../llama.cpp/src/llama-kv-cache.h` | KV cache class |
|
||||
| `../../llama.cpp/src/llama-kv-cache.cpp` | KV cache impl (find_slot, apply_ubatch, shift) |
|
||||
| `../../llama.cpp/src/llama-kv-cells.h` | cell bookkeeping |
|
||||
| `../../llama.cpp/src/llama-kv-cache-iswa.h` | SWA dual cache |
|
||||
| `../../llama.cpp/src/llama-memory-recurrent.h` | recurrent state memory |
|
||||
| `../../llama.cpp/src/llama-graph.h` | graph inputs, params, reuse |
|
||||
| `../../llama.cpp/src/llama-graph.cpp` | graph building blocks |
|
||||
| `../../llama.cpp/src/llama-vocab.cpp` | tokenizers |
|
||||
| `../../llama.cpp/src/unicode-data.cpp` | unicode tables |
|
||||
| `../../llama.cpp/src/llama-sampler.cpp` | samplers |
|
||||
| `../../llama.cpp/src/llama-grammar.cpp` | GBNF grammar sampling |
|
||||
| `../../llama.cpp/src/llama-quant.cpp` | quantization |
|
||||
| `../../llama.cpp/src/llama-cparams.h` | `LLAMA_MAX_SEQ` |
|
||||
| `../../llama.cpp/common/speculative.h` | speculative decoding API |
|
||||
| `../../llama.cpp/common/chat.h` | chat templates / tool calls |
|
||||
| `../../llama.cpp/common/jinja/README.md` | Jinja engine |
|
||||
| `../../llama.cpp/common/peg-parser.cpp` | PEG output parser |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd.h` | mtmd C API incl. audio generation |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd-helper.h` | helper API (eval chunks, gen_audio) |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd.cpp` | tokenizer/encoder glue, `mtmd_gen_audio_process` |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd-helper.cpp` | media decoding, `llama_decode` chunk driver |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd-helper-gen.cpp` | TTS stateful pipelines |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd-audio.cpp` | mel/FFT preprocessing |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd-image.cpp` | image preprocessing |
|
||||
| `../../llama.cpp/tools/mtmd/clip.cpp` | encoder graphs and execution |
|
||||
| `../../llama.cpp/tools/mtmd/clip.h` | clip API |
|
||||
| `../../llama.cpp/tools/mtmd/clip-impl.h` | projector type enum |
|
||||
| `../../llama.cpp/tools/mtmd/models/` | per-model encoder/generator graphs |
|
||||
| `../../llama.cpp/tools/mtmd/models/qwen3tts-gen.cpp` | code predictor + on-graph sampling |
|
||||
| `../../llama.cpp/tools/mtmd/models/qwen3tts-spkenc.cpp` | ECAPA-TDNN speaker encoder |
|
||||
| `../../llama.cpp/tools/mtmd/models/pockettts-gen.cpp` | flow-matching latent generator |
|
||||
| `../../llama.cpp/tools/mtmd/models/pockettts-seanet.cpp` | Mimi/SEANet codec |
|
||||
| `../../llama.cpp/tools/mtmd/models/whisper-enc.cpp` | Whisper encoder |
|
||||
| `../../llama.cpp/tools/mtmd/models/siglip.cpp` | SigLIP vision encoder |
|
||||
| `../../llama.cpp/tools/mtmd/models/mobilenetv5.cpp` | Gemma3n vision tower |
|
||||
| `../../llama.cpp/tools/mtmd/mtmd-cli.cpp` | multimodal CLI |
|
||||
| `../../llama.cpp/tools/mtmd/README.md` | multimodal usage |
|
||||
| `../../llama.cpp/tools/mtmd/README-dev.md` | libmtmd design + TTS porting checklist |
|
||||
| `../../llama.cpp/tools/tts/tts.cpp` | TTS CLI |
|
||||
| `../../llama.cpp/tools/tts/README.md` | TTS usage |
|
||||
| `../../llama.cpp/tools/server/server.cpp` | route registration |
|
||||
| `../../llama.cpp/tools/server/server-context.cpp` | slots, `update_slots`, endpoints impl |
|
||||
| `../../llama.cpp/tools/server/server-queue.cpp` | task queue |
|
||||
| `../../llama.cpp/tools/server/server-http.cpp` | HTTP/SSE |
|
||||
| `../../llama.cpp/tools/server/server-models.cpp` | router mode |
|
||||
| `../../llama.cpp/tools/server/README.md` | server docs/endpoints |
|
||||
| `../../llama.cpp/tools/server/tests/README.md` | server pytest |
|
||||
| `../../llama.cpp/tools/cli/README.md` | CLI args |
|
||||
| `../../llama.cpp/tools/rpc/rpc-server.cpp` | RPC server |
|
||||
| `../../llama.cpp/tools/rpc/README.md` | RPC usage |
|
||||
| `../../llama.cpp/app/llama.cpp` | unified `llama` binary |
|
||||
| `../../llama.cpp/examples/simple/simple.cpp` | minimal decode loop |
|
||||
| `../../llama.cpp/examples/batched/batched.cpp` | multi-sequence batching |
|
||||
| `../../llama.cpp/examples/parallel/parallel.cpp` | serving simulation |
|
||||
| `../../llama.cpp/examples/diffusion/diffusion-cli.cpp` | diffusion LM driver |
|
||||
| `../../llama.cpp/examples/diffusion/README.md` | diffusion params |
|
||||
| `../../llama.cpp/examples/llama.android/lib/src/main/cpp/ai_chat.cpp` | Android JNI |
|
||||
| `../../llama.cpp/examples/llama.android/lib/src/main/cpp/CMakeLists.txt` | Android lib build |
|
||||
| `../../llama.cpp/examples/llama.android/lib/src/main/java/com/arm/aichat/InferenceEngine.kt` | Kotlin API |
|
||||
| `../../llama.cpp/examples/llama.swiftui/llama.cpp.swift/LibLlama.swift` | Swift wrapper |
|
||||
| `../../llama.cpp/examples/llama.swiftui/README.md` | iOS sample |
|
||||
| `../../llama.cpp/build-xcframework.sh` | Apple xcframework build |
|
||||
| `../../llama.cpp/docs/xcframework.md` | xcframework usage |
|
||||
| `../../llama.cpp/docs/build.md` | build docs (all backends, KleidiAI, OpenCL, Android) |
|
||||
| `../../llama.cpp/docs/android.md` | Android build |
|
||||
| `../../llama.cpp/docs/ops.md` | op support matrix |
|
||||
| `../../llama.cpp/docs/ops/CPU.csv` | CPU op CSV |
|
||||
| `../../llama.cpp/docs/ops/Vulkan.csv` | Vulkan op CSV |
|
||||
| `../../llama.cpp/docs/multimodal.md` | multimodal models |
|
||||
| `../../llama.cpp/docs/multi-gpu.md` | split modes, tensor parallel |
|
||||
| `../../llama.cpp/docs/speculative.md` | speculative decoding |
|
||||
| `../../llama.cpp/docs/backend/OPENCL.md` | Adreno OpenCL |
|
||||
| `../../llama.cpp/docs/backend/snapdragon/README.md` | Hexagon/Snapdragon build |
|
||||
| `../../llama.cpp/docs/backend/snapdragon/windows.md` | Windows on Snapdragon |
|
||||
| `../../llama.cpp/docs/backend/snapdragon/developer.md` | Hexagon dev notes |
|
||||
| `../../llama.cpp/docs/backend/ET.md` | ET backend |
|
||||
| `../../llama.cpp/docs/backend/SYCL.md` | SYCL |
|
||||
| `../../llama.cpp/docs/backend/VirtGPU.md` | VirtGPU |
|
||||
| `../../llama.cpp/docs/build-riscv64-spacemit.md` | RISC-V |
|
||||
| `../../llama.cpp/docs/development/HOWTO-add-model.md` | adding models |
|
||||
| `../../llama.cpp/docs/development/debugging-tests.md` | test debugging |
|
||||
| `../../llama.cpp/docs/autoparser.md` | output parser |
|
||||
| `../../llama.cpp/tests/CMakeLists.txt` | test registration |
|
||||
| `../../llama.cpp/tests/test-backend-ops.cpp` | per-op backend tests |
|
||||
| `../../llama.cpp/ci/README.md` | CI overview |
|
||||
| `../../llama.cpp/ci/run.sh` | ggml-ci script |
|
||||
| `../../llama.cpp/convert_hf_to_gguf.py` | conversion entry |
|
||||
| `../../llama.cpp/conversion/base.py` | conversion base classes |
|
||||
| `../../llama.cpp/conversion/qwen3tts.py` | Qwen3-TTS conversion |
|
||||
| `../../llama.cpp/conversion/pockettts.py` | Pocket-TTS conversion |
|
||||
| `../../llama.cpp/conversion/wavtokenizer.py` | WavTokenizer conversion |
|
||||
| `../../llama.cpp/gguf-py/gguf/gguf_writer.py` | GGUF writer |
|
||||
| `../../llama.cpp/gguf-py/gguf/constants.py` | GGUF keys/arch constants |
|
||||
| `../../llama.cpp/gguf-py/README.md` | gguf-py docs |
|
||||
| `../../llama.cpp/scripts/create_ops_docs.py` | ops.md generator |
|
||||
| `../../llama.cpp/scripts/snapdragon/build.py` | Snapdragon build helper |
|
||||
| `../../llama.cpp/grammars/README.md` | GBNF guide |
|
||||
| `../../llama.cpp/vendor/` | vendored third-party libs |
|
||||
| `../../llama.cpp/CONTRIBUTING.md` | contribution/testing rules |
|
||||
@@ -0,0 +1,561 @@
|
||||
# MNN (alibaba/MNN) — Engine Analysis
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Local path | `/data/zhoutaichang/embedding_infer/MNN` |
|
||||
| Remote | `https://github.com/alibaba/MNN.git` (origin) |
|
||||
| Branch / HEAD | `master` @ `bef71b9756a2c77549eddbe33eb97290e3b16602` (2026-09-04, "[Vulkan:Perf] Optimize INT4 cooperative matrix path") |
|
||||
| Dirty status | clean (`git status --porcelain` empty) |
|
||||
| Submodules | none (`.gitmodules` absent; third-party code is vendored under `3rd_party/` or fetched by CMake at build time — see §11) |
|
||||
| Engine version | 3.6.1 ([MNNDefine.h](../../MNN/include/MNN/MNNDefine.h)) |
|
||||
| Analysis date | 2026-09-08 |
|
||||
| Reading restrictions | Per [AGENTS.md](../../MNN/AGENTS.md), `schema/private/` and `source/internal/` were **not** read or cited. |
|
||||
|
||||
Evidence labels: **[Verified]** = code/doc read; **[Inferred]** = reasoned from structure; **[Proposal]** = design suggestion; **[Unknown]** = could not determine. Links are relative to `analysis/engines/`.
|
||||
|
||||
---
|
||||
|
||||
## 0. Summary
|
||||
|
||||
- **What it is** [Verified]: MNN is a C++11 inference engine (no training focus, `-fno-rtti -fno-exceptions`) with two APIs — the low-level Session API (`Interpreter → createSession → runSession`) and the recommended Express/Module API (`Module::load → onForward(VARP)`) — plus a separate transformer stack (`transformers/llm/engine`, `transformers/diffusion/engine`) built on top of Module. See [AGENTS.md](../../MNN/AGENTS.md), [Interpreter.hpp](../../MNN/include/MNN/Interpreter.hpp), [Module.hpp](../../MNN/include/MNN/expr/Module.hpp).
|
||||
- **Execution runtime vs serving stack** [Verified]: the core is a single-graph, single-request runtime. There is no request scheduler, no continuous batching, no paging. Serving is thin: `mls_server` (`POST /chat/completions`, SSE) and `apps/mnncli` (`/v1/chat/completions`, `/v1/messages`) wrap one `Llm` behind a mutex ([mls_server.cpp](../../MNN/transformers/llm/engine/app/mls_server.cpp), [mnncli_server.cpp](../../MNN/apps/mnncli/src/mnncli_server.cpp)).
|
||||
- **Model format** [Verified]: FlatBuffers `.mnn` ([MNN.fbs](../../MNN/schema/default/MNN.fbs), 186 `OpType` enumerators) with large weights split into a sidecar `.mnn.weight` addressed by `Op.externalPath` + per-op `external:[offset,size,...]`. Converter frontends: ONNX, TF, TFLite, Caffe, TorchScript, MNN-JSON ([cli.cpp](../../MNN/tools/converter/source/common/cli.cpp)).
|
||||
- **Backends in tree** [Verified]: 16 dirs under [source/backend](../../MNN/source/backend): `cpu` (+`arm82` fp16, x86 AVX2/AVX512-VNNI, RISC-V RVV, KleidiAI, SME2, bf16), `metal`, `opencl`, `vulkan` (image or buffer variant chosen at build), `opengl`, `cuda`, `musa`, `tensorrt`, `coreml`, `nnapi`, `qnn`, `hiai`, `hexagon`, `rknn` (plugin op), `neuropilot` (stub only in OSS). No LoongArch, no WebGPU.
|
||||
- **Heterogeneous fallback** [Verified]: every non-CPU `Session` carries a CPU "backup" backend; per-op `Backend::onCreate` returning `nullptr` triggers a retry on CPU inside `_createExecutions` ([Pipeline.cpp#L583-L596](../../MNN/source/core/Pipeline.cpp#L583)), with `WrapExecution` inserting layout/device copies. Only works when the runtime's `CompilerType` is `Compiler_Loop` (full geometry decomposition); NPU-style backends (`CoreML`, `NNAPI`, `QNN`, `HIAI`, `TensorRT`) compile the whole subgraph and are exempt from wrapping.
|
||||
- **Kernel dispatch** [Verified]: per-backend `addCreator(OpType, Creator*)` maps; CPU registers 75 op types, Metal 36, OpenCL 39 (×buffer/image), Vulkan buffer 35, CUDA ~44. Geometry decomposition lowers shape-manipulation ops into `OpType_Raster` regions so backends only need a small compute subset ([GeometryComputer.cpp](../../MNN/source/geometry/GeometryComputer.cpp), [docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)).
|
||||
- **LLM quantization** [Verified]: exporter default W4 block-64 asymmetric with fp16 scales; W2/W3/W4/W8/FP16 supported (W2/W3 need ARMv8.6 i8mm); optional HQQ/AWQ/GPTQ/OmniQuant/SmoothQuant ([llmexport.py](../../MNN/transformers/llm/export/llmexport.py)). Runtime dequant is fused into the GEMM under `MNN_LOW_MEMORY` (weight-dequant float GEMM with `memory=normal`, or dynamic-int8-activation GEMM with `memory=low`) ([ConvolutionFloatFactory.cpp](../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp)). KV cache: int8 / TQ3 / TQ4 on CPU, int8 on Metal, optional mmap-to-disk.
|
||||
- **LLM engine** [Verified]: `Llm` (text), `Omni : Embedding : Llm` (vision/audio/talker), `Talker : Llm` (speech). Decode is a plain autoregressive loop in `ArGeneration::generate` ([generate.cpp#L44](../../MNN/transformers/llm/engine/src/speculative_decoding/generate.cpp#L44)) with optional lookahead / MTP / EAGLE / DFlash speculative strategies. Modules are cloned per (seq-len-bucket, all_logits) key to avoid resizing (`mPrefillKey=100`, decode key 1).
|
||||
- **TTS is first-class** [Verified]: (1) in-engine Qwen2.5-Omni thinker→talker→DiT(flow ODE)→BigVGAN and Qwen3-TTS talker→code-predictor→speech-decoder (single waveform callback after all frames, `Talker::generateQwen3TTS` in `omni.cpp`) while Qwen2.5-Omni streams **chunked PCM** through `setWavformCallback(const float*, size_t, bool last)` and a two-thread async DiT/vocoder pipeline ([omni.cpp](../../MNN/transformers/llm/engine/src/omni.cpp)); (2) a native `MNNTTSSDK` (Bert-VITS2, Supertonic, Piper) under [apps/frameworks/mnn_tts](../../MNN/apps/frameworks/mnn_tts); (3) a full sherpa-onnx port (VITS/Kokoro/Matcha/HiFiGAN + Zipformer ASR) under [apps/frameworks/sherpa-mnn](../../MNN/apps/frameworks/sherpa-mnn).
|
||||
- **VLA is absent** [Verified]: exhaustive grep for VLA/OpenVLA/π0/SmolVLA/LeRobot/diffusion-policy/proprio/action-head yields zero real hits. Building blocks exist: 16 vision-encoder exporters (Qwen2/2.5/3/3.5-VL, Gemma3/4, InternVL, MiniCPM-V, SmolVLM/Idefics3, LFM2-VL, MobileCLIP, ...), M-RoPE, DiT/flow-matching runtime (SD3.5/Sana/Wan2.1 T2V, token2wav DiT).
|
||||
- **Edge profile** [Verified]: minimal `libMNN.so` links only pthread/dl (Linux) or log/m/android (Android); OpenCL/Vulkan are `dlopen`ed by default; repo-reported core size ≈ 800 KB (Android armv7a, `c++_shared`), 12 MB full iOS static lib; `MNN_BUILD_MINI` strips geometry/shape for fixed-shape models. Platforms with build scripts: Android (NDK, API ≥21 arm64 / 14 armv7), iOS/macOS (CocoaPods `MNN.podspec`, fat framework), Linux/Windows/OpenHarmony, WASM (emcmake), RISC-V (SpacemiT K3), CUDA sm60–120 (incl. sm87 Jetson Orin), Qualcomm QNN/Hexagon, Huawei HIAI, Rockchip RKNN, Apple ANE via CoreML.
|
||||
- **No JIT / graph capture** [Verified]: kernels are AOT (Metal source strings compiled at first use with a persisted tuning cache; Vulkan SPIR-V arrays; OpenCL online-compiled with binary+tuning cache; CUDA nvcc). No `cudaGraph`, no `MTLIndirectCommandBuffer`. GPU batching is controlled by `OP_ENCODER_NUMBER_FOR_COMMIT`.
|
||||
- **Bindings** [Verified]: C++ only in core (no C API); Python `MNN.llm` (`response/generate/…`, no TTS callback), Android JNI + Kotlin apps, iOS ObjC++/Swift wrapper with audio-out callback.
|
||||
- **Concurrency model** [Verified]: one `Llm` = one `Executor` + one `RuntimeManager`; thread-safe only if one thread drives it ([test_multi_instance.cpp](../../MNN/transformers/llm/engine/test/test_multi_instance.cpp)). Cancellation of text decode has no engine API — apps step `generate(1)` themselves; `USER_CANCEL` is set only when the wav callback returns `false`.
|
||||
- **Top facts for edge TTS/VLA**: TTS with RTF reporting exists today (chunk-streamed for Qwen2.5-Omni, single-callback for Qwen3-TTS, on CPU/GPU, DiT forced fp32 on GPU); the DiT ODE loop is host-driven Express code, so a diffusion action head would be implementable the same way; the missing pieces for VLA are proprio/action I/O and any low-latency control loop.
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Export["Export / conversion (host)"]
|
||||
HF["HF checkpoint"] --> LLMEXP["llmexport.py<br/>utils/model_mapper.py, vision.py, audio.py, talker.py, token2wav.py"]
|
||||
ONNX["ONNX / TF / TFLite / Caffe / TorchScript"] --> CONV["MNNConvert<br/>tools/converter (optimizer/merge, postconvert)"]
|
||||
LLMEXP --> CONV
|
||||
GGUF["GGUF / safetensors"] --> G2M["gguf2mnn.py / safetensors2mnn.py"] --> CONV
|
||||
CONV --> MNNF["llm.mnn + llm.mnn.weight + config.json + tokenizer.txt + embeddings_bf16.bin<br/>(+ visual.mnn, audio.mnn, talker.mnn, dit.mnn, bigvgan.mnn ...)"]
|
||||
end
|
||||
subgraph App["Application layer"]
|
||||
DEMO["llm_demo / qwen3_tts_demo / llm_bench"]
|
||||
MLS["mls_server / mnncli (HTTP, SSE)"]
|
||||
PY["pymnn MNN.llm"]
|
||||
AND["Android MnnLlmChat (JNI) / iOS MNNLLMChat / TaoAvatar"]
|
||||
end
|
||||
subgraph LLM["transformers/llm/engine"]
|
||||
LLMC["Llm (llm.cpp): load, forwardRaw, generate, response, prompt cache"]
|
||||
OMNI["Omni : Embedding : Llm (omni.cpp): vision/audio encoders, M-RoPE"]
|
||||
TALK["Talker : Llm: codec LM -> DiT -> BigVGAN / speech_decoder<br/>async token2wav workers, wav callback"]
|
||||
TOK["Tokenizer (SP / tiktoken / BERT / HF BPE / Pipeline) + jinja.hpp"]
|
||||
SAMP["Sampler (greedy/topK/topP/minP/tfs/typical/penalty/mixed)"]
|
||||
GEN["Generation strategies: AR, Lookahead, MTP, EAGLE, DFlash"]
|
||||
KVM["KVMeta (add/remove/reserve, prefix file)"]
|
||||
DISK["DiskEmbedding (bf16/int8/int4/int2/int3 rows from file)"]
|
||||
end
|
||||
subgraph Express["express/ (Module API)"]
|
||||
MOD["Module::load / clone -> NetModule -> PipelineModule -> StaticModule"]
|
||||
EXEC["Executor / RuntimeManager (shared Runtime, hints, cache, external file)"]
|
||||
EXPR["Expr / VARP lazy graph, ComputeCache"]
|
||||
end
|
||||
subgraph Core["source/core (Session API)"]
|
||||
INTERP["Interpreter -> Schedule::schedule"]
|
||||
SESS["Session (ModeGroup, resize/run)"]
|
||||
PIPE["Pipeline: encode (shape+geometry) -> allocMemory (create Execution, wrap copies) -> execute"]
|
||||
GEO["GeometryComputer (-> Raster/Loop) + SizeComputer"]
|
||||
BUF["BufferAllocator (Eager / Defer), KVCacheManager"]
|
||||
BK["Backend / Runtime registry (MNNInsertExtraRuntimeCreator)"]
|
||||
end
|
||||
subgraph Backends["source/backend/*"]
|
||||
CPU["CPU (+arm82 fp16, AVX2/512, RVV, KleidiAI, SME2, bf16)"]
|
||||
GPU["Metal | OpenCL | Vulkan | OpenGL | CUDA | MUSA | TensorRT"]
|
||||
NPU["CoreML | NNAPI | QNN | HIAI | Hexagon | RKNN(plugin) | NeuroPilot(stub)"]
|
||||
end
|
||||
MNNF --> LLMC
|
||||
DEMO --> LLMC
|
||||
MLS --> LLMC
|
||||
PY --> LLMC
|
||||
AND --> LLMC
|
||||
LLMC --> OMNI --> TALK
|
||||
LLMC --> TOK
|
||||
LLMC --> SAMP
|
||||
LLMC --> GEN
|
||||
LLMC --> KVM
|
||||
LLMC --> DISK
|
||||
LLMC --> MOD
|
||||
MOD --> EXEC
|
||||
MOD --> EXPR
|
||||
MOD --> SESS
|
||||
INTERP --> SESS --> PIPE
|
||||
PIPE --> GEO
|
||||
PIPE --> BUF
|
||||
PIPE --> BK
|
||||
BK --> CPU
|
||||
BK --> GPU
|
||||
BK --> NPU
|
||||
GPU -. "unsupported op -> CPU backup" .-> CPU
|
||||
```
|
||||
|
||||
### Key modules
|
||||
|
||||
| Module | Path | Responsibility | Key symbols |
|
||||
|---|---|---|---|
|
||||
| Public headers | [include/MNN](../../MNN/include/MNN) | Session API, tensor, forward types, Express/Module API | `Interpreter`, `Tensor`, `MNNForwardType`, `BackendConfig`, `ScheduleConfig`, `Express::Module`, `Express::Executor`, `VARP` |
|
||||
| Core scheduling | [source/core/Schedule.cpp](../../MNN/source/core/Schedule.cpp) | Pick backend type, build per-pipeline op lists, mark I/O tensors | `Schedule::schedule`, `Schedule::getAppropriateType`, `Schedule::BackendCache`, `OpCacheInfo` |
|
||||
| Session | [source/core/Session.cpp](../../MNN/source/core/Session.cpp) | Owns pipelines, translates `SessionMode`/`HintMode` into `RuntimeHint`, resize/run | `Session::Session`, `Session::resize`, `Session::run`, `ModeGroup::setHint`, `createPipelineBackend` |
|
||||
| Pipeline | [source/core/Pipeline.cpp](../../MNN/source/core/Pipeline.cpp) | shape+geometry → command buffer → execution creation (with CPU fallback) → memory plan → execute | `Pipeline::encode`, `Pipeline::allocMemory`, `_createExecutions`, `_InsertCopy`, `Pipeline::execute`, `_pushTuningTask` |
|
||||
| Backend abstraction | [source/core/Backend.hpp](../../MNN/source/core/Backend.hpp), [Backend.cpp](../../MNN/source/core/Backend.cpp) | Backend/Runtime interfaces, creator registry, hints | `Backend::onCreate/onAcquire/onCopyBuffer/onSync`, `Runtime`, `RuntimeHint`, `MNNInsertExtraRuntimeCreator`, `registerBackend` |
|
||||
| Memory | [source/core/BufferAllocator.hpp](../../MNN/source/core/BufferAllocator.hpp) | Static/dynamic pools, mmap-backed allocators | `EagerBufferAllocator`, `DeferBufferAllocator`, `BufferAllocator::createMmap` |
|
||||
| Copy/wrap | [source/core/WrapExecution.cpp](../../MNN/source/core/WrapExecution.cpp) | Insert device/layout conversion between executions | `WrapExecution::needWrap`, `makeCopyExecution` |
|
||||
| Geometry | [source/geometry](../../MNN/source/geometry) | Decompose ops into Raster/Loop primitives | `GeometryComputer`, `GeometryComputerUtils::shapeComputeAndGeometryTransform`, `registerGeometryOps` |
|
||||
| Shape | [source/shape](../../MNN/source/shape) | Output-shape inference per op | `SizeComputer`, `SizeComputerSuite`, `REGISTER_SHAPE` |
|
||||
| KV cache core | [source/core/KVMeta.hpp](../../MNN/source/core/KVMeta.hpp), [KVCacheManager.hpp](../../MNN/source/core/KVCacheManager.hpp) | Host-side KV cursor; base KV storage w/ disk mmap | `KVMeta::sync`, `KVCacheConfig`, `KVCacheManager::onAlloc/onRealloc` |
|
||||
| CPU backend | [source/backend/cpu](../../MNN/source/backend/cpu) | Kernels, thread pool, core detection, attention, KV cache | `CPUBackend::addCreator/onCreate`, `CPURuntime`, `ThreadPool`, `MNNGetCPUInfo`, `CPUAttention`, `CPUKVCacheManager`, `CoreFunctions` |
|
||||
| ARM fp16 | [source/backend/arm82](../../MNN/source/backend/arm82) | fp16 variant of CPU backend (NC8HW8) | `Arm82Backend`, `Arm82Functions::init` |
|
||||
| GPU backends | [metal](../../MNN/source/backend/metal), [opencl](../../MNN/source/backend/opencl), [vulkan](../../MNN/source/backend/vulkan), [cuda](../../MNN/source/backend/cuda) | Device runtimes, kernels, tuning caches | `MetalBackend`, `OpenCLBackend`, `VulkanBackend`, `CUDABackend` |
|
||||
| NPU backends | [coreml](../../MNN/source/backend/coreml), [nnapi](../../MNN/source/backend/nnapi), [qnn](../../MNN/source/backend/qnn), [hiai](../../MNN/source/backend/hiai), [hexagon](../../MNN/source/backend/hexagon), [rknn](../../MNN/source/backend/rknn) | Subgraph compilation / delegation | `CoreMLBackend::buildModel`, `NNAPIBackend::buildModel`, `QnnBackend::finalizeGraph`, `NPUBackend::bulidIRModelAndLoad`, `registerRKNNPlugin` |
|
||||
| Express | [express/](../../MNN/express) | Lazy VARP graph, Module implementations, RuntimeManager | `Executor::RuntimeManager::createRuntimeManager`, `StaticModule`, `PipelineModule`, `Executor::ComputeCache` |
|
||||
| Converter | [tools/converter](../../MNN/tools/converter) | Frontends + optimizer passes + weight quant/coding | `MNNConverter.cpp`, `cli.cpp`, `PostConverter.cpp`, `merge/FuseAttention.cpp`, `WeightQuantAndCoding.cpp` |
|
||||
| LLM export | [transformers/llm/export](../../MNN/transformers/llm/export) | HF → MNN (text/vision/audio/talker/token2wav/draft models) | `llmexport.py`, `utils/model_mapper.py`, `utils/vision.py`, `utils/audio.py`, `utils/talker.py`, `utils/token2wav.py`, `gguf2mnn.py` |
|
||||
| LLM engine | [transformers/llm/engine](../../MNN/transformers/llm/engine) | Runtime for LLM/VLM/omni/TTS/embedding/reranker | `Llm`, `Omni`, `Talker`, `Embedding`, `LlmConfig`, `Tokenizer`, `Sampler`, `Generation*`, `DiskEmbedding` |
|
||||
| Diffusion engine | [transformers/diffusion/engine](../../MNN/transformers/diffusion/engine) | SD1.5 (PNDM), SD3.5 (flow), Sana (flow, Qwen3 text enc), Wan2.1 T2V | `Diffusion::createDiffusion`, `StableDiffusion`, `DiffusionSD35`, `SanaDiffusion`, `WanDiffusion` |
|
||||
| CV / audio libs | [tools/cv](../../MNN/tools/cv), [tools/audio](../../MNN/tools/audio) | OpenCV-like ops on VARP; wav I/O, fbank/whisper/conformer/usm mel | `MNN::CV::resize/imread`, `MNN::AUDIO::load/save/whisper_fbank/conformer_fbank/usm_fbank` |
|
||||
| Apps | [apps/](../../MNN/apps) | Android/iOS chat apps, TTS SDK, sherpa port, mnncli server, Sana app | `MnnLlmChat`, `MnnTaoAvatar`, `mnn_tts`, `sherpa-mnn`, `mnncli` |
|
||||
| Python | [pymnn/](../../MNN/pymnn) | Bindings: expr/nn/cv/audio/llm/reranker | `pymnn/src/llm.h` (`PyMNNLLM_methods`) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Device layer
|
||||
|
||||
### Backend abstraction and selection
|
||||
- [Verified] `MNNForwardType` ([MNNForwardType.h](../../MNN/include/MNN/MNNForwardType.h)): `CPU=0, METAL=1, CUDA=2, OPENCL=3, AUTO=4, NN=5 (NNAPI on Android / CoreML on iOS / QNN online), OPENGL=6, VULKAN=7, USER_0=8 (HIAI), USER_1=9 (TensorRT), HEXAGON=10, USER_3=11, ALL=12, CPU_EXTENSION=13, MNN_MEMORY_AHARDWAREBUFFER=14, MUSA=15`; offline-convert pseudo-types `MNN_CONVERT_QNN=32, MNN_CONVERT_NEUROPILOT=33, MNN_CONVERT_COREML=34`. `MNNGpuMode` bit-flags: `MNN_GPU_TUNING_{NONE,HEAVY,WIDE,NORMAL,FAST}`, `MNN_GPU_MEMORY_BUFFER (1<<6)`, `MNN_GPU_MEMORY_IMAGE (1<<7)`, `MNN_GPU_RECORD_OP (1<<8)`, `MNN_GPU_RECORD_BATCH (1<<9)` — these are OR'ed into `ScheduleConfig::numThread` for GPU backends (the LLM engine does `numThread |= 64 | 512` for OpenCL, [llm.cpp#L232](../../MNN/transformers/llm/engine/src/llm.cpp#L232)).
|
||||
- [Verified] `BackendConfig` = `MemoryMode {Normal, High, Low}`, `PowerMode {Normal, High, Low}`, `PrecisionMode {Normal, High, Low, Low_BF16}`, and a **union** of `void* sharedContext` / `size_t flags`. `RuntimeStatus` probes: `STATUS_SUPPORT_FP16`, `_DOT_PRODUCT`, `_POWER_LOW`, `_SIMD_GROUP_REDUCE`, `_FUSED_PROJ`.
|
||||
- [Verified] `class Backend` ([Backend.hpp](../../MNN/source/core/Backend.hpp)): pure virtuals `onCreate(inputs, outputs, op)` (returns `Execution*` or `nullptr` = unsupported), `onResizeBegin/End`, `onExecuteBegin/End`, `onAcquire(tensor, StorageType)`, `onClearBuffer`, `onCopyBuffer(src, dst)`; helpers `onAcquireBuffer/onReleaseBuffer` store a `MemObj` in the tensor descriptor; `StorageType {STATIC, DYNAMIC, DYNAMIC_SEPERATE, DYNAMIC_IN_EXECUTION}`; GPU hooks `onMapTensor/onUnmapTensor/onSync(MapType, toCpu, tensor)`; `getMetaPtr/setMetaPtr` (KV meta). `class Runtime`: `CompilerType {Compiler_Geometry, Compiler_Origin, Compiler_Loop}` (default Loop), `AllocatorType {Defer, Eager}`, `onCreate(config, origin)`, `onReset`, `onGabageCollect(level)`, `onSetCache/onGetCache`, `onSetCachePath`, `onMeasure`, async-work API (`setAsyncWork/waitAsyncWork/mCancelled`), `onConcurrencyBegin/End`.
|
||||
- [Verified] `RuntimeHint` ([Backend.hpp](../../MNN/source/core/Backend.hpp)) carries all tunables: `memoryAllocatorType`, `winogradMemoryUsed=3`, `cpuDecreaseRate=50`, `dynamicQuantOption`, `attentionOption=8`, `kvcacheSizeLimit=-1`, `kvcacheDirPath`, `prefixcacheDirPath`, `midMemoryPath`, `weightMemoryPath`, `mmapFileSize=1024 MB`, `useCachedMmap`, `encorderNumForCommit=10`, `initThreadNumber`, `useArmSme2Cores`, `enableKleidiAI`, `cpuIds`, `divisionRatio=41`. `Session::ModeGroup::setHint` ([Session.cpp#L77](../../MNN/source/core/Session.cpp#L77)) is the single translation point from `Interpreter::HintMode`.
|
||||
- [Verified] Registry: `registerBackend()` in [Backend.cpp](../../MNN/source/core/Backend.cpp) (under `std::call_once`) always registers CPU and conditionally CoreML (`MNN_COREML_ENABLED`), NNAPI, QNN, RKNN plugin, Hexagon, OpenCL, Metal, NeuroPilot. Vulkan, OpenGL, CUDA, MUSA, TensorRT, HIAI self-register via file-scope static initializers (e.g. [VulkanRuntime.cpp](../../MNN/source/backend/vulkan/runtime/VulkanRuntime.cpp), [cuda/Register.cpp](../../MNN/source/backend/cuda/Register.cpp), [GLBackend.cpp](../../MNN/source/backend/opengl/GLBackend.cpp)). `MNNGetExtraRuntimeCreator` instantiates a probe `Runtime` when the creator was registered with `needCheck=true` (OpenCL, Vulkan, HIAI) so a missing driver degrades gracefully.
|
||||
- [Verified] Discovery/selection: `Schedule::getAppropriateType` ([Schedule.cpp#L111](../../MNN/source/core/Schedule.cpp#L111)) resolves `MNN_FORWARD_AUTO` through the fixed priority list `HIAI → CoreML/NN → TensorRT → CUDA → OpenCL → Metal → Vulkan → CPU`, falls back to `config.backupType` if the creator is missing, and demotes OpenCL+`Power_Low` if `STATUS_SUPPORT_POWER_LOW` is false. `RuntimeFactory::create` is a one-liner over the registry ([RuntimeFactory.cpp](../../MNN/source/core/RuntimeFactory.cpp)).
|
||||
|
||||
### Memory allocation and pools
|
||||
- [Verified] [BufferAllocator.hpp](../../MNN/source/core/BufferAllocator.hpp): `EagerBufferAllocator` (free-list `multimap<size,Node>`, 64-byte alignment via `MNN_MEMORY_ALIGN_DEFAULT` in [MNNMemoryUtils.h](../../MNN/source/core/MNNMemoryUtils.h), per-thread groups) and `DeferBufferAllocator` (symbolic plan then `compute()/apply()` into one arena). Factories `createDefault`, `createMmap(dir, prefix, postfix, autoRemove, syncValid)`, `createRecurse(parent)`.
|
||||
- [Verified] CPU: `CPURuntime` owns one static Eager pool and `MNN_CPU_MAX_BUFFER_INDEX=2` dynamic buffers; `createDynamicBufferAlloctor` returns Defer vs Eager based on `hint().memoryAllocatorType` (`HintMode::MEM_ALLOCATOR_TYPE`) ([CPUBackend.cpp](../../MNN/source/backend/cpu/CPUBackend.cpp)). `hint().weightMemoryPath` swaps the static pool for an mmap-backed one (this is `use_mmap` in the LLM config → `EXTERNAL_WEIGHT_DIR`); `midMemoryPath` does the same for dynamic buffers (`EXTERNAL_FEATUREMAP_DIR`).
|
||||
- [Verified] `SessionMode` memory knobs: `Session_Memory_Collect` runs `onGabageCollect(0)` after every `allocMemory` ([Session.cpp#L363](../../MNN/source/core/Session.cpp#L363)); `Session_Memory_Cache` keeps static memory. `Session_Input_User` (what `Module::Config::shapeMutable=true` maps to) keeps inputs user-owned and stages copies through `inputTensorCopyCache` in `Pipeline::_copyInputs`.
|
||||
- [Verified] Express default: `Executor::getGlobalExecutor()` creates a 1-thread CPU runtime with `memoryAllocatorType=0` (Defer) ([Executor.cpp#L154](../../MNN/express/Executor.cpp#L154)).
|
||||
|
||||
### Host↔device transfers and synchronization
|
||||
- [Verified] [Tensor.hpp](../../MNN/include/MNN/Tensor.hpp): `copyFromHostTensor`, `copyToHostTensor`, `map/unmap(MapType, DimensionType)`, `wait(MapType, finish)`. `Tensor::map` calls `Backend::onSync` then `onMapTensor`, falling back to a temp host copy when mapping is unsupported ([Tensor.cpp](../../MNN/source/core/Tensor.cpp)).
|
||||
- [Verified] Metal ([MetalBackend.mm](../../MNN/source/backend/metal/MetalBackend.mm)): one `MTLCommandQueue`, a rolling `MTLCommandBuffer`, `mEncoderCount` reset at `onExecuteBegin`, encoder flushed/committed every `hint().encorderNumForCommit` ops (`OP_ENCODER_NUMBER_FOR_COMMIT`), `onSync(toCpu)` = flush + commit + `waitUntilCompleted`; `onAcquire` handles `DYNAMIC_IN_EXECUTION` from an execution pool; tuned threadgroup tables persisted through `onGetCache/onSetCache` (`MetalTuneLevel` default `Wide`, schema [MetalCache.fbs](../../MNN/source/backend/metal/schema/MetalCache.fbs)).
|
||||
- [Verified] OpenCL ([OpenCLBackend.cpp](../../MNN/source/backend/opencl/core/OpenCLBackend.cpp)): `setGpuMode` decodes BUFFER/IMAGE (AUTO → BUFFER on Mali/Intel, IMAGE elsewhere) and tuning level; record-queue (`MNN_GPU_RECORD_OP/BATCH`) when the device supports it; `onSync(toCpu)` = `commandQueue().finish()`; SVM fine-grain buffers under `MNN_OPENCL_SVM_ENABLE`; binary+tuning cache in a FlatBuffer (`CLCache` schema under [opencl/core/runtime](../../MNN/source/backend/opencl/core/runtime)). The buffer path can be compiled out with `MNN_OPENCL_SIZE_CUT`.
|
||||
- [Verified] Vulkan buffer variant ([VulkanBackend.cpp](../../MNN/source/backend/vulkan/buffer/backend/VulkanBackend.cpp)): `mDirect = !(gpuMode & MNN_GPU_RECORD_BATCH)`; all command buffers submitted in one `vkQueueSubmit` + single `VulkanFence` in `_finish()`; `onCopyBuffer` stages through a host-visible buffer; dynamic pool is an `EagerBufferAllocator` aligned to `nonCoherentAtomSize`. Image variant selected by `MNN_VULKAN_IMAGE=ON` (default) at build time.
|
||||
|
||||
### Portability — what is actually in tree
|
||||
|
||||
| Backend dir | Forward type | Registration | CMake gate | Delegation granularity / notes |
|
||||
|---|---|---|---|---|
|
||||
| [cpu](../../MNN/source/backend/cpu) | CPU / CPU_EXTENSION | `registerCPURuntimeCreator` | always | per-op; `Compiler_Loop` |
|
||||
| [arm82](../../MNN/source/backend/arm82) | CPU_EXTENSION (fp16) | `registerArm82RuntimeCreator` → `Arm82Functions::init` | `MNN_ARM82` (ON) | chosen when `Precision_Low` and `fp16arith` |
|
||||
| [metal](../../MNN/source/backend/metal) | METAL | `registerMetalRuntimeCreator` | `MNN_METAL` + APPLE | per-op |
|
||||
| [opencl](../../MNN/source/backend/opencl) | OPENCL | `registerOpenCLRuntimeCreator` (needCheck) | `MNN_OPENCL` | per-op; dlopen unless `MNN_USE_SYSTEM_LIB` |
|
||||
| [vulkan](../../MNN/source/backend/vulkan) | VULKAN | static init (needCheck) | `MNN_VULKAN`, `MNN_VULKAN_IMAGE` | per-op; dlopen `libvulkan` |
|
||||
| [opengl](../../MNN/source/backend/opengl) | OPENGL | static init | `MNN_OPENGL` | per-op; links GLESv3/EGL |
|
||||
| [cuda](../../MNN/source/backend/cuda) | CUDA | static init | `MNN_CUDA` (+`MNN_CUDA_QUANT/BF16/TUNE_PARAM`) | per-op; sm 60–120 |
|
||||
| [musa](../../MNN/source/backend/musa) | MUSA=15 | static init | `MNN_MUSA` (+ stub/compat/native) | Moore Threads |
|
||||
| [tensorrt](../../MNN/source/backend/tensorrt) | USER_1 | static init | `MNN_TENSORRT` | whole subgraph at `onResizeEnd` |
|
||||
| [coreml](../../MNN/source/backend/coreml) | NN | `registerCoreMLRuntimeCreator` | `MNN_COREML` | whole subgraph: `buildModel()` at `onResizeEnd`, `invokeModel()` at `onExecuteEnd` |
|
||||
| [nnapi](../../MNN/source/backend/nnapi) | NN | `registerNNAPIRuntimeCreator` | `MNN_NNAPI` (API ≥ 29) | whole graph via `ANeuralNetworksCompilation`; `Compiler_Geometry` |
|
||||
| [qnn](../../MNN/source/backend/qnn) | NN (online) / CONVERT_QNN (offline) | `registerQNNRuntimeCreator` | `MNN_QNN`, `MNN_QNN_ONLINE_FINALIZE`, `MNN_QNN_CONVERT_MODE` | `Compiler_Origin`; graph finalized once; context binary cached via `onGetCache/onSetCache`; SDK via [prepare_qnn_deps.sh](../../MNN/prepare_qnn_deps.sh) |
|
||||
| [hiai](../../MNN/source/backend/hiai) | USER_0 | static init (needCheck) | `MNN_NPU` (not an `option()`) | whole graph `bulidIRModelAndLoad`; `Compiler_Origin` |
|
||||
| [hexagon](../../MNN/source/backend/hexagon) | HEXAGON=10 | `registerHexagon` (probes DSP lib) | `MNN_HEXAGON` | per-op over decomposed stream (`Compiler_Loop`), forced 1 thread |
|
||||
| [rknn](../../MNN/source/backend/rknn) | (CPU + Plugin op) | `registerRKNNPlugin` | `MNN_RKNN` + `MNN_WITH_PLUGIN` | one plugin op wraps a whole `.rknn` file |
|
||||
| [neuropilot](../../MNN/source/backend/neuropilot) | CONVERT_NEUROPILOT=33 | — | forced OFF ([CMakeLists.txt](../../MNN/source/backend/neuropilot/CMakeLists.txt)) | not shipped in OSS |
|
||||
|
||||
- [Verified] NNAPI and CoreML both claim `MNN_FORWARD_NN`; `MNNInsertExtraRuntimeCreator` rejects duplicates, so only one can be linked per build [Inferred from registry assert].
|
||||
- [Verified] CPU ISA paths: `source/backend/cpu/arm/arm64` (NEON, `*_ARMV82_*` sdot, `*_ARMV86_*` i8mm/smmla, `low_memory/` W2/W3/W4 GEMM, `sme2_asm/`), `arm32`, `x86_x64/{sse,avx,avxfma,avx512}` (AVX512-VNNI split), `riscv/rvv` (+ SpacemiT IME2), `kleidiai/` (fetched v1.16.0 by [cmake/KleidiAI.cmake](../../MNN/cmake/KleidiAI.cmake)), `bf16/`. Runtime feature detection `MNNGetCPUInfo()` ([CPURuntime.cpp](../../MNN/source/backend/cpu/CPURuntime.cpp)): `fp16arith, dot, i8mm, sve2, sme2, rvv/zvfh, groups (big/little clusters by cpufreq), smeCoreNumber, perfCoreNumber`.
|
||||
|
||||
### Multi-backend fallback (unsupported op on GPU)
|
||||
- [Verified] `Session::createPipelineBackend` ([Session.cpp#L26](../../MNN/source/core/Session.cpp#L26)) creates `cache.first` (target) and, unless the target is CPU, a separate CPU `cache.second` with `flags=4` (`MNN_CPU_USE_DEFAULT_BACKEND`).
|
||||
- [Verified] `_createExecutions` ([Pipeline.cpp#L538](../../MNN/source/core/Pipeline.cpp#L538)): try `OpCommonUtils::createExecutionWithExternal(mBackend, …)`; if `nullptr`, `// Try Backup` on `mBackupBackend`; if still null → `NOT_SUPPORT`. `_InsertCopy` then walks every command's inputs/outputs and, when `WrapExecution::needWrap(t, curBackend)` is true, appends an `OpType_Copy` command built by `WrapExecution::makeCopyExecution` (const tensors cached via `copyConstCache`). `needWrap` returns false for `MNN_FORWARD_NN` and types ≥ `MNN_CONVERT_QNN`, and for CPU↔CPU_EXTENSION only when `bytes` or `pack` (on NC4HW4 data) differ ([WrapExecution.cpp](../../MNN/source/core/WrapExecution.cpp)).
|
||||
- [Verified] Coarser fallback under `Session_Backend_Auto`: `Pipeline::allocMemory` asks `Runtime::onMeasure` per op; if any reports `initCostLong` it prints "Turn back to cpu", starts `_pushTuningTask` asynchronously and swaps the whole pipeline backend to CPU ([Pipeline.cpp#L1079](../../MNN/source/core/Pipeline.cpp#L1079)).
|
||||
- [Verified] Only `Compiler_Loop` runtimes get op-level CPU fallback ([docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)); NPU backends compile the whole graph and refuse (return null) unsupported ops.
|
||||
|
||||
### Runtime sharing
|
||||
- [Verified] `Executor::RuntimeManager::createRuntimeManager(config)` ([Executor.cpp#L334](../../MNN/express/Executor.cpp#L334)) calls `_getOrCreateRuntime(type, …)` on the current `Executor` — an existing runtime of the same forward type is **reused** (`onReset` with new thread count), and `mRuntime.second` (CPU backup) is always the executor's CPU runtime. `StaticModule::clone` reuses the RuntimeManager's runtime and shared const tensors, so N cloned Modules share one thread pool/allocator/tuning cache while owning separate `Backend`/`Session` objects ([StaticModule.cpp](../../MNN/express/module/StaticModule.cpp)). `Executor::newExecutor` + `ExecutorScope` gives a fully independent runtime (used by the Talker's async worker threads).
|
||||
|
||||
### CPU threading and core binding
|
||||
- [Verified] `MNN_USE_THREAD_POOL=ON` (default) → own pool ([ThreadPool.cpp](../../MNN/source/backend/cpu/ThreadPool.cpp)): pools keyed by affinity mask, `MNN_THREAD_POOL_MAX_TASKS=2` concurrent task slots, sleeping-worker bitmask. `CPURuntime::_resetThreadPool` may return fewer threads than requested. `CPURuntime::_validateCpuIds` forbids mixing little and big cores; `Power_Low` → little cluster, `Power_High` → highest clusters, `Power_Normal` → no pinning; `MNNSetSchedAffinity` via raw `sched_setaffinity` syscall. Work split by `computeDivideSizes` weights little cores with `CPU_LITTLECORE_DECREASE_RATE` (default 50%).
|
||||
|
||||
---
|
||||
|
||||
## 3. Kernel layer
|
||||
|
||||
### Operator set and registration
|
||||
- [Verified] [MNN.fbs](../../MNN/schema/default/MNN.fbs): `enum OpType` with 186 entries; transformer block `Attention=299, FmhaV2=300, Fmhca=301, SeqLen2Spatial=302, SplitGeLU=303, GroupNorm=304, LinearAttention=305, RoPE=306, FusedLinear=307, GatedRMSNorm=308`; quant ops `ConvInt8=513, Int8ToFloat=514, DepthwiseConvInt8=515, FloatToInt8=517, EltwiseInt8=518, DynamicQuant=155`; control `While=600, If=601, LayerNorm=603, GridSample=604`; escape hatches `Plugin=256`, `Extra=512`; `Stft=156` is the only audio-ish op. `table Op { inputIndexes, main:OpParameter, name, outputIndexes, type, defaultDimentionFormat=NHWC, externalPath }`; `table Net { bizCode, extraTensorDescribe, extraInfo, oplists, outputName, preferForwardType, sourceType, tensorName, tensorNumber, usage (INFERENCE/TRAIN/INFERENCE_STATIC), subgraphs, mnn_uuid }`. `AttentionParam { kv_cache, kv_shared_layer, layer_index, kv_shared_layer_index, mhq_quant, output_c4, attnScale }`, `RoPEParam`, `FusedLinearParam`, `LinearAttentionParam` (gate folding), `LoopParam/RegionCommand` (the Loop IR). Other schema files: [CaffeOp.fbs](../../MNN/schema/default/CaffeOp.fbs) (`IDSTQuan`, `Convolution2D`), [TensorflowOp.fbs](../../MNN/schema/default/TensorflowOp.fbs), [Tensor.fbs](../../MNN/schema/default/Tensor.fbs) (`MNN_DATA_FORMAT {NCHW, NHWC, NC4HW4, NHWC4}`, `Blob.external`), [UserDefine.fbs](../../MNN/schema/default/UserDefine.fbs), [TFQuantizeOp.fbs](../../MNN/schema/default/TFQuantizeOp.fbs), [Type.fbs](../../MNN/schema/default/Type.fbs), [ExtraInfo.fbs](../../MNN/schema/default/ExtraInfo.fbs), [TrainInfo.fbs](../../MNN/schema/default/TrainInfo.fbs).
|
||||
- [Verified] Registration per backend: CPU `REGISTER_CPU_OP_CREATOR(name, OpType)` → `CPUBackend::addCreator` into a `std::map<OpType, Creator*>`; generated [CPUOPRegister.cpp](../../MNN/source/backend/cpu/CPUOPRegister.cpp) registers 75 op types (incl. `Attention`, `LinearAttention`, `RoPE`, `Stft`, `Plugin`, `ConvInt8`, `DynamicQuant`, `While`). Metal [MetalOPRegister.mm](../../MNN/source/backend/metal/MetalOPRegister.mm) 36 (transformer ops under `MNN_SUPPORT_TRANSFORMER_FUSE`). OpenCL keys by `pair<OpType, GpuMemObject>` (buffer vs image) — [OpenCLOPRegister.cpp](../../MNN/source/backend/opencl/core/OpenCLOPRegister.cpp) 66 entries / 39 types. Vulkan buffer 35 / image 26 (self-registering statics). CUDA ~44 (static `CUDACreatorRegister`). NNAPI 23, CoreML 22, QNN 41.
|
||||
|
||||
### Dispatch mechanism
|
||||
- [Verified] `Execution` ([Execution.hpp](../../MNN/source/core/Execution.hpp)): `onResize`, `onExecute` (pure), `onClone` (capability probe when `dst==nullptr`), plus a string-keyed `insertExtraCreator/searchExtraCreator`. `CPUBackend::onCreate` ([CPUBackend.cpp](../../MNN/source/backend/cpu/CPUBackend.cpp)) rewrites `Convolution` → `ConvInt8` when I/O `quantAttr.type == DT_INT8` and the conv has quantized weights, then looks up `gCreator`.
|
||||
- [Verified] Precision → CPU variant (`CPURuntime::onCreate`): `Precision_Low` + `fp16arith` → `Arm82Backend` (pack 8, bytes 2); `Precision_Low_BF16` → BF16 core functions; `flags==4` → plain CPU; else `AVX2Backend` on x86. All variants share the `CoreFunctions` table ([CommonOptFunction.h](../../MNN/source/backend/cpu/compute/CommonOptFunction.h): `pack`, `bytes`, `supportFp16arith/SDot/I8mm/SME2`, kernel pointers) — so **NC4HW4 is really NC{pack}HW{pack}**: 4 (fp32 NEON), 8 (fp16 / AVX2), 16 (AVX512).
|
||||
- [Verified] The flow per op at resize: `SizeComputer::computeOutputSize` → `GeometryComputer::onCompute` (or identity `DefaultGeometryComputer`) → `Backend::onCreate` → `Execution::onResize`; at run: `Execution::onExecute` in `Pipeline::execute` ([Pipeline.cpp#L1167](../../MNN/source/core/Pipeline.cpp#L1167)).
|
||||
|
||||
### Geometry decomposition
|
||||
- [Verified] [GeometryComputer.hpp](../../MNN/source/geometry/GeometryComputer.hpp): `GeometryComputer::Context` holds one pre-built `OpType_Raster` op used for every emitted region command; `registerGeometryComputer(computer, types, CompilerType)`; two tables (`Compiler_Geometry`, `Compiler_Loop`), `Compiler_Origin` bypasses geometry. 51 files (`GeometryConv2D`, `GeometryBatchMatMul`, `GeometryReshape`, `GeometryPermute`, `GeometryBinary`, `GeometryReduce`, `GeometryLayernorm`, `GeometryFusedProj`, `GeometryGatedRMSNorm`, …), 84 registrations in [GeometryOPRegister.cpp](../../MNN/source/geometry/GeometryOPRegister.cpp). `GEOMETRY_COMPUTE_MASK` bits: `FUSEREGION`, `FUSEREGION_MULTI`, `USELOOP`, `OPENCACHE`.
|
||||
- [Verified] Contract ([docs/contribute/op.md](../../MNN/docs/contribute/op.md), [docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)): schema → shape → (optional) geometry → per-backend execution; if geometry is written, no per-backend kernel is required. [Inferred] Minimal backend residue: `Raster`, `Convolution(+Depthwise)`, `MatMul`, `BinaryOp`, `UnaryOp`, `Pooling`, `Reduction`, `Softmax`, `ReLU/PReLU`, `Scale`, `While/Loop` — the intersection of all backend registries.
|
||||
- [Verified] `Usage_INFERENCE_STATIC` models (converted with `--saveStaticModel`) skip shape+geometry entirely (`needComputeGeometry=false` in `Schedule::schedule`; `Pipeline::encode` copies commands verbatim) — this is what `MNN_SKIPBUILD_GEOMETRY`/`MNN_BUILD_MINI` rely on.
|
||||
|
||||
### Shape inference / dynamic shapes
|
||||
- [Verified] [SizeComputer.hpp](../../MNN/source/shape/SizeComputer.hpp): `onComputeSize`, `onComputeFlops`, `REGISTER_SHAPE(_INPUTS)` (ops whose shape depends on input *content*, e.g. `Stft`, mark `mNeedContentInputIndex`); 82 files, 122 registrations ([ShapeRegister.cpp](../../MNN/source/shape/ShapeRegister.cpp)); transformer shapes gated by `MNN_SUPPORT_TRANSFORMER_FUSE`.
|
||||
- [Verified] Runtime shape change: `Interpreter::resizeTensor` → `Session::setNeedResize`; `Session::resize()` ([Session.cpp#L306](../../MNN/source/core/Session.cpp#L306)) re-runs `Pipeline::encode` (shape+geometry) then `allocMemory` (execution creation cached in `executionCache`, memory re-planned). `Session_Resize_Check/Fix` (`Module::traceOrOptimize`) records a resize trace and freezes it. For Express Modules, `shapeMutable=true` → `Session_Input_User`.
|
||||
|
||||
### Layout
|
||||
- [Verified] `Tensor::DimensionType {TENSORFLOW (NHWC), CAFFE (NCHW), CAFFE_C4 (NC4HW4)}`; `NativeInsideDescribe::dimensionFormat` defaults to `MNN_DATA_FORMAT_NC4HW4` ([TensorUtils.hpp](../../MNN/source/core/TensorUtils.hpp)); `Region` is the raster primitive. `Arm82Backend` reinterprets NC4HW4 as NC8HW8 ([Arm82Backend.hpp](../../MNN/source/backend/arm82/Arm82Backend.hpp)). Layout conversion between ops/backends is automatic via `WrapExecution` (§2).
|
||||
|
||||
### Quantization formats and where dequant happens
|
||||
- [Verified] Classic PTQ int8: [tools/quantization](../../MNN/tools/quantization) (`quantized.out`, [calibration.cpp](../../MNN/tools/quantization/calibration.cpp): keys `quant_bits`, `feature_quantize_method` (KL/ADMM/EMA), `weight_quantize_method`, `feature_clamp_value`, `weight_clamp_value`, `winogradOpt`, `skip_quant_op_names`). Weights stored as `IDSTQuan { buffer, alpha, type (1 int8 / 2 sparse / 3 fp16 / 4 weightInt8 / 8 external low-bit), quantScale, scaleIn/Out, aMaxOrBits, aMin, readType, alphaFp16, scaleStorage }` ([CaffeOp.fbs](../../MNN/schema/default/CaffeOp.fbs)) and decoded by `ConvolutionCommon::load` ([ConvolutionCommon.cpp](../../MNN/source/core/ConvolutionCommon.cpp)) into `Int8Common { weight, alpha, canUseInt4/Int3/Int2, asymmetric }`.
|
||||
- [Verified] LLM weight quant (export): `--quant_bit {2,3,4,8,16}` (default 4), `--quant_block` (default 64; 0 = per-channel), `--lm_quant_bit/--lm_quant_block`, `--visual_quant_bit/--visual_quant_block`, `--embed_bit {16,8,4}`, `--sym`, `--scale_bit {16,32}`, `--awq`, `--hqq`, `--gptq_path`, `--omni`, `--smooth`, `--calib_data`, `--act_bit {8,16}`, `--quant_config` (per-op JSON) — [llmexport.py](../../MNN/transformers/llm/export/llmexport.py); packing in [utils/mnn_converter.py](../../MNN/transformers/llm/export/utils/mnn_converter.py) (`quant`, `write_header`, `build_weight`) and [utils/mnn_utils.py](../../MNN/transformers/llm/export/utils/mnn_utils.py) (`repack_low_bits`). MNNConvert equivalents: `--weightQuantBits`, `--weightQuantBlock`, `--weightQuantAsymmetric`, `--hqq`, `--weightQuantScaleBit` ([cli.cpp](../../MNN/tools/converter/source/common/cli.cpp)).
|
||||
- [Verified] Runtime dequant on CPU (`MNN_LOW_MEMORY`, forced ON by `MNN_BUILD_LLM`): `ConvolutionFloatFactory::_createUnit` ([ConvolutionFloatFactory.cpp](../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp)) — with `Memory_Low` → `DenseConvInt8TiledExecutor(isDynamicQuant=true)` (activations quantized to int8 per forward: `BatchAsyDynamicQuant`/`BatchSymDynamicQuant`, `DYNAMIC_QUANT_OPTIONS` 1 = per-tensor, 2 = block-quant input), else `DenseConvolutionTiledExecutor` where block-wise scale/bias pointers are threaded into the packed GEMM (`selectLowMemoryMatmulFunc`) — no materialized float weights. ARM64 kernels: [arm64/low_memory](../../MNN/source/backend/cpu/arm/arm64/low_memory) (`MNNGemmInt8AddBiasScale_ARMV82_w{2,3,4}_Unit.S`, `_ARMV86_w{2,3,4}_`, `MNNDynamicQuantFP32_Pack{4,8}.S`). GPU: Metal `MetalConvolution1x1.mm` accepts `mDequantBits ∈ {2,3,4,8}` (Q4 16-byte GEMV); OpenCL `ConvBufLowMemoryExecution`/`ConvLowMemoryExecution` + `gemm_int.cl`/`gemv_conv1x1_buf.cl`; Vulkan buffer `gemv_dequant_int4.comp`, `dynamic_w8a8_coop_gemm_fused.comp`, `VulkanConv1x1CoopA8`; CUDA `weight_only_quant/ConvFpAIntBExecution.cu` + CUTLASS int8.
|
||||
- [Verified] fp16: `MNN_ARM82` (ON; `Precision_Low`), bf16 `MNN_SUPPORT_BF16` (OFF). KleidiAI: `MNN_KLEIDIAI` (ON) but used only if `RuntimeHint::enableKleidiAI` (`CPU_ENABLE_KLEIDIAI`, default from `MNN_KLEIDIAI_DEFAULT_ON`=OFF) and the conv is 1×1 stride-1. SME2: `MNN_SME2` (ON), hints `CPU_SME2_INSTRUCTIONS`, `CPU_SME2_NEON_DIVISION_RATIO` (default 41).
|
||||
- [Verified] Attention/KV quant: `RuntimeHint::attentionOption` (default 8): `%8` → 0 none / 1 K int8 / 2 K+V int8 / 3 K TQ3 / 4 K+V TQ3 / 5 K TQ4 / 6 K+V TQ4; `/8` → flash attention on/off ([CPUAttention.cpp](../../MNN/source/backend/cpu/CPUAttention.cpp), `KVQuantMode` in [CPUKVCacheManager.hpp](../../MNN/source/backend/cpu/CPUKVCacheManager.hpp)). LLM config key `attention_mode` (legacy `quant_qkv`). Metal supports only int8 KV and flash attention for `head_dim ∈ {64,128,256}` ([docs/transformers/llm.md](../../MNN/docs/transformers/llm.md)); other GPU backends ignore `attention_mode` [Verified, repo doc].
|
||||
- [Verified] Fused transformer ops: `MNN_SUPPORT_TRANSFORMER_FUSE`; converter merge passes in [optimizer/merge](../../MNN/tools/converter/source/optimizer/merge) (`FuseAttention`, `FuseFmhaV2`, `FuseFmhca`, `FuseGeLu`, `FuseSplitGeLu`, `FuseGroupNorm`, `FuseLayerNorm{,V2,V3,RMS,RMSGamm,WithoutGammBeta}`, `MergeDynamicQuantV1/V2`, `ConvertMatMulToConv2D`) and [postconvert/FuseTransformerC4.cpp](../../MNN/tools/converter/source/optimizer/postconvert/FuseTransformerC4.cpp); `RoPE`, `FusedLinear` (QKV / gate-up / LN+proj), `LinearAttention` gate folding, `GatedRMSNorm` are emitted by the exporter (`--disable_fuse_qkv_proj`, `--disable_fuse_gate_up_proj`, `--disable_fuse_ln_proj`, `--disable_transformer_c4`). Cross-layer KV sharing: `AttentionParam.kv_shared_layer_index` → `Execution::onClone` from a registry in `_createExecutions`.
|
||||
|
||||
### Compilation / JIT / graph capture
|
||||
- [Verified] No machine-code JIT. `codegen/` ([codegen/CMakeLists.txt](../../MNN/codegen/CMakeLists.txt): `MNN_CODEGEN_OPENCL/METAL/CUDA`, all OFF) is source-level op fusion emitted at `Session::resize` when `Session_Codegen_Enable`. Metal shaders are embedded C strings ([AllShader.cpp](../../MNN/source/backend/metal/AllShader.cpp)) compiled by the OS with pipeline-state + tuning cache; Vulkan SPIR-V pre-generated ([buffer/compiler](../../MNN/source/backend/vulkan/buffer/compiler)); OpenCL online-built with `CLCache`; CUDA AOT `.cu`. No `cudaGraph`/`MTLIndirectCommandBuffer` anywhere. `Interpreter::setCacheFile` / `RuntimeManager::setCache` persist tuning across runs (LLM engine writes `tmp_path/mnn_cachefile.bin` for non-CPU backends).
|
||||
|
||||
### Fallback when op unsupported
|
||||
- [Verified] Per-op CPU backup (§2); if CPU also lacks the op → `NOT_SUPPORT` at resize. Unresolved `OpType_Extra` is a converter-time failure (only Metal registers an `Extra` creator; CPU has none).
|
||||
|
||||
### Extension points: custom op / backend
|
||||
- [Verified] Plugin ops: `MNN_WITH_PLUGIN`; [include/MNN/plugin](../../MNN/include/MNN/plugin) (`PluginKernel.hpp`: `CPUComputeKernel::init/compute/resize`, `REGISTER_PLUGIN_COMPUTE_KERNEL(name, kernel)`; `PluginShapeInference.hpp`: `REGISTER_PLUGIN_OP(name, inferShapeKernel)`), string-keyed; `OpType_Plugin { type:string, attr }` executed by [CPUPlugin.cpp](../../MNN/source/backend/cpu/CPUPlugin.cpp). RKNN and offline-QNN reuse this path. No `registerCustomOp` API exists.
|
||||
- [Verified] Native op: schema + `REGISTER_SHAPE` + optional geometry + `REGISTER_{CPU,METAL,OPENCL}_OP_CREATOR` / `VulkanBackend::addCreator` ([docs/contribute/op.md](../../MNN/docs/contribute/op.md)); converter `--customOpLibs`/`--allowCustomOp`.
|
||||
- [Verified] New backend: subclass `Backend` + `Runtime`, register via `MNNInsertExtraRuntimeCreator(type, creator, needCheck)` ([docs/contribute/backend.md](../../MNN/docs/contribute/backend.md)).
|
||||
|
||||
---
|
||||
|
||||
## 4. Model runner
|
||||
|
||||
### Model loading / export / conversion
|
||||
- [Verified] `.mnn` = FlatBuffer `Net`; `Interpreter::createFromFile/createFromBuffer`, `createSession(ScheduleConfig{type, numThread|gpuMode, backendConfig, backupType, saveTensors, path})`, `setSessionMode`, `setSessionHint`, `setCacheFile`, `setExternalFile`, `resizeTensor/resizeSession`, `runSession(WithCallBack)`, `updateSessionToModel` ([Interpreter.hpp](../../MNN/include/MNN/Interpreter.hpp)). `SessionMode` values 0–17 (`Session_Debug/Release`, `Input_Inside/User`, `Output_Inside/User`, `Resize_Direct/Defer`, `Backend_Fix/Auto`, `Memory_Collect/Cache`, `Codegen_Disable/Enable`, `Resize_Check/Fix`, `Module_Forward_Separate/Combine`); `HintMode` 0–17 (see §2); `ExternalPathType {KVCACHE_DIR, FEATUREMAP_DIR, WEIGHT_DIR, NPU_FILE_DIR, PREFIXCACHE_DIR}`.
|
||||
- [Verified] Module API: `Module::load(inputs, outputs, file|buffer, rtMgr, Config{dynamic=false, shapeMutable=true, rearrange=false, backend, base})` ([Module.hpp](../../MNN/include/MNN/expr/Module.hpp)); implementation `NetModule → PipelineModule (sub-graphs, If/While/MoE modules) → StaticModule (wraps a Session)` ([Module.cpp](../../MNN/express/module/Module.cpp), [PipelineModule.hpp](../../MNN/express/module/PipelineModule.hpp), [StaticModule.cpp](../../MNN/express/module/StaticModule.cpp)). `Module::load` auto-sets the external file to `<file>.weight` if none set. `Module::clone(module, shareParams)` shares weights, new Session. Lazy Express eval: `Executor::LazyMode {LAZY_FULL, LAZY_CONTENT, LAZY_COMPUTE_ONCE}`; `Executor::ComputeCache` in [express/Utils.hpp](../../MNN/express/Utils.hpp).
|
||||
- [Verified] External weights: `Op.externalPath` + `Convolution2D.external=[offset, weight_bytes, bias_bytes]`, `Blob.external=[offset, bytes]`; writer [writeFb.cpp](../../MNN/tools/converter/source/common/writeFb.cpp) (`--saveExternalData`, auto for large models, threshold `externalTreshold=64 KB` in [config.hpp](../../MNN/tools/converter/include/config.hpp)); loaded lazily by `FileLoader` at execution creation (`createExecutionWithExternal`).
|
||||
- [Verified] Converter: [MNNConverter.cpp](../../MNN/tools/converter/source/MNNConverter.cpp) → `Cli::convertModel`; frontends `onnx/`, `tensorflow/`, `tflite/`, `caffe/`, `torch/` (TorchScript, `MNN_BUILD_TORCH`); flags `--fp16`, `--weightQuantBits`, `--weightQuantBlock`, `--hqq`, `--optimizeLevel 0/1/2`, `--optimizePrefer`, `--keepInputFormat`, `--saveStaticModel`, `--saveExternalData`, `--transformerFuse`, `--transformerFuseC4`, `--transformerFuseQkvProj/GateUpProj/LnProj`, `--alignDenormalizedValue`, `--detectSparseSpeedUp`, `--customOpLibs`, `--allowCustomOp`, `--testdir/--thredhold`, `--JsonFile`, `--rknn`. Pass pipeline in [PostConverter.cpp](../../MNN/tools/converter/source/optimizer/PostConverter.cpp) (`RemoveUnusefulOp → … → MergeBNToConvolution → … → AddTensorFormatConverter → ReIndexTensor`), 41 merge passes, 33 `onnxextra/` rewrites.
|
||||
- [Verified] GGUF import ([gguf2mnn.py](../../MNN/transformers/llm/export/gguf2mnn.py)): F16, Q4_0, Q4_1, Q4_K, Q5_0, Q5_1, Q6_K, Q8_0 → MNN low-bit blobs (others assert); [safetensors2mnn.py](../../MNN/transformers/llm/export/safetensors2mnn.py) for pre-quantized/plain safetensors.
|
||||
- [Verified] LLM export pipeline ([llmexport.py](../../MNN/transformers/llm/export/llmexport.py): `export_talker → export_vision → export_audio → export_eagle → export_dflash → export_language → export_mtp → export_tokenizer → export_config`) produces `llm.mnn`, `llm.mnn.weight`, `llm_config.json`, `config.json` (runtime defaults: `backend_type=cpu, thread_num=4, precision=low, memory=low, sampler_type=mixed, temperature=0.8, top_k=40, top_p=0.9, min_p=0.05, …`), `tokenizer.txt` (or `.mtok` binary), `embeddings_bf16.bin` / `embeddings_int{8,4}.bin`, optional `visual.mnn`, `audio.mnn`, `talker.mnn`, `predit.mnn`, `dit.mnn`, `bigvgan.mnn`, `spk_dict.mnn`, `code_predictor.mnn`, `speech_decoder.mnn`, `speaker_encoder.mnn`, `mtp.mnn`, EAGLE/DFlash draft files, `ple_embeddings_bf16.bin` (Gemma4). Supported families in [model_mapper.py](../../MNN/transformers/llm/export/utils/model_mapper.py): llama, qwen/qwen2/qwen3/qwen3_moe/qwen3_5(_moe/_text), qwen2_vl/qwen2_5_vl/qwen3_vl(_moe), qwen2_audio, qwen3_asr, qwen2_5_omni, qwen3_tts, internlm, mobilellm, baichuan, deepseek-vl, llama4_text, mimo, poi_qwen2_mtp, chatglm/chatglm2, phi/phi-msft, internvl_chat, gemma2/gemma3(_text)/gemma4, openelm, idefics3/smolvlm, llava_qwen2 (FastVLM), hunyuan_v1_dense/hunyuan_vl, gpt_oss, minicpm/minicpmv, funaudiochat, glm_ocr, lfm2/lfm2_moe/lfm2_vl/lfm2_audio. NPU export helpers: [export/npu](../../MNN/transformers/llm/export/npu) (`generate_llm_qnn.py`, `generate_llm_mlda.py`) driving [compilefornpu.cpp](../../MNN/tools/cpp/compilefornpu.cpp).
|
||||
|
||||
### Execution lifecycle (init → warmup → step)
|
||||
- [Verified] `Llm::createLLM(config.json)` → `Omni` if `is_visual || is_audio || has_talker` else `Llm` ([llm.cpp#L88](../../MNN/transformers/llm/engine/src/llm.cpp#L88)). `Llm::load()` ([llm.cpp#L319](../../MNN/transformers/llm/engine/src/llm.cpp#L319)): check files → clear module pool before re-creating runtime (Backend lifetime) → `initRuntime()` (ScheduleConfig from `backend_type/thread_num/precision/power/memory`, `RuntimeManager`, hints from `setRuntimeHint`) → `Tokenizer::createTokenizer` → merge `context.json` into jinja context → `DiskEmbedding` (+ PLE) → chat template → `Sampler` → `Module::load({input_ids, attention_mask, position_ids, logits_index[, deepstack_embeds, ple_embeddings]}, {logits[, talker_embeds][, hidden_states]})` with `shapeMutable = !(opencl|vulkan|npu)`, `rearrange=true`, external file = `llm_weight` → `setSpeculativeConfig` (spec only if the model declares a dimensioned `logits_index` input) → `GenerationStrategyFactory::create` → clone decode module (key `<1,false>`) and spec module (`<draft+1,true>`) → preallocate mask/position VARPs → `mGenerationStrategy->load` → status `RUNNING`.
|
||||
- [Verified] **No implicit warmup**. Optional `Llm::tuning(OP_ENCODER_NUMBER, candidates)` benchmarks commit granularity on Metal/CPU (skipped on OpenCL) then resets KV; demos call it with `{1,5,10,20,30,50,100}`.
|
||||
- [Verified] Step: `Llm::forwardRaw(hidden, mask, pos, extra)` ([llm.cpp#L569](../../MNN/transformers/llm/engine/src/llm.cpp#L569)) picks module by `(seqLenKey = inDecode ? seq_len : mPrefillKey(100), allLogits)`, chooses `logitsAllIdx`/`logitsLastIdx`, `onForward`, validates outputs, optional `Tensor::wait` when `async=false`, then `mMeta->sync()`. `forwardVec` sets `mMeta->add = seq_len` and splits into `size_limit` blocks for chunked prefill.
|
||||
- [Verified] Runtime hints actually set by the LLM engine ([llm.cpp#L181](../../MNN/transformers/llm/engine/src/llm.cpp#L181)): `INIT_THREAD_NUMBER=4`, `MEM_ALLOCATOR_TYPE=0`, `ATTENTION_OPTION`, `USE_CACHED_MMAP`, `DYNAMIC_QUANT_OPTIONS`, `KVCACHE_INFO` (ptr), `CPU_SME2_NEON_DIVISION_RATIO`, `MMAP_FILE_SIZE`, `OP_ENCODER_NUMBER_FOR_COMMIT` (0 during prefill / 512 during decode on OpenCL); external paths `KVCACHE_DIR` (`kvcache_mmap`), `PREFIXCACHE_DIR`, `WEIGHT_DIR` (`use_mmap`), `NPU_FILE_DIR`. Never set: `KVCACHE_SIZE_LIMIT`, `CPU_CORE_IDS`, `CPU_ENABLE_KLEIDIAI`.
|
||||
|
||||
### Dynamic shapes
|
||||
- [Verified] Strategy is **bucketed module clones, not per-step resize**: prefill of any length maps to key 100 (the Session inside resizes on shape change since `Session_Input_User`), decode always key 1 (no resize), spec-verify key `draft+1`. Mask/position VARPs are reused when dims match (`needNewVar`). Chunked prefill (`size_limit: [block, 1]`) keeps shapes to a handful of values, needed on OpenCL/Vulkan/NPU where `shapeMutable=false`.
|
||||
|
||||
### State & KV cache
|
||||
- [Verified] `KVMeta` ([KVMeta.hpp](../../MNN/source/core/KVMeta.hpp)): `block=4096, previous, remove, reserve[] (pairs begin/len), n_reserve, add, file_name, file_flag {NoChange, PendingWrite, PendingRead}, seqlen_in_disk, layer_index, layer_nums, attn_scale, spec_block`; `sync()` applies `previous = previous - remove + add + Σreserve_len` after each forward. `Llm::setKVCacheInfo`, `eraseHistory(begin,end)` (partial rollback via reserve segments), `reset()` (`remove = previous`), `reuse_kv` (multi-turn KV retention), `prompt_cache` (text-prefix diff → only the token delta is prefilled, [llm.cpp#L1167](../../MNN/transformers/llm/engine/src/llm.cpp#L1167)), `setPrefixCacheFile` (KV prefix to disk; **requires `kvcache_mmap=true`**).
|
||||
- [Verified] Storage: [KVCacheManager.hpp](../../MNN/source/core/KVCacheManager.hpp) (`KVCacheConfig { mKVCacheDir, mPrefixCacheDir, mExpandChunk=64, mBlockNum=1, mKvAlignNum }`, key layout `{numhead, [maxlen/hP, headdim, hP]}` (or int8 `[maxlen/hP8, headdim/lP8, hP8, lP8]`), value `{numhead, [headdim/hP, maxlen, hP]}`, mmap FDs) and [CPUKVCacheManager.cpp](../../MNN/source/backend/cpu/CPUKVCacheManager.cpp) (`onAlloc/onRealloc` grow by `mExpandChunk` on demand, `expandKVCacheInMem/InDisk`, `moveKVCacheFromMemToDisk` when `kvcacheSizeLimit` exceeded, quant via `MNNQuantAttentionKey/Value`). Layout is contiguous per head — **no paging, no eviction policy**; the KV is a growable array trimmed only by explicit `remove/reserve`. Per-`Llm` instance (each cloned Module has its own attention executions → own KV). Metal/OpenCL/Vulkan/CUDA have their own attention executions ([MetalAttention.mm](../../MNN/source/backend/metal/MetalAttention.mm), [AttentionBufExecution.cpp](../../MNN/source/backend/opencl/execution/buffer/AttentionBufExecution.cpp), [VulkanAttention.cpp](../../MNN/source/backend/vulkan/buffer/execution/VulkanAttention.cpp), [AttentionExecution.cu](../../MNN/source/backend/cuda/execution/AttentionExecution.cu)).
|
||||
- [Verified] Embedding lookup is host-side from a file (`DiskEmbedding`, [diskembedding.cpp](../../MNN/transformers/llm/engine/src/diskembedding.cpp): bf16/int8/int4/int2/int3 rows, buffered `FileLoader` reads, tie-embeddings read from inside `llm.mnn.weight`), then fed as `input_embeds`.
|
||||
|
||||
### Sampling and speculative decoding
|
||||
- [Verified] [sampler.cpp](../../MNN/transformers/llm/engine/src/sampler.cpp): `greedy, temperature, topK, topP, minP, tfs, typical, penalty (repetition/presence/frequency + n-gram), mixed` pipeline (`penalty` forced first), `logit_bias`, `banned_tokens`, GPU top-k prefilter.
|
||||
- [Verified] [speculative_decoding](../../MNN/transformers/llm/engine/src/speculative_decoding): `ArGeneration` (plain AR), `LookaheadGeneration` (n-gram lookahead, `draft_predict_length`, `ngram_match_maxlen`, `lookup_file`), `MtpGeneration` (`mtp.mnn`, needs `hidden_states`), `EagleGeneration` (tree drafting: `eagle_model/eagle_fc/eagle_d2t`, `eagle_depth=3`, `eagle_topk`), `DFlashGeneration` (block diffusion-style drafts: `dflash_model/fc/kvmat`, `dflash_block_size=8`). Verification = greedy longest prefix (`Generation::draftVerify`), KV rollback via `KVMeta.reserve/spec_block`.
|
||||
|
||||
### Multimodal stages
|
||||
- [Verified] `Omni` ([omni.cpp](../../MNN/transformers/llm/engine/src/omni.cpp)): separate `mProcessorRuntimeManager` from the `mllm` sub-config (can put encoders on a different backend/thread count); `visual.mnn` (`mVisionModule`) with per-family preprocess `defaultVisionProcess` (CV resize+normalize), `qwen2VisionProcess` (smart-resize to pixel budget, patch 14 / 16 for Qwen3-VL, window mask), `gemma4VisionProcess`, `smolvlmVisionProcess`, `minicpmVisionProcess`, `hunyuanVisionProcess`, `qwenVideoProcess` (fps sampling, temporal patch 2, `video_max_frames=768`); `audio.mnn` (`mAudioModule`) fed by `whisper_fbank` / `conformer_fbank` / `usm_fbank` (16 kHz; Qwen2-Audio clipped to 3000 frames), audio window mask for 2-input encoders; M-RoPE position streams (`MropeInfo`); prompt syntax `<img>…</img>`, `<audio>…</audio>`, `<video>…</video>` or `MultimodalPrompt{images, audios, videos}`; embeddings spliced into pad-token runs in `Omni::embedding`.
|
||||
- [Verified] `Talker` (§9) for speech output; `Embedding`/reranker (`sentence_embeddings` output pooled inside the graph, [embedding.cpp](../../MNN/transformers/llm/engine/src/embedding.cpp), [reranker.hpp](../../MNN/transformers/llm/engine/include/llm/reranker.hpp)).
|
||||
- [Verified] Diffusion ([diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/diffusion.hpp)): `DiffusionModelType {STABLE_DIFFUSION_1_5, STABLE_DIFFUSION_TAIYI_CHINESE, SANA_DIFFUSION, WAN2_1_T2V}`; `StableDiffusion` = text_encoder → unet (PNDM/PLMS, [scheduler.hpp](../../MNN/transformers/diffusion/engine/src/scheduler.hpp)) → vae_decoder with module release by `memoryMode` and `Session_Resize_Check/Fix` to freeze the denoise-loop resize plan; `DiffusionSD35` (3 text encoders, DiT transformer, **flow matching** `step_flow_match`); `SanaDiffusion` (Qwen3-0.6B text encoder, DiT, flow matching, img2img; [sana_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/sana_diffusion.hpp)); `WanDiffusion` (T2V, `runVideo`, frame dump; [wan_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/wan_diffusion.hpp), [wan_diffusion_demo.cpp](../../MNN/transformers/diffusion/engine/wan_diffusion_demo.cpp)). Export: [diffusion/export](../../MNN/transformers/diffusion/export). Build: `MNN_BUILD_DIFFUSION` (forces `MNN_LOW_MEMORY`, `MNN_SUPPORT_TRANSFORMER_FUSE`, `MNN_BUILD_OPENCV`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Scheduler
|
||||
|
||||
### (a) Request / token / pipeline-level scheduling
|
||||
- **Request scheduling: N/A** [Verified]. One `Llm` object serves one conversation at a time; there is no queue, no batching across requests, no continuous batching, no prefill/decode disaggregation. `mls_server` and `mnncli` serialize HTTP requests with a `std::mutex` around `llm->response` ([mls_server.cpp](../../MNN/transformers/llm/engine/app/mls_server.cpp)). Batch dim is fixed to 1 in the exported graph (`input_ids` `[1, seq]`; [llmexport.py](../../MNN/transformers/llm/export/llmexport.py) patches the batch dim).
|
||||
- **Concurrency** [Verified]: N independent `Llm` instances in N threads work (each has its own `Executor`/`RuntimeManager`; [test_multi_instance.cpp](../../MNN/transformers/llm/engine/test/test_multi_instance.cpp)); `ExecutorScope` makes the executor thread-local. Runtimes created from the *global* executor share the CPU runtime and thread pool (`_getOrCreateRuntime`). `LlmContext::mutex` only protects token vectors for external readers.
|
||||
- **Prefill vs decode** [Verified]: prefill = one `forwardVec` over all prompt embeddings (optionally chunked by `size_limit` block size — `Llm::generate(ids)` loops `generate(chunk_embeds, 0)`), decode = `ArGeneration::generate` loop of `sample → is_stop → tokenizer_decode → stream to os → forwardVec({token})` ([generate.cpp#L44](../../MNN/transformers/llm/engine/src/speculative_decoding/generate.cpp#L44)). Stage-specific tuning: `Llm::switchMode(Prefill|Decode)` toggles OpenCL commit batching.
|
||||
- **Multi-stage pipelines** [Verified]: (1) `Omni::responseInterleaved` interleaves one thinker decode step with one talker step in a single loop (`while (!thinker_done || !talker_done)`, [omni.cpp#L2286](../../MNN/transformers/llm/engine/src/omni.cpp#L2286)); (2) `Talker` async token2wav: main thread runs the codec LM and enqueues `WavChunk`s (60 codec tokens + 24/12 context) → `ditWorkerLoop` thread (own `Executor::newExecutor`, cloned `mPreDit/mDit`) → `mMelQueue` → `vocoderWorkerLoop` thread (cloned BigVGAN) → wav callback ([omni.cpp#L2944](../../MNN/transformers/llm/engine/src/omni.cpp#L2944)). Enabled only when the talker's backend is CPU and a processor runtime distinct from the talker runtime exists; GPU backends run token2wav synchronously. `Talker::finalize` waits up to 60 s for the last chunk.
|
||||
- **Cancellation / abort** [Verified]: `LlmStatus::USER_CANCEL` exists and stops every generation loop, but the engine sets it in exactly one place — when the wav callback returns `false` ([omni.cpp#L3403](../../MNN/transformers/llm/engine/src/omni.cpp#L3403)). Text decode has no cancel API; `response()` blocks until stop/max tokens/`timeout_ms`. Apps work around this by calling `response(..., max_new_tokens=0)` (prefill only) and stepping `llm->generate(1)` per token under their own flag ([llm_session.cpp#L300](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp#L300)). `timeout_ms` config aborts with `TIMEOUT` after prefill or during decode. Async resize tuning can be cancelled through `Runtime::mCancelled`.
|
||||
- **Backpressure** [Verified]: none for text (stream is an `std::ostream*`, writes are synchronous). For audio, the wav callback's return value is the only backpressure/abort signal; the async queues are unbounded `std::queue`s.
|
||||
|
||||
### (b) Graph / operator / thread-level scheduling
|
||||
- [Verified] `Schedule::schedule` ([Schedule.cpp#L291](../../MNN/source/core/Schedule.cpp#L291)): builds `ScheduleInfo` = const-tensor init (with external weight loader) + one `PipelineInfo` per `ScheduleConfig` (multi-path sessions supported via `createMultiPathSession`), ops kept in model order (`generateScheduleGraph` filters by `path`/`saveTensors`), marks INPUT/OUTPUT usage, computes `needInputContentForShape`, and for static models disables shape/geometry.
|
||||
- [Verified] `Pipeline::encode` → `shapeComputeAndGeometryTransform` (shape + geometry + const folding, with `OpResizeCache` for shape-change replay) → `allocMemory` (`_createExecutions` → `_InsertCopy` → `_SetTensorBackend` → per-execution `onResize` between `onResizeBegin/End`, memory planned by the Defer/Eager allocator with `_recycleDynamicMemory`) → `execute` (linear walk of `executeBuffer.command` calling `onExecute`, between `onExecuteBegin/End` on both backends). Op order is topological as emitted by the converter; there is no reordering or multi-stream assignment.
|
||||
- [Verified] Thread pools: CPU `ThreadPool` per affinity mask, `numThread` per `ScheduleConfig`, big/little weighting, SME2 core selection (`useArmSme2Cores`, `divisionRatio`). GPU "streams": one Metal command queue with commit every N encoders; one OpenCL command queue with optional record queue; one Vulkan submit per `onExecuteEnd`; no multi-stream overlap. Async: `Runtime::setAsyncWork` for background tuning; `INIT_THREAD_NUMBER` for parallel module load.
|
||||
|
||||
---
|
||||
|
||||
## 6. I/O layer
|
||||
|
||||
- **Image preprocessing** [Verified]: `CV::ImageProcess` ([ImageProcess.hpp](../../MNN/include/MNN/ImageProcess.hpp): 15 formats incl. NV21/NV12/I420, `Filter {NEAREST, BILINEAR, BICUBIC}`, affine `Matrix`, mean/normal, `convert(src → Tensor|raw)`, `draw`), kernels in [ImageProcessFunction.cpp](../../MNN/source/backend/cpu/compute/ImageProcessFunction.cpp). OpenCV-like `MNN::CV` on VARP ([tools/cv/include/cv/cv.hpp](../../MNN/tools/cv/include/cv/cv.hpp): `resize/warpAffine/cvtColor/GaussianBlur/threshold/findContours/imread/imwrite/solvePnP…`), `MNN_BUILD_OPENCV`, `MNN_IMGCODECS` (stb-based).
|
||||
- **Audio preprocessing** [Verified]: `MNN::AUDIO` ([audio.hpp](../../MNN/tools/audio/include/audio/audio.hpp)): `load/save` (wav), `hamming_window/hann_window`, `melscale_fbanks`, `spectrogram`, `mel_spectrogram`, `fbank`, `whisper_fbank`, `conformer_fbank` (NeMo-style), `usm_fbank` (Gemma4), resamplers (linear, soxr-HQ); `MNN_BUILD_AUDIO`. Python exposes all but the two fork additions ([pymnn/src/audio.h](../../MNN/pymnn/src/audio.h)).
|
||||
- **Tokenization** [Verified]: in-house [tokenizer.cpp](../../MNN/transformers/llm/engine/src/tokenizer/tokenizer.cpp): `tokenizer.txt` with magic 430 and type `SENTENCEPIECE | TIKTOIKEN | BERT | HUGGINGFACE | PIPELINE` (`.mtok` binary: `Normalizer/PreTokenizer/TokenizerModel/TokenDecoder`, added tokens, byte-level BPE); special/stop tokens in the file; `is_stop()`; chat templates rendered by a vendored single-header Jinja2 engine [jinja.hpp](../../MNN/transformers/llm/engine/src/tokenizer/jinja.hpp) (jinja.cpp, not minja) with tools support; template/eos/bos read from the tokenizer file or `config.json["jinja"]`; `stripThinkBlocks` for prompt-cache stability ([prompt_cache_utils.hpp](../../MNN/transformers/llm/engine/src/prompt_cache_utils.hpp)).
|
||||
- **Streaming** [Verified]: text tokens are written to `std::ostream*` per token (apps subclass `std::streambuf`, e.g. [llm_stream_buffer.hpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_stream_buffer.hpp), with UTF-8 reassembly); no token callback API. Audio: `setWavformCallback(bool(const float* pcm, size_t n, bool last))`, chunked at ~120 mel frames / 60 codec tokens (Qwen2.5-Omni) or once at the end (Qwen3-TTS, single `speech_decoder` call). Perf counters in `LlmContext` (`prefill_us, decode_us, sample_us, ttfa_us, audio_us`).
|
||||
- **Buffering** [Verified]: disk-streamed embeddings; mmap weights (`use_mmap` → `EXTERNAL_WEIGHT_DIR`, `mmap_size`, `use_cached_mmap`); KV to disk (`kvcache_mmap`); OpenCL/Metal tuning cache file.
|
||||
- **Postprocessing** [Verified]: detokenize per token; wav writing is the caller's job (`MNN::AUDIO::save(path, VARP, 24000)` in [qwen3_tts_demo.cpp](../../MNN/transformers/llm/engine/demo/qwen3_tts_demo.cpp)); diffusion writes PNG via `MNN::CV::imwrite`.
|
||||
- **Application boundaries** [Verified]: C++ (`Interpreter`, `Express`, `Llm`; no C API beyond [MNNSharedContext.h](../../MNN/include/MNN/MNNSharedContext.h)); Python `MNN.llm` ([pymnn/src/llm.h](../../MNN/pymnn/src/llm.h): `create/load/forward/generate/response/get_context/set_config/erase_history/txt_embedding/create_lora…`, GIL released in `response`; no TTS); HTTP `mls` (`POST /chat/completions`, SSE) and `mnncli` (`/v1/models`, `/v1/chat/completions`, Anthropic `/v1/messages`); CLIs `llm_demo`, `llm_bench`, `embedding_demo`, `reranker_demo`, `qwen3_tts_demo`, `multi_lora_demo`, `rollback_demo`, `tokenizer_demo`, `apply_template`, `llm_logits_diff` ([demo/](../../MNN/transformers/llm/engine/demo)); Android JNI + Kotlin (`MnnLlmChat`, `MnnTaoAvatar`), iOS ObjC++ wrapper ([LLMInferenceEngineWrapper.h](../../MNN/apps/iOS/MNNLLMChat/MNNLLMiOS/InferenceEngine/LLMInferenceEngineWrapper.h) with `setAudioWaveformCallback`), engine iOS demo ([engine/ios](../../MNN/transformers/llm/engine/ios)); OpenHarmony build only (no app); no WASM/JS binding (build recipe only).
|
||||
|
||||
---
|
||||
|
||||
## 7. Execution flow
|
||||
|
||||
### (a) Text LLM: one `response()` call
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App
|
||||
participant Llm as Llm (llm.cpp)
|
||||
participant Tok as Tokenizer+jinja
|
||||
participant Emb as DiskEmbedding
|
||||
participant Mod as Module pool (StaticModule/Session)
|
||||
participant Pipe as Pipeline/Backend
|
||||
participant Attn as CPUAttention+KVCacheManager
|
||||
participant Gen as ArGeneration
|
||||
participant Samp as Sampler
|
||||
App->>Llm: response(ChatMessages, os, end_with, max_new)
|
||||
Llm->>Tok: apply_chat_template(messages) -> prompt
|
||||
Llm->>Llm: prompt_cache? diff vs mCachedPromptText -> delta tokens
|
||||
Llm->>Tok: tokenizer_encode(prompt|delta)
|
||||
Llm->>Llm: generate_init(os): reset ctx; if !reuse_kv: mMeta.remove=previous
|
||||
Llm->>Emb: embedding(ids) -> VARP [seq,1,hidden]
|
||||
Llm->>Llm: forwardVec: mMeta.add=seq; gen_attention_mask; gen_position_ids
|
||||
Llm->>Mod: select key (100, all_logits) -> onForward({embeds,mask,pos,logits_index})
|
||||
Mod->>Pipe: Session resize (shape+geometry, exec cache) if dims changed; execute
|
||||
Pipe->>Attn: OpType_Attention onExecute: onAlloc(KVMeta) grows KV by expandChunk; quant K/V; flash-attn blocks
|
||||
Pipe-->>Mod: logits (last token)
|
||||
Mod-->>Llm: outputs; mMeta.sync() (previous += add - remove)
|
||||
Llm->>Gen: generate(max_new) [OpenCL: OP_ENCODER_NUMBER_FOR_COMMIT=512]
|
||||
loop until stop token / max_new / USER_CANCEL / TIMEOUT
|
||||
Gen->>Samp: sample(logits) -> token
|
||||
Gen->>Llm: is_stop(token)? (sets NORMAL_FINISHED)
|
||||
Gen->>Tok: tokenizer_decode(token)
|
||||
Gen->>App: os << piece (flush)
|
||||
Gen->>Llm: forwardVec({token}) -> module key (1,false), mMeta.add=1
|
||||
Llm->>Mod: onForward (no resize; KV append)
|
||||
Mod-->>Gen: next logits
|
||||
end
|
||||
Gen-->>Llm: status MAX_TOKENS_FINISHED | NORMAL_FINISHED
|
||||
Llm->>Llm: updateCachedPromptText (prompt_cache)
|
||||
Llm-->>App: return (output_tokens in LlmContext)
|
||||
```
|
||||
|
||||
### (b) TTS: Qwen2.5-Omni thinker → talker → DiT → BigVGAN (streaming)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App
|
||||
participant Omni as Omni (omni.cpp)
|
||||
participant Thinker as Llm modules (llm.mnn)
|
||||
participant Talker as Talker (talker.mnn)
|
||||
participant Q as WavChunk queues
|
||||
participant DiT as ditWorkerLoop (predit.mnn + dit.mnn, own Executor)
|
||||
participant Voc as vocoderWorkerLoop (bigvgan.mnn clone)
|
||||
App->>Omni: setWavformCallback(cb)
|
||||
App->>Omni: response(prompt, os)
|
||||
Omni->>Talker: generate_init()
|
||||
Omni->>Thinker: prefill (+vision/audio embeddings via visual.mnn/audio.mnn)
|
||||
Thinker-->>Omni: logits + talker_embeds -> Talker.addTalkerEmbeds
|
||||
loop thinker decode (text streamed to os)
|
||||
Omni->>Thinker: forwardVec(token)
|
||||
Thinker-->>Omni: talker_embeds[i]
|
||||
end
|
||||
App->>Omni: generateWavform() (or interleaved: talker steps inside the text loop)
|
||||
Omni->>Talker: generate(): stepPrefill(concat(talker_embeds[0], text_bos+codec_pad, talker_embeds[1]+codec_bos))
|
||||
loop codec tokens (max talker_max_new_tokens) until 8292/8294
|
||||
Talker->>Talker: stepForward: embedding(prev)+talker_embeds[i]|text_pad -> forward -> sample (penalty,topK,topP,temp)
|
||||
Talker->>Q: trySubmitChunkAsync: when >= 24+60+12 tokens push WavChunk(codec_tokens, noise slice)
|
||||
end
|
||||
Talker->>Q: finalize(): push last chunk, wait mWavLastDone (<=60s)
|
||||
Q->>DiT: pop WavChunk
|
||||
DiT->>DiT: mPreDit({cond,spk,code}) -> code_embeds,rope,mask; dit_steps-1 ODE steps (Euler or RK4) of mDit({x,code_embeds,rope,mask,t}) -> mel[80, 2*codec]
|
||||
DiT->>Q: push chunk with mel (left/right padding sliced)
|
||||
Q->>Voc: pop mel chunk; concat into mMelBuffer
|
||||
Voc->>Voc: mBigvgan(mel) -> waveform; trim 8*240 samples context each side
|
||||
Voc->>App: cb(pcm_ptr, n_samples, is_last) (return false => USER_CANCEL)
|
||||
Omni-->>App: prints talker prefill/decode, token2wav time, tts rtf
|
||||
```
|
||||
|
||||
[Verified] Qwen3-TTS variant (`Omni::generateTTS` → `Talker::generateQwen3TTS`, [omni.cpp#L3222](../../MNN/transformers/llm/engine/src/omni.cpp#L3222)): prompt embedder (codec prefix + text + tts tokens, optional ECAPA speaker embedding from `--ref_audio`) → per-frame loop: talker → first code (2048 + EOS) → `code_predictor` autoregressively over 16 groups → `codec_embedder` next input → after all frames one `speech_decoder({codes[1,16,frames]}) → waveform` → single callback with `last=true`. Not chunk-streamed.
|
||||
|
||||
---
|
||||
|
||||
## 8. Tests and examples
|
||||
|
||||
- [Verified] Unit tests: [test/](../../MNN/test) (`op/` 103 files, `expr/` 21, `speed/` 17, `grad/`, `core/`, `plugin/`, `model/`, `cv/`, `kleidiai/`, `sharedmem/`); runner `./run_test.out [test_name] [backend 0=CPU,3=OpenCL] [precision] [thread/gpuMode] [flag] [memory]` ([test/main.cpp](../../MNN/test/main.cpp)); `MNN_TEST_SKIP` env; attention/KV tests [AttentionTest.cpp](../../MNN/test/op/AttentionTest.cpp), GEMM/GEMV microbenchmarks [GemmSpeed.cpp](../../MNN/test/speed/GemmSpeed.cpp), [GemvBWTest.cpp](../../MNN/test/speed/GemvBWTest.cpp).
|
||||
- [Verified] Driver [test.sh](../../MNN/test.sh) (`static | local | android <serial> [filter]`) with stages in [test_stages.json](../../MNN/test_stages.json) (`unit → lowmem → smokeA → smokeB → bench → llm`, filters `cpu|opencl-image|opencl-buffer|vulkan|lowmem|llm`, documented driver-bug skip rationales for Mali/OpenCL-image); [docs/testing.md](../../MNN/docs/testing.md); skill [skills/test-ci/SKILL.md](../../MNN/skills/test-ci/SKILL.md) incl. real-device iPhone LLM bench ([ios_llm_bench.sh](../../MNN/transformers/llm/engine/ios/ios_llm_bench.sh)).
|
||||
- [Verified] Model/tool tests: [tools/cpp](../../MNN/tools/cpp) (`MNNV2Basic`, `ModuleBasic`, `backendTest`, `testModel`, `timeProfile`, `GetMNNInfo`, `MNN2QNNModel`, `compilefornpu`); converter round-trips [tools/script](../../MNN/tools/script) (`testMNNFromOnnx.py/Tf/Tflite/Torch`, `testPTQ.py`, `testTransformerC4Switches.py`); pymnn [pymnn/test](../../MNN/pymnn/test).
|
||||
- [Verified] LLM: [engine/test](../../MNN/transformers/llm/engine/test) (`test_tokenizer`, `test_omni_video_prompt`, `test_multi_instance`), evals [transformers/llm/eval](../../MNN/transformers/llm/eval) (perplexity, C-Eval), [llm_bench.cpp](../../MNN/transformers/llm/engine/tools/llm_bench.cpp) (`-p 512` pp, `-n 128` tg, `-pg`), `bench_ttfa.py` (time to first audio), nightly [llm_nightly.sh](../../MNN/transformers/llm/benchmark/llm_nightly.sh).
|
||||
- [Verified] CI ([.github/workflows](../../MNN/.github/workflows)): `linux.yml` builds 3 configs and runs `run_test.out` (incl. LLM/vision/audio/OpenCL/Vulkan-enabled build); `macos.yml`, `windows.yml`, `android.yml` (build-only), `ios.yml`, pymnn wheels, `code-format.yml`. **No on-device GPU/NPU CI in GitHub Actions.**
|
||||
- [Verified] Examples: [demo/exec](../../MNN/demo/exec) (pictureRecognition, segment, multiPose, transformerDemo, expressDemo…), [benchmark/](../../MNN/benchmark) (`benchmark.out`, Android script), apps: [MnnLlmChat](../../MNN/apps/Android/MnnLlmChat) (LLM/VLM/audio-in/audio-out/diffusion, model market HF+ModelScope+Modelers), [MnnTaoAvatar](../../MNN/apps/Android/MnnTaoAvatar) (LLM+ASR+TTS+A2BS+NNR avatar), [iOS MNNLLMChat](../../MNN/apps/iOS/MNNLLMChat), [apps/sana](../../MNN/apps/sana), [apps/mnncli](../../MNN/apps/mnncli), TTS SDK demo [mnn_tts/demo/android](../../MNN/apps/frameworks/mnn_tts/demo/android).
|
||||
|
||||
---
|
||||
|
||||
## 9. TTS relevance
|
||||
|
||||
**What exists** [Verified]
|
||||
- **Engine-native speech output** (`Talker : Llm`, [omni.hpp](../../MNN/transformers/llm/engine/src/omni.hpp)):
|
||||
- Qwen2.5-Omni: `talker.mnn` (codec-token LM conditioned on thinker hidden states), `spk_dict.mnn` (speakers `Chelsie`, `Ethan`), `predit.mnn` + `dit.mnn` (flow-matching DiT, `dit_steps=5`, `dit_solver` 1 = Euler / 4 = RK4, forced `Precision_High` on GPU), `bigvgan.mnn` vocoder; 50 codec tokens ≈ 1 s (repo doc); chunked streaming (60-token DiT chunks, 24/12 context, vocoder 8/8 context at 240× upsample), async two-thread token2wav on CPU, interleaved thinker/talker mode (`interleaved=true`), RTF reporting.
|
||||
- Qwen3-TTS (`talker_type=qwen3_tts`): talker + `code_predictor` (16 groups × 2048) + `codec_embedder` + `speech_decoder` (upsample 1920) + `speaker_encoder` (ECAPA-TDNN; **reference audio required** — voice cloning only); languages `auto/chinese/english/german/italian/portuguese/spanish/japanese/korean/french/russian`; API `Llm::generateTTS(text, language, max_new_tokens, ref_audio)`; demo [qwen3_tts_demo.cpp](../../MNN/transformers/llm/engine/demo/qwen3_tts_demo.cpp).
|
||||
- Export: [utils/talker.py](../../MNN/transformers/llm/export/utils/talker.py) (`Qwen2_5OmniTalker`, `Qwen3TTSTalker`), [utils/token2wav.py](../../MNN/transformers/llm/export/utils/token2wav.py) (`DiTBlock`, `DiTAttention`, `DitRotary`, BigVGAN anti-alias surgery, `ECAPA_TDNN`, `Qwen2_5OmniToken2Wav`, `Qwen3TTSToken2Wav`).
|
||||
- Config keys ([llmconfig.hpp#L201](../../MNN/transformers/llm/engine/src/llmconfig.hpp#L201)): `has_talker, talker_model/weight/embedding_file, talker_type, predit_model, dit_model, bigvgan_model, spk_dict, talker_speaker, talker_max_new_tokens, dit_steps, dit_solver, interleaved, code_predictor_*, speech_decoder_*, speaker_encoder_*, tts_{bos,eos,pad}_token_id`.
|
||||
- App integration: Android `LlmSession_setWavformCallbackNative` writes `output.wav` @24 kHz and plays chunks ([llm_session.cpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp), `AudioChunksPlayer.kt`); iOS `setAudioWaveformCallback`; docs [llm.md](../../MNN/docs/transformers/llm.md) (AudioQueue playback example).
|
||||
- **Native TTS SDK** ([apps/frameworks/mnn_tts](../../MNN/apps/frameworks/mnn_tts)): `MNNTTSSDK::Process(text) → (sample_rate, Audio)` with backends Bert-VITS2 (Chinese/English G2P, BERT front-ends, `TTSGenerator`), Supertonic (duration predictor / text encoder / vector estimator / vocoder, multi-voice styles), Piper (espeak-ng phonemizer; currently commented out); Android service + JNI, iOS, desktop. Shipped models `bert-vits2-MNN` (1.39 GB) and `supertonic-tts-mnn` (464 MB) in [model_market.json](../../MNN/apps/Android/MnnLlmChat/app/src/main/assets/model_market.json).
|
||||
- **sherpa-mnn** ([apps/frameworks/sherpa-mnn](../../MNN/apps/frameworks/sherpa-mnn)): sherpa-onnx ported onto MNN: offline TTS VITS / Kokoro / Matcha + HiFiGAN, MeloTTS/Piper lexicons, streaming Zipformer ASR, VAD, speaker ID, C/Kotlin/Python APIs. Used by both Android apps for voice chat (ASR in, TTS out).
|
||||
- **Audio input (ASR side)**: `audio.mnn` encoders for Qwen2-Audio, Qwen3-ASR (`qwen3_asr`), Qwen2.5-Omni audio, FunAudioChat, LFM2-Audio, Gemma4 audio ([utils/audio.py](../../MNN/transformers/llm/export/utils/audio.py)); `MNN::AUDIO` front-ends (whisper/conformer/usm); Whisper itself is not a shipped pipeline (only its fbank).
|
||||
- **Ops**: `OpType_Stft` (shape/CPU kernel, [ShapeStft.cpp](../../MNN/source/shape/ShapeStft.cpp)); everything else (mel, iSTFT, vocoder convs/transposed convs/snake activations) is expressed as ordinary graph ops after export.
|
||||
|
||||
**What's missing** [Verified unless noted]
|
||||
- No CosyVoice / Fish-Speech / F5-TTS / SparkTTS / ChatTTS / Kokoro-native (only via sherpa) / SNAC / EnCodec / DAC / WavTokenizer; no Qwen3-Omni.
|
||||
- Qwen3-TTS path is not chunk-streamed (single `speech_decoder` call) and has no preset speakers (ref audio mandatory).
|
||||
- No sample-rate conversion on output; fixed 24 kHz assumption in demos.
|
||||
- No engine-level audio backpressure/queue bounds; no `USER_CANCEL` for text other than via wav callback.
|
||||
- Python binding lacks `setWavformCallback`/`generateTTS`.
|
||||
- [Proposal] For vLLM-Omni-style stage separation, the natural seams are `Talker::stepForward` (codec LM) and the `WavChunk` queues (DiT / vocoder) — they are already separate Modules with their own RuntimeManager/Executor, so they could be placed on CPU vs GPU or different cores.
|
||||
|
||||
---
|
||||
|
||||
## 10. VLA relevance
|
||||
|
||||
**What exists** [Verified]
|
||||
- Vision encoders exportable with dynamic resolution + M-RoPE: `Qwen2/2.5/3/3.5-VL`, `Qwen2.5-Omni`, `Qwen-VL`, `DeepSeek-VL`, `InternVL`, `GLM-OCR`, `Gemma3/4`, `Idefics3/SmolVLM`, `HunyuanVL`, `MiniCPM-V`, `MobileCLIP`, `LFM2-VL`, `FastVLM (llava_qwen2)` ([utils/vision.py](../../MNN/transformers/llm/export/utils/vision.py)); runtime preprocessing in `Omni::*VisionProcess` incl. video frame sampling; window attention masks; separate processor runtime (`mllm` config) so the encoder can run on another backend.
|
||||
- Structured prompt input: `MultimodalPrompt{images (raw RGB buffers), audios (waveform), videos (frames+timestamps)}`; `input_embeds` overload of `response/generate` lets a host inject arbitrary continuous embeddings (e.g. proprioception projected on the host) [Verified API; use for proprio is a Proposal].
|
||||
- Diffusion/flow machinery: `DiT` blocks + host-driven ODE loop (`Talker::ditForward`), SD3.5/Sana flow matching, Wan2.1 T2V; generic `Module` API can run any exported action-head graph; `Session_Resize_Check/Fix` keeps repeated fixed-shape denoise steps allocation-free.
|
||||
- Low-latency primitives: decode key-1 module (no resize), `reuse_kv`, `eraseHistory` (rollback), KV quant, fp16 ARM, KleidiAI/SME2, NPU delegation (QNN/HIAI/CoreML) for static-shape encoders.
|
||||
|
||||
**What's missing** [Verified]
|
||||
- Zero VLA/robotics code: no OpenVLA / π0 / SmolVLA / RDT / Octo / LeRobot, no action tokenizer/detokenizer (e.g. FAST), no action chunking, no proprio embedding path, no continuous-action head, no diffusion-policy scheduler, no control-loop/timer, no ROS/serial I/O.
|
||||
- No multi-request batching (action chunk sampling with CFG would need batch-2 graphs exported explicitly).
|
||||
- [Proposal] A VLA on MNN would be: export vision tower via `utils/vision.py`, export the LLM with `hidden_states=true` (already used by MTP/EAGLE) to expose the last hidden state, export the action head (MLP or DiT) as a separate `.mnn` Module, and drive the flow-matching loop on the host exactly as `Talker::ditForward` does.
|
||||
|
||||
---
|
||||
|
||||
## 11. Edge deployment profile
|
||||
|
||||
- **Build system** [Verified]: CMake ≥ 3.6, C++11 default (C++17 for CUDA+transformer-fuse, C++20 for MSVC+QNN), Google-style, `-fno-rtti -fno-exceptions`, `-fvisibility=hidden -ffunction-sections -fdata-sections`, `-O3` (no `-Os`), `--gc-sections -s` only for `MNN_BUILD_FOR_ANDROID_COMMAND` ([CMakeLists.txt](../../MNN/CMakeLists.txt)). Key options (defaults): `MNN_BUILD_SHARED_LIBS=ON`, `MNN_SEP_BUILD=ON` (separate `libMNN_CL/Vulkan/Express/OpenCV/Audio/llm/diffusion.so`), `MNN_USE_THREAD_POOL=ON`, `MNN_ARM82=ON`, `MNN_KLEIDIAI=ON` (`MNN_KLEIDIAI_DEFAULT_ON=OFF`), `MNN_SME2=ON`, `MNN_AVX2=ON`, `MNN_AVX512=OFF`, `MNN_USE_RVV=OFF`, `MNN_METAL/OPENCL/VULKAN/OPENGL/CUDA/MUSA/TENSORRT/COREML/NNAPI/QNN/RKNN/HEXAGON=OFF`, `MNN_LOW_MEMORY=OFF` (forced ON by `MNN_BUILD_LLM`), `MNN_SUPPORT_TRANSFORMER_FUSE=OFF` (forced ON by LLM/diffusion), `MNN_BUILD_LLM=OFF`, `MNN_BUILD_LLM_OMNI=OFF` (forces OpenCV+Audio+imgcodecs), `MNN_BUILD_DIFFUSION=OFF`, `MNN_BUILD_OPENCV=OFF`, `MNN_IMGCODECS=OFF`, `MNN_BUILD_AUDIO=OFF`, `MNN_WITH_PLUGIN=OFF`, `MNN_BUILD_MINI=OFF` (= `MNN_SKIPBUILD_GEOMETRY` + `MNN_REDUCE_SIZE`), `MNN_SUPPORT_BF16=OFF`, `MNN_USE_SYSTEM_LIB=OFF` (dlopen OpenCL/Vulkan), `MNN_JNI=OFF`, `MNN_BUILD_CONVERTER=OFF`, `MNN_BUILD_TEST=OFF`. `MNN_CPU_WEIGHT_DEQUANT_GEMM` was removed (comment in CMakeLists; superseded by `MNN_LOW_MEMORY`). Docs/code drift: [docs/compile/cmake.md](../../MNN/docs/compile/cmake.md) lists `MNN_USE_SSE` default OFF (code ON) and options no longer present (`MNN_METALLIB_SOURCE`, `MNN_VULKAN_DEBUG`).
|
||||
- **Binary size** [repo-reported, not reproduced]: [README.md](../../MNN/README.md) — iOS full static lib armv7+arm64 ≈ 12 MB (+2 MB executable growth), Android core `.so` ≈ 800 KB (armv7a, `c++_shared`), `MNN_BUILD_MINI` ≈ −25%; [docs/intro/about.md](../../MNN/docs/intro/about.md) — trimmed iOS static ≈ 6.1 MB (+600 KB). Op pruning: `GetMNNInfo` → `tools/script/prue_mnn_ops.py` rewrites `CPUOPRegister.cpp`/`GeometryOPRegister.cpp` ([docs/faq.md](../../MNN/docs/faq.md)).
|
||||
- **Runtime dependencies** [Verified]: minimal `libMNN`: Linux `-pthread -ldl`; Android `log m android`; OHOS `libhilog_ndk.z.so`; flatbuffers header-only vendored; protobuf converter-only; OpenGL is the only GPU backend hard-linked (`GLESv3`, `EGL`). Vendored [3rd_party](../../MNN/3rd_party): flatbuffers 1.10, half, stb (imageHelper), OpenCL headers, rapidjson, protobuf (converter), musa_compat. Fetched at build: KleidiAI v1.16.0, CUTLASS (CUDA), oneDNN (opt), googletest (cv/audio tests), QNN SDK zip ([prepare_qnn_deps.sh](../../MNN/prepare_qnn_deps.sh)). Additional LLM deps: cpp-httplib (vendored), jinja.cpp (vendored), OpenSSL only for `mls`.
|
||||
- **Platforms** [Verified]: Android NDK arm64-v8a (API 21, [project/android/build_64.sh](../../MNN/project/android/build_64.sh): OpenCL + LLM + diffusion) and armeabi-v7a (API 14, [package_scripts/android/build.sh](../../MNN/package_scripts/android/build.sh)); iOS ≥ 11 via [MNN.podspec](../../MNN/MNN.podspec) / `MNN.framework` ([package_scripts/ios/buildiOS.sh](../../MNN/package_scripts/ios/buildiOS.sh), Metal + ARM82, thread pool off); macOS arm64+x86_64 fat framework; Linux x86/arm64 (cross-compile toolchain [project/cross-compile](../../MNN/project/cross-compile)); Windows MSVC/clang incl. ARM64; OpenHarmony ([project/harmony/build_64.sh](../../MNN/project/harmony/build_64.sh)); WASM via emcmake ([docs/compile/engine.md](../../MNN/docs/compile/engine.md), LLM-on-WASM recipe in llm.md); RISC-V RVV + SpacemiT K3 IME2 ([riscv/CMakeLists.txt](../../MNN/source/backend/cpu/riscv/CMakeLists.txt), perf note [riscv_k3_ime2_asymmetric_w4b64.md](../../MNN/docs/perf/riscv_k3_ime2_asymmetric_w4b64.md)); no LoongArch.
|
||||
- **Accelerators** [Verified]: NVIDIA CUDA sm 60/61/62/70/72/75/80/86/**87**/89/120 ([cuda/CMakeLists.txt](../../MNN/source/backend/cuda/CMakeLists.txt); sm87 = Jetson Orin [Inferred], CUDA ≥ 11.4), TensorRT (`MNN_TRT_DYNAMIC` dlopen); Apple Metal (Metal4 tensor ops via `MNN_METAL_TENSOR`), CoreML/ANE (`MNN_FORWARD_NN`); Android OpenCL (dlopen search list for Mali/Adreno/PowerVR/pocl in [OpenCLWrapper.cpp](../../MNN/source/backend/opencl/core/runtime/OpenCLWrapper.cpp)), Vulkan (dlopen [vulkan_wrapper.cpp](../../MNN/source/backend/vulkan/runtime/vulkan_wrapper.cpp)), NNAPI (API ≥ 29), Qualcomm QNN/HTP (online: static-shape generic `.mnn`, `MNN_FORWARD_NN`; offline: `MNN2QNNModel` → `model_<soc>_<arch>.bin` + plugin `.mnn` run on CPU type with `MNN_WITH_PLUGIN`; SoC table 8 Gen1/2/3/Elite = 36/69, 43/73, 57/75, 69/79 in [docs/inference/npu.md](../../MNN/docs/inference/npu.md); LLM-on-QNN export [llm.md](../../MNN/docs/transformers/llm.md)), Hexagon HVX direct (`libMNN_htpops*.so` prebuilt with Hexagon SDK, [hexagon/README.md](../../MNN/source/backend/hexagon/README.md)), Huawei HIAI (`MNN_NPU`, prebuilt DDK libs), Rockchip RKNN (`MNNConvert --rknn` → wrapper `.mnn` + `.rknn`, fp32 output only, host copies), MediaTek NeuroPilot not shipped; Moore Threads MUSA.
|
||||
- **Model conversion path & constraints** [Verified]: generic CNN/transformer: `MNNConvert -f ONNX|TF|TFLITE|CAFFE|TORCH --modelFile … --MNNModel … [--fp16|--weightQuantBits N --weightQuantBlock B] [--saveExternalData] [--saveStaticModel]`; LLM: `llmexport.py --path <hf> --export mnn [--quant_bit 4 --quant_block 64 --hqq]`, needs `MNNConvert` (or pymnn) and produces fixed batch-1 graphs; fixed-shape backends (OpenCL/Vulkan/NPU) require `shapeMutable=false` + chunked prefill; QNN needs static shapes per (chunk, history) bucket; NPU export scripts are Qualcomm/MLDA specific; TTS needs `MNN_BUILD_AUDIO` (+ `LLM_SUPPORT_AUDIO` define) and vision needs `MNN_BUILD_OPENCV` (+`MNN_IMGCODECS`). Pre-converted models published on ModelScope `MNN` org and HF `taobao-mnn` ([docs/transformers/models.md](../../MNN/docs/transformers/models.md)).
|
||||
- **Python wheel** [Verified]: `pip install MNN` (repo-reported); source build `pymnn/pip_package` with deps string incl. `llm`, `opencl`, `vulkan`, `cuda`, `torch` ([docs/compile/pymnn.md](../../MNN/docs/compile/pymnn.md)); console tools `mnnconvert`, `mnnquant`.
|
||||
|
||||
---
|
||||
|
||||
## 12. Limitations and unknowns
|
||||
|
||||
- [Verified] No request-level scheduler, batching, or paged KV: one request per `Llm`, batch dim 1 baked into exported graphs.
|
||||
- [Verified] No engine API to cancel a running text `response()`; apps must step `generate(1)` themselves. `USER_CANCEL` only via wav callback.
|
||||
- [Verified] KV cache is a contiguous growable array per attention execution; no eviction beyond explicit `remove/reserve`; disk spill only with `kvcache_mmap`/`KVCACHE_SIZE_LIMIT` (the LLM engine never sets the size limit hint).
|
||||
- [Verified] Flash attention / KV quant knobs are CPU-first: Metal supports int8 KV + FA with head_dim ∈ {64,128,256}; OpenCL/Vulkan/CUDA ignore `attention_mode` (repo doc). TQ3/TQ4 KV quant is CPU-only.
|
||||
- [Verified] OpenCL/Vulkan image mode lacks `LinearAttention` (falls back to CPU); Vulkan low-bit kernels exist only in the buffer variant.
|
||||
- [Verified] W2/W3 weights need ARMv8.6 i8mm + fp16 hardware; W4 GEMV on Metal accepts only 2/3/4/8-bit.
|
||||
- [Verified] `Llm::tuning` is skipped on OpenCL; `Llm::load` performs no warmup, so first-token latency includes shader compile / tuning unless a cache file exists.
|
||||
- [Verified] Talker async token2wav only on CPU backend; DiT forced to fp32 on GPU (fp16 collapses the ODE).
|
||||
- [Verified] Qwen3-TTS requires reference audio (no preset speakers) and produces audio only after all frames; Qwen2.5-Omni speakers limited to Chelsie/Ethan.
|
||||
- [Verified] No Whisper/SenseVoice/Paraformer pipeline in the engine (ASR delegated to sherpa-mnn Zipformer or Qwen-audio encoders).
|
||||
- [Verified] No VLA/robotics code; no action tokenizer; no proprio path.
|
||||
- [Verified] NeuroPilot backend is a stub in OSS; HIAI/NPU option undeclared (`-DMNN_NPU=ON` by hand); NNAPI and CoreML cannot coexist (same forward type).
|
||||
- [Verified] No JIT/graph capture; GPU op launch overhead mitigated only by commit batching / record queues.
|
||||
- [Verified] Python binding has no TTS/audio-out API; no C API; no WASM binding (build recipe only); no on-device CI in GitHub Actions.
|
||||
- [Verified] Docs drift: cmake.md defaults vs code (`MNN_USE_SSE`, `MNN_AVX512_VNNI`), stale option names; `docs/inference/npu.md` omits Hexagon/NeuroPilot.
|
||||
- [Unknown] Actual latency/RTF numbers on target edge devices — only repo-reported figures exist (e.g. Metal prefill +3.6–3.9% note [metal_prefill_optimization_summary.md](../../MNN/docs/perf/metal_prefill_optimization_summary.md); MnnLlmChat README speed-up claims vs llama.cpp/fastllm); none reproduced here.
|
||||
- [Unknown] Behaviour of `schema/private/` / `source/internal/` features (model auth, metrics under `MNN_INTERNAL`) — not read per AGENTS.md.
|
||||
- [Unknown] Whether the bounded-thread `ThreadPool` (`MNN_THREAD_POOL_MAX_TASKS=2`) limits running vision encoder and talker concurrently on the same runtime — the Talker avoids it by creating a fresh `Executor` per worker thread.
|
||||
- [Inferred] Multi-model pipelines (encoder on NPU, LLM on CPU, vocoder on GPU) are feasible today via per-Module `RuntimeManager`s, but every cross-backend hop is a synchronous host copy (`onSync` + `copyToHostTensor`).
|
||||
|
||||
---
|
||||
|
||||
## 13. Reference index
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| [../../MNN/AGENTS.md](../../MNN/AGENTS.md) | Repo guidance; restricted dirs; architecture summary |
|
||||
| [../../MNN/README.md](../../MNN/README.md) | Overview; repo-reported size numbers; news (Qwen3.5, Qwen3-VL, Omni) |
|
||||
| [../../MNN/CMakeLists.txt](../../MNN/CMakeLists.txt) | All top-level build options, platform flags, link deps |
|
||||
| [../../MNN/MNN.podspec](../../MNN/MNN.podspec) | CocoaPods spec (iOS 11+) |
|
||||
| [../../MNN/prepare_qnn_deps.sh](../../MNN/prepare_qnn_deps.sh) | Downloads QNN SDK headers/libs |
|
||||
| [../../MNN/test.sh](../../MNN/test.sh), [../../MNN/test_stages.json](../../MNN/test_stages.json) | CI/test driver and stage matrix |
|
||||
| [../../MNN/include/MNN/MNNDefine.h](../../MNN/include/MNN/MNNDefine.h) | Version 3.6.1 |
|
||||
| [../../MNN/include/MNN/MNNForwardType.h](../../MNN/include/MNN/MNNForwardType.h) | Forward types, GPU mode flags, BackendConfig, RuntimeStatus |
|
||||
| [../../MNN/include/MNN/Interpreter.hpp](../../MNN/include/MNN/Interpreter.hpp) | Session API, SessionMode, HintMode, ExternalPathType |
|
||||
| [../../MNN/include/MNN/Tensor.hpp](../../MNN/include/MNN/Tensor.hpp) | Tensor, DimensionType, map/wait |
|
||||
| [../../MNN/include/MNN/ImageProcess.hpp](../../MNN/include/MNN/ImageProcess.hpp) | Image preprocessing API |
|
||||
| [../../MNN/include/MNN/MNNSharedContext.h](../../MNN/include/MNN/MNNSharedContext.h) | Only `extern "C"` header (shared GPU context) |
|
||||
| [../../MNN/include/MNN/expr/Module.hpp](../../MNN/include/MNN/expr/Module.hpp) | Module::load/Config/clone |
|
||||
| [../../MNN/include/MNN/expr/Executor.hpp](../../MNN/include/MNN/expr/Executor.hpp) | Executor, RuntimeManager, LazyMode |
|
||||
| [../../MNN/include/MNN/plugin](../../MNN/include/MNN/plugin) | Plugin op kernel/shape registration |
|
||||
| [../../MNN/source/core/Backend.hpp](../../MNN/source/core/Backend.hpp), [Backend.cpp](../../MNN/source/core/Backend.cpp) | Backend/Runtime/RuntimeHint; creator registry |
|
||||
| [../../MNN/source/core/RuntimeFactory.cpp](../../MNN/source/core/RuntimeFactory.cpp) | Runtime creation |
|
||||
| [../../MNN/source/core/Schedule.cpp](../../MNN/source/core/Schedule.cpp) | Backend type resolution, op scheduling |
|
||||
| [../../MNN/source/core/Session.cpp](../../MNN/source/core/Session.cpp) | Session lifecycle, hint translation, backup backend |
|
||||
| [../../MNN/source/core/Pipeline.cpp](../../MNN/source/core/Pipeline.cpp) | encode/allocMemory/execute, CPU fallback, copies, tuning |
|
||||
| [../../MNN/source/core/WrapExecution.cpp](../../MNN/source/core/WrapExecution.cpp) | Cross-backend/layout copy insertion |
|
||||
| [../../MNN/source/core/Execution.hpp](../../MNN/source/core/Execution.hpp) | Execution interface |
|
||||
| [../../MNN/source/core/BufferAllocator.hpp](../../MNN/source/core/BufferAllocator.hpp) | Eager/Defer allocators, mmap allocator |
|
||||
| [../../MNN/source/core/MNNMemoryUtils.h](../../MNN/source/core/MNNMemoryUtils.h) | 64-byte alignment |
|
||||
| [../../MNN/source/core/Tensor.cpp](../../MNN/source/core/Tensor.cpp) | Host/device copy and map |
|
||||
| [../../MNN/source/core/TensorUtils.hpp](../../MNN/source/core/TensorUtils.hpp) | NC4HW4 descriptor, Region |
|
||||
| [../../MNN/source/core/ConvolutionCommon.cpp](../../MNN/source/core/ConvolutionCommon.cpp) | Quantized weight decode |
|
||||
| [../../MNN/source/core/KVMeta.hpp](../../MNN/source/core/KVMeta.hpp), [KVCacheManager.hpp](../../MNN/source/core/KVCacheManager.hpp) | KV cursor and storage base |
|
||||
| [../../MNN/source/geometry/GeometryComputer.hpp](../../MNN/source/geometry/GeometryComputer.hpp), [GeometryComputer.cpp](../../MNN/source/geometry/GeometryComputer.cpp), [GeometryOPRegister.cpp](../../MNN/source/geometry/GeometryOPRegister.cpp) | Geometry decomposition |
|
||||
| [../../MNN/source/shape/SizeComputer.hpp](../../MNN/source/shape/SizeComputer.hpp), [ShapeRegister.cpp](../../MNN/source/shape/ShapeRegister.cpp), [ShapeStft.cpp](../../MNN/source/shape/ShapeStft.cpp) | Shape inference |
|
||||
| [../../MNN/source/backend/cpu/CPUBackend.cpp](../../MNN/source/backend/cpu/CPUBackend.cpp), [CPUOPRegister.cpp](../../MNN/source/backend/cpu/CPUOPRegister.cpp), [CPURuntime.cpp](../../MNN/source/backend/cpu/CPURuntime.cpp), [ThreadPool.cpp](../../MNN/source/backend/cpu/ThreadPool.cpp) | CPU backend, dispatch, core detection, thread pool |
|
||||
| [../../MNN/source/backend/cpu/CPUAttention.cpp](../../MNN/source/backend/cpu/CPUAttention.cpp), [CPUKVCacheManager.hpp](../../MNN/source/backend/cpu/CPUKVCacheManager.hpp), [CPUKVCacheManager.cpp](../../MNN/source/backend/cpu/CPUKVCacheManager.cpp) | Attention, KV quant, disk KV |
|
||||
| [../../MNN/source/backend/cpu/CPUPlugin.cpp](../../MNN/source/backend/cpu/CPUPlugin.cpp) | Plugin op execution |
|
||||
| [../../MNN/source/backend/cpu/compute/CommonOptFunction.h](../../MNN/source/backend/cpu/compute/CommonOptFunction.h) | CoreFunctions table (pack/bytes/kernels) |
|
||||
| [../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp](../../MNN/source/backend/cpu/compute/ConvolutionFloatFactory.cpp) | Low-memory conv/GEMM selection |
|
||||
| [../../MNN/source/backend/cpu/compute/ImageProcessFunction.cpp](../../MNN/source/backend/cpu/compute/ImageProcessFunction.cpp) | Image preprocessing kernels |
|
||||
| [../../MNN/source/backend/cpu/arm/arm64/low_memory](../../MNN/source/backend/cpu/arm/arm64/low_memory) | W2/W3/W4 GEMM and dynamic quant asm |
|
||||
| [../../MNN/source/backend/cpu/riscv/CMakeLists.txt](../../MNN/source/backend/cpu/riscv/CMakeLists.txt) | RISC-V RVV / SpacemiT build |
|
||||
| [../../MNN/source/backend/arm82/Arm82Backend.hpp](../../MNN/source/backend/arm82/Arm82Backend.hpp) | fp16 backend (NC8HW8) |
|
||||
| [../../MNN/source/backend/metal/MetalBackend.mm](../../MNN/source/backend/metal/MetalBackend.mm), [MetalOPRegister.mm](../../MNN/source/backend/metal/MetalOPRegister.mm), [MetalAttention.mm](../../MNN/source/backend/metal/MetalAttention.mm), [AllShader.cpp](../../MNN/source/backend/metal/AllShader.cpp), [MetalCache.fbs](../../MNN/source/backend/metal/schema/MetalCache.fbs) | Metal backend |
|
||||
| [../../MNN/source/backend/opencl/core/OpenCLBackend.cpp](../../MNN/source/backend/opencl/core/OpenCLBackend.cpp), [OpenCLOPRegister.cpp](../../MNN/source/backend/opencl/core/OpenCLOPRegister.cpp), [OpenCLWrapper.cpp](../../MNN/source/backend/opencl/core/runtime/OpenCLWrapper.cpp), [AttentionBufExecution.cpp](../../MNN/source/backend/opencl/execution/buffer/AttentionBufExecution.cpp) | OpenCL backend |
|
||||
| [../../MNN/source/backend/vulkan/runtime/VulkanRuntime.cpp](../../MNN/source/backend/vulkan/runtime/VulkanRuntime.cpp), [vulkan_wrapper.cpp](../../MNN/source/backend/vulkan/runtime/vulkan_wrapper.cpp), [buffer/backend/VulkanBackend.cpp](../../MNN/source/backend/vulkan/buffer/backend/VulkanBackend.cpp), [VulkanAttention.cpp](../../MNN/source/backend/vulkan/buffer/execution/VulkanAttention.cpp), [buffer/compiler](../../MNN/source/backend/vulkan/buffer/compiler) | Vulkan backend |
|
||||
| [../../MNN/source/backend/cuda/Register.cpp](../../MNN/source/backend/cuda/Register.cpp), [cuda/CMakeLists.txt](../../MNN/source/backend/cuda/CMakeLists.txt), [AttentionExecution.cu](../../MNN/source/backend/cuda/execution/AttentionExecution.cu) | CUDA backend |
|
||||
| [../../MNN/source/backend/opengl/GLBackend.cpp](../../MNN/source/backend/opengl/GLBackend.cpp) | OpenGL backend |
|
||||
| [../../MNN/source/backend/coreml](../../MNN/source/backend/coreml), [nnapi](../../MNN/source/backend/nnapi), [qnn](../../MNN/source/backend/qnn), [hiai](../../MNN/source/backend/hiai), [hexagon](../../MNN/source/backend/hexagon), [hexagon/README.md](../../MNN/source/backend/hexagon/README.md), [rknn](../../MNN/source/backend/rknn), [neuropilot/CMakeLists.txt](../../MNN/source/backend/neuropilot/CMakeLists.txt), [musa](../../MNN/source/backend/musa), [tensorrt](../../MNN/source/backend/tensorrt) | NPU / other accelerator backends |
|
||||
| [../../MNN/express/Executor.cpp](../../MNN/express/Executor.cpp), [Utils.hpp](../../MNN/express/Utils.hpp), [module/Module.cpp](../../MNN/express/module/Module.cpp), [module/StaticModule.cpp](../../MNN/express/module/StaticModule.cpp), [module/PipelineModule.hpp](../../MNN/express/module/PipelineModule.hpp) | Express runtime and Module implementations |
|
||||
| [../../MNN/schema/default/MNN.fbs](../../MNN/schema/default/MNN.fbs), [CaffeOp.fbs](../../MNN/schema/default/CaffeOp.fbs), [TensorflowOp.fbs](../../MNN/schema/default/TensorflowOp.fbs), [Tensor.fbs](../../MNN/schema/default/Tensor.fbs), [UserDefine.fbs](../../MNN/schema/default/UserDefine.fbs), [TFQuantizeOp.fbs](../../MNN/schema/default/TFQuantizeOp.fbs), [Type.fbs](../../MNN/schema/default/Type.fbs), [ExtraInfo.fbs](../../MNN/schema/default/ExtraInfo.fbs), [TrainInfo.fbs](../../MNN/schema/default/TrainInfo.fbs) | Model schema |
|
||||
| [../../MNN/codegen/CMakeLists.txt](../../MNN/codegen/CMakeLists.txt) | Source-level fusion codegen options |
|
||||
| [../../MNN/cmake/KleidiAI.cmake](../../MNN/cmake/KleidiAI.cmake) | KleidiAI fetch/integration |
|
||||
| [../../MNN/tools/converter/source/MNNConverter.cpp](../../MNN/tools/converter/source/MNNConverter.cpp), [common/cli.cpp](../../MNN/tools/converter/source/common/cli.cpp), [common/writeFb.cpp](../../MNN/tools/converter/source/common/writeFb.cpp), [include/config.hpp](../../MNN/tools/converter/include/config.hpp), [optimizer/PostConverter.cpp](../../MNN/tools/converter/source/optimizer/PostConverter.cpp), [optimizer/merge](../../MNN/tools/converter/source/optimizer/merge), [postconvert/FuseTransformerC4.cpp](../../MNN/tools/converter/source/optimizer/postconvert/FuseTransformerC4.cpp) | Converter |
|
||||
| [../../MNN/tools/quantization/calibration.cpp](../../MNN/tools/quantization/calibration.cpp) | Offline int8 calibration |
|
||||
| [../../MNN/tools/cpp](../../MNN/tools/cpp), [compilefornpu.cpp](../../MNN/tools/cpp/compilefornpu.cpp) | Test/benchmark/NPU-compile tools |
|
||||
| [../../MNN/tools/script](../../MNN/tools/script) | Converter/model test scripts, op pruning |
|
||||
| [../../MNN/tools/cv/include/cv/cv.hpp](../../MNN/tools/cv/include/cv/cv.hpp) | OpenCV-like API |
|
||||
| [../../MNN/tools/audio/include/audio/audio.hpp](../../MNN/tools/audio/include/audio/audio.hpp) | Audio front-end API |
|
||||
| [../../MNN/transformers/llm/export/llmexport.py](../../MNN/transformers/llm/export/llmexport.py), [gguf2mnn.py](../../MNN/transformers/llm/export/gguf2mnn.py), [safetensors2mnn.py](../../MNN/transformers/llm/export/safetensors2mnn.py), [utils/model_mapper.py](../../MNN/transformers/llm/export/utils/model_mapper.py), [utils/vision.py](../../MNN/transformers/llm/export/utils/vision.py), [utils/audio.py](../../MNN/transformers/llm/export/utils/audio.py), [utils/talker.py](../../MNN/transformers/llm/export/utils/talker.py), [utils/token2wav.py](../../MNN/transformers/llm/export/utils/token2wav.py), [utils/mnn_converter.py](../../MNN/transformers/llm/export/utils/mnn_converter.py), [utils/mnn_utils.py](../../MNN/transformers/llm/export/utils/mnn_utils.py), [export/npu](../../MNN/transformers/llm/export/npu) | LLM/VLM/TTS export |
|
||||
| [../../MNN/transformers/llm/engine/include/llm/llm.hpp](../../MNN/transformers/llm/engine/include/llm/llm.hpp), [reranker.hpp](../../MNN/transformers/llm/engine/include/llm/reranker.hpp) | Public LLM API |
|
||||
| [../../MNN/transformers/llm/engine/src/llm.cpp](../../MNN/transformers/llm/engine/src/llm.cpp), [llmconfig.hpp](../../MNN/transformers/llm/engine/src/llmconfig.hpp), [omni.hpp](../../MNN/transformers/llm/engine/src/omni.hpp), [omni.cpp](../../MNN/transformers/llm/engine/src/omni.cpp), [embedding.cpp](../../MNN/transformers/llm/engine/src/embedding.cpp), [diskembedding.cpp](../../MNN/transformers/llm/engine/src/diskembedding.cpp), [sampler.cpp](../../MNN/transformers/llm/engine/src/sampler.cpp), [kvmeta.hpp](../../MNN/transformers/llm/engine/src/kvmeta.hpp), [prompt_cache_utils.hpp](../../MNN/transformers/llm/engine/src/prompt_cache_utils.hpp) | LLM engine core |
|
||||
| [../../MNN/transformers/llm/engine/src/speculative_decoding](../../MNN/transformers/llm/engine/src/speculative_decoding), [generate.cpp](../../MNN/transformers/llm/engine/src/speculative_decoding/generate.cpp) | AR / lookahead / MTP / EAGLE / DFlash generation |
|
||||
| [../../MNN/transformers/llm/engine/src/tokenizer/tokenizer.cpp](../../MNN/transformers/llm/engine/src/tokenizer/tokenizer.cpp), [jinja.hpp](../../MNN/transformers/llm/engine/src/tokenizer/jinja.hpp) | Tokenizers, chat templates |
|
||||
| [../../MNN/transformers/llm/engine/demo](../../MNN/transformers/llm/engine/demo), [qwen3_tts_demo.cpp](../../MNN/transformers/llm/engine/demo/qwen3_tts_demo.cpp) | CLI demos |
|
||||
| [../../MNN/transformers/llm/engine/app/mls_server.cpp](../../MNN/transformers/llm/engine/app/mls_server.cpp) | Minimal OpenAI-style HTTP server |
|
||||
| [../../MNN/transformers/llm/engine/tools/llm_bench.cpp](../../MNN/transformers/llm/engine/tools/llm_bench.cpp), [engine/test](../../MNN/transformers/llm/engine/test), [test_multi_instance.cpp](../../MNN/transformers/llm/engine/test/test_multi_instance.cpp), [engine/ios](../../MNN/transformers/llm/engine/ios), [ios_llm_bench.sh](../../MNN/transformers/llm/engine/ios/ios_llm_bench.sh) | LLM bench/tests/iOS demo |
|
||||
| [../../MNN/transformers/llm/eval](../../MNN/transformers/llm/eval), [benchmark/llm_nightly.sh](../../MNN/transformers/llm/benchmark/llm_nightly.sh) | Evals and nightly |
|
||||
| [../../MNN/transformers/diffusion/engine/include/diffusion/diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/diffusion.hpp), [sana_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/sana_diffusion.hpp), [wan_diffusion.hpp](../../MNN/transformers/diffusion/engine/include/diffusion/wan_diffusion.hpp), [src/scheduler.hpp](../../MNN/transformers/diffusion/engine/src/scheduler.hpp), [wan_diffusion_demo.cpp](../../MNN/transformers/diffusion/engine/wan_diffusion_demo.cpp), [diffusion/export](../../MNN/transformers/diffusion/export) | Diffusion engine |
|
||||
| [../../MNN/apps/Android/MnnLlmChat](../../MNN/apps/Android/MnnLlmChat), [llm_session.cpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp), [llm_stream_buffer.hpp](../../MNN/apps/Android/MnnLlmChat/app/src/main/cpp/llm_stream_buffer.hpp), [model_market.json](../../MNN/apps/Android/MnnLlmChat/app/src/main/assets/model_market.json) | Android chat app |
|
||||
| [../../MNN/apps/Android/MnnTaoAvatar](../../MNN/apps/Android/MnnTaoAvatar) | Avatar app (LLM+ASR+TTS+A2BS+NNR) |
|
||||
| [../../MNN/apps/iOS/MNNLLMChat](../../MNN/apps/iOS/MNNLLMChat), [LLMInferenceEngineWrapper.h](../../MNN/apps/iOS/MNNLLMChat/MNNLLMiOS/InferenceEngine/LLMInferenceEngineWrapper.h) | iOS chat app |
|
||||
| [../../MNN/apps/frameworks/mnn_tts](../../MNN/apps/frameworks/mnn_tts), [mnn_tts/demo/android](../../MNN/apps/frameworks/mnn_tts/demo/android) | Native TTS SDK (Bert-VITS2 / Supertonic / Piper) |
|
||||
| [../../MNN/apps/frameworks/sherpa-mnn](../../MNN/apps/frameworks/sherpa-mnn) | sherpa-onnx port (TTS/ASR/VAD) |
|
||||
| [../../MNN/apps/mnncli](../../MNN/apps/mnncli), [mnncli_server.cpp](../../MNN/apps/mnncli/src/mnncli_server.cpp) | CLI + OpenAI/Anthropic-compatible server |
|
||||
| [../../MNN/apps/sana](../../MNN/apps/sana) | Sana image-edit app |
|
||||
| [../../MNN/pymnn/src/llm.h](../../MNN/pymnn/src/llm.h), [pymnn/src/audio.h](../../MNN/pymnn/src/audio.h), [pymnn/test](../../MNN/pymnn/test) | Python bindings |
|
||||
| [../../MNN/project/android/build_64.sh](../../MNN/project/android/build_64.sh), [project/harmony/build_64.sh](../../MNN/project/harmony/build_64.sh), [project/cross-compile](../../MNN/project/cross-compile), [package_scripts/android/build.sh](../../MNN/package_scripts/android/build.sh), [package_scripts/ios/buildiOS.sh](../../MNN/package_scripts/ios/buildiOS.sh) | Platform build/packaging scripts |
|
||||
| [../../MNN/demo/exec](../../MNN/demo/exec), [benchmark](../../MNN/benchmark) | Classic demos and benchmark |
|
||||
| [../../MNN/test](../../MNN/test), [test/main.cpp](../../MNN/test/main.cpp), [test/op/AttentionTest.cpp](../../MNN/test/op/AttentionTest.cpp), [test/speed/GemmSpeed.cpp](../../MNN/test/speed/GemmSpeed.cpp), [test/speed/GemvBWTest.cpp](../../MNN/test/speed/GemvBWTest.cpp) | Unit tests |
|
||||
| [../../MNN/.github/workflows](../../MNN/.github/workflows) | CI |
|
||||
| [../../MNN/skills/test-ci/SKILL.md](../../MNN/skills/test-ci/SKILL.md) | Test/CI agent skill |
|
||||
| [../../MNN/docs/contribute/op.md](../../MNN/docs/contribute/op.md), [docs/contribute/backend.md](../../MNN/docs/contribute/backend.md) | Extension recipes |
|
||||
| [../../MNN/docs/transformers/llm.md](../../MNN/docs/transformers/llm.md), [models.md](../../MNN/docs/transformers/models.md) | LLM export/config/backends/TTS docs; model hubs |
|
||||
| [../../MNN/docs/inference/npu.md](../../MNN/docs/inference/npu.md) | NPU backends (QNN/CoreML/NNAPI/HIAI/RKNN) |
|
||||
| [../../MNN/docs/compile/cmake.md](../../MNN/docs/compile/cmake.md), [engine.md](../../MNN/docs/compile/engine.md), [pymnn.md](../../MNN/docs/compile/pymnn.md) | Build docs |
|
||||
| [../../MNN/docs/intro/about.md](../../MNN/docs/intro/about.md), [docs/faq.md](../../MNN/docs/faq.md), [docs/testing.md](../../MNN/docs/testing.md) | Size claims, pruning, testing |
|
||||
| [../../MNN/docs/perf/metal_prefill_optimization_summary.md](../../MNN/docs/perf/metal_prefill_optimization_summary.md), [riscv_k3_ime2_asymmetric_w4b64.md](../../MNN/docs/perf/riscv_k3_ime2_asymmetric_w4b64.md) | Repo-reported perf notes |
|
||||
| [../../MNN/3rd_party](../../MNN/3rd_party) | Vendored deps |
|
||||
@@ -0,0 +1,220 @@
|
||||
# Mobile NPU simulation with the Qualcomm AI Runtime SDK (Hexagon HTP), and the CPU speed-ups it triggered
|
||||
|
||||
Goal set on 2026-09-10: install Qualcomm's AI Runtime SDK with its Hexagon HTP (NPU) simulator, use it to stand in for mobile devices, and make the branch beat the other engines on CPU and on mobile. This page records what was installed, what runs on the simulator, what the simulator can and cannot tell us, and the CPU changes measured on the way. Raw artifacts: `tools/qairt/` (SDK, runtime libs, Python env notes), `analysis/experiments/edge_engine_compare/results/cpu_opt/` (CPU runs), the session scratch directory for ONNX/DLC files (not committed, 460 MB each).
|
||||
|
||||
## 1. What was installed and how
|
||||
|
||||
| Item | Where | Notes |
|
||||
|---|---|---|
|
||||
| Qualcomm AI Runtime **Community** SDK 2.37.0.250724 (formerly QNN / AI Engine Direct) | `tools/qairt/qairt/2.37.0.250724/` | Downloaded from Qualcomm's software center without a login (the URL ExecuTorch's `backends/qualcomm/scripts/qnn_config.sh` pins); 1.35 GB zip. Contains the converters (`qairt-converter`, `qairt-quantizer`), `qnn-net-run`, `qnn-context-binary-generator`, profiling readers, and the backends for x86 (`libQnnCpu.so`, `libQnnHtp.so` = **HTP functional simulator + offline graph preparation**, `libQnnHtpQemu.so` = HTP QEMU backend, id 13), Android aarch64 and Hexagon v66–v79 skels. |
|
||||
| LLVM runtime libraries the SDK binaries link against | `tools/qairt/runtime_libs/` (`libc++.so.1`, `libc++abi.so.1`, `libunwind.so.1`) | Not on this host and no root: unpacked from Ubuntu jammy `libc++1-14`, `libc++abi1-14` and `libunwind-18` `.deb` files with `dpkg-deb -x`; used via `LD_LIBRARY_PATH`. |
|
||||
| Python 3.10 environment for the SDK tools | `.venvs/qairt` (uv) | The SDK supports 3.8/3.10 only. Its `check-python-dependency` installer needs `setuptools<81` (for `pkg_resources`) and does **not** install `onnx`; `onnx==1.16.2`, `onnxruntime==1.18.1`, `protobuf<5` were added. |
|
||||
| ONNX exporter dependency | `onnxscript` in the CUDA venv | The TorchScript exporter fails on `aten::diff` in the decoder; the dynamo exporter (`torch.onnx.export(..., dynamo=True)`) works. |
|
||||
| Qualcomm AI Hub | not installed | It is a cloud service (real devices behind an API token), not a local simulator; nothing on this host can stand in for it. |
|
||||
|
||||
Traps hit (all fixed in the scripts under `analysis/experiments/edge_engine_compare/` and `tools/qairt/`): converter rejects ONNX `Reshape` with `allowzero=1` (dynamo emits it; cleared), rejects the `IsNaN`→`Where` guards the dynamo exporter puts after every softmax (stripped), and the HTP graph preparation fails on an **int Gather from a float table** (`node_embedding`, the RVQ codebook lookup) with `could not create op: q::Gather`, so the codebook lookup was moved to the host and the HTP graph takes the summed codebook embeddings as a float input.
|
||||
|
||||
## 2. What runs on the HTP simulator
|
||||
|
||||
Pipeline (scripts: `scratchpad/qnn/export_htp_decoder.py`, `htp_pipeline.sh`, `run_qnn.sh`):
|
||||
|
||||
1. `WindowedCode2Wav` (branch `vllm_omni/edge/decoder_export.py`, context 72 frames + chunk 25) → `HtpBody` variant whose input is `quantized [1, 512, 97]` (RVQ sum done on the host) → ONNX (dynamo, opset 18, 458 MB fp32).
|
||||
2. `qairt-converter --float_bitwidth 16` → DLC (fp16), 229 Reshape / 62 FullyConnected / 55 StridedSlice / 45 Transpose / 17 RmsNorm / 16 MatMul / 31 Conv2d / 6 ConvTranspose / 8 Softmax / ElementWiseNeuron (SnakeBeta) — every op accepted by the converter.
|
||||
3. `qnn-net-run --backend libQnnHtp.so` on x86 with the HTP backend extensions (`dsp_arch v75`, `O=3`, `vtcm_mb 8`): graph preparation for HTP v75 **succeeds** (RmsNorm and Gelu were flagged "no properties registered" during prepare but did not stop it), execution runs on the functional simulator.
|
||||
|
||||
Parity: the same DLC on the QNN **CPU** backend (fp32) reproduces the torch decoder bit-for-bit (SNR 107.6 dB, max |diff| 7e-6 over 48,000 samples) — the conversion is correct. The HTP fp16 execution of the 2-second decoder chunk was still running on the functional simulator when this document was written (3 h of wall time, 5 cores): the chunk is 481 GFLOP and the simulator executed the 2.67 GFLOP predictor body at ≈16 MFLOP/s, so the run should finish ≈8.5 h after its 01:38 start. A background chain runs `parity_dec_htp.py` against the torch reference when it exits and writes `analysis/experiments/edge_engine_compare/qnn/run/decoder_htp_parity.txt` (plus the profiling records). The same graph on the QEMU backend is not possible on this host (below).
|
||||
|
||||
**Code predictor on the HTP simulator (real weights, complete).** Because the 15-pass code predictor is the CPU bottleneck (§3), it is the component that matters most on a phone NPU. Its 5-layer body (all 56 tensors loaded from the checkpoint, fused qkv/gate-up) was exported the same way (`export_predictor_htp.py`, input `x [1, 17, 1024]`), converted to fp32 and fp16 DLC (`Mul 82 / Add 46 / Reshape 35 / MatMul …`, all accepted), and run:
|
||||
|
||||
| backend | result |
|
||||
|---|---|
|
||||
| QNN CPU backend, fp32 DLC | SNR **111.2 dB** vs torch, 41 ms per inference on 16 host threads (reference kernels) |
|
||||
| **HTP v75 functional simulator, fp16 DLC** | graph preparation 2.3 s, execution completes; SNR **50.7 dB** / corr 0.999996 vs the fp32 torch body, i.e. fp16 HTP math is faithful for the predictor |
|
||||
| HTP profiler (`--profiling_level detailed`, HTP profiling reader) | per-op records are produced (e.g. `rms_norm_0`, `node_linear` …) but their "cycles" are simulator cycles (3.3e11 for one pass), not Hexagon device cycles |
|
||||
|
||||
**HTP QEMU backend (`libQnnHtpQemu.so`, backend id 13).** Tried for a second, closer-to-silicon data point. It refuses the HTP backend-extension config (`Unable to find HTP interface provider`) and without it fails at context creation (`err 0x3e8`); its strings explain why: it only loads *offline-prepared* context binaries and launches a Hexagon emulator subprocess whose command line must come from `QNN_HTP_QEMU_ARGS`, i.e. it needs the `qemu-hexagon` binary and the Hexagon skel/`libQnnHtpV75Skel.so` from the **Hexagon SDK**, which is a separate Qualcomm Package Manager download (account-gated) and not part of the community AI Runtime SDK. So on this host the HTP path is validated functionally (prepare + execute on the x86 simulator), not cycle-accurately.
|
||||
|
||||
Artifacts: `analysis/experiments/edge_engine_compare/qnn/` (scripts, converter logs, run logs, profiling records, execution metadata); the ONNX/DLC files (150–460 MB) stay in the session scratch directory.
|
||||
|
||||
What the simulator does and does not give:
|
||||
- It is a **functional** simulator ("x86 Linux host backend serves as a functional simulator for the hardware accelerator and supports graph preparation", SDK backend docs): op support, graph preparation for a chosen Hexagon arch, fp16/int8 numerics, and the context binary that would be deployed. Wall time on x86 is not device time (one 2 s chunk takes minutes here).
|
||||
- Per-op **cycle counts** come from the HTP profiler only when running on the device (the docs' detailed profiling "provides per op profiling result by cycle counts"); on the x86 simulator the profiling reader returns the host-side timeline. A device-time number therefore still needs a phone or Qualcomm AI Hub. The QEMU backend (`libQnnHtpQemu.so`) is exercised the same way (see below).
|
||||
|
||||
## 3. CPU: making the branch faster than llama.cpp on the same host
|
||||
|
||||
Starting point (§5 of [edge_device_simulation.md](edge_device_simulation.md)): 134 ms per 80 ms frame at 12+4 threads, of which 97 ms were the fifteen code-predictor sub-steps and 10 ms the sampler; llama.cpp's native path costs 85–97 ms per frame plus a 55 ms/frame serial vocoder (RTF 1.9–2.0).
|
||||
|
||||
What was found and changed (branch commit `6ce8de1b`):
|
||||
|
||||
| finding | change | effect (0.6B, 12+4 threads, 12 prompts) |
|
||||
|---|---|---|
|
||||
| each of the 15 sub-steps re-ran the 5-layer predictor over the whole prefix (up to 17 positions), no KV cache | `forward_cached`: per-layer KV cache, 2-token prefill then one token per sub-step; default on CPU | sub-step 6.4 → 4.7 ms; frame 134 → 108 ms; RTF 1.7 → 1.38 |
|
||||
| the sub-step is per-operator overhead, not bandwidth (1.6 ms floor); OpenMP thread count irrelevant (4/2 threads: no change); per-submodule `torch.compile` worse (+2 ms, guard overhead per call) | shape-static `forward_static_step` (tensor position, boolean mask) compiled as one graph | micro-benchmark 70 → 58 ms per frame for the loop; engine run: see table below |
|
||||
| bf16 M=1 GEMV kernels carry ~30 µs fixed overhead each; int8-dynamic Linear has none and halves the bytes | dynamic int8 quantization of every predictor Linear + lm heads on CPU (fp32 activations), default on | micro-benchmark 70 → 36 ms per frame for the loop |
|
||||
| vLLM's stock sampler spends 9 ms per step in `compiled_random_sample` (a `torch.compile`'d kernel) on CPU | talker `sample()` with `prefer_model_sampler` on CPU: same math eagerly (penalties, temperature, top-k, exponential noise with per-request generators) | 9 → <0.5 ms per step |
|
||||
|
||||
Engine runs (`benchmarks/tts/stream_latency_bench.py --step-stats`, 0.6B CustomVoice, 12 prompts, NUMA node 1 pinned; per-frame numbers are the stage-0 engine step and its parts):
|
||||
|
||||
| variant | threads | talker step ms/frame | talker fwd | predictor loop (15 sub-steps) | sampler | TTFA p50 | playback-start p50 | RTF |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| before (re-prefill predictor, stock sampler) | 12+4 | 134 | 25 | 97 (6.4/sub-step) | 10 | 273 ms | 4.9 s | 1.7 |
|
||||
| KV-cached predictor | 12+4 | 108 | 26 | 71 (4.7) | 9 | 251 ms | 3.6 s | 1.38 |
|
||||
| + predictor 4 / 2 OpenMP threads | 12+4 | 108 | 25 | 71 (4.6–4.7) | 9 | 241–254 ms | 3.5–3.6 s | 1.36–1.39 |
|
||||
| + per-submodule torch.compile (rejected) | 12+4 | 131–139 | 26 | 94–101 (6.2–6.7) | 9 | 266–272 ms | 4.7 s | 1.62–1.63 |
|
||||
| + compiled shape-static step (bf16) + fast sampler | 12+4 | 114 | 26 | 84 (5.6) | 0.6 | 240 ms | 3.1 s | 1.27 |
|
||||
| + fast CPU sampler (bf16 KV-cached predictor) | 12+4 | 99 | 25 | 71 (4.7) | 0.6 | 245 ms | 3.6 s | 1.39 |
|
||||
| + int8-dynamic predictor + fast sampler (rejected for fidelity, below) | 12+4 | 68 | 26 | 38 (2.5) | 0.7 | 212 ms | 2.0 s | 0.98 |
|
||||
| int8-dynamic predictor, 24+8 threads (rejected for fidelity, below) | 32 | 59 | 21 | 34 (2.2) | 0.6 | 175 ms | 1.7 s | 0.80 |
|
||||
| same, 6+2 threads | 8 | 144 | 39 | 102 (6.8) | 0.6 | 372 ms | 5.8 s | 1.92 |
|
||||
| **fp32 predictor + compiled static step + fast sampler (faithful; the new default)** | 12+4 | **93** | 25 | **64 (4.3)** | 0.6 | **225 ms** | 2.7 s | **1.15** |
|
||||
| same, 24+8 threads | 32 | **77** | 21 | 53 (3.5) | 0.7 | **201 ms** | 1.9 s | **0.88** |
|
||||
| int8-dynamic predictor, Base checkpoint with reference speaker (x-vector) | 12+4 | 80 | 26 | 38 (2.5) | 0.7 | 696 ms (incl. per-request x-vector) | 2.6 s | 1.12 (was 1.77) |
|
||||
| **fp32 default, Base checkpoint with reference speaker (x-vector), the row the llama.cpp comparison uses** | 12+4 | **109** (mean; p90 158) | 25 | 69 (4.6) | <1 | 686 ms (incl. per-request x-vector) | 3.0 s | **1.27** (was 1.77) |
|
||||
| same, 24+8 threads | 32 | **91** (mean; p90 111) | 21 | 56 (3.7) | <1 | 486 ms | 2.2 s | **0.99** (was 1.7) |
|
||||
|
||||
Folding the per-sub-step lm head, top-k Gumbel sampling, embedding and projection into a second compiled graph (branch commit `3085f501`) changes little on this host (sub-step 4.25 → 4.16 ms); the compiled fp32 transformer step itself is now the cost, i.e. fp32 weight traffic (4.8 GB per frame for the 15 passes), which is why the 6+2-thread class stays at 145–148 ms per frame (RTF 1.9): there the answer is a real int8/int4 GEMV kernel (PyTorch's `_weight_int8pack_mm` on this build is an unoptimized reference, 10× slower than fp32) or llama.cpp-style quantized weights, not more compilation.
|
||||
|
||||
Against llama.cpp on the same host and threads (native C++, Q4_0/f16 GGUF, [edge_engine_comparison.md](edge_engine_comparison.md) §4): 85–97 ms per frame for talker + predictor plus a 55 ms/frame serial vocoder, RTF 1.9–2.0 at 8–32 threads. With the faithful fp32 predictor the branch generates a frame in 77–93 ms including sampling, with the vocoder in a concurrent stage: **RTF 0.88 (real time) on 32 threads and 1.15 on 16 threads, against llama.cpp's 1.9–2.0 at any thread count**, i.e. 1.7–2.3× faster end-to-end, and at parity per talker frame at 16 threads (93 vs 85–97 ms). The remaining terms are the talker forward (21–26 ms, vLLM's bf16 CPU backend) and the eager glue in each sub-step (now compiled; 2 % gain).
|
||||
|
||||
Fidelity, measured in situ (`VLLM_OMNI_PREDICTOR_PARITY_CHECK=1`: every sub-step also runs the untouched bf16 predictor on the same inputs and compares logits):
|
||||
|
||||
GPU regression check after the CPU changes (all of them are gated on the CPU platform: `_setup_cpu_fast_path` and `prefer_model_sampler` are no-ops on CUDA): 0.6B CustomVoice, edge profile, one H200 through the scheduler, 12 prompts: TTFA p50 44.5 ms, playback-start 44.5 ms, stall 0, RTF 0.086, init 77.6 s — inside the 40.7–54 ms / 0.079–0.097 band of the 2026-09-09 edge-profile runs (`results/gpu_sanity/`).
|
||||
|
||||
| predictor variant vs bf16 reference | top-1 agreement | top-5 overlap | KL(ref‖test) | max \|logit diff\| |
|
||||
|---|---|---|---|---|
|
||||
| fp32 weights, compiled static step (control: only bf16 rounding differs) | 93.8 % | 95.3 % | 0.0015 | 0.23 |
|
||||
| torch.ao dynamic int8 (per-tensor int8 activations, per-channel int8 weights) | **22.2 %** | – | – | 7.0 |
|
||||
| int8 transformer body, fp32 lm heads (`ao-nohead`) | 23.7 % | 33.3 % | 0.76 | 7.0 |
|
||||
|
||||
So the fast int8 rows above are **not usable as-is**: per-tensor activation quantization of the residual stream destroys the residual-codebook predictions even though the same quantization is exact on a random model (the real activations have outliers). torchao's dynamic-activation path is faithful in eager mode but numerically wrong under `torch.compile` on CPU (cosine 0.02 vs fp32, it caused 160 s runaway utterances), and its weight-only path is faithful but slow (78 ms per frame). The shipped default is therefore **fp32 weights + compiled static step** (`VLLM_OMNI_PREDICTOR_INT8=fp32`); int8 stays opt-in for measurement.
|
||||
|
||||
## 4. Mobile: what can be claimed, and what cannot
|
||||
|
||||
What can be claimed for mobile after this session, and what cannot:
|
||||
|
||||
- **Claimable**: both halves of the on-device TTS path now exist as HTP-ready artifacts with verified numerics — the Code2Wav decoder (fp16 DLC, HTP graph prepared for v75, bit-exact on the QNN CPU backend) and the code predictor body (fp16 DLC, executed on the HTP simulator at 50.7 dB vs fp32). The talker backbone (28 layers, a standard Qwen3 decoder) is the same class of graph as the predictor and the remaining export; the codec-token protocol (WP5) and the branch's decoder export already give the phone client its contract. No other engine in this study runs Qwen3-TTS on a Hexagon NPU today (llama.cpp and MNN target the CPU/GPU on Android), so on the NPU the branch's artifacts are ahead in *deployability*.
|
||||
- **Not claimable**: device speed. The x86 HTP backend is a functional simulator; its profiler reports simulator cycles, and Qualcomm AI Hub (real devices) is a cloud service that needs an account/API token this host does not have. A "faster than MNN/llama.cpp on a phone" statement therefore cannot be measured here; the honest projection is qualitative: the predictor's 15 sequential passes are exactly the kind of small-batch, launch-bound work an NPU with a prepared context binary executes in a few milliseconds each, which is what would turn the CPU-side 63–100 ms predictor loop into a sub-frame cost.
|
||||
- **Next concrete steps**: (1) export the talker decode step with a static KV cache the same way; (2) run all three graphs on a Snapdragon device or AI Hub for cycle-accurate numbers; (3) an int8 GEMV kernel for the CPU predictor at 8 threads (the only CPU class where llama.cpp's Q4_0 talker is still faster per frame).
|
||||
|
||||
## 5. Real Snapdragon phones through Qualcomm AI Hub (2026-09-10, afternoon)
|
||||
|
||||
With the user's AI Hub token the same graphs were compiled to QNN context binaries and profiled on physical devices in Qualcomm's farm: **Samsung Galaxy S24 (Snapdragon 8 Gen 3)** and **Samsung Galaxy S25 (Snapdragon 8 Elite for Galaxy)**, every operator placed on the Hexagon NPU (`--compute_unit npu`; AI Hub confirms 100 % NPU placement for all builds). Times are AI Hub's `estimated_inference_time` (median of on-device runs). Scripts, job ids and the raw profile JSON (per-op cycles) are in `analysis/experiments/edge_engine_compare/qnn/aihub/`. Int-weight builds use AI Hub's compile-time quantization with random calibration unless marked *calibrated*; they are timing rows, the calibrated rows carry the fidelity.
|
||||
|
||||
### 5.1 Per-component device timings
|
||||
|
||||
| graph (real weights) | build | Galaxy S24 | Galaxy S25 | note |
|
||||
|---|---|---|---|---|
|
||||
| code predictor body, 17 tokens (§2 graph) | fp16 | 2.73 ms | 2.18 ms | 300/300 ops on NPU; the x86 simulator needed ~3 min for the same pass |
|
||||
| code predictor cached step, 1 token, 16-slot cache | fp16 | 2.59 ms | 2.26 ms | same as the 17-token body: weight-streaming bound (160 MB fp16 per step) |
|
||||
| same | w8a16 | 1.32 ms | 1.16 ms | ×15 sub-steps = **20 / 17 ms per frame** |
|
||||
| same | w4a16 | 1.27 ms | 1.11 ms | no gain over int8: half the step is fixed overhead |
|
||||
| talker decode step, 28 layers, 256-token cache, v1 export (stacked cache + select, expanded GQA) | fp16 | 39.7 ms | 36.9 ms | only 22–24 % of cycles in the linear layers |
|
||||
| same v1 | w8a16 / w4a16 | 30.0 / 30.2 ms | 23.6 / 23.6 ms | int4 = int8: overhead-bound |
|
||||
| **talker decode step, v2 export** (per-layer cache I/O, GQA by reshaping q, no cache select) | fp16 | **17.9 ms** | **15.8 ms** | 72 % of cycles in the linear layers, i.e. bandwidth-bound |
|
||||
| same v2 | **w8a16** | **12.1 ms** | **9.35 ms** | |
|
||||
| Code2Wav decoder chunk, 25 frames (2.0 s) with 72-frame context | fp16 | 5,197 ms | 4,539 ms | **208 / 182 ms per emitted frame**: 76 % of cycles in the Snake activation (mul·sin·pow) on 96–192 ch × 62k–186k samples |
|
||||
| same | w8a16 | 4,350 ms | 4,876 ms | 16-bit activations keep the slow elementwise path |
|
||||
| same | int8 (w8a8, random calibration) | **301 ms** | **266 ms** | 12.0 / 10.6 ms per emitted frame; 17× faster than fp16 |
|
||||
| same, 24-frame context (49-frame window) | int8 (random calibration) | 140 ms | 118 ms | 2.2× faster, but **not a valid optimization**: truncating the context to 24 frames costs 15.9 dB against the 72-frame output on the host (48 frames: 23.0 dB, 36: 19.6, 16: 12.2), so the window is doing real work. The fp16 build of this export also failed AI Hub's context-binary conversion (exit code 15). |
|
||||
|
||||
Memory: every build peaks at 100–250 MB of working memory on top of the memory-mapped context binary (talker fp16 context ≈ 0.9 GB, int8 ≈ 0.45 GB; decoder fp16 ≈ 0.2 GB).
|
||||
|
||||
### 5.1b The NPU is the wrong unit for the vocoder (Adreno GPU, same phone)
|
||||
|
||||
The vocoder profile said the cost was elementwise math on sample-rate tensors, which is a bandwidth problem, not a matrix problem — so the same graphs were compiled to TFLite and profiled on the **Adreno GPU** and the CPU of the Galaxy S25:
|
||||
|
||||
| graph, fp16/fp32 TFLite | Hexagon NPU (QNN) | Adreno GPU | phone CPU |
|
||||
|---|---|---|---|
|
||||
| Code2Wav decoder chunk (2.0 s) | 4,539 ms | **542 ms** | 2,735 ms |
|
||||
| → per emitted 80 ms frame | 182 ms | **21.7 ms** | 109 ms |
|
||||
| talker decode step (v2) | **15.8 ms** | 29.9 ms | 29.8 ms |
|
||||
| code predictor cached step | **2.25 ms** | 4.33 ms | 4.82 ms |
|
||||
|
||||
The GPU runs the vocoder **8.4× faster than the NPU**, and the NPU runs the transformer steps ~2× faster than the GPU. The right deployment is therefore heterogeneous, and it maps onto the branch's existing two-stage pipeline exactly: **stage 0 (talker + code predictor) on the NPU, stage 1 (Code2Wav) on the GPU**, which on this phone already run in separate processes on the server.
|
||||
|
||||
Fidelity and precision on the GPU. The TFLite GPU delegate runs fp16 by default; `--tflite_options` controls that, and the choice is a real one (Galaxy S25, same graph, same input):
|
||||
|
||||
| GPU configuration | per 2 s chunk | per emitted frame | SNR vs torch fp32 |
|
||||
|---|---|---|---|
|
||||
| default (fp16 tensors) | 542–572 ms | 21.7–22.9 ms | 27.5 dB (corr 0.9991) |
|
||||
| `allow_fp32_as_fp16=false` | 1,145 ms | **45.8 ms** | **108.4 dB** (corr 0.99999999999) |
|
||||
| `gpu_inference_priority1=…MAX_PRECISION` | 2,882 ms | 115.3 ms | 106.6 dB — slower, no more accurate |
|
||||
| `qnn_gpu_precision=kGpuFp32` / `kGpuFp16` | 569–580 ms | 22.8–23.2 ms | 27.5 dB — the flag changed nothing here |
|
||||
| ONNX Runtime on the GPU (`--target_runtime onnx`) | 3,063 ms | 122.5 ms | 115.6 dB |
|
||||
| (reference) TFLite on the phone CPU | 2,735 ms | 109 ms | 106.6 dB |
|
||||
|
||||
So the vocoder can run **numerically exact on the Adreno GPU at 45.8 ms per frame** — still 4× faster than fp16 on the NPU — and there is no reason to accept the fp16 delegate's −27 dB noise floor unless the extra 24 ms per frame matters.
|
||||
|
||||
### 5.1c Which mobile *runtime*, on the same graphs and the same phone
|
||||
|
||||
The other half of "compared with other engines": llama.cpp cannot reach a phone's accelerators for this model at all (no Hexagon backend, CPU-only vocoder), so on-device the meaningful comparison is between the runtimes that can execute these graphs. All rows are the same two exported graphs on the Galaxy S25, fp16 unless noted, per 80 ms frame:
|
||||
|
||||
| runtime | talker step, NPU | talker step, GPU | talker step, CPU | vocoder, NPU | vocoder, GPU | vocoder, CPU |
|
||||
|---|---|---|---|---|---|---|
|
||||
| QNN context binary | **15.8 ms** | – | – | 182 ms | – | – |
|
||||
| TensorFlow Lite | 16.3 ms | 29.9 ms | 29.8 ms | fails to run after compiling | **21.7 ms** (fp16) / **45.8 ms** (fp32, exact) | 109 ms |
|
||||
| ONNX Runtime | 16.6 ms | – | 33.1 ms | out of device memory | 122.5 ms | 124.3 ms |
|
||||
|
||||
Reading: on the NPU all three runtimes land within 5 % of each other, because they all end up in QNN — the runtime choice does not matter there, the *export layout* does (§5.1). The runtimes separate on the other units, and the one decision worth making is the vocoder's: TensorFlow Lite on the GPU is 2.7× faster than ONNX Runtime on the same GPU and 8.4× faster than QNN on the NPU. Neither TFLite nor ONNX Runtime can run the vocoder on the NPU at all (one fails after compiling, the other exceeds device memory), so the QNN context binary is the only NPU vocoder path — and it is the slowest option of all.
|
||||
|
||||
### 5.1d Beyond the two phones: laptop, automotive and IoT Snapdragons
|
||||
|
||||
The same two graphs on other Snapdragon classes in the AI Hub farm (talker step as a QNN context binary on the NPU,
|
||||
vocoder as TFLite on the GPU):
|
||||
|
||||
| device (class) | talker step, NPU | vocoder, GPU (per emitted frame) |
|
||||
|---|---|---|
|
||||
| Galaxy S25 — 8 Elite (phone) | 15.8 ms | 21.7 ms |
|
||||
| Galaxy S24 — 8 Gen 3 (phone) | 17.9 ms | – |
|
||||
| Snapdragon X Elite CRD (Windows laptop) | 19.6 ms | TFLite is not a supported target for this device |
|
||||
| SA8775P ADP (automotive) | 23.8 ms | 68.0 ms |
|
||||
| Dragonwing RB3 Gen 2 — QCS6490 (low-end IoT) | rejected: "Tensor 'x' has a floating-point type which is not supported by the targeted device" | 124.3 ms |
|
||||
|
||||
Reading: the transformer step is within 1.5× across a phone, a laptop and an automotive board, so the NPU story
|
||||
generalises; the QCS6490 tier is integer-only, which is exactly the case where the (currently broken) int8 path would
|
||||
be mandatory rather than optional. The vocoder's GPU cost scales with the GPU tier, and only the two phone GPUs are
|
||||
fast enough to keep it under the 80 ms frame budget.
|
||||
|
||||
### 5.2 Per-frame budget on the phone (measured components, 80 ms frame)
|
||||
|
||||
| configuration (Galaxy S25) | talker | predictor ×15 | vocoder | serial | RTF | pipelined | RTF |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| everything on the NPU, fp16 | 15.8 | 33.9 | 182 | 232 ms | 2.9 | 182 ms | 2.3 |
|
||||
| NPU steps + **GPU vocoder, fp32 (exact)** | 15.8 | 33.9 | **45.8** | 96 ms | 1.19 | **49.7 ms** | **0.62** |
|
||||
| NPU steps + GPU vocoder, fp16 delegate | 15.8 | 33.9 | 21.7 | 71 ms | 0.89 | **49.7 ms** | **0.62** |
|
||||
| everything on the GPU (fp16) | 29.9 | 65.0 | 21.7 | 117 ms | 1.5 | 95 ms | 1.2 |
|
||||
|
||||
The pipelined column is what the branch's architecture gives: the talker stage and the Code2Wav stage are separate engines exchanging codec tokens, so on a phone they overlap and the frame cost is the slower stage, not the sum. In that mode the talker stage is the bottleneck at 49.7 ms, the vocoder is free either way, and **the exact-precision vocoder costs nothing** — so the deliverable configuration is talker + predictor on the NPU in fp16 and Code2Wav on the GPU in fp32, at **RTF 0.62 on a Snapdragon 8 Elite, real time with a 1.6× margin, with no quantization anywhere**.
|
||||
|
||||
Earlier all-int8 numbers (37–44 ms per frame) are not deliverable: every automatic integer build here is numerically broken (§5.3). They remain as an upper bound on what a proper quantization effort could add.
|
||||
|
||||
Not in the budget: host-side embedding lookups, sampling and the RVQ codebook sum (sub-millisecond on a phone CPU), and per-call dispatch overhead (0.1–0.3 ms × 17 calls per frame on QNN). Each graph was measured in isolation, so these are sums or maxima of medians, not a pipeline measurement.
|
||||
|
||||
### 5.3 Fidelity on device
|
||||
|
||||
On-device inference jobs with the same inputs as the x86 runs (`decoder_quant_chain.py`, outputs in `aihub/out_*.npy`):
|
||||
|
||||
| graph, build | device | SNR vs torch fp32 | corr |
|
||||
|---|---|---|---|
|
||||
| Code2Wav decoder chunk, fp16 | Galaxy S25 / S24 | **39.2 / 39.1 dB** | 0.99994 |
|
||||
| code predictor body, fp16 | Galaxy S25 | **48.3 dB** | 0.999993 (x86 simulator: 50.7 dB) |
|
||||
| code predictor body, w8a16 / w4a16, AI Hub compile-time quantization (random calibration) | Galaxy S25 | 1.0 / 0.3 dB | 0.61 / 0.54: the random-data activation ranges are wrong, so these timing builds are not fidelity evidence |
|
||||
| code predictor body, w8a16 / w8a8, quantize job calibrated on 6 draws of the input distribution (held-out test input) | Galaxy S25 | **3.1 / 0.7 dB** | 0.77 / 0.39; profiles at 2.09 / 1.23 ms |
|
||||
| Code2Wav decoder, int8 (w8a8) calibrated on the test window (AI Hub quantize job, min–max ranges) | S25 / S24 | **−3.3 dB** | 0.00 (real-looking signal, uncorrelated with the reference) |
|
||||
| same, calibrated on 7 held-out windows of real codec tokens (696 frames from the WP5 codec-stream runs) | S25 / S24 | **−4.3 / −4.3 dB** | 0.00 (same failure; the calibrated build profiles at 282 ms on the S25) |
|
||||
|
||||
So **fp16 on the Hexagon NPU is faithful and every integer build produced here is not**:
|
||||
|
||||
- fp16: decoder 39 dB (the fp16 accumulation limit of a 481-GFLOP vocoder; the same DLC on the QNN CPU backend was 107 dB), predictor 48 dB. Both are usable.
|
||||
- The vocoder cannot take 8-bit activations at all: per-tensor ranges cannot hold the Snake activation (`x + sin²(ax)/a`) on sample-rate tensors, and calibrating on real codec-token windows does not help (−4.3 dB, output uncorrelated with the reference). The w8a16 build keeps the fidelity story open but runs as slowly as fp16 (4.4–4.9 s per chunk), so it buys nothing.
|
||||
- The code predictor is damaged by **int8 weights alone**: calibrated w8a16 (int8 weights, int16 activations, correct activation ranges) reaches only 3.1 dB / corr 0.77. This is the same failure the CPU experiment found for torch.ao per-tensor int8 (22 % top-1 agreement, §3) and it is a property of the model, not of the runtime: the predictor's residual heads need per-channel scales or quantization-aware training.
|
||||
|
||||
The x86 simulator run of the fp16 decoder (§2) was stopped after 15 h once this device parity existed.
|
||||
|
||||
### 5.4 Reading
|
||||
|
||||
- **The export mattered more than the device.** The first talker export spent 70 % of its NPU cycles on KV-cache plumbing; the v2 layout (per-layer cache I/O, grouped-query attention by reshaping the query) is 2.2× faster and bandwidth-bound. That layout is what the branch's HTP export should emit.
|
||||
- **The unit mattered even more.** The Hexagon NPU is 8.4× *slower* than the Adreno GPU on the vocoder (whose cost is elementwise Snake math on sample-rate tensors) and ~2× *faster* on the transformer steps. Sending everything to the NPU — the obvious reading of "run it on the phone's AI accelerator" — is the worst of the three placements measured.
|
||||
- **What runs faithfully today**: talker + predictor on the NPU (fp16, 39–48 dB) and Code2Wav on the GPU in **fp32** (108 dB, i.e. exact), ≈96 ms per 80 ms frame serially and **≈50 ms pipelined** on a Snapdragon 8 Elite. That is real time with a 1.6× margin, with no quantization and no precision compromise. The export code is in `vllm_omni/edge/qnn_export.py` (transformer steps) and `vllm_omni/edge/decoder_export.py` (vocoder).
|
||||
- **What still needs work**: a stateful streaming vocoder would cut the GPU's 21.7 ms further (the chunk synthesizes 3.9× more samples than it emits, and simply shortening the context is not valid — 24 frames of context costs 15.9 dB); int8 needs per-channel scales or QAT to be usable at all.
|
||||
- **Against the other engines on mobile**: llama.cpp and MNN have no Hexagon NPU path for Qwen3-TTS, and llama.cpp's CPU-only design extrapolates above RTF 2 on a phone's CPU cores (1.9–2.0 on a 16–32-thread Xeon). The branch is the only engine here with a validated on-device accelerator path, and with the NPU/GPU split it is real time on the phone while llama.cpp is not.
|
||||
@@ -0,0 +1,524 @@
|
||||
# ncnn (Tencent/ncnn) — Engine Analysis
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Local path | `/data/zhoutaichang/embedding_infer/ncnn` |
|
||||
| Remote | `https://github.com/Tencent/ncnn.git` (origin) |
|
||||
| Branch | `master` |
|
||||
| HEAD | `3b7bdba7fc8aea8fd46779533eee027df77c639d` — "Fix SIGFPE with missing or incomplete CPU topology (#6967)", 2026-09-07 |
|
||||
| Dirty status | clean (`git status --porcelain` empty) |
|
||||
| Submodules | **not initialized**: `glslang` (780b6a20…) and `python/pybind11` (d03662f0…) show `-` in `git submodule status` |
|
||||
| Latest tag in checkout | `20260526` (README links the 20260526 release archives) |
|
||||
| Analysis date | 2026-09-08 |
|
||||
| Repo AGENTS.md / CLAUDE.md | none present (checked `AGENTS.md`, `CLAUDE.md`, `.github/copilot-instructions.md`) |
|
||||
|
||||
Evidence labels used below: **[Verified]** read in code/docs, **[Inferred]** reasoned from structure, **[Proposal]** design suggestion, **[Unknown]** could not determine.
|
||||
|
||||
---
|
||||
|
||||
## 0. Summary
|
||||
|
||||
- **What it is.** ncnn is a dependency-free C++ inference *library* (not a serving stack): a static graph runtime with a `Net` (graph + weights) and `Extractor` (per-request lazy evaluator) API, hand-optimized CPU kernels (ARM NEON/fp16/bf16/dotprod/i8mm/SVE, x86 SSE2→AVX512-FP16/VNNI/BF16, RISC-V V/Zfh, MIPS MSA, LoongArch LSX/LASX) and a Vulkan compute GPU backend. [Verified] ([README](../../ncnn/README.md), [src/net.h](../../ncnn/src/net.h), [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt))
|
||||
- **No serving layer.** There is no HTTP server, request queue, scheduler, or continuous batching; the "runtime" is `Net::load_param/load_model` + `Extractor::input/extract`. Request-level scheduling is entirely the application's job. [Verified] (§5)
|
||||
- **Graph execution is lazy pull-based.** `Extractor::extract(blob)` recursively runs `NetPrivate::forward_layer` on the producer chain until the requested blob is materialized; unrequested branches never run. [Verified] ([src/net.cpp#L123](../../ncnn/src/net.cpp#L123), [#L2871](../../ncnn/src/net.cpp#L2871))
|
||||
- **Backends in tree:** CPU (with runtime ISA dispatch) and Vulkan only. No CUDA, Metal, CoreML, OpenCL, NNAPI, QNN. Apple GPUs are reached through MoltenVK (`libMoltenVK.dylib` is dlopen'ed by the in-house loader). [Verified] ([src/simplevk.cpp#L558](../../ncnn/src/simplevk.cpp#L558), [FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md))
|
||||
- **Transformer/LLM support is real and recent (2025–2026).** Layers `MultiHeadAttention`, `SDPA` (with GQA and flash-attention style CPU/Vulkan kernels), `RotaryEmbed`, `RMSNorm`, `Embed`, `Gemm`, `GLU`, `GELU` exist with arch-specific kernels; an in-layer **KV cache** (`7=1` param, `cache_k/v_in/out` blobs, `Extractor::set_kvcache_allocator`, `set_kvcache_max_seqlen_hint`, `extract(..., type=1)`) is documented and tested. [Verified] ([docs/developer-guide/kvcache.md](../../ncnn/docs/developer-guide/kvcache.md), [src/layer/sdpa.cpp](../../ncnn/src/layer/sdpa.cpp), [src/net.h#L193](../../ncnn/src/net.h#L193))
|
||||
- **LLM benchmark harness ships in-tree**: `benchncnn_llm` measures 256-token prefill and cached single-token decode for Qwen2.5-0.5B, Qwen3-0.6B, MiniCPM4-0.5B, Hunyuan-0.5B, TinyLlama-1.1B, Llama3.2-1B, Youtu-LLM-2B decoders (param files only; weights are random). [Verified] ([benchmark/benchncnn_llm.cpp](../../ncnn/benchmark/benchncnn_llm.cpp), [benchmark/CMakeLists.txt](../../ncnn/benchmark/CMakeLists.txt))
|
||||
- **LLM weight quantization exists**: block-quantized Gemm/MHA (W4/W6 weight-only, W8A8 dynamic per-block on CPU) via `ncnnllm2table`/`ncnnllm2int` (minmax/mseclip/awq/gptq), gated by `NCNN_WEIGHT_QUANT`; plus classic PTQ int8 via `ncnn2table`/`ncnn2int8`. [Verified] ([docs/how-to-use-and-FAQ/quantized-int8-inference.md](../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md), [tools/quantize](../../ncnn/tools/quantize))
|
||||
- **TTS exists as an example, not a runtime**: `examples/piper.cpp` runs a VITS (Piper) pipeline split into 5 ncnn nets (enc_p, emb_g, dp, flow, dec) with custom layers registered from the app, a dictionary-based English phonemizer and WAV writer. `examples/whisper.cpp` runs Whisper ASR (fbank net → encoder → cached decoder with beam search). Audio ops `Spectrogram`/`InverseSpectrogram` (STFT/iSTFT) and 1D conv/deconv exist. [Verified] ([examples/piper.cpp](../../ncnn/examples/piper.cpp), [examples/whisper.cpp](../../ncnn/examples/whisper.cpp), [src/layer/spectrogram.h](../../ncnn/src/layer/spectrogram.h))
|
||||
- **No tokenizer, no sampler, no vocoder library.** Whisper example carries a minimal reverse-vocab tokenizer; Piper carries a dict phonemizer. Nothing generic. [Verified]
|
||||
- **VLA: nothing specific.** Vision encoders (CNN/ViT) are first-class; MLP/Gemm heads trivial; diffusion/flow policies would require the host to loop denoise steps calling one Extractor per step (no scheduler code). [Verified]/[Inferred] (§10)
|
||||
- **Model format**: text `.param` + flat `.bin` (magic 7767517), produced mainly by **pnnx** (PyTorch/TorchScript/ONNX → ncnn; pip installable) with fp16 weights by default; legacy converters for Caffe/ONNX/Darknet/MXNet/Keras/TF-MLIR. [Verified] ([docs/developer-guide/param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md), [tools/pnnx/README.md](../../ncnn/tools/pnnx/README.md))
|
||||
- **Dynamic shapes** are natural: every `Mat` carries its own dims; layers compute output shapes at forward time; param shape hints (`30=`) are optional; `Reshape` supports runtime **expressions** over input shapes. [Verified] ([docs/developer-guide/expression.md](../../ncnn/docs/developer-guide/expression.md))
|
||||
- **Threading** is OpenMP-inside-kernels (`#pragma omp parallel for num_threads(opt.num_threads)`), with big.LITTLE affinity via `set_cpu_powersave`, a bundled minimal OpenMP runtime (`simpleomp`) and a bundled Vulkan loader (`simplevk`) so no external runtime deps are needed. [Verified] ([src/cpu.h](../../ncnn/src/cpu.h), [src/simpleomp.h](../../ncnn/src/simpleomp.h), [docs/developer-guide/vulkan-driver-loader.md](../../ncnn/docs/developer-guide/vulkan-driver-loader.md))
|
||||
- **Vulkan shaders are GLSL embedded as source and compiled to SPIR-V at runtime** by the in-tree glslang, with a persistent `PipelineCache` (SPIR-V + driver blob) to cut startup. [Verified] ([cmake/ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake), [docs/developer-guide/vulkan-pipeline-cache.md](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md))
|
||||
- **Edge portability is the strongest asset**: 60 CMake toolchains (Android, iOS/tvOS/visionOS/watchOS via ios.toolchain, HarmonyOS, Jetson, QNX, ESP32, RISC-V bare-metal, HiSilicon, Loongson, MIPS, PowerPC, Windows XP, WebAssembly), 45 CI workflows, `NCNN_SIMPLESTL/SIMPLEOMP/SIMPLEOCV/SIMPLEMATH` shims, per-layer `WITH_LAYER_x` opt-out for size. [Verified] ([toolchains](../../ncnn/toolchains), [docs/how-to-use-and-FAQ/build-minimal-library.md](../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md))
|
||||
- **Batching**: `NCNN_BATCH` adds an `n` dimension to `Mat`; layers lacking `support_batch` are looped per item by the Net. KV cache supports **batch size 1 only**. [Verified] ([src/net.cpp#L662](../../ncnn/src/net.cpp#L662), [#L2882](../../ncnn/src/net.cpp#L2882))
|
||||
|
||||
---
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph App["Application (C++/C/Python/JNI)"]
|
||||
A1[Mat::from_pixels / audio PCM / token ids] --> A2[Net.load_param + load_model]
|
||||
A2 --> A3[Extractor.input / extract]
|
||||
end
|
||||
|
||||
subgraph Runtime["ncnn core (src/)"]
|
||||
N[Net / NetPrivate<br/>layers[], blobs[], custom registry] --> E[Extractor / ExtractorPrivate<br/>blob_mats[], blob_mats_gpu[], Option]
|
||||
E -->|lazy pull| FL[NetPrivate::forward_layer]
|
||||
FL --> CL[convert_layout<br/>fp16/bf16 cast + elempack]
|
||||
CL --> DL[do_forward_layer<br/>batch loop, lightmode inplace]
|
||||
DL --> L[Layer_final<br/>layer_cpu / layer_vulkan]
|
||||
O[Option<br/>num_threads, use_fp16_*, use_int8_*, use_bf16_storage, use_packing_layout, allocators, kvcache_*] --> FL
|
||||
P[ParamDict / ModelBin / DataReader] --> N
|
||||
AL[Allocator / PoolAllocator / UnlockedPoolAllocator] --> E
|
||||
end
|
||||
|
||||
subgraph CPU["CPU backend"]
|
||||
L --> LC[layer/*.cpp naive]
|
||||
L --> LA[layer/arm|x86|riscv|mips|loongarch<br/>runtime ISA dispatch]
|
||||
CPUI[cpu.cpp: cpu_support_*, big.LITTLE, affinity, omp] --> LA
|
||||
end
|
||||
|
||||
subgraph GPU["Vulkan backend (NCNN_VULKAN)"]
|
||||
L --> LV[layer/vulkan/*_vulkan.cpp]
|
||||
LV --> PL[Pipeline / PipelineCache<br/>glslang GLSL->SPIR-V at runtime]
|
||||
LV --> CMD[VkCompute / VkTransfer<br/>record_upload/download/pipeline, submit_and_wait]
|
||||
CMD --> VD[VulkanDevice / GpuInfo<br/>queues, VkAllocator pools]
|
||||
VD --> SVK[simplevk loader<br/>dlopen libvulkan / MoltenVK / ICD]
|
||||
end
|
||||
|
||||
subgraph Tools["Offline tools"]
|
||||
T1[pnnx: PyTorch/ONNX -> .ncnn.param/.bin] --> A2
|
||||
T2[ncnnoptimize: fuse/eliminate, fp16 weights] --> A2
|
||||
T3[ncnn2table/ncnn2int8, ncnnllm2table/ncnnllm2int] --> A2
|
||||
end
|
||||
```
|
||||
|
||||
**Key modules** [Verified]
|
||||
|
||||
| Module | Path | Responsibility | Key symbols |
|
||||
|---|---|---|---|
|
||||
| Graph container | [src/net.h](../../ncnn/src/net.h), [src/net.cpp](../../ncnn/src/net.cpp) | Parse param, create layers (overwrite → vulkan → cpu → custom), load weights, create pipelines, upload to GPU, own local pool allocators | `Net`, `NetPrivate`, `Net::load_param` (L1306), `Net::load_model` (L2022), `Net::register_custom_layer` (L1215), `create_extractor` |
|
||||
| Per-request executor | [src/net.cpp#L2676](../../ncnn/src/net.cpp#L2676) | Holds `blob_mats` (+GPU), per-extractor `Option`; lazy DAG evaluation; unpack/cast outputs | `Extractor`, `ExtractorPrivate`, `Extractor::extract` (L2871), `NetPrivate::forward_layer` (L123 CPU, L192 Vulkan), `convert_layout` (L358), `do_forward_layer` (L633) |
|
||||
| Layer base + registry | [src/layer.h](../../ncnn/src/layer.h), [src/layer.cpp](../../ncnn/src/layer.cpp) | Virtual op interface, capability flags, factory tables | `Layer`, `support_*` flags (L45–90), `forward/forward_inplace`, `Layer_final` (L201), `layer_registry`, `create_layer_cpu/naive/vulkan`, `DEFINE_LAYER_CREATOR` (L199) |
|
||||
| Tensor | [src/mat.h](../../ncnn/src/mat.h) | N-D tensor with `elemsize/elempack/cstep/n`; pixel conversion | `Mat` (L50), `VkMat` (L389), `VkImageMat` (L559), `from_pixels*` (L293–307), `substract_mean_normalize` (L335), `batch()` (L239) |
|
||||
| Options | [src/option.h](../../ncnn/src/option.h), [src/option.cpp](../../ncnn/src/option.cpp) | All runtime knobs incl. precision, allocators, kv cache | `Option` |
|
||||
| Allocators | [src/allocator.h](../../ncnn/src/allocator.h), [src/allocator.cpp](../../ncnn/src/allocator.cpp) | Aligned malloc, host pools, Vulkan buffer/image pools | `fastMalloc` (L56), `Allocator` (L142), `PoolAllocator` (L151), `UnlockedPoolAllocator` (L180), `VkAllocator` (L267), `VkBlobAllocator`, `VkWeightAllocator`, `VkStagingAllocator`, `VkAndroidHardwareBufferImageAllocator` (L405) |
|
||||
| CPU topology | [src/cpu.h](../../ncnn/src/cpu.h), [src/cpu.cpp](../../ncnn/src/cpu.cpp) | ISA feature detection (ruapu), core counts, big.LITTLE, affinity, OpenMP wrappers, denormals | `cpu_support_arm_*`, `cpu_support_x86_*`, `get_big_cpu_count`, `set_cpu_powersave`, `set_cpu_thread_affinity`, `set_kmp_blocktime`, `set_flush_denormals` |
|
||||
| Vulkan device | [src/gpu.h](../../ncnn/src/gpu.h), [src/gpu.cpp](../../ncnn/src/gpu.cpp) | Instance/device creation, capability query, queue + allocator pools, shader compile | `create_gpu_instance` (gpu.cpp L2919), `GpuInfo`, `VulkanDevice`, `get_gpu_device`, `compile_spirv_module` |
|
||||
| Vulkan commands | [src/command.h](../../ncnn/src/command.h), [src/command.cpp](../../ncnn/src/command.cpp) | Command buffer recording/submission, staging copies | `VkCompute::record_upload/record_download/record_pipeline/submit_and_wait`, `VkTransfer` |
|
||||
| Vulkan pipelines | [src/pipeline.h](../../ncnn/src/pipeline.h), [src/pipelinecache.h](../../ncnn/src/pipelinecache.h) | Shader module/pipeline objects, on-disk cache | `Pipeline`, `PipelineCache::save_cache/load_cache/get_pipeline` |
|
||||
| Vulkan loader | [src/simplevk.h](../../ncnn/src/simplevk.h), [src/simplevk.cpp](../../ncnn/src/simplevk.cpp) | In-house loader (no libvulkan link) | `load_vulkan_driver` |
|
||||
| Param/weights I/O | [src/paramdict.h](../../ncnn/src/paramdict.h), [src/modelbin.h](../../ncnn/src/modelbin.h), [src/datareader.h](../../ncnn/src/datareader.h) | Key=value params, weight blobs with type flags, stream abstraction | `ParamDict`, `ModelBin`, `DataReader` |
|
||||
| Operator impls | [src/layer](../../ncnn/src/layer) (110 registered types), [src/layer/arm](../../ncnn/src/layer/arm), [src/layer/x86](../../ncnn/src/layer/x86), [src/layer/vulkan](../../ncnn/src/layer/vulkan), [src/layer/riscv](../../ncnn/src/layer/riscv), [src/layer/mips](../../ncnn/src/layer/mips), [src/layer/loongarch](../../ncnn/src/layer/loongarch) | Naive reference + arch kernels + GLSL shaders | e.g. `SDPA`, `MultiHeadAttention`, `Gemm`, `Convolution` |
|
||||
| Shims | [src/simpleomp.h](../../ncnn/src/simpleomp.h), [src/simplestl.h](../../ncnn/src/simplestl.h), [src/simpleocv.h](../../ncnn/src/simpleocv.h), [src/simplemath.h](../../ncnn/src/simplemath.h) | Minimal OpenMP runtime, STL, OpenCV-like image I/O, libm | — |
|
||||
| C API | [src/c_api.h](../../ncnn/src/c_api.h) | Flat C bindings incl. kv cache options | `ncnn_net_*`, `ncnn_extractor_*`, `ncnn_option_set_kvcache_allocator` |
|
||||
| Python | [python/src/main.cpp](../../ncnn/python/src/main.cpp) | pybind11 bindings for Net/Extractor/Mat/Option/Layer/Vulkan | `py::class_<Extractor>` (L959) |
|
||||
| Converters | [tools/pnnx](../../ncnn/tools/pnnx), [tools/onnx](../../ncnn/tools/onnx), [tools/ncnnoptimize.cpp](../../ncnn/tools/ncnnoptimize.cpp), [tools/quantize](../../ncnn/tools/quantize) | Model export, offline fusion, quantization | `pnnx`, `onnx2ncnn`, `ncnnoptimize`, `ncnn2table`, `ncnn2int8`, `ncnnllm2table`, `ncnnllm2int` |
|
||||
| Build glue | [cmake/ncnn_add_layer.cmake](../../ncnn/cmake/ncnn_add_layer.cmake), [cmake/ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake), [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt) | Layer registry generation, ISA-variant source generation, shader embedding | `ncnn_add_layer`, `ncnn_add_arch_opt_layer`, `ncnn_add_shader` |
|
||||
|
||||
---
|
||||
|
||||
## 2. Device layer
|
||||
|
||||
### Backends actually in tree [Verified]
|
||||
- **CPU**: naive C++ in `src/layer/*.cpp` plus arch dirs: arm (56 `*_arm.cpp` + ISA variants `_asimdhp/_asimddp/_asimdfhm/_bf16/_i8mm/_vfpv4`), x86 (59 base + `_avx2/_fma/_fma4/_xop/_f16c/_avx512*/_avxvnni*/_avxneconvert`), riscv (52), mips (56), loongarch (56). Counts from `ls src/layer/<arch>/*_<arch>.cpp`.
|
||||
- **Vulkan GPU**: 64 `*_vulkan.cpp` layer implementations and 300 `.comp` shaders under [src/layer/vulkan/shader](../../ncnn/src/layer/vulkan/shader). Enabled by `NCNN_VULKAN` (default **OFF**) ([CMakeLists.txt#L82](../../ncnn/CMakeLists.txt#L82)).
|
||||
- **Not present**: CUDA, Metal, CoreML, OpenCL, NNAPI, QNN/Hexagon, TensorRT. A grep for those names over `src/` only hits Vulkan loader/portability code (`VK_KHR_portability_subset`, MoltenVK dlopen). The FAQ explicitly states Vulkan was chosen over CUDA/OpenCL/Metal for portability ([FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md), "why using vulkan over cuda/opencl/metal").
|
||||
|
||||
### Device discovery / selection [Verified]
|
||||
- CPU: `get_cpu_count / get_big_cpu_count / get_little_cpu_count / get_physical_*` ([src/cpu.h#L124](../../ncnn/src/cpu.h#L124)); big.LITTLE is derived from sysfs `cpufreq` max frequency (`get_max_freq_khz`, [src/cpu.cpp#L1486](../../ncnn/src/cpu.cpp#L1486)) and topology files (`thread_siblings`, [src/cpu.cpp#L961](../../ncnn/src/cpu.cpp#L961)). ISA features: `cpu_support_arm_asimdhp/asimddp/bf16/i8mm/sve*`, `cpu_support_x86_avx2/avx512_*`, `cpu_support_riscv_v/zfh/zvfh`, `cpu_riscv_vlenb` via bundled [src/ruapu.h](../../ncnn/src/ruapu.h). `Option::num_threads` defaults to `get_physical_big_cpu_count()` ([src/option.cpp](../../ncnn/src/option.cpp)).
|
||||
- Vulkan: lazy singleton `create_gpu_instance()` under a mutex ([src/gpu.cpp#L2919](../../ncnn/src/gpu.cpp#L2919)); loader order: env `VK_ICD_FILENAMES`/`NCNN_VULKAN_DRIVER` → system `libvulkan.so/.dylib/vulkan-1.dll` → direct driver file search (`libvulkan_radeon.so`, `libMoltenVK.dylib`, Android HAL `HMI`) ([src/simplevk.cpp#L287](../../ncnn/src/simplevk.cpp#L287), [#L558](../../ncnn/src/simplevk.cpp#L558), [vulkan-driver-loader.md](../../ncnn/docs/developer-guide/vulkan-driver-loader.md)). Each physical device gets a `GpuInfo` with a `rough_score` (device type, subgroup size, fp16/int8/coopmat support, memory) used to choose `get_default_gpu_index()` ([src/gpu.cpp#L1297](../../ncnn/src/gpu.cpp#L1297), [#L3624](../../ncnn/src/gpu.cpp#L3624)). `Net::set_vulkan_device(int|VulkanDevice*)` or `opt.vulkan_device_index` selects a device before `load_param` ([src/net.cpp#L1338](../../ncnn/src/net.cpp#L1338)). If no valid device, `opt.use_vulkan_compute` is silently set to false (CPU fallback) ([src/net.cpp#L1348](../../ncnn/src/net.cpp#L1348)).
|
||||
- Capability sanitization: at `load_param`, unsupported `use_fp16_packed/storage/arithmetic`, `use_int8_*`, `use_bf16_*`, `use_cooperative_matrix`, `use_subgroup_ops` are cleared from the Net option based on `GpuInfo` ([src/net.cpp#L1350–1372](../../ncnn/src/net.cpp#L1350)). `GpuInfo` exposes ~50 extension flags including `VK_KHR_cooperative_matrix`, `VK_NV_cooperative_matrix2`, `VK_KHR_shader_bfloat16`, `VK_EXT_shader_float8`, `VK_ANDROID_external_memory_android_hardware_buffer` ([src/gpu.h#L327–378](../../ncnn/src/gpu.h#L327)).
|
||||
|
||||
### Memory allocation / pools [Verified]
|
||||
- Host: `fastMalloc` = aligned alloc with `NCNN_MALLOC_ALIGN` and 64-byte over-read pad ([src/allocator.h#L56](../../ncnn/src/allocator.h#L56)). `PoolAllocator` (mutex) / `UnlockedPoolAllocator` keep `budgets`/`payouts` lists; reuse if a free block is ≥ size and within `size_compare_ratio`; drop when > `size_drop_threshold` (default 10) ([src/allocator.cpp#L98](../../ncnn/src/allocator.cpp#L98)). Option roles: `blob_allocator` (named blobs, lifetime = Extractor), `workspace_allocator` (per-layer scratch), `kvcache_allocator` (must differ from blob allocator, [src/net.cpp#L2876](../../ncnn/src/net.cpp#L2876)). With `use_local_pool_allocator` (default true) the Net lazily creates its own pools at `load_model` ([src/net.cpp#L2136](../../ncnn/src/net.cpp#L2136)) and clones outputs out of them on `extract` so the Net can be destroyed early ([src/net.cpp#L3060](../../ncnn/src/net.cpp#L3060)).
|
||||
- Device: `VkAllocator` family: `VkBlobAllocator` (activations), `VkWeightAllocator` (device-local weights; `use_weights_in_host_memory` flips to host-visible), `VkStagingAllocator`/`VkWeightStagingAllocator` (host-visible staging), with `flush/invalidate` for non-coherent memory ([src/allocator.h#L267–395](../../ncnn/src/allocator.h#L267)). `VulkanDevice::acquire_blob_allocator/acquire_staging_allocator` hand out pooled allocators; an Extractor acquires them on first `extract` if not set ([src/net.cpp#L2915](../../ncnn/src/net.cpp#L2915)). Memory type choice via `find_memory_index(required, preferred, preferred_not)` and `is_mappable/is_coherent/is_device_local` ([src/gpu.h#L465](../../ncnn/src/gpu.h#L465)). Android `AHardwareBuffer` zero-copy import allocator exists (API ≥ 26) ([src/allocator.h#L405](../../ncnn/src/allocator.h#L405), [use-ncnn-with-android-hardware-buffer.md](../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md)).
|
||||
- Model weights: `Net::load_model(const unsigned char*)` references external memory without copying; `opt.use_mapped_model_loading` mmaps the `.bin` on Linux/Android/OHOS/Windows/Apple ([src/net.cpp#L2261](../../ncnn/src/net.cpp#L2261)).
|
||||
|
||||
### Host↔device transfers and synchronization [Verified]
|
||||
- All GPU work is recorded into a `VkCompute` command buffer: `record_upload` (staging copy + optional fp32→fp16 cast + packing), `record_download`, `record_clone`, `record_pipeline(pipeline, bindings, push constants, dispatcher)`, then `submit_and_wait()` (fence wait) ([src/command.h#L22–87](../../ncnn/src/command.h#L22)). Weights are uploaded once at `load_model` via `VkTransfer`, committed every 256 MB ([src/net.cpp#L2115](../../ncnn/src/net.cpp#L2115)).
|
||||
- Inside a Vulkan extract, the Net records the whole reachable subgraph into one command buffer; it force-submits when a CPU-only layer needs a downloaded input, or when `pending_dispatch_total()` exceeds a threshold scaled by `rough_score` (32 KB … 8 MB "for avoiding driver timeout") ([src/net.cpp#L250–283](../../ncnn/src/net.cpp#L250)). Final output is downloaded and waited on in `Extractor::extract` ([src/net.cpp#L2940](../../ncnn/src/net.cpp#L2940)). Barriers are inserted automatically by `barrier_readwrite` bookkeeping ([src/command.h#L82](../../ncnn/src/command.h#L82)).
|
||||
- Queues: `VulkanDevice::acquire_queue/reclaim_queue` pool per family; `GpuInfo::compute_queue_count()` is documented as the practical max number of concurrent extractors ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md), "batch inference"). Zero-copy `VkMat::mapped()` on unified-memory devices and cross-net `VkMat` chaining through a shared `VkCompute` are documented there too.
|
||||
- Timestamps/profiling: `NCNN_BENCHMARK` build records per-layer GPU timestamps via query pool ([src/net.cpp#L2952](../../ncnn/src/net.cpp#L2952)).
|
||||
|
||||
### Portability [Verified]
|
||||
- OS: Linux, Android (NDK ≥ r18b; Vulkan needs API 24 with system loader, or lower with simplevk), iOS/macOS/Catalyst/tvOS/visionOS/watchOS, Windows (incl. XP), HarmonyOS (`__OHOS__`), QNX, WebAssembly (Emscripten), bare-metal RISC-V/ESP32 — evidenced by [toolchains](../../ncnn/toolchains) and [.github/workflows](../../ncnn/.github/workflows) (android.yml, ios.yml, harmonyos.yml, web-assembly.yml, esp32.yml, elf-riscv64.yml, windows-arm.yml, linux-ppc64.yml, linux-loongarch64.yml …).
|
||||
- Vulkan platform matrix (repo doc): Intel/AMD/NVIDIA on Windows/Linux, Qualcomm/ARM Mali on Android, Apple on mac/iOS via MoltenVK ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md)). Jetson: aarch64 CPU + Tegra Vulkan driver ([docs/how-to-build/how-to-build.md#L66](../../ncnn/docs/how-to-build/how-to-build.md#L66), [toolchains/jetson.toolchain.cmake](../../ncnn/toolchains/jetson.toolchain.cmake)).
|
||||
|
||||
---
|
||||
|
||||
## 3. Kernel layer
|
||||
|
||||
### Operator set & registration [Verified]
|
||||
- 110 layer types registered via `ncnn_add_layer(Class)` in [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt) (two marked OFF by default: `ArgMax`, `SPP`). Full list with params in [docs/developer-guide/operators.md](../../ncnn/docs/developer-guide/operators.md) and [operation-param-weight-table.md](../../ncnn/docs/developer-guide/operation-param-weight-table.md). Transformer/audio-relevant: `MultiHeadAttention`, `SDPA`, `RotaryEmbed`, `RMSNorm`, `LayerNorm`, `GroupNorm`, `Embed`, `Gemm`, `MatMul`, `Einsum`, `GLU`, `GELU`, `Softmax`, `Convolution1D/DepthWise1D/Deconvolution1D`, `Pooling1D`, `Spectrogram`, `InverseSpectrogram`, `LSTM/GRU/RNN`, `GridSample`, `DeformableConv2D`, `Interp`, `PixelShuffle`, `CumulativeSum`.
|
||||
- Registration is **build-time code generation**: `ncnn_add_layer` (macro at [cmake/ncnn_add_layer.cmake#L82](../../ncnn/cmake/ncnn_add_layer.cmake#L82)) appends `#include` + `DEFINE_LAYER_CREATOR(Class)` lines into generated `layer_declaration.h` and a `layer_registry` table per ISA level (`layer_registry_arm82`, `_avx512`, …) into `layer_registry.h` (templates: [src/layer_declaration.h.in](../../ncnn/src/layer_declaration.h.in), [src/layer_registry.h.in](../../ncnn/src/layer_registry.h.in), [src/layer_type_enum.h.in](../../ncnn/src/layer_type_enum.h.in)). ISA variants are produced by rewriting the base arch source with `cmake/ncnn_generate_<opt>_source.cmake` (renaming the class, e.g. `Convolution_arm_asimdhp`) and compiling with the ISA flags ([cmake/ncnn_add_layer.cmake#L2](../../ncnn/cmake/ncnn_add_layer.cmake#L2)). `WITH_LAYER_<name>=OFF` removes a layer and its kernels (registry gets a null creator).
|
||||
- `layer_to_index(name)` is a linear strcmp scan ([src/layer.cpp#L150](../../ncnn/src/layer.cpp#L150)); `create_layer_cpu(index)` picks the best registry table at runtime based on `cpu_support_*` when `NCNN_RUNTIME_CPU=ON` (see generated `layer_registry.h` consumers in [src/layer.cpp](../../ncnn/src/layer.cpp) after L400) [Verified for mechanism; exact selection code lives in the generated portion].
|
||||
|
||||
### Dispatch mechanism (op → backend impl) [Verified]
|
||||
1. `Net::load_param` creates each layer as: overwrite-builtin registry → `create_layer_vulkan` (if `use_vulkan_compute` and device) → `create_layer_cpu` → custom registry; missing type is a hard error ([src/net.cpp#L1400–1420](../../ncnn/src/net.cpp#L1400)).
|
||||
2. `create_layer_vulkan/cpu` return a `Layer_final` wrapper holding `layer_cpu` and optionally `layer_vulkan`; `load_param/create_pipeline` try the Vulkan layer first and **delete it and fall back to CPU** if `support_vulkan` becomes false after `load_param` (layer-specific param not supported on GPU) ([src/layer.cpp#L280–344](../../ncnn/src/layer.cpp#L280)). The Net additionally recreates a CPU layer when the Vulkan variant rejects the params or the layer's `featmask` disables Vulkan ([src/net.cpp#L1536](../../ncnn/src/net.cpp#L1536)).
|
||||
3. At forward time, `Layer_final::forward(Mat…)` → `layer_cpu`, `forward(VkMat…)` → `layer_vulkan` ([src/layer.cpp#L359–400](../../ncnn/src/layer.cpp#L359)). The Vulkan `forward_layer` inspects `layer->support_vulkan` per layer: GPU layers get `record_upload` of any host-only inputs; CPU-only layers get `record_download` + forced `submit_and_wait`, then run on host ([src/net.cpp#L215–248](../../ncnn/src/net.cpp#L215)). Mixed CPU/GPU graphs therefore work automatically at the cost of syncs.
|
||||
4. Inside an arch kernel, further dispatch is by data type: e.g. `Clip_arm::forward_inplace` checks `elembits()==16 && opt.use_fp16_storage` → fp16 path, bf16 path, else fp32 ([layer-support-behavior.md](../../ncnn/docs/developer-guide/layer-support-behavior.md)). `SDPA_arm` declares `support_fp16_storage = cpu_support_arm_asimdhp()` and `support_bf16_storage = true` ([src/layer/arm/sdpa_arm.cpp#L14](../../ncnn/src/layer/arm/sdpa_arm.cpp#L14)).
|
||||
5. Layout negotiation is done by the Net, not the kernel: `convert_layout` casts fp32→fp16/bf16 when `layer->support_fp16_storage/support_bf16_storage` and the option is on, and re-packs to the arch-optimal `elempack` (4 NEON, 8 AVX, 16 AVX-512, `vlenb/4` RVV) unless `support_any_packing` ([src/net.cpp#L358–470](../../ncnn/src/net.cpp#L358); [element-packing.md](../../ncnn/docs/developer-guide/element-packing.md)). KV-cache blobs skip layout conversion ([src/net.cpp#L364](../../ncnn/src/net.cpp#L364)).
|
||||
6. Per-layer overrides: `31=<featmask>` bits disable fp16 arithmetic/storage, bf16, int8, vulkan, sgemm, winograd, threading for that layer ([docs/developer-guide/layer-feat-mask.md](../../ncnn/docs/developer-guide/layer-feat-mask.md), [src/net.cpp#L100](../../ncnn/src/net.cpp#L100)).
|
||||
|
||||
### Quantization formats & where dequant happens [Verified]
|
||||
- **Weight storage flags** in `.bin`: `0` fp32, `0x01306B47` fp16, `0x01348B83` bf16, other = quantized int8 with table ([param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md)). pnnx writes fp16 weights by default (`fp16=1`, [tools/pnnx/src/save_ncnn.cpp#L240](../../ncnn/tools/pnnx/src/save_ncnn.cpp#L240)); `ModelBin` converts to the layer's requested type at load.
|
||||
- **Activation precision**: fp16 storage/arithmetic on ARMv8.2 (`asimdhp`), VFPv4 fp16 storage on ARMv7, bf16 storage on any ARM/x86 (`NCNN_BF16`), fp16/bf16/int8 packed+storage+arithmetic on Vulkan, chosen by `Option` flags and clamped by device caps. Casting happens in `convert_layout` at layer boundaries and in `Extractor::extract` back to fp32 unless `type=1` ([src/net.cpp#L2985–3055](../../ncnn/src/net.cpp#L2985)).
|
||||
- **PTQ int8** (`NCNN_INT8`): `ncnn2table` (KL/ACIQ, from images or `.npy`, also static weight scales for RNN/GRU/LSTM/MHA/Embed) → `ncnn2int8` rewrites weights and adds `8=`/`18=` int8 scale params; `Quantize/Dequantize/Requantize` layers exist and conv/innerproduct/gemm/mha/sdpa/embed have `forward_int8` paths; dequant to fp32 happens inside those layers or at `extract` (int8 output cast, [src/net.cpp#L3043](../../ncnn/src/net.cpp#L3043)). x86 int8 uses AVX-VNNI/AVX512-VNNI/AVX-VNNI-INT8 kernels; ARM uses dotprod/i8mm ([src/layer/arm/convolution_arm_i8mm.cpp](../../ncnn/src/layer/arm/convolution_arm_i8mm.cpp), [src/layer/x86/convolution_x86_avx512vnni.cpp](../../ncnn/src/layer/x86/convolution_x86_avx512vnni.cpp)).
|
||||
- **LLM block quant** (`NCNN_WEIGHT_QUANT`, default ON): `Gemm`/`MultiHeadAttention` `quantize_term = bits*100 + input_scale*10 + block_code` (bits 4/6/8, block 32/64/128); W4/W6 are weight-only with fp32 activations, W8 is dynamic W8A8 per-block int32 accumulate, CPU only; when enabled the layer turns off packing/fp16/bf16/Vulkan (`support_vulkan=false`) ([src/layer/multiheadattention.cpp#L84–100](../../ncnn/src/layer/multiheadattention.cpp#L84), [src/layer/gemm.h](../../ncnn/src/layer/gemm.h), [quantized-int8-inference.md](../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md)). Tools: [tools/quantize/ncnnllm2table.cpp](../../ncnn/tools/quantize/ncnnllm2table.cpp), [tools/quantize/ncnnllm2int.cpp](../../ncnn/tools/quantize/ncnnllm2int.cpp); test [tests/test_gemm_block_quant.cpp](../../ncnn/tests/test_gemm_block_quant.cpp).
|
||||
|
||||
### Compilation / JIT / graph capture [Verified]
|
||||
- CPU: no JIT; all kernels are AOT C++/intrinsics/inline asm, selected at runtime by ISA table. Winograd/sgemm/packing variants are selected in each layer's `create_pipeline` based on shape hints and `Option` (e.g. `use_winograd23/43/63_convolution`).
|
||||
- Vulkan: GLSL `.comp` sources are preprocessed into hex headers at build time ([cmake/ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake)) and **compiled to SPIR-V at runtime** by in-tree glslang with option-dependent macros (`compile_spirv_module`, [src/gpu.h#L572](../../ncnn/src/gpu.h#L572); glslang linked privately, [src/CMakeLists.txt#L341](../../ncnn/src/CMakeLists.txt#L341)). Specialization constants + push constants parameterize shapes; `PipelineCache` deduplicates pipelines in-process and persists SPIR-V plus driver `VkPipelineCache` blobs to a file ([docs/developer-guide/vulkan-pipeline-cache.md](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md)). No command-buffer replay/graph capture across extracts: each `extract` records a fresh `VkCompute`.
|
||||
- No runtime graph optimization: fusions (conv+bn, conv+activation, etc.) are done offline by `ncnnoptimize` ([tools/ncnnoptimize.cpp#L42–77](../../ncnn/tools/ncnnoptimize.cpp#L42)) or pnnx (`optlevel`). `grep fuse src/net.cpp` finds nothing.
|
||||
|
||||
### Backend coverage gaps (computed from file presence) [Verified]
|
||||
Computed by checking, for each of the 110 `ncnn_add_layer(...)` entries in [src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt), whether `src/layer/vulkan/<name>_vulkan.cpp` / `src/layer/arm/<name>_arm.cpp` exist.
|
||||
|
||||
| Backend | Layers **without** a dedicated kernel (run naive C++ on CPU, or CPU fallback for Vulkan) |
|
||||
|---|---|
|
||||
| Vulkan (64/110 covered) | ArgMax, Bias, BNLL, **Embed**, Exp, Input, Log, MVN, Power, Proposal, ROIPooling, SPP, Threshold, Tile, **RNN, LSTM, GRU**, Squeeze, ExpandDims, DetectionOutput, YoloDetectionOutput, Yolov3DetectionOutput, PSROIPooling, ROIAlign, StatisticsPooling, **Pooling1D, ConvolutionDepthWise1D, Deconvolution1D, DeconvolutionDepthWise1D**, Convolution3D, ConvolutionDepthWise3D, Pooling3D, Deconvolution3D, DeconvolutionDepthWise3D, **MatMul, Einsum, GLU**, DeformableConv2D, Fold, **GridSample**, CumulativeSum, CopyTo, Diag, **Spectrogram, InverseSpectrogram**, Flip |
|
||||
| ARM NEON (56/110 covered) | ArgMax, BNLL, **Embed**, Exp, Input, Log, MemoryData, MVN, Power, Proposal, Reduction, ROIPooling, Split, SPP, Threshold, Tile, Squeeze, ExpandDims, Normalize, Permute, PriorBox, DetectionOutput, Reorg, Yolo*DetectionOutput, PSROIPooling, ROIAlign, Noop, DeepCopy, StatisticsPooling, Softplus, **Pooling1D, ConvolutionDepthWise1D, Deconvolution1D, DeconvolutionDepthWise1D**, Convolution3D/DepthWise3D, Pooling3D, Deconvolution3D/DepthWise3D, **Einsum, GLU**, DeformableConv2D, Fold, Unfold, **GridSample**, CumulativeSum, CopyTo, Diag, CELU, Shrink, **Spectrogram, InverseSpectrogram**, Flip |
|
||||
|
||||
Implications for TTS/VLA: `Embed`, `GLU`, depthwise/transposed 1-D convs (common in vocoders), `Spectrogram`, and RNNs always run on CPU — on a Vulkan graph each such layer forces a download + `submit_and_wait` ([src/net.cpp#L237](../../ncnn/src/net.cpp#L237)). `SDPA`, `MultiHeadAttention`, `Gemm`, `RMSNorm`, `LayerNorm`, `RotaryEmbed`, `Convolution1D`, `Deconvolution` are covered on both ARM and Vulkan. [Verified]
|
||||
|
||||
### Precision / performance option matrix [Verified]
|
||||
Defaults from [src/option.cpp](../../ncnn/src/option.cpp); effects from [src/net.cpp](../../ncnn/src/net.cpp) and layer `support_*` checks.
|
||||
|
||||
| Option | Default | CPU effect | Vulkan effect |
|
||||
|---|---|---|---|
|
||||
| `use_fp16_storage` / `use_fp16_arithmetic` / `use_fp16_packed` | true | ARMv8.2 (`asimdhp`), ARMv7 VFPv4 storage, RISC-V Zfh/Zvfh: activations cast to fp16 between layers that `support_fp16_storage` | fp16 SSBO storage, fp16 math, packed fp16x4; cleared if device lacks support |
|
||||
| `use_bf16_storage` | false | Any ARM/x86/LoongArch with `NCNN_BF16`: bf16 activations (takes precedence over fp16 when set, [src/net.cpp#L400](../../ncnn/src/net.cpp#L400)) | `use_bf16_packed/storage` need `VK_KHR_shader_bfloat16` |
|
||||
| `use_int8_inference` (+`use_int8_*`) | true | Runs `forward_int8` on layers carrying int8 scales | int8 storage/arithmetic shaders where present |
|
||||
| `use_packing_layout` | true | elempack 4/8/16 SIMD layout negotiated per layer | elempack 4 (`support_vulkan_packing`) |
|
||||
| `use_winograd_convolution` / `use_sgemm_convolution` (+`winograd23/43/63`) | true | Conv algorithm choice at `create_pipeline` (more memory) | Winograd/GEMM shaders incl. cooperative-matrix `_cm` variants |
|
||||
| `use_cooperative_matrix`, `use_subgroup_ops`, `use_shader_local_memory` | true | — | Tensor-core style matmul (`VK_KHR_cooperative_matrix`/NV), subgroup reductions, shared memory (discrete GPUs only) |
|
||||
| `lightmode` | true | Free intermediates early, inplace when refcount==1 | same |
|
||||
| `use_local_pool_allocator` | true | Net-owned pools when app passes none | — |
|
||||
| `num_threads`, `openmp_blocktime` (20 ms), `flush_denormals` (3=DAZ+FTZ) | big-core count | OpenMP width, spin time, denormal handling around each extract | CPU-side upload/cast still threaded |
|
||||
| `kvcache_allocator`, `kvcache_max_seqlen_hint`, `kvcache_vkallocator` | null / 0 | KV cache placement and initial reservation | device-resident cache |
|
||||
| `use_mapped_model_loading`, `use_weights_in_host_memory` | false | mmap `.bin` | keep GPU weights in host-visible memory (iGPU/UMA) |
|
||||
| `vulkan_device_index` | -1 (auto by `rough_score`) | — | device choice |
|
||||
|
||||
### Fallback when op unsupported [Verified]
|
||||
- Unknown layer type in param → `load_param` fails ("layer %s not exists or registered", [src/net.cpp#L1417](../../ncnn/src/net.cpp#L1417)); the app must register it via `register_custom_layer` first (Piper example does this).
|
||||
- Vulkan-unsupported layer/param → transparent CPU execution with upload/download around it ([src/net.cpp#L215](../../ncnn/src/net.cpp#L215); [FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md) "layers without vulkan support").
|
||||
- Arch kernel missing → naive `src/layer/<name>.cpp` is compiled and registered in its place ([cmake/ncnn_add_layer.cmake#L39](../../ncnn/cmake/ncnn_add_layer.cmake#L39)).
|
||||
- Precision unsupported by a layer → Net converts inputs to fp32/elempack 1 for it (`support_*` false).
|
||||
|
||||
### Extension points [Verified]
|
||||
- Custom op: subclass `ncnn::Layer`, implement `load_param/load_model/forward(_inplace)`, set `one_blob_only/support_inplace/support_packing…`, then `net.register_custom_layer("Type", creator, destroyer, userdata)` (string or index) — tutorial in [how-to-implement-custom-layer-step-by-step.md](../../ncnn/docs/developer-guide/how-to-implement-custom-layer-step-by-step.md), [add-custom-layer.zh.md](../../ncnn/docs/developer-guide/add-custom-layer.zh.md); pnnx exports unknown modules as `moduleop` so names round-trip ([tools/pnnx/README.md](../../ncnn/tools/pnnx/README.md)). Python can register custom layers too ([python/src/main.cpp#L1046](../../ncnn/python/src/main.cpp#L1046)).
|
||||
- Overwriting a builtin op with your own implementation uses the same API (`create_overwrite_builtin_layer`, [src/net.cpp#L1215](../../ncnn/src/net.cpp#L1215)).
|
||||
- Custom Vulkan op: add shader, `upload_model`, `create_pipeline`, `forward(VkMat…)` ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md) "add vulkan compute support to layer"; GLSL helper macros in [docs/developer-guide/glsl-extension.md](../../ncnn/docs/developer-guide/glsl-extension.md)).
|
||||
- Custom allocator: subclass `Allocator`/`VkAllocator` ([custom-allocator.md](../../ncnn/docs/developer-guide/custom-allocator.md)).
|
||||
- Low-level op API: create/forward a single layer without a Net ([low-level-operation-api.md](../../ncnn/docs/developer-guide/low-level-operation-api.md)).
|
||||
- New backend (e.g. NPU): **no plugin interface**; you would add a third `layer_xxx` slot to `Layer_final` and a third registry, mirroring how Vulkan is wired in [src/layer.cpp#L201](../../ncnn/src/layer.cpp#L201) and [src/net.cpp](../../ncnn/src/net.cpp). [Inferred]
|
||||
|
||||
---
|
||||
|
||||
## 4. Model runner
|
||||
|
||||
### Formats & conversion pipeline [Verified]
|
||||
- Runtime format: `model.param` (text: magic `7767517`, `layer_count blob_count`, then `Type name in out blobs... k=v`) + `model.bin` (concatenated 32-bit-aligned weight blobs with optional type flag) ([param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md)). Binary param (`.param.bin`) + `ncnn2mem` header for string-less builds ([tools/ncnn2mem.cpp](../../ncnn/tools/ncnn2mem.cpp)); loading table in [ncnn-load-model.md](../../ncnn/docs/how-to-use-and-FAQ/ncnn-load-model.md).
|
||||
- Recommended path: PyTorch → `pip install pnnx` → `pnnx.export(model, "m.pt", (x,))` → `m.ncnn.param/bin` + `m_ncnn.py` ([README](../../ncnn/README.md)). pnnx also loads ONNX and TNN ([tools/pnnx/src/load_onnx.cpp](../../ncnn/tools/pnnx/src/load_onnx.cpp), [load_tnn.cpp](../../ncnn/tools/pnnx/src/load_tnn.cpp)); 253 ncnn lowering passes in [tools/pnnx/src/pass_ncnn](../../ncnn/tools/pnnx/src/pass_ncnn) including `F_scaled_dot_product_attention.cpp`, `nn_MultiheadAttention.cpp`, `nn_RMSNorm.cpp`, `fuse_convert_rotaryembed.cpp`, `torch_stft.cpp`, `torchaudio_F_spectrogram.cpp`. CLI options `inputshape`, `inputshape2` (dynamic dim discovery), `fp16`, `optlevel`, `moduleop`, `customop` ([tools/pnnx/README.md#L85](../../ncnn/tools/pnnx/README.md#L85)).
|
||||
- Legacy converters: [tools/onnx/onnx2ncnn.cpp](../../ncnn/tools/onnx/onnx2ncnn.cpp), [tools/caffe](../../ncnn/tools/caffe), [tools/darknet](../../ncnn/tools/darknet), [tools/mxnet](../../ncnn/tools/mxnet), [tools/keras](../../ncnn/tools/keras), [tools/tensorflow](../../ncnn/tools/tensorflow), [tools/mlir](../../ncnn/tools/mlir). Post-processing: [tools/ncnnoptimize.cpp](../../ncnn/tools/ncnnoptimize.cpp) (fusion, dead-op removal, fp16/fp32 storage choice), [tools/ncnnmerge.cpp](../../ncnn/tools/ncnnmerge.cpp).
|
||||
- LLM decoders: export with sequence length 1, then **patch the param** to add `cache_k_in/v_in/k_out/v_out` blobs + `7=1` on every MHA/SDPA and set `Gemm 7=0` (dynamic M) — reference Python in [kvcache.md](../../ncnn/docs/developer-guide/kvcache.md); `benchncnn_llm` further rewrites `Gemm 18=<quantize_term>` in memory for block-quant variants ([benchmark/benchncnn_llm.cpp#L72](../../ncnn/benchmark/benchncnn_llm.cpp#L72)).
|
||||
|
||||
### Execution lifecycle [Verified]
|
||||
1. **Init**: `Net net; net.opt.* = …; net.load_param(path|mem|AAsset|DataReader); net.load_model(...)`. `load_param` creates layers and resolves blob shape hints; `load_model` reads weights, calls `Layer::create_pipeline(opt)` (weight repacking, winograd transform, GPU pipeline creation) and `upload_model` for Vulkan ([src/net.cpp#L2022–2170](../../ncnn/src/net.cpp#L2022)). Option changes must precede loading (comments in [src/option.h](../../ncnn/src/option.h)).
|
||||
2. **Warmup**: no explicit API; first `extract` triggers pool allocator growth and (Vulkan) SPIR-V compile/pipeline creation unless a `PipelineCache` file is preloaded. `benchncnn` uses 8 warmup loops by convention ([benchmark/benchncnn_llm.cpp#L42](../../ncnn/benchmark/benchncnn_llm.cpp#L42)).
|
||||
3. **Step**: `Extractor ex = net.create_extractor(); ex.input(name|idx, Mat); ex.extract(name|idx, out[, type])`. `Extractor` copies `net.opt` at creation; `set_light_mode`, `set_blob_allocator`, `set_workspace_allocator`, `set_kvcache_allocator`, `set_kvcache_max_seqlen_hint`, `set_*_vkallocator` per extractor ([src/net.h#L167–256](../../ncnn/src/net.h#L167)). Light mode (default) releases intermediate blobs as soon as consumed and does inplace ops when the refcount is 1 ([src/net.cpp#L644](../../ncnn/src/net.cpp#L644)). Extract also sets OpenMP blocktime and denormal flags around the run ([src/net.cpp#L2896](../../ncnn/src/net.cpp#L2896)).
|
||||
4. **Teardown**: `Extractor::clear`, `Net::clear` (destroys pipelines, frees allocators), `ncnn::destroy_gpu_instance()`.
|
||||
|
||||
### Dynamic shapes [Verified]
|
||||
- Tensors carry shape at runtime; layers derive output shapes in `forward`. Param shape hints (`30=` on Input/any layer) only feed `create_pipeline` heuristics ([src/net.cpp#L1488](../../ncnn/src/net.cpp#L1488)); they are not enforced.
|
||||
- `Reshape 6="expr"` and `Slice` accept expressions over input dims (`0w`, `1c`, `+`, `*`, `max`, …) resolved on the host at forward time, avoiding shape-op subgraphs ([docs/developer-guide/expression.md](../../ncnn/docs/developer-guide/expression.md), [src/expression.h](../../ncnn/src/expression.h)).
|
||||
- Gemm `7=` constant-M flag must be 0 for variable sequence length; MHA/SDPA accept arbitrary `seqlen` (`Mat::h`). `Input` layers accept any size; Piper feeds a variable-length phoneme id vector and gets variable-length audio ([examples/piper.cpp#L529](../../ncnn/examples/piper.cpp#L529)).
|
||||
- Batch: `Mat::n` (NCNN_BATCH ON by default); the Net loops layers that lack `support_batch` per batch element and validates consistent `n` across inputs ([src/net.cpp#L662–712](../../ncnn/src/net.cpp#L662), [#L789](../../ncnn/src/net.cpp#L789)). Test: [tests/test_mat_batch.cpp](../../ncnn/tests/test_mat_batch.cpp).
|
||||
|
||||
### State & KV cache [Verified]
|
||||
- **Allocation & layout**: cache is a `Mat(head_dim, seqlen, num_kv_head)` whose `cstep` reserves extra sequence capacity; `Mat::h` is the logical length. `SDPA::kvcache_capacity` reserves `max_seqlen_hint` on first allocation, else `max(16, …)` up to +256, then grows geometrically by ×1.5; `create_or_grow_kvcache` reuses in place if `new_seqlen <= capacity` and the cache came from the kv allocator, otherwise allocates and copies valid history per head ([src/layer/sdpa.cpp#L28–92](../../ncnn/src/layer/sdpa.cpp#L28); same in [src/layer/multiheadattention.cpp#L230](../../ncnn/src/layer/multiheadattention.cpp#L230)). Layout is backend-private (head-contiguous on CPU, device-resident `VkMat` on Vulkan); Vulkan uses `sdpa_kvcache_append.comp` / `sdpa_kvcache_copy.comp` shaders ([src/layer/vulkan/shader/sdpa_kvcache_append.comp](../../ncnn/src/layer/vulkan/shader/sdpa_kvcache_append.comp)).
|
||||
- **Paging / eviction**: none. The cache is a contiguous per-layer buffer; no paged attention, no sliding window, no eviction policy. Growth = realloc + memcpy. [Verified]
|
||||
- **Reuse across calls**: consume-and-replace: `ex.input(cache_in_idx, cache); cache.release(); ex.extract(cache_out_idx, cache, 1)`; `type=1` preserves fp16/bf16/packing/allocator/capacity. A session-owned `UnlockedPoolAllocator` with `size_compare_ratio=0` (or `VkAllocator`) must be set on every Extractor of the session and differ from the blob allocator; shallow `Mat` copies are not snapshots, so beam search must clone ([kvcache.md](../../ncnn/docs/developer-guide/kvcache.md), [src/net.cpp#L2876](../../ncnn/src/net.cpp#L2876)). Whisper's beam search keeps a `std::vector<ncnn::Mat> kvcache` per beam ([examples/whisper.cpp#L291](../../ncnn/examples/whisper.cpp#L291)).
|
||||
- **Cross-attention**: MHA with separate q and k/v inputs reuses a static cache without appending (encoder–decoder), SDPA always appends ([kvcache.md](../../ncnn/docs/developer-guide/kvcache.md) §2).
|
||||
- **Attention mask / RoPE** are model inputs: `benchncnn_llm` builds a causal `attention_mask (dst_seqlen × cur_seqlen)` with −10000 and `cos/sin` caches on the host per step ([benchmark/benchncnn_llm.cpp#L285–309](../../ncnn/benchmark/benchncnn_llm.cpp#L285)); `RotaryEmbed` takes `(x, cos_cache, sin_cache)` ([src/layer/rotaryembed.cpp#L19](../../ncnn/src/layer/rotaryembed.cpp#L19)).
|
||||
- **Other recurrent state**: LSTM/GRU/RNN take/return hidden-state blobs; RVM example threads 4 recurrent tensors between frames ([examples/rvm.cpp#L197](../../ncnn/examples/rvm.cpp#L197)).
|
||||
- Tests: [tests/test_sdpa_kvcache.cpp](../../ncnn/tests/test_sdpa_kvcache.cpp), [tests/test_sdpa_kvcache_session.cpp](../../ncnn/tests/test_sdpa_kvcache_session.cpp), [tests/test_multiheadattention_kvcache.cpp](../../ncnn/tests/test_multiheadattention_kvcache.cpp), [tests/test_multiheadattention_kvcache_allocator.cpp](../../ncnn/tests/test_multiheadattention_kvcache_allocator.cpp); perf: [tests/perf/perf_sdpa_kvcache.cpp](../../ncnn/tests/perf/perf_sdpa_kvcache.cpp), [perf_sdpa_decode.cpp](../../ncnn/tests/perf/perf_sdpa_decode.cpp), [perf_sdpa_prefill.cpp](../../ncnn/tests/perf/perf_sdpa_prefill.cpp).
|
||||
|
||||
### Multimodal stages [Verified]
|
||||
- ncnn has no notion of stages; each stage is a separate `Net`, glued by host code. Whisper: `fbank` net (waveform → log-mel, uses Spectrogram-based graph [Inferred]) → `encoder` → `embed_token` + `embed_position` → `decoder` (self-attn with dynamic cache, cross-attn with static encoder k/v) → `proj_out` ([examples/whisper.cpp#L310–318](../../ncnn/examples/whisper.cpp#L310)). Piper: `enc_p` → `emb_g` (speaker) → `dp` (duration predictor with rational-quadratic spline custom layer) → host `path_attention` (length regulation + noise) → `flow` → `dec` (HiFi-GAN-style vocoder inside VITS) → PCM ([examples/piper.cpp#L518–675](../../ncnn/examples/piper.cpp#L518)).
|
||||
- Vision: `Mat::from_pixels_resize` + `substract_mean_normalize` → detector net → host NMS/decode (e.g. [examples/yolo11.cpp#L352](../../ncnn/examples/yolo11.cpp#L352)); Vulkan zero-copy camera path via AHB import ([docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md](../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md)).
|
||||
- Diffusion: no sampler/scheduler code in tree. [Verified by absence: grep for "diffusion|ddim|scheduler" in src/examples returns nothing relevant]
|
||||
|
||||
---
|
||||
|
||||
## 5. Scheduler
|
||||
|
||||
### (a) Request / token / pipeline-level scheduling — **N/A** [Verified]
|
||||
- There is no request queue, no batch former, no continuous batching, no prefill/decode scheduler, no multi-stage pipeline orchestrator. The unit of work is one synchronous `Extractor::extract` call on the calling thread. Prefill vs decode is purely the host choosing `seqlen>1` vs `seqlen=1` inputs against the same Net ([benchmark/benchncnn_llm.cpp#L406–469](../../ncnn/benchmark/benchncnn_llm.cpp#L406), [kvcache.md](../../ncnn/docs/developer-guide/kvcache.md) §5).
|
||||
- Concurrency model: a `Net` is immutable after load and can be shared by many threads, each creating its own `Extractor` (documented pattern with per-thread `UnlockedPoolAllocator` + shared locked workspace allocator, [custom-allocator.md](../../ncnn/docs/developer-guide/custom-allocator.md); Vulkan "batch inference" with `omp parallel for` over extractors bounded by `compute_queue_count()`, [vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md)). An `Extractor` itself is single-threaded state (`blob_mats` vector) and not safe to share. [Inferred]
|
||||
- Cancellation/abort: none; a running `extract` cannot be interrupted. Backpressure: none (host's job). Timeouts: only the Vulkan chunked-submit heuristic to avoid GPU driver watchdogs ([src/net.cpp#L250](../../ncnn/src/net.cpp#L250)).
|
||||
- Batching across requests: `NCNN_BATCH` allows `n>1` in a single extract but the KV cache rejects `n>1` ([src/net.cpp#L2882](../../ncnn/src/net.cpp#L2882)), so LLM decode is strictly one sequence per extract.
|
||||
|
||||
### (b) Graph / operator / thread-level scheduling [Verified]
|
||||
- **Op ordering**: depth-first recursion from the requested blob to its producers (`forward_layer` recursion over `layer->bottoms`), i.e. topological order induced by param order; memoized in `blob_mats` so shared subgraphs run once per Extractor ([src/net.cpp#L123–150](../../ncnn/src/net.cpp#L123)). Multiple `extract` calls on one Extractor reuse computed blobs (e.g. Piper extracts `out0/out1/out2` from one run).
|
||||
- **Intra-op threading**: OpenMP `parallel for` inside kernels with `num_threads(opt.num_threads)`; per-layer `featmask` bit 7 forces single thread; `set_omp_num_threads`, `set_kmp_blocktime` (default 20 ms spin), `set_cpu_powersave(0|1|2)` binds to all/little/big cores (Android/Linux), `set_cpu_thread_affinity(CpuSet)` ([src/cpu.h#L145–170](../../ncnn/src/cpu.h#L145); [openmp-best-practice.md](../../ncnn/docs/how-to-use-and-FAQ/openmp-best-practice.md)). `NCNN_SIMPLEOMP` provides a minimal LLVM-ABI OpenMP runtime supporting only `parallel for num_threads` ([src/simpleomp.h](../../ncnn/src/simpleomp.h)); `NCNN_THREADS=OFF` builds single-threaded (WASM CI does this).
|
||||
- **Inter-op parallelism**: none on CPU (sequential layer execution). On Vulkan, layers are recorded back-to-back into one command buffer with automatic barriers; no multi-queue/multi-stream overlap inside one extract. Overlap is achievable only by running several Extractors on several threads/queues.
|
||||
- **Stream assignment**: one `VkCompute` (compute queue) per extract; `VkTransfer` (transfer queue if `unified_compute_transfer_queue()` false) for weight upload ([src/gpu.h#L257–264](../../ncnn/src/gpu.h#L257)).
|
||||
- **Memory scheduling**: light mode frees blobs after last use; pool allocators recycle; no static memory planning.
|
||||
- **A53/A55 tuning**: `use_a53_a55_optimized_kernel` auto-set from the current thread's core ([src/option.cpp](../../ncnn/src/option.cpp), [docs/developer-guide/arm-a53-a55-dual-issue.md](../../ncnn/docs/developer-guide/arm-a53-a55-dual-issue.md)).
|
||||
|
||||
---
|
||||
|
||||
## 6. I/O layer
|
||||
|
||||
- **Image preprocessing** [Verified]: `Mat::from_pixels / from_pixels_resize / from_pixels_roi / from_pixels_roi_resize` with `PIXEL_RGB/BGR/GRAY/RGBA/BGRA` and `X2Y` conversions, stride support, `to_pixels*`, `substract_mean_normalize`, `from_android_bitmap*` ([src/mat.h#L255–335](../../ncnn/src/mat.h#L255)); resize/rotate/affine/drawing helpers ([src/mat_pixel_resize.cpp](../../ncnn/src/mat_pixel_resize.cpp), [src/mat_pixel_rotate.cpp](../../ncnn/src/mat_pixel_rotate.cpp), [src/mat_pixel_affine.cpp](../../ncnn/src/mat_pixel_affine.cpp), [src/mat_pixel_drawing.cpp](../../ncnn/src/mat_pixel_drawing.cpp); doc [efficient-roi-resize-rotate.md](../../ncnn/docs/how-to-use-and-FAQ/efficient-roi-resize-rotate.md)). `NCNN_SIMPLEOCV` supplies `cv::Mat/imread/imwrite` stand-ins ([src/simpleocv.h](../../ncnn/src/simpleocv.h)) backed by [src/stb_image.h](../../ncnn/src/stb_image.h). Vulkan YCbCr camera import shader [src/convert_ycbcr.comp](../../ncnn/src/convert_ycbcr.comp).
|
||||
- **Audio preprocessing** [Verified]: no audio I/O library. `Spectrogram` (n_fft, hop, win, hann/hamming, center/pad, power, onesided, normalized) and `InverseSpectrogram` layers implement STFT/iSTFT in-graph with a naive DFT (cos/sin loops) ([src/layer/spectrogram.cpp](../../ncnn/src/layer/spectrogram.cpp), [src/layer/inversespectrogram.cpp](../../ncnn/src/layer/inversespectrogram.cpp)); Whisper reads 16-bit WAV by hand (`load_wav_samples`, [examples/whisper.cpp#L829](../../ncnn/examples/whisper.cpp#L829)) and Piper writes a WAV header by hand ([examples/piper.cpp#L677](../../ncnn/examples/piper.cpp#L677)).
|
||||
- **Tokenization** [Verified]: none in the library. Whisper example: plain reverse-vocab list (`whisper_vocab.txt`) with hard-coded special-token ids ([examples/whisper.cpp#L31–200](../../ncnn/examples/whisper.cpp#L31)); Piper: binary word→phoneme-id dictionary + rule-based sentence splitting (`simple_phonemize`, [examples/piper.cpp#L323](../../ncnn/examples/piper.cpp#L323)); `benchncnn_llm` feeds random embeddings (no tokenizer). `Embed` layer maps ids→vectors in-graph ([src/layer/embed.h](../../ncnn/src/layer/embed.h)).
|
||||
- **Streaming** [Verified]: no streaming API. Token streaming falls out naturally from the per-step decode loop (the app owns the loop). Audio streaming/chunking is not provided; Piper synthesizes the whole utterance then normalizes/clips to int16 ([examples/piper.cpp#L646](../../ncnn/examples/piper.cpp#L646)). Chunked/streaming vocoding would need the model to be exported in a chunkable form. [Inferred]
|
||||
- **Buffering** [Verified]: `Mat` refcounted buffers; `Extractor::input` stores a shallow reference (no copy) ([src/net.cpp#L2861](../../ncnn/src/net.cpp#L2861)); external memory `Mat(w,h,c,data)` constructors allow zero-copy input.
|
||||
- **Postprocessing** [Verified]: host code in examples (argmax/greedy in whisper `run_decoder_*`, beam search with log-softmax, NMS in detectors). No sampling library (temperature/top-p) in tree.
|
||||
- **Application boundaries** [Verified]:
|
||||
- C++ API: [src/net.h](../../ncnn/src/net.h), [src/mat.h](../../ncnn/src/mat.h), [src/option.h](../../ncnn/src/option.h).
|
||||
- C API (`NCNN_C_API`, default ON): [src/c_api.h](../../ncnn/src/c_api.h) incl. `ncnn_mat_create_*_batch`, `ncnn_option_set_kvcache_*`, `ncnn_pipelinecache_*`.
|
||||
- Python (`pip install ncnn`, pybind11): `ncnn.Net/Extractor/Mat/Option/Layer/PoolAllocator/VulkanDevice…`, `extract(name, type)` returning `(ret, Mat)`, kv-cache setters ([python/src/main.cpp#L959](../../ncnn/python/src/main.cpp#L959), [python/README.md](../../ncnn/python/README.md)); a model zoo of detection/pose/classification wrappers ([python/ncnn/model_zoo](../../ncnn/python/ncnn/model_zoo)).
|
||||
- Android: `AAssetManager` loaders ([src/net.h#L110](../../ncnn/src/net.h#L110)), `Mat::from_android_bitmap`, AHB Vulkan import; the JNI demo project moved out of tree ([examples/squeezencnn/README.md](../../ncnn/examples/squeezencnn/README.md)).
|
||||
- iOS: `Info.plist` + `ios.toolchain.cmake`; MoltenVK for GPU.
|
||||
- HTTP server / CLI serving: none. Only benchmark/example binaries ([benchmark/benchncnn.cpp](../../ncnn/benchmark/benchncnn.cpp)).
|
||||
|
||||
---
|
||||
|
||||
## 7. Execution flow
|
||||
|
||||
### (a) Text LLM decode with KV cache (as in `benchncnn_llm` / kvcache.md) [Verified]
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App
|
||||
participant Net as Net(decoder)
|
||||
participant Ex as Extractor
|
||||
participant NP as NetPrivate::forward_layer
|
||||
participant SDPA as SDPA layer (per block)
|
||||
participant Gemm as Gemm/RMSNorm/RoPE layers
|
||||
participant Proj as Net(proj_out)
|
||||
|
||||
App->>Net: load_param(decoder.param patched 7=1, Gemm 7=0) ; load_model(bin)
|
||||
Note over Net: create_pipeline per layer, (Vulkan: upload weights, compile SPIR-V)
|
||||
App->>App: prefill: tokens -> embeds Mat(hidden, seqlen), causal mask, cos/sin cache
|
||||
App->>Ex: create_extractor(); set_kvcache_allocator(pool); set_kvcache_max_seqlen_hint(N)
|
||||
App->>Ex: input(in0..in3), input(cache_k_in_i, empty) for all i
|
||||
App->>Ex: extract(cache_k_out_i, cache[i], type=1) for all i
|
||||
Ex->>NP: forward_layer(producer of cache_k_out_i)
|
||||
NP->>Gemm: RMSNorm -> Gemm(q,k,v) -> RotaryEmbed
|
||||
NP->>SDPA: forward(q,k,v,mask,past_k,past_v)
|
||||
SDPA->>SDPA: create_or_grow_kvcache (reserve hint) ; append ; flash-attn QK^T softmax V
|
||||
SDPA-->>NP: out, cache_k_out, cache_v_out (private layout)
|
||||
NP-->>Ex: blob_mats filled (memoized)
|
||||
App->>Ex: extract("out0", hidden) (already computed for last block, no re-run)
|
||||
App->>Proj: Extractor.input(last row).extract("out0", logits)
|
||||
App->>App: argmax/sample -> next token
|
||||
loop decode step
|
||||
App->>Ex: new Extractor; input(embed of 1 token, mask (past+1 x 1), cos/sin row)
|
||||
App->>Ex: input(cache_k_in_i, cache[i]); cache[i].release()
|
||||
App->>Ex: extract(cache_k_out_i, cache[i], 1) ; extract("out0")
|
||||
Note over SDPA: cache reused in place when seqlen <= capacity, else grow x1.5 + memcpy
|
||||
App->>Proj: logits -> next token
|
||||
end
|
||||
```
|
||||
|
||||
### (b) TTS / ASR path — Whisper ASR (encoder–decoder with cross-attention cache) and Piper TTS [Verified]
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App as whisper.cpp
|
||||
participant FB as Net(fbank)
|
||||
participant Enc as Net(encoder)
|
||||
participant Emb as Net(embed_token/position)
|
||||
participant Dec as Net(decoder, MHA 7=1)
|
||||
participant PO as Net(proj_out)
|
||||
|
||||
App->>App: load_wav_samples(16-bit PCM) -> Mat waveform
|
||||
App->>FB: Extractor.input("in0", waveform).extract("out0", input_features)
|
||||
Note over FB: log-mel via Spectrogram-based graph [Inferred]
|
||||
App->>Enc: input(input_features).extract("out0", encoder_states)
|
||||
App->>App: prompt ids (SOT, lang, task, no_timestamps)
|
||||
App->>Emb: ids -> token_embeds ; positions -> position_embeds ; sum
|
||||
App->>Dec: input(in0=embeds, in1=encoder_states, in2=attention_mask)
|
||||
App->>Dec: extract(cache_k/v_out_i, kv[i], type=1) for all MHA ; extract("out0")
|
||||
Note over Dec: self-attn MHA appends to dynamic cache; cross-attn MHA uses static k/v from encoder_states
|
||||
App->>PO: last state -> logits ; log_softmax ; beam search candidates keep own kv vector
|
||||
loop until EOT
|
||||
App->>Dec: input(1 token embeds, encoder_states, mask, kv[i]) ; extract(kv_out, "out0")
|
||||
App->>PO: logits -> beam update
|
||||
end
|
||||
App->>App: detokenize via reverse vocab -> text
|
||||
|
||||
Note over App: Piper TTS (examples/piper.cpp): text -> simple_phonemize (dict) -> Net enc_p (custom relative_embeddings layers) -> Net emb_g(speaker) -> Net dp (custom spline layer, noise) -> host path_attention (durations, noise) -> Net flow -> Net dec (vocoder) -> normalize -> int16 PCM -> WAV 22050 Hz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Tests and examples
|
||||
|
||||
- **Unit tests** [Verified]: 181 `tests/test_*.cpp` (per-layer randomized tests comparing naive vs optimized vs Vulkan via [tests/testutil.h](../../ncnn/tests/testutil.h); `_oom` variants exercise allocation failure). Build with `-DNCNN_BUILD_TESTS=ON`; each becomes a CTest via [cmake/run_test.cmake](../../ncnn/cmake/run_test.cmake) ([tests/CMakeLists.txt](../../ncnn/tests/CMakeLists.txt)). Non-layer tests: `test_c_api`, `test_cpu`, `test_expression`, `test_modelbin`, `test_paramdict`, `test_mat_batch`, `test_mat_pixel*`, `test_squeezenet`, and with Vulkan `test_command`, `test_pipeline_cache`. Transformer-specific: `test_sdpa*`, `test_multiheadattention*` (incl. `_kvcache`, `_block_quant`), `test_rotaryembed`, `test_rmsnorm`, `test_gemm_*` (incl. `_block_quant`, `_nt_int8`), `test_embed`, `test_spectrogram`, `test_inversespectrogram`, `test_glu`, `test_lstm/gru`.
|
||||
- **How to run (per repo docs/CI)** [Verified]:
|
||||
```shell
|
||||
# library + tests (CPU)
|
||||
mkdir build && cd build
|
||||
cmake -DNCNN_BUILD_TESTS=ON .. # add -DNCNN_VULKAN=ON for GPU tests (needs glslang submodule)
|
||||
cmake --build . -j $(nproc)
|
||||
ctest --output-on-failure -j $(nproc) # exact invocation used by CI: .github/workflows/linux-x64-cpu-gcc.yml
|
||||
# model benchmark (CPU, 4 big-core threads, 4 loops, cooldown) / GPU device 0
|
||||
./benchmark/benchncnn 4 4 2 -1 1
|
||||
./benchmark/benchncnn 8 1 2 0 0
|
||||
# LLM prefill/decode tokens-per-second table
|
||||
./benchmark/benchncnn_llm [loop count] [num threads] [powersave] [gpu device] [cooling down]
|
||||
# examples (need OpenCV or -DNCNN_SIMPLEOCV=ON) — piper needs en_*.ncnn.param/bin + en-word_id.bin in cwd
|
||||
./examples/piper "Hello World" 0 out.wav
|
||||
./examples/whisper <wav> ... # needs whisper_vocab.txt and exported nets
|
||||
```
|
||||
Sources: [docs/how-to-build/how-to-build.md#L40](../../ncnn/docs/how-to-build/how-to-build.md#L40), [.github/workflows/linux-x64-cpu-gcc.yml#L53](../../ncnn/.github/workflows/linux-x64-cpu-gcc.yml#L53), [benchmark/README.md](../../ncnn/benchmark/README.md), [examples/piper.cpp#L718](../../ncnn/examples/piper.cpp#L718). Note this checkout cannot build Vulkan or Python until `git submodule update --init` (glslang, pybind11).
|
||||
- **Perf micro-benchmarks** [Verified]: [tests/perf](../../ncnn/tests/perf) (`perf_sdpa_prefill/decode/kvcache`, conv, gemm-ish ops).
|
||||
- **Model benchmarks** [Verified]: `benchncnn` (36 built-in params incl. `vision_transformer.param`, int8 variants; usage `./benchncnn [loops] [threads] [powersave] [gpu] [cooldown] param=… shape=…`) and `benchncnn_llm` (7 LLM decoders, prints prefill/decode tokens/s; `[loop] [threads] [powersave] [gpu] [cooldown]`) ([benchmark/README.md](../../ncnn/benchmark/README.md), [benchmark/benchncnn_llm.cpp#L643](../../ncnn/benchmark/benchncnn_llm.cpp#L643)). README carries repo-reported timing tables for many boards (Jetson AGX Orin/Orin Nano/Nano/TX2 NX, RK3588, Snapdragon 8xx/X Elite, Raspberry Pi 3/4/5, Apple A7, Loongson…) — repo-reported, not reproduced.
|
||||
- **Examples** [Verified] ([examples/CMakeLists.txt](../../ncnn/examples/CMakeLists.txt)): vision — squeezenet (+C API), yolov2…yolo11 (det/seg/pose/obb/cls), yoloworld, yolox, nanodet, scrfd, retinaface, arcface, simplepose, yolact, rvm (video matting w/ recurrent state), ppocrv5 (OCR), p2pnet, mobilenet/squeezenet SSD; audio — [piper.cpp](../../ncnn/examples/piper.cpp) (TTS), [whisper.cpp](../../ncnn/examples/whisper.cpp) (ASR). Python examples mirror detection models ([python/examples](../../ncnn/python/examples)). Mobile app demos are external (nihui/ncnn-android-*), only a README remains in [examples/squeezencnn](../../ncnn/examples/squeezencnn).
|
||||
- **CI** [Verified]: 45 workflows in [.github/workflows](../../ncnn/.github/workflows): per-OS/arch CPU & GPU (Linux x64 GPU via SwiftShader/lavapipe [Inferred]), Android (armeabi-v7a/arm64-v8a/x86/x86_64/riscv64, Vulkan ON), iOS/macOS/Catalyst/tvOS/visionOS/watchOS, HarmonyOS, WebAssembly (3 variants incl. simpleomp threads), ESP32, RISC-V ELF, Windows (MSVC/clang/mingw/XP/ARM), Intel SDE for AVX-512 variants, code-format, CodeQL, coverage, `compare-binary-size` (reports `libncnn.so` delta for x86_64/armhf/aarch64 on PRs), pnnx and python wheel builds.
|
||||
|
||||
---
|
||||
|
||||
## 9. TTS relevance
|
||||
|
||||
**What exists** [Verified]
|
||||
- End-to-end TTS example: Piper/VITS split into 5 nets with 3 app-registered custom layers, speaker embedding, duration predictor, flow, and neural vocoder (inside `dec`), 22.05 kHz int16 output; Vulkan enabled per net (`opt.use_vulkan_compute = true`) ([examples/piper.cpp](../../ncnn/examples/piper.cpp)). Export recipe references nihui/ncnn-android-piper `export_ncnn.py` (external).
|
||||
- ASR counterpart with the full encoder–decoder + KV-cache machinery (Whisper tiny…large-v3-turbo) ([examples/whisper.cpp](../../ncnn/examples/whisper.cpp)).
|
||||
- Audio ops: `Spectrogram`/`InverseSpectrogram` (STFT/iSTFT; pnnx maps `torch.stft/istft`, `torchaudio.functional.spectrogram/inverse_spectrogram` — [tools/pnnx/src/pass_ncnn/torch_stft.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torch_stft.cpp), [torchaudio_F_inverse_spectrogram.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torchaudio_F_inverse_spectrogram.cpp)), `Convolution1D/ConvolutionDepthWise1D/Deconvolution1D/DeconvolutionDepthWise1D` (with ARM fp16 and Vulkan kernels: [src/layer/arm/convolution1d_arm_asimdhp.cpp](../../ncnn/src/layer/arm/convolution1d_arm_asimdhp.cpp), [src/layer/vulkan/convolution1d_vulkan.cpp](../../ncnn/src/layer/vulkan/convolution1d_vulkan.cpp)), `Pooling1D`, `LSTM/GRU`, `LayerNorm/GroupNorm/InstanceNorm`, `Gemm/MatMul`, `MultiHeadAttention/SDPA` with cache — sufficient for VITS, HiFi-GAN/Vocos-style vocoders, Tacotron-like RNNs and transformer TTS decoders.
|
||||
- Transformer-decoder TTS (e.g. token-based LLM-TTS such as CosyVoice/Fish-Speech-style AR heads) can reuse the LLM decode loop with `SDPA 7=1`; speech-token → mel/flow-matching and vocoder stages would be additional nets. [Inferred]
|
||||
|
||||
**What is missing** [Verified unless noted]
|
||||
- No tokenizer/G2P/phonemizer library (only an English dictionary hack in the example; comment says "works for english only", [examples/piper.cpp#L325](../../ncnn/examples/piper.cpp#L325)).
|
||||
- No audio I/O, resampling, mel filterbank utility, or vocoder library; no streaming/chunked synthesis API; no FFT acceleration (Spectrogram is a naive DFT [Verified from cos/sin loops], likely slow for long inputs [Inferred]).
|
||||
- No sampler (temperature/top-k/top-p/repetition penalty) or beam search utility outside the Whisper example.
|
||||
- No neural audio codec (EnCodec/SNAC/DAC) implementations or examples; codebook lookup can be done with `Embed` [Inferred].
|
||||
- No batched multi-utterance decode (KV cache is batch-1).
|
||||
- Piper export requires patches to the upstream repo and an external script (not in tree).
|
||||
|
||||
---
|
||||
|
||||
## 10. VLA relevance
|
||||
|
||||
**What exists** [Verified]
|
||||
- Vision encoders: CNN backbones are the historical core (winograd/sgemm/int8/fp16 conv on ARM/x86/Vulkan); ViT is exercised by `benchmark/models/vision_transformer.param` and `MultiHeadAttention/LayerNorm/GELU/Gemm` kernels ([benchmark/models/vision_transformer.param](../../ncnn/benchmark/models/vision_transformer.param)); `Interp`, `GridSample`, `DeformableConv2D`, `ROIAlign` for spatial ops. Camera zero-copy on Android via AHB.
|
||||
- Structured / proprioceptive inputs: any number of `Input` blobs (1-D `Mat`), `Concat`, `Gemm/InnerProduct`, `Embed` for discrete state; expression-based reshape for variable shapes.
|
||||
- Action heads: MLP (`InnerProduct`/`Gemm` + activations), Gaussian/mixture heads (host post-process), autoregressive token heads via the LLM path (`Embed` → decoder with `SDPA 7=1` → `Gemm` proj).
|
||||
- Diffusion/flow policy building blocks: `Convolution1D`/`GroupNorm`/`Mish`/`SiLU`(Swish) (Diffusion Policy U-Net 1D), transformer blocks, `Embed` for timestep tables, `RotaryEmbed`. Each denoise step = one `Extractor` call with `(x_t, t_embed, cond)` inputs; host implements the DDIM/flow-matching update. [Inferred]
|
||||
- Low-latency loop support: light mode + pooled allocators, `set_cpu_powersave(2)` big-core pinning, `openmp_blocktime` tuning, Vulkan `PipelineCache` to avoid first-run shader compilation, mmap model loading, `VkMat` chaining across nets without downloads ([vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md)).
|
||||
|
||||
**What is missing** [Verified unless noted]
|
||||
- No VLA/robotics example, no diffusion-policy or flow-matching sampler, no action chunking utilities.
|
||||
- No multi-input scheduling: vision encoder and language decoder are separate Nets executed serially by the host; no async overlap primitive beyond threads + separate Extractors [Inferred].
|
||||
- No paged/shared KV cache across parallel action samples (batch-1 cache); sampling K action candidates needs K cache clones.
|
||||
- No NPU/DSP delegates (Hexagon/QNN, Ethos, RKNPU, Jetson DLA/CUDA); GPU only via Vulkan, so Jetson runs CPU or Tegra-Vulkan, not TensorRT.
|
||||
- Vision-language projector/token merging (e.g. SigLIP+MLP) must be exported as ordinary Gemm graphs; multimodal embedding concatenation is host code [Inferred].
|
||||
|
||||
---
|
||||
|
||||
## 11. Edge deployment profile
|
||||
|
||||
- **Build system** [Verified]: CMake ≥ 2.8.12 (3.10 policies), C++03/11 compatible core (`host.gcc-c++03.toolchain.cmake` exists), no runtime third-party deps. Key options ([CMakeLists.txt#L61–110](../../ncnn/CMakeLists.txt#L61)): `NCNN_VULKAN` (OFF), `NCNN_SIMPLEVK` (ON), `NCNN_SYSTEM_GLSLANG` (OFF), `NCNN_OPENMP` (ON), `NCNN_SIMPLEOMP/SIMPLESTL/SIMPLEOCV/SIMPLEMATH` (OFF), `NCNN_THREADS` (ON), `NCNN_RUNTIME_CPU` (ON), `NCNN_INT8` (ON), `NCNN_WEIGHT_QUANT` (ON), `NCNN_BF16` (ON), `NCNN_BATCH` (ON), `NCNN_C_API` (ON), `NCNN_PLATFORM_API` (ON), `NCNN_PIXEL*` (ON), `NCNN_STDIO/STRING` (ON), `NCNN_SHARED_LIB` (OFF), `NCNN_DISABLE_RTTI/EXCEPTION` (ON when tools/examples off), `NCNN_BUILD_TOOLS/EXAMPLES/BENCHMARK/TESTS`, `NCNN_PYTHON`, per-ISA `NCNN_ARM82/ARM82DOT/ARM84BF16/ARM84I8MM/ARM86SVE*`, `NCNN_AVX2/AVX512*/AVXVNNI*`, `NCNN_RVV/ZFH/ZVFH/XTHEADVECTOR`, `NCNN_MSA/LSX/LASX`, `WITH_LAYER_<name>`.
|
||||
- **Binary size** [Verified/Unknown]: the repo does not publish absolute sizes; CI `compare-binary-size` computes `libncnn.so` deltas per PR for x86_64/armhf/aarch64 ([.github/workflows/compare-binary-size.yml](../../ncnn/.github/workflows/compare-binary-size.yml)). The size cheatsheet ([build-minimal-library.md](../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md)) lists reductions: disable RTTI/exceptions, Vulkan, STDIO/STRING, BF16/INT8/WEIGHT_QUANT, pixel functions, OpenMP, unused ISA kernels, runtime dispatch, unused layers, and STL (`simplestl`, `-nodefaultlibs -fno-builtin -nostdinc++`). Absolute numbers: [Unknown].
|
||||
- **Dependencies** [Verified]: none at runtime. Optional: OpenMP runtime (or simpleomp), Vulkan driver (loaded dynamically; no libvulkan link with simplevk), glslang (submodule, statically linked when `NCNN_VULKAN=ON`; not initialized in this checkout), OpenCV or simpleocv for examples/tools, protobuf for legacy converters, libtorch for building pnnx from source (pip wheel avoids it).
|
||||
- **Platforms** [Verified]:
|
||||
- Embedded NVIDIA/Jetson: aarch64 toolchain + Tegra Vulkan ([how-to-build.md#L66](../../ncnn/docs/how-to-build/how-to-build.md#L66)); benchmark tables for AGX Orin/Orin Nano/Nano/TX2 NX (repo-reported). No CUDA/TensorRT.
|
||||
- ARM CPU: ARMv7 (NEON, VFPv4 fp16), ARMv8/8.2/8.4/8.6 (fp16, dotprod, fhm, bf16, i8mm, SVE/SVE2 flags), A53/A55 dual-issue tuning, big.LITTLE affinity; toolchains for Cortex-A cross builds, HiSilicon Hi35xx, Rockchip, Allwinner, Raspberry Pi ([toolchains](../../ncnn/toolchains)).
|
||||
- Apple: macOS/iOS/tvOS/visionOS/watchOS builds via [toolchains/ios.toolchain.cmake](../../ncnn/toolchains/ios.toolchain.cmake) and CI; GPU only through MoltenVK (Vulkan→Metal), no native Metal/CoreML/ANE.
|
||||
- Android: NDK builds for all ABIs incl. riscv64, Vulkan on Adreno/Mali (API 24+ with system loader; simplevk allows <24), `AAssetManager` loading, AHB zero-copy (API 26+), JNI bitmap helpers. Prebuilt release archives are built with `-DANDROID_PLATFORM=android-19/21 -DNCNN_VULKAN=ON` ([build-android.cmd](../../ncnn/build-android.cmd)). No NNAPI/QNN.
|
||||
- HarmonyOS (`ohos.toolchain.cmake`, `__OHOS__` mmap path), QNX, Windows XP/ARM, WebAssembly (single-thread or simpleomp+pthreads, SIMD128 via SSE2 path), ESP32/RISC-V bare metal (`riscv64-unknown-elf`, XuanTie C906/C907/C908/C910, SpacemiT K1), LoongArch, MIPS, PowerPC.
|
||||
- **Model conversion path & constraints** [Verified]: PyTorch → pnnx (needs `inputshape` for shape propagation; `inputshape2` to mark dynamic dims; unsupported modules → `moduleop` + custom layer; fp16 weights default) → optional `ncnnoptimize` (skip if pnnx) → optional `ncnn2table/ncnn2int8` or `ncnnllm2table/ncnnllm2int`. ONNX also via pnnx (preferred) or `onnx2ncnn`. Constraints: batch dim removed by pnnx (`Mat` is CHW-ish, WHC ordering in tools), ops must exist among the 110 types, control flow must be unrolled or hosted, LLM decoders need param patching for cache and dynamic-M Gemm, Vulkan-unsupported layers (e.g. `Embed`, `RNN/LSTM/GRU`, `Spectrogram` — no `*_vulkan.cpp` in [src/layer/vulkan](../../ncnn/src/layer/vulkan)) fall back to CPU with sync cost.
|
||||
|
||||
---
|
||||
|
||||
## 12. Limitations and unknowns
|
||||
|
||||
1. [Verified] No request scheduler, batching server, or streaming API; all orchestration is application code.
|
||||
2. [Verified] KV cache is per-layer contiguous with geometric growth and batch size 1; no paging, sliding window, prefix sharing or eviction.
|
||||
3. [Verified] Attention mask and RoPE cos/sin tables are host-provided inputs each step; positions are implicit in the mask shape.
|
||||
4. [Verified] Vulkan shaders compile at runtime (glslang); first-run latency unless a `PipelineCache` file is shipped; cache is not portable across devices/drivers/builds ([vulkan-pipeline-cache.md#L288](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md#L288)).
|
||||
5. [Verified] No CUDA/Metal/CoreML/OpenCL/NNAPI/QNN backends; NPUs on phones and Jetson tensor cores are unreachable except through Vulkan cooperative-matrix shaders (15 `*_cm.comp` shaders incl. `sdpa_fa_cm.comp`, `gemm_cm.comp`).
|
||||
6. [Verified] W4/W6/W8 block-quantized Gemm/MHA run on CPU only and disable fp16/bf16/packing/Vulkan for those layers.
|
||||
7. [Verified] Piper/Whisper export scripts and Android demo apps live in external nihui/* repos; Piper needs a patched upstream.
|
||||
8. [Verified] `Spectrogram` uses a direct DFT (no FFT library); [Inferred] cost grows as O(n_fft²) per frame.
|
||||
9. [Verified] glslang and pybind11 submodules are not initialized in this checkout, so Vulkan and Python builds are not possible from it as-is.
|
||||
10. [Unknown] Absolute library binary sizes per configuration (only CI deltas exist).
|
||||
11. [Unknown] Real throughput of `benchncnn_llm` on target edge boards; README LLM section describes the harness but no LLM numbers were found in the tables reviewed.
|
||||
12. [Unknown] Thread-safety guarantees of `VulkanDevice` allocator/queue pools under heavy multi-thread use beyond the documented `compute_queue_count()` bound.
|
||||
13. [Inferred] Multi-Net pipelines (encoder → decoder) pay a host round-trip unless both are Vulkan and chained with `VkMat` + shared `VkCompute`; CPU-fallback layers inside a GPU graph force `submit_and_wait`.
|
||||
14. [Inferred] Very large vocab `Gemm` proj_out on CPU dominates decode time for small LLMs; no fused sampling.
|
||||
15. [Proposal] For a vLLM-Omni edge backend, wrap `Net`+`Extractor` per stage behind an executor that owns the kv-cache allocator per sequence, keeps a `PipelineCache` per device, and pins decode threads to big cores; use `extract(type=1)` handles as opaque cache objects.
|
||||
|
||||
---
|
||||
|
||||
## 13. Reference index
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| [../../ncnn/README.md](../../ncnn/README.md) | Project overview, quick start (pnnx → ncnn), release links |
|
||||
| [../../ncnn/CMakeLists.txt](../../ncnn/CMakeLists.txt) | Top-level build options and ISA feature flags |
|
||||
| [../../ncnn/build-android.cmd](../../ncnn/build-android.cmd) | Flags used for prebuilt Android archives |
|
||||
| [../../ncnn/src/net.h](../../ncnn/src/net.h) | `Net`/`Extractor` public API incl. kv-cache setters |
|
||||
| [../../ncnn/src/net.cpp](../../ncnn/src/net.cpp) | Graph load, layer creation/fallback, lazy forward, layout conversion, batch loop, Vulkan submit heuristics |
|
||||
| [../../ncnn/src/layer.h](../../ncnn/src/layer.h) | `Layer` interface, `support_*` flags, creator macros |
|
||||
| [../../ncnn/src/layer.cpp](../../ncnn/src/layer.cpp) | Registry lookup, `Layer_final` CPU/Vulkan wrapper |
|
||||
| [../../ncnn/src/layer_declaration.h.in](../../ncnn/src/layer_declaration.h.in) / [layer_registry.h.in](../../ncnn/src/layer_registry.h.in) / [layer_type_enum.h.in](../../ncnn/src/layer_type_enum.h.in) | Generated registry templates |
|
||||
| [../../ncnn/src/mat.h](../../ncnn/src/mat.h) | `Mat`/`VkMat`/`VkImageMat`, pixel conversion API, batch dim |
|
||||
| [../../ncnn/src/option.h](../../ncnn/src/option.h) / [option.cpp](../../ncnn/src/option.cpp) | Runtime options and defaults |
|
||||
| [../../ncnn/src/allocator.h](../../ncnn/src/allocator.h) / [allocator.cpp](../../ncnn/src/allocator.cpp) | Host/Vulkan allocators and pools |
|
||||
| [../../ncnn/src/cpu.h](../../ncnn/src/cpu.h) / [cpu.cpp](../../ncnn/src/cpu.cpp) | ISA detection, topology, affinity, OpenMP wrappers |
|
||||
| [../../ncnn/src/ruapu.h](../../ncnn/src/ruapu.h) | Bundled ISA detection header |
|
||||
| [../../ncnn/src/gpu.h](../../ncnn/src/gpu.h) / [gpu.cpp](../../ncnn/src/gpu.cpp) | Vulkan instance/device/GpuInfo, shader compile |
|
||||
| [../../ncnn/src/simplevk.h](../../ncnn/src/simplevk.h) / [simplevk.cpp](../../ncnn/src/simplevk.cpp) | In-house Vulkan loader (dlopen, MoltenVK, Android HAL) |
|
||||
| [../../ncnn/src/command.h](../../ncnn/src/command.h) / [command.cpp](../../ncnn/src/command.cpp) | `VkCompute`/`VkTransfer` recording and submission |
|
||||
| [../../ncnn/src/pipeline.h](../../ncnn/src/pipeline.h) / [pipelinecache.h](../../ncnn/src/pipelinecache.h) | Vulkan pipeline objects and persistent cache |
|
||||
| [../../ncnn/src/paramdict.h](../../ncnn/src/paramdict.h) / [modelbin.h](../../ncnn/src/modelbin.h) / [datareader.h](../../ncnn/src/datareader.h) | Param/weight parsing abstractions |
|
||||
| [../../ncnn/src/expression.h](../../ncnn/src/expression.h) | Runtime shape expression evaluator |
|
||||
| [../../ncnn/src/c_api.h](../../ncnn/src/c_api.h) | C API |
|
||||
| [../../ncnn/src/simpleomp.h](../../ncnn/src/simpleomp.h) / [simplestl.h](../../ncnn/src/simplestl.h) / [simpleocv.h](../../ncnn/src/simpleocv.h) / [simplemath.h](../../ncnn/src/simplemath.h) | Minimal runtime shims |
|
||||
| [../../ncnn/src/stb_image.h](../../ncnn/src/stb_image.h) | Image decoding for simpleocv |
|
||||
| [../../ncnn/src/convert_ycbcr.comp](../../ncnn/src/convert_ycbcr.comp) | Camera YCbCr → RGB Vulkan shader |
|
||||
| [../../ncnn/src/mat_pixel_resize.cpp](../../ncnn/src/mat_pixel_resize.cpp) / [mat_pixel_rotate.cpp](../../ncnn/src/mat_pixel_rotate.cpp) / [mat_pixel_affine.cpp](../../ncnn/src/mat_pixel_affine.cpp) / [mat_pixel_drawing.cpp](../../ncnn/src/mat_pixel_drawing.cpp) | Pixel preprocessing utilities |
|
||||
| [../../ncnn/src/CMakeLists.txt](../../ncnn/src/CMakeLists.txt) | Layer registration list, shader embedding, glslang link |
|
||||
| [../../ncnn/src/layer](../../ncnn/src/layer) | Naive layer implementations (110 types) |
|
||||
| [../../ncnn/src/layer/sdpa.h](../../ncnn/src/layer/sdpa.h) / [sdpa.cpp](../../ncnn/src/layer/sdpa.cpp) | SDPA with GQA and kv cache growth logic |
|
||||
| [../../ncnn/src/layer/multiheadattention.h](../../ncnn/src/layer/multiheadattention.h) / [multiheadattention.cpp](../../ncnn/src/layer/multiheadattention.cpp) | MHA with cache, int8 and block-quant paths |
|
||||
| [../../ncnn/src/layer/rotaryembed.h](../../ncnn/src/layer/rotaryembed.h) / [rotaryembed.cpp](../../ncnn/src/layer/rotaryembed.cpp) | RoPE from host cos/sin caches |
|
||||
| [../../ncnn/src/layer/embed.h](../../ncnn/src/layer/embed.h) | Token embedding lookup |
|
||||
| [../../ncnn/src/layer/gemm.h](../../ncnn/src/layer/gemm.h) | Gemm params incl. block quantization |
|
||||
| [../../ncnn/src/layer/spectrogram.h](../../ncnn/src/layer/spectrogram.h) / [spectrogram.cpp](../../ncnn/src/layer/spectrogram.cpp) / [inversespectrogram.h](../../ncnn/src/layer/inversespectrogram.h) / [inversespectrogram.cpp](../../ncnn/src/layer/inversespectrogram.cpp) | STFT / iSTFT layers |
|
||||
| [../../ncnn/src/layer/arm](../../ncnn/src/layer/arm) / [x86](../../ncnn/src/layer/x86) / [riscv](../../ncnn/src/layer/riscv) / [mips](../../ncnn/src/layer/mips) / [loongarch](../../ncnn/src/layer/loongarch) | Arch-optimized kernels |
|
||||
| [../../ncnn/src/layer/arm/sdpa_arm.cpp](../../ncnn/src/layer/arm/sdpa_arm.cpp) | ARM SDPA (fp16/bf16 storage flags) |
|
||||
| [../../ncnn/src/layer/arm/convolution_arm_i8mm.cpp](../../ncnn/src/layer/arm/convolution_arm_i8mm.cpp) / [convolution1d_arm_asimdhp.cpp](../../ncnn/src/layer/arm/convolution1d_arm_asimdhp.cpp) | ISA-variant kernel examples |
|
||||
| [../../ncnn/src/layer/x86/convolution_x86_avx512vnni.cpp](../../ncnn/src/layer/x86/convolution_x86_avx512vnni.cpp) | x86 int8 VNNI kernel |
|
||||
| [../../ncnn/src/layer/vulkan](../../ncnn/src/layer/vulkan) / [shader](../../ncnn/src/layer/vulkan/shader) | Vulkan layer impls and 300 GLSL shaders |
|
||||
| [../../ncnn/src/layer/vulkan/shader/sdpa_kvcache_append.comp](../../ncnn/src/layer/vulkan/shader/sdpa_kvcache_append.comp) | GPU kv-cache append shader |
|
||||
| [../../ncnn/src/layer/vulkan/convolution1d_vulkan.cpp](../../ncnn/src/layer/vulkan/convolution1d_vulkan.cpp) | 1-D conv on GPU |
|
||||
| [../../ncnn/cmake/ncnn_add_layer.cmake](../../ncnn/cmake/ncnn_add_layer.cmake) / [ncnn_add_shader.cmake](../../ncnn/cmake/ncnn_add_shader.cmake) / [run_test.cmake](../../ncnn/cmake/run_test.cmake) | Build macros |
|
||||
| [../../ncnn/toolchains](../../ncnn/toolchains) / [jetson.toolchain.cmake](../../ncnn/toolchains/jetson.toolchain.cmake) / [ios.toolchain.cmake](../../ncnn/toolchains/ios.toolchain.cmake) | Cross-compile toolchains |
|
||||
| [../../ncnn/tools/pnnx](../../ncnn/tools/pnnx) / [README.md](../../ncnn/tools/pnnx/README.md) / [src/save_ncnn.cpp](../../ncnn/tools/pnnx/src/save_ncnn.cpp) / [src/load_onnx.cpp](../../ncnn/tools/pnnx/src/load_onnx.cpp) / [src/load_tnn.cpp](../../ncnn/tools/pnnx/src/load_tnn.cpp) / [src/pass_ncnn](../../ncnn/tools/pnnx/src/pass_ncnn) | PyTorch/ONNX → ncnn exporter |
|
||||
| [../../ncnn/tools/pnnx/src/pass_ncnn/torch_stft.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torch_stft.cpp) / [torchaudio_F_inverse_spectrogram.cpp](../../ncnn/tools/pnnx/src/pass_ncnn/torchaudio_F_inverse_spectrogram.cpp) | Audio op lowering |
|
||||
| [../../ncnn/tools/onnx/onnx2ncnn.cpp](../../ncnn/tools/onnx/onnx2ncnn.cpp) | Legacy ONNX converter |
|
||||
| [../../ncnn/tools/caffe](../../ncnn/tools/caffe) / [darknet](../../ncnn/tools/darknet) / [mxnet](../../ncnn/tools/mxnet) / [keras](../../ncnn/tools/keras) / [tensorflow](../../ncnn/tools/tensorflow) / [mlir](../../ncnn/tools/mlir) | Other converters |
|
||||
| [../../ncnn/tools/ncnnoptimize.cpp](../../ncnn/tools/ncnnoptimize.cpp) / [ncnnmerge.cpp](../../ncnn/tools/ncnnmerge.cpp) / [ncnn2mem.cpp](../../ncnn/tools/ncnn2mem.cpp) | Offline graph optimization / packaging |
|
||||
| [../../ncnn/tools/quantize](../../ncnn/tools/quantize) / [ncnnllm2table.cpp](../../ncnn/tools/quantize/ncnnllm2table.cpp) / [ncnnllm2int.cpp](../../ncnn/tools/quantize/ncnnllm2int.cpp) | PTQ int8 and LLM block quantization tools |
|
||||
| [../../ncnn/benchmark/benchncnn.cpp](../../ncnn/benchmark/benchncnn.cpp) / [benchncnn_llm.cpp](../../ncnn/benchmark/benchncnn_llm.cpp) / [CMakeLists.txt](../../ncnn/benchmark/CMakeLists.txt) / [README.md](../../ncnn/benchmark/README.md) / [models/vision_transformer.param](../../ncnn/benchmark/models/vision_transformer.param) | Benchmarks and repo-reported results |
|
||||
| [../../ncnn/examples/CMakeLists.txt](../../ncnn/examples/CMakeLists.txt) | Example targets |
|
||||
| [../../ncnn/examples/piper.cpp](../../ncnn/examples/piper.cpp) | Piper/VITS TTS example |
|
||||
| [../../ncnn/examples/whisper.cpp](../../ncnn/examples/whisper.cpp) | Whisper ASR with kv cache and beam search |
|
||||
| [../../ncnn/examples/yolo11.cpp](../../ncnn/examples/yolo11.cpp) / [rvm.cpp](../../ncnn/examples/rvm.cpp) / [squeezencnn/README.md](../../ncnn/examples/squeezencnn/README.md) | Vision / recurrent-state / Android pointers |
|
||||
| [../../ncnn/tests/CMakeLists.txt](../../ncnn/tests/CMakeLists.txt) / [testutil.h](../../ncnn/tests/testutil.h) | Test harness |
|
||||
| [../../ncnn/tests/test_sdpa_kvcache.cpp](../../ncnn/tests/test_sdpa_kvcache.cpp) / [test_sdpa_kvcache_session.cpp](../../ncnn/tests/test_sdpa_kvcache_session.cpp) / [test_multiheadattention_kvcache.cpp](../../ncnn/tests/test_multiheadattention_kvcache.cpp) / [test_multiheadattention_kvcache_allocator.cpp](../../ncnn/tests/test_multiheadattention_kvcache_allocator.cpp) / [test_gemm_block_quant.cpp](../../ncnn/tests/test_gemm_block_quant.cpp) / [test_mat_batch.cpp](../../ncnn/tests/test_mat_batch.cpp) | Transformer/kv-cache/batch tests |
|
||||
| [../../ncnn/tests/perf](../../ncnn/tests/perf) / [perf_sdpa_kvcache.cpp](../../ncnn/tests/perf/perf_sdpa_kvcache.cpp) / [perf_sdpa_decode.cpp](../../ncnn/tests/perf/perf_sdpa_decode.cpp) / [perf_sdpa_prefill.cpp](../../ncnn/tests/perf/perf_sdpa_prefill.cpp) | Micro-benchmarks |
|
||||
| [../../ncnn/python/src/main.cpp](../../ncnn/python/src/main.cpp) / [python/README.md](../../ncnn/python/README.md) / [python/ncnn/model_zoo](../../ncnn/python/ncnn/model_zoo) / [python/examples](../../ncnn/python/examples) | Python bindings |
|
||||
| [../../ncnn/.github/workflows](../../ncnn/.github/workflows) / [compare-binary-size.yml](../../ncnn/.github/workflows/compare-binary-size.yml) | CI matrix, binary-size tracking |
|
||||
| [../../ncnn/docs/developer-guide/kvcache.md](../../ncnn/docs/developer-guide/kvcache.md) | KV cache design and usage |
|
||||
| [../../ncnn/docs/developer-guide/layer-support-behavior.md](../../ncnn/docs/developer-guide/layer-support-behavior.md) | Meaning of `support_*` flags |
|
||||
| [../../ncnn/docs/developer-guide/layer-feat-mask.md](../../ncnn/docs/developer-guide/layer-feat-mask.md) | Per-layer feature mask `31=` |
|
||||
| [../../ncnn/docs/developer-guide/expression.md](../../ncnn/docs/developer-guide/expression.md) | Dynamic shape expressions |
|
||||
| [../../ncnn/docs/developer-guide/element-packing.md](../../ncnn/docs/developer-guide/element-packing.md) | elempack layout |
|
||||
| [../../ncnn/docs/developer-guide/param-and-model-file-structure.md](../../ncnn/docs/developer-guide/param-and-model-file-structure.md) | File formats |
|
||||
| [../../ncnn/docs/developer-guide/operators.md](../../ncnn/docs/developer-guide/operators.md) / [operation-param-weight-table.md](../../ncnn/docs/developer-guide/operation-param-weight-table.md) | Operator/param reference |
|
||||
| [../../ncnn/docs/developer-guide/custom-allocator.md](../../ncnn/docs/developer-guide/custom-allocator.md) | Allocator roles and concurrency patterns |
|
||||
| [../../ncnn/docs/developer-guide/low-level-operation-api.md](../../ncnn/docs/developer-guide/low-level-operation-api.md) | Single-layer usage |
|
||||
| [../../ncnn/docs/developer-guide/how-to-implement-custom-layer-step-by-step.md](../../ncnn/docs/developer-guide/how-to-implement-custom-layer-step-by-step.md) / [add-custom-layer.zh.md](../../ncnn/docs/developer-guide/add-custom-layer.zh.md) | Custom layer tutorials |
|
||||
| [../../ncnn/docs/developer-guide/vulkan-driver-loader.md](../../ncnn/docs/developer-guide/vulkan-driver-loader.md) / [vulkan-pipeline-cache.md](../../ncnn/docs/developer-guide/vulkan-pipeline-cache.md) / [glsl-extension.md](../../ncnn/docs/developer-guide/glsl-extension.md) | Vulkan internals |
|
||||
| [../../ncnn/docs/developer-guide/arm-a53-a55-dual-issue.md](../../ncnn/docs/developer-guide/arm-a53-a55-dual-issue.md) | Little-core kernel tuning |
|
||||
| [../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md](../../ncnn/docs/how-to-use-and-FAQ/quantized-int8-inference.md) | PTQ and LLM block quant workflow |
|
||||
| [../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md](../../ncnn/docs/how-to-use-and-FAQ/build-minimal-library.md) | Binary size reduction |
|
||||
| [../../ncnn/docs/how-to-use-and-FAQ/openmp-best-practice.md](../../ncnn/docs/how-to-use-and-FAQ/openmp-best-practice.md) | Threading/affinity guidance |
|
||||
| [../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md](../../ncnn/docs/how-to-use-and-FAQ/vulkan-notes.md) / [FAQ-ncnn-vulkan.md](../../ncnn/docs/how-to-use-and-FAQ/FAQ-ncnn-vulkan.md) | Vulkan usage, platform matrix, fallback |
|
||||
| [../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md](../../ncnn/docs/how-to-use-and-FAQ/use-ncnn-with-android-hardware-buffer.md) | Zero-copy camera input |
|
||||
| [../../ncnn/docs/how-to-use-and-FAQ/ncnn-load-model.md](../../ncnn/docs/how-to-use-and-FAQ/ncnn-load-model.md) / [efficient-roi-resize-rotate.md](../../ncnn/docs/how-to-use-and-FAQ/efficient-roi-resize-rotate.md) | Loading and preprocessing docs |
|
||||
| [../../ncnn/docs/how-to-build/how-to-build.md](../../ncnn/docs/how-to-build/how-to-build.md) | Per-platform build instructions (Jetson, Android, iOS, WASM, HarmonyOS, ESP32…) |
|
||||
@@ -0,0 +1,50 @@
|
||||
# Repository inventory
|
||||
|
||||
Verified on 2026-09-08 (UTC+8 machine clock) with `git rev-parse`, `git status --porcelain`, `git remote -v`, and `git submodule status`. No checkout was modified. The analysis root is `/data/zhoutaichang/embedding_infer/`; `/home/zhoutaichang/embedding_infer` is the same directory (same inode), so both paths refer to one set of checkouts.
|
||||
|
||||
## Expected vs actual
|
||||
|
||||
| Engine | Local path | Remote | Expected branch / HEAD | Actual branch | Actual HEAD | Working tree | Last commit (author date) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| vLLM | [`../vllm`](../vllm) | `https://github.com/vllm-project/vllm.git` | main / 51da0ca66 | `main` | `51da0ca66c8065619c79e35dff97aa99aeaf5644` | clean | 2026-09-07 15:15 UTC, "[CI/Build] Unskip ColQwen3 multimodal pooling tests on Transformers v5 (#55588)" |
|
||||
| vLLM-Omni | [`../vllm-omni`](../vllm-omni) | `https://github.com/vllm-project/vllm-omni.git` | main / 7be014bc | `main` | `7be014bce6374f06c95b703763bdbac4c6198f31` | clean | 2026-09-07 14:50 UTC, "[Bugfix][NPU] Fix MiniMax-H3 INT8 quantization dispatch (#6876)" |
|
||||
| llama.cpp | [`../llama.cpp`](../llama.cpp) | `https://github.com/ggml-org/llama.cpp.git` | master / e71b80510 | `master` | `e71b80510c848c00175924ecf3c40333ccae8eb5` | clean | 2026-09-07 16:28 +0200, Revert "CUDA: size routed MoE MMQ N-tiles ... (#24546)" (#28551) |
|
||||
| ncnn | [`../ncnn`](../ncnn) | `https://github.com/Tencent/ncnn.git` | master / 3b7bdba7 | `master` | `3b7bdba7fc8aea8fd46779533eee027df77c639d` | clean | 2026-09-07 19:42 +0800, "Fix SIGFPE with missing or incomplete CPU topology (#6967)" |
|
||||
| MNN | [`../MNN`](../MNN) | `https://github.com/alibaba/MNN.git` | master / bef71b97 | `master` | `bef71b9756a2c77549eddbe33eb97290e3b16602` | clean | 2026-09-04 17:39 +0800, "[Vulkan:Perf] Optimize INT4 cooperative matrix path" |
|
||||
| ExecuTorch | [`../executorch`](../executorch) | `https://github.com/pytorch/executorch.git` | main / cfc6ecd516 | `main` | `cfc6ecd5166bec70c5fde2bad836fed26018fe52` | clean | 2026-09-07 08:47 -0700, "Qualcomm AI Engine Direct - Enable QNN Windows ARM64 build in CI (#22438)" |
|
||||
|
||||
**Result: all six checkouts match the expected branch and commit exactly, and all six working trees are clean.** No discrepancies in branch or HEAD.
|
||||
|
||||
## Discrepancies and caveats (not corrected, reported only)
|
||||
|
||||
1. **Submodules are not initialized** in the checkouts that declare them (`git submodule status` prints a leading `-`):
|
||||
- ncnn: `glslang`, `python/pybind11`. Consequence: a Vulkan-enabled build of ncnn from this tree would need `git submodule update --init` (not run). CPU-only analysis is unaffected.
|
||||
- executorch: 21 submodules including `backends/xnnpack/third-party/XNNPACK`, `backends/vulkan/third-party/*`, `backends/mlx/third-party/mlx`, `extension/llm/tokenizers`, `third-party/{flatbuffers,flatcc,googletest,gflags,json,ao,ios-cmake}`, `kernels/optimized/third-party/eigen`, `shim`, `backends/cadence/utils/FACTO`. Consequence: the tokenizer implementation used by ExecuTorch's LLM runner (`extension/llm/tokenizers`) is **not present on disk**, so statements about it in [engines/executorch.md](engines/executorch.md) are limited to how it is called, not how it is implemented.
|
||||
- vLLM, vLLM-Omni, llama.cpp, MNN declare no submodules at these commits (empty `git submodule status`).
|
||||
2. **Installed Python packages differ from the analyzed checkouts.** The machine's Python 3.12 has `vllm 0.28.0` installed as a wheel in `~/.local/lib/python3.12/site-packages`, and `vllm-omni 0.28.0rc2.dev22+gbe335a86f` installed editable from `/home/zhoutaichang/feature/vllm-omni-main` (commit `be335a86f`), **not** from `../vllm-omni` (`7be014bc`). Any runtime experiment therefore exercises the installed versions, not the analyzed commits, unless a separate environment is created (which would be a dependency installation and was not done without permission). See [experiments.md](experiments.md).
|
||||
3. **Host is x86_64, not ARM.** Native ARM CPU, Apple Silicon, and mobile GPU/NPU measurements cannot be taken on this machine; only x86 CPU and NVIDIA L20X data-center GPU runs are possible here. Edge numbers in this study are therefore either repo-reported (cited) or marked as unknown.
|
||||
|
||||
## Tree size snapshot
|
||||
|
||||
Rough sizes to calibrate reading effort (from `du -sh`, excluding `.git`); see per-engine reports for module maps.
|
||||
|
||||
| Engine | Primary language(s) | Notable top-level dirs |
|
||||
|---|---|---|
|
||||
| vLLM | Python, CUDA/C++ (`csrc/`) | `vllm/v1`, `vllm/model_executor`, `vllm/platforms`, `vllm/multimodal`, `vllm/entrypoints`, `csrc/`, `docs/` |
|
||||
| vLLM-Omni | Python | `vllm_omni/{engine,core,worker,diffusion,model_executor,entrypoints,distributed,config,host_weight_runtime}`, `examples/`, `docs/`, `recipes/` |
|
||||
| llama.cpp | C/C++ | `ggml/`, `src/`, `include/`, `tools/` (server, mtmd, tts), `examples/`, `gguf-py/`, `convert_hf_to_gguf.py` |
|
||||
| ncnn | C++ (+GLSL shaders) | `src/` (`layer/{arm,x86,vulkan,...}`), `tools/` (pnnx, quantize), `python/`, `examples/`, `docs/` |
|
||||
| MNN | C++ (+Metal/OpenCL/Vulkan/CUDA), Python | `source/{core,backend,geometry,shape}`, `express/`, `schema/default`, `tools/converter`, `transformers/{llm,diffusion}`, `apps/`, `pymnn/` |
|
||||
| ExecuTorch | Python (AOT), C++ (runtime) | `exir/`, `runtime/`, `backends/`, `kernels/`, `extension/`, `examples/`, `devtools/`, `docs/source` |
|
||||
|
||||
## Per-repo agent guidance honored during analysis
|
||||
|
||||
- `vllm/AGENTS.md` (via `CLAUDE.md`): no system `pip`; use `uv`. No packages were installed for this study.
|
||||
- `llama.cpp/AGENTS.md`: fully autonomous agents must not contribute; this study is read-only analysis and proposes no PRs.
|
||||
- `MNN/AGENTS.md`: `schema/private/` and `source/internal/` are restricted and were not read or referenced.
|
||||
- `executorch/AGENTS.md`: points to `.claude/` reference docs and `.wiki/`; used as documentation sources.
|
||||
|
||||
## Related material outside the six checkouts
|
||||
|
||||
- GPU scheduling guide: `/home/zhoutaichang/feature/GPU调度工具快速上手.pdf` (read; procedures summarized in [experiments.md](experiments.md)).
|
||||
- Other vLLM-Omni worktrees exist under `/home/zhoutaichang/feature/` (`vllm-omni-main`, `vllm-omni-abot`, `vllm-omni-pr6463`, ...). They were **not** analyzed; only `../vllm-omni` at `7be014bc` is in scope.
|
||||
@@ -0,0 +1,484 @@
|
||||
# vLLM (vllm-project/vllm) — Engine Analysis
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| Local path | `/data/zhoutaichang/embedding_infer/vllm` |
|
||||
| Remote | `https://github.com/vllm-project/vllm.git` (origin) |
|
||||
| Branch | `main` |
|
||||
| HEAD | `51da0ca66c8065619c79e35dff97aa99aeaf5644` — 2026-09-07 "[CI/Build] Unskip ColQwen3 multimodal pooling tests on Transformers v5 (#55588)" |
|
||||
| `git describe --tags` | `v0.28.1rc0-500-g51da0ca66` (newest tag in checkout: `v0.29.0rc4`) |
|
||||
| Working tree | clean (`git status --porcelain` empty) |
|
||||
| Submodules | none registered (`git submodule status` prints nothing); kernel deps are fetched by CMake `FetchContent` at build time |
|
||||
| Installed vs checkout | `pip show vllm` reports **0.28.0** installed in this machine's Python, while the checkout is ~500 commits past `v0.28.1rc0` (post-0.29.0rc4). Importing `vllm` from inside the checkout fails (`No module named 'vllm._C_stable_libtorch'`) because the C++/CUDA extensions are not built here. All statements below are about the **checkout**, not the installed wheel. |
|
||||
| Analysis date | 2026-09-08 |
|
||||
| Method | Read-only source/doc inspection (no builds, no execution). Evidence tags: [Verified] / [Inferred] / [Proposal] / [Unknown]. |
|
||||
|
||||
Reading guidance followed: `AGENTS.md` (`CLAUDE.md` is just `@AGENTS.md`) — no private/internal dirs are excluded for this repo.
|
||||
|
||||
## 0. Summary
|
||||
|
||||
- [Verified] vLLM is a Python-first LLM **serving stack** (OpenAI-compatible HTTP/gRPC/Python API) plus an **execution runtime** built on PyTorch: request scheduling + paged KV cache in an "EngineCore" process, model execution in worker processes, and hand-written CUDA/C++/Triton kernels bound through `torch.ops`. There is no standalone C/C++ runtime binary; PyTorch (`torch==2.13.0`) is a hard dependency ([pyproject.toml](../../vllm/pyproject.toml), [requirements/common.txt](../../vllm/requirements/common.txt)).
|
||||
- [Verified] Only the **V1 engine** exists in-tree: `AsyncLLM`/`LLMEngine` → `EngineCoreClient` (ZMQ + msgspec) → `EngineCore` (`Scheduler` + `KVCacheManager`) → `Executor` → `Worker` → `GPUModelRunner` ([vllm/v1/engine/async_llm.py](../../vllm/vllm/v1/engine/async_llm.py), [vllm/v1/engine/core.py](../../vllm/vllm/v1/engine/core.py)).
|
||||
- [Verified] In-tree platforms: **CUDA, ROCm, XPU (Intel GPU), CPU (x86/ARM/PowerPC/s390x/RISC-V, macOS CPU)** and a **Zen-CPU** variant. **TPU is a thin shim** that imports the out-of-tree `tpu_inference` package. Everything else (Ascend NPU, Apple Metal/MLX, Gaudi, ...) is an out-of-tree platform plugin discovered via the `vllm.platform_plugins` entry-point group ([vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py), [vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py)).
|
||||
- [Verified] Backend substitution has three documented extension points: platform plugin (`Platform` subclass returning a `worker_cls`, attention backend, device communicator), out-of-tree models (`ModelRegistry.register_model` via `vllm.general_plugins`), and out-of-tree layer/op implementations (`CustomOp.register_oot`, `AttentionBackendEnum.CUSTOM`, and the newer `vllm.ir` op/impl registry) ([docs/design/plugin_system.md](../../vllm/docs/design/plugin_system.md), [vllm/model_executor/custom_op.py](../../vllm/vllm/model_executor/custom_op.py), [vllm/ir/op.py](../../vllm/vllm/ir/op.py)).
|
||||
- [Verified] Request scheduling is a single token-budget loop with **no explicit prefill/decode phases** (chunked prefill, prefix caching, spec decode all fall out of `num_computed_tokens` vs `num_tokens_with_spec`), FCFS or priority queues, preemption-by-recompute, and an `AsyncScheduler` subclass that overlaps step N+1 scheduling with step N execution ([vllm/v1/core/sched/scheduler.py](../../vllm/vllm/v1/core/sched/scheduler.py#L535)).
|
||||
- [Verified] KV cache is block-paged with hash-based prefix caching (sha256 default), LRU eviction via a doubly linked free list, and per-layer-type managers (full, sliding-window, chunked-local, Mamba, circular buffer) coordinated by a hybrid coordinator ([vllm/v1/core/block_pool.py](../../vllm/vllm/v1/core/block_pool.py), [vllm/v1/core/single_type_kv_cache_manager.py](../../vllm/vllm/v1/core/single_type_kv_cache_manager.py)).
|
||||
- [Verified] Execution uses `torch.compile` (piecewise Dynamo/Inductor, "VLLM_COMPILE" mode) plus a CUDA-graph dispatcher with `NONE/PIECEWISE/FULL/FULL_DECODE_ONLY/FULL_AND_PIECEWISE` modes; all compilation/capture happens at startup, never on a live request ([vllm/config/compilation.py](../../vllm/vllm/config/compilation.py), [vllm/v1/cudagraph_dispatcher.py](../../vllm/vllm/v1/cudagraph_dispatcher.py), [docs/design/cuda_graphs.md](../../vllm/docs/design/cuda_graphs.md)).
|
||||
- [Verified] A second, experimental **Model Runner V2** (`vllm/v1/worker/gpu/`) is now the default when Triton is available and no unsupported feature is requested; it is "async-first" and Triton-heavy ([vllm/config/vllm.py](../../vllm/vllm/config/vllm.py#L657), [docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md)).
|
||||
- [Verified] **CPU backend** is real and ARM-aware: NEON/BF16/I8MM code paths in `csrc/cpu`, oneDNN + Arm Compute Library GEMMs, KleidiAI-backed dynamic-4-bit W4A8 kernels through `torch.ops.aten._dyn_quant_matmul_4bit`, thread binding, tcmalloc preload, aarch64 wheels and `arm64` Docker images; validated on AWS Graviton3 per docs ([cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake), [vllm/v1/attention/backends/cpu_attn.py](../../vllm/vllm/v1/attention/backends/cpu_attn.py#L481), [docs/getting_started/installation/cpu.arm.inc.md](../../vllm/docs/getting_started/installation/cpu.arm.inc.md)).
|
||||
- [Verified] **TTS: none.** There is no speech-synthesis model, vocoder, codec decoder, or audio-output endpoint. What exists is audio **input** (Whisper, Ultravox, Qwen2-Audio, Voxtral, Kimi-Audio, MOSS-Audio, omni "thinker" halves, etc.), `/v1/audio/transcriptions|translations`, and a WebSocket `/v1/realtime` streaming-ASR path with a `SupportsRealtime` model interface ([vllm/model_executor/models/registry.py](../../vllm/vllm/model_executor/models/registry.py), [vllm/entrypoints/speech_to_text/realtime/serving.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/serving.py)).
|
||||
- [Verified] **VLA: one model.** `OpenVLAForActionPrediction` (DINOv2+SigLIP backbone, projector, Llama LM, discretized action-token vocabulary) is registered; no proprioceptive input channel, no diffusion/flow action head, no closed-loop control API. `DiffusionGemmaForBlockDiffusion` shows the runner can host a non-autoregressive block-diffusion *text* decoder ([vllm/model_executor/models/openvla.py](../../vllm/vllm/model_executor/models/openvla.py), [vllm/model_executor/models/diffusion_gemma.py](../../vllm/vllm/model_executor/models/diffusion_gemma.py)).
|
||||
- [Verified] Jetson is recognized (`/etc/nv_tegra_release` check, non-NVML CUDA platform, `8.7`/`11.0` in supported CUDA archs, Thor `sm_110a` troubleshooting note) but there is no Jetson-specific build target, docs page, or CI job ([vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py#L76), [CMakeLists.txt](../../vllm/CMakeLists.txt#L112), [docs/usage/troubleshooting.md](../../vllm/docs/usage/troubleshooting.md)).
|
||||
- [Verified] Apple Silicon: CPU-only in-tree (FP16/FP32, BF16 if `FEAT_BF16`, build from source, CI smoke-tested on macOS runners); GPU via the out-of-tree `vllm-metal` (MLX) plugin. No Android/Vulkan/OpenCL/QNN/NNAPI code anywhere in tree ([docs/getting_started/installation/cpu.apple.inc.md](../../vllm/docs/getting_started/installation/cpu.apple.inc.md), [docs/getting_started/installation/gpu.apple.inc.md](../../vllm/docs/getting_started/installation/gpu.apple.inc.md)).
|
||||
- [Verified] GGUF and bitsandbytes quantization have been **moved out of tree** to plugins (`vllm-gguf-plugin`, `vllm-bnb-plugin`); in-tree formats are AWQ/GPTQ(+Marlin), FP8, compressed-tensors (INT8 W8A8, W4A8 int/fp8, ...), ModelOpt (FP4/MXFP8), MXFP4, TorchAO, Quark, INC, and an "online" method ([vllm/model_executor/layers/quantization/__init__.py](../../vllm/vllm/model_executor/layers/quantization/__init__.py), [docs/features/quantization/gguf.md](../../vllm/docs/features/quantization/gguf.md)).
|
||||
- [Verified] Deployment footprint is server-class: 3+ processes (API server, engine core, one worker per device), ZMQ/msgspec IPC, FastAPI/uvicorn, `transformers>=5.10`, `torch 2.13`; wheel size CI cap is 500 MB ([docs/design/arch_overview.md](../../vllm/docs/design/arch_overview.md#L67), [.buildkite/check-wheel-size.py](../../vllm/.buildkite/check-wheel-size.py)).
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph FE["Frontend process(es)"]
|
||||
CLI["vllm CLI<br/>vllm/entrypoints/cli"] --> API["OpenAI-compatible FastAPI server<br/>vllm/entrypoints/launchers/api_server<br/>routers: chat/completions/responses/embeddings/<br/>audio transcriptions/translations/realtime WS"]
|
||||
PY["LLM (offline)<br/>vllm/entrypoints/llm.py"] --> LLME["LLMEngine (sync)<br/>vllm/v1/engine/llm_engine.py"]
|
||||
RS["Rust frontend (experimental)<br/>rust/"] -.ZMQ.-> CORE
|
||||
API --> ALLM["AsyncLLM<br/>vllm/v1/engine/async_llm.py"]
|
||||
ALLM --> IP["InputProcessor + Renderer<br/>tokenize, MM preprocess, validate"]
|
||||
ALLM --> OP["OutputProcessor + IncrementalDetokenizer"]
|
||||
IP --> CC["EngineCoreClient (Inproc / Sync MP / Async MP)<br/>vllm/v1/engine/core_client.py"]
|
||||
LLME --> CC
|
||||
end
|
||||
CC -- "ZMQ + msgspec" --> CORE
|
||||
subgraph EC["EngineCore process (1 per DP rank)"]
|
||||
CORE["EngineCoreProc.run_busy_loop<br/>vllm/v1/engine/core.py"] --> SCH["Scheduler / AsyncScheduler<br/>vllm/v1/core/sched/"]
|
||||
SCH --> KVM["KVCacheManager → KVCacheCoordinator → BlockPool<br/>prefix cache, LRU free list"]
|
||||
SCH --> ECM["EncoderCacheManager"]
|
||||
SCH --> SOM["StructuredOutputManager<br/>xgrammar/guidance/outlines/lmfe"]
|
||||
CORE --> EXE["Executor (Uniproc / Multiproc / Ray)<br/>vllm/v1/executor/"]
|
||||
end
|
||||
EXE -- "shm MessageQueue / RPC" --> W
|
||||
subgraph WK["Worker process (1 per device)"]
|
||||
W["Worker (gpu_worker / cpu_worker / xpu_worker)"] --> MR["GPUModelRunner V1 (gpu_model_runner.py)<br/>or Model Runner V2 (vllm/v1/worker/gpu/)"]
|
||||
MR --> CG["CudagraphDispatcher + torch.compile backend<br/>vllm/v1/cudagraph_dispatcher.py, vllm/compilation/"]
|
||||
MR --> MODEL["nn.Module from ModelRegistry<br/>vllm/model_executor/models/"]
|
||||
MODEL --> LAYERS["Layers: Attention, Linear, MoE, RMSNorm, RoPE, quant<br/>vllm/model_executor/layers/"]
|
||||
LAYERS --> ATTN["AttentionBackend (FA/FlashInfer/Triton/FlexAttention/<br/>ROCm/CPU/...) via AttentionBackendEnum"]
|
||||
LAYERS --> OPS["CustomOp dispatch → forward_cuda/cpu/xpu/hip/oot<br/>vllm.ir ops → priority impls (vllm_c/native/...)"]
|
||||
OPS --> CSRC["torch.ops._C / _C_stable_libtorch / _moe_C<br/>csrc/, csrc/cpu/, csrc/libtorch_stable/"]
|
||||
MR --> SMP["Sampler (vllm/v1/sample/)"]
|
||||
W --> KVT["KV cache tensors; KVConnector / KV offload"]
|
||||
end
|
||||
PLAT["current_platform (vllm/platforms)<br/>builtin: cuda, rocm, xpu, cpu, zen_cpu, tpu(shim)<br/>entry-point group vllm.platform_plugins"] -. "worker_cls, attn backend,<br/>communicator, dtype/quant checks" .-> W
|
||||
PLAT -.-> ATTN
|
||||
```
|
||||
|
||||
Key modules:
|
||||
|
||||
| Module | Path | Responsibility | Key symbols |
|
||||
| --- | --- | --- | --- |
|
||||
| Async engine front-end | [vllm/v1/engine/async_llm.py](../../vllm/vllm/v1/engine/async_llm.py) | Request admission, streaming outputs, abort/pause, streaming-input sessions | `AsyncLLM`, `generate`, `_add_streaming_input_request`, `abort` |
|
||||
| Sync engine | [vllm/v1/engine/llm_engine.py](../../vllm/vllm/v1/engine/llm_engine.py) | Offline `step()` loop used by `LLM` | `LLMEngine.step`, `add_request` |
|
||||
| Engine core | [vllm/v1/engine/core.py](../../vllm/vllm/v1/engine/core.py) | Owns scheduler, KV init, busy loop, ZMQ I/O threads | `EngineCore.step`, `step_with_batch_queue`, `_initialize_kv_caches`, `EngineCoreProc.run_busy_loop` |
|
||||
| Core client | [vllm/v1/engine/core_client.py](../../vllm/vllm/v1/engine/core_client.py) | In-proc vs multiproc clients, request/response transport | `EngineCoreClient.make_client`, `InprocClient` |
|
||||
| Input/output processing | [vllm/v1/engine/input_processor.py](../../vllm/vllm/v1/engine/input_processor.py), [vllm/v1/engine/output_processor.py](../../vllm/vllm/v1/engine/output_processor.py), [vllm/v1/engine/detokenizer.py](../../vllm/vllm/v1/engine/detokenizer.py) | Validation, MM cache injection, incremental detokenization, stop strings | `InputProcessor.process_inputs`, `OutputProcessor.process_outputs`, `FastIncrementalDetokenizer` |
|
||||
| Scheduler | [vllm/v1/core/sched/scheduler.py](../../vllm/vllm/v1/core/sched/scheduler.py), [vllm/v1/core/sched/async_scheduler.py](../../vllm/vllm/v1/core/sched/async_scheduler.py), [vllm/v1/core/sched/request_queue.py](../../vllm/vllm/v1/core/sched/request_queue.py) | Token-budget scheduling, preemption, encoder budget, spec-decode slots | `Scheduler.schedule`, `update_from_output`, `_preempt_request`, `FCFSRequestQueue`, `PriorityRequestQueue` |
|
||||
| KV cache manager | [vllm/v1/core/kv_cache_manager.py](../../vllm/vllm/v1/core/kv_cache_manager.py), [vllm/v1/core/kv_cache_coordinator.py](../../vllm/vllm/v1/core/kv_cache_coordinator.py), [vllm/v1/core/block_pool.py](../../vllm/vllm/v1/core/block_pool.py), [vllm/v1/core/kv_cache_utils.py](../../vllm/vllm/v1/core/kv_cache_utils.py) | Block allocation, prefix-cache lookup, eviction, events | `KVCacheManager.allocate_slots`, `get_computed_blocks`, `BlockPool.cache_full_blocks`, `FreeKVCacheBlockQueue`, `hash_block_tokens` |
|
||||
| KV cache specs | [vllm/v1/kv_cache_interface.py](../../vllm/vllm/v1/kv_cache_interface.py), [vllm/v1/kv_cache_layout.py](../../vllm/vllm/v1/kv_cache_layout.py) | Per-layer spec types, physical layouts | `FullAttentionSpec`, `SlidingWindowSpec`, `MambaSpec`, `KVCacheConfig`, `KVCacheLayout` |
|
||||
| Executor | [vllm/v1/executor/abstract.py](../../vllm/vllm/v1/executor/abstract.py), [vllm/v1/executor/multiproc_executor.py](../../vllm/vllm/v1/executor/multiproc_executor.py), [vllm/v1/executor/uniproc_executor.py](../../vllm/vllm/v1/executor/uniproc_executor.py) | Spawns workers, broadcasts `execute_model`/`sample_tokens` | `Executor.get_class`, `MultiprocExecutor`, `WorkerProc` |
|
||||
| Workers | [vllm/v1/worker/worker_base.py](../../vllm/vllm/v1/worker/worker_base.py), [vllm/v1/worker/gpu_worker.py](../../vllm/vllm/v1/worker/gpu_worker.py), [vllm/v1/worker/cpu_worker.py](../../vllm/vllm/v1/worker/cpu_worker.py), [vllm/v1/worker/xpu_worker.py](../../vllm/vllm/v1/worker/xpu_worker.py) | Device init, memory profiling, warmup/graph capture, sleep mode | `Worker.init_device`, `determine_available_memory`, `compile_or_warm_up_model`, `CPUWorker` |
|
||||
| Model runner V1 | [vllm/v1/worker/gpu_model_runner.py](../../vllm/vllm/v1/worker/gpu_model_runner.py), [vllm/v1/worker/cpu_model_runner.py](../../vllm/vllm/v1/worker/cpu_model_runner.py) | Persistent batch, input prep, MM encoder execution, forward, sampling, KV alloc | `GPUModelRunner.execute_model`, `_dummy_run`, `capture_model`, `initialize_kv_cache`, `load_model`, `CPUModelRunner` |
|
||||
| Model runner V2 | [vllm/v1/worker/gpu/model_runner.py](../../vllm/vllm/v1/worker/gpu/model_runner.py), [vllm/v1/worker/gpu/README.md](../../vllm/vllm/v1/worker/gpu/README.md) | Async-first rewrite; fixed-row persistent state, Gumbel sampling | `GPUModelRunner.execute_model`, `prepare_inputs`, `sample_tokens` |
|
||||
| Attention | [vllm/v1/attention/backend.py](../../vllm/vllm/v1/attention/backend.py), [vllm/v1/attention/backends/registry.py](../../vllm/vllm/v1/attention/backends/registry.py), [vllm/v1/attention/selector.py](../../vllm/vllm/v1/attention/selector.py) | Backend ABC, enum registry with overrides, selection | `AttentionBackend`, `AttentionMetadataBuilder`, `AttentionBackendEnum`, `get_attn_backend` |
|
||||
| Platforms | [vllm/platforms/interface.py](../../vllm/vllm/platforms/interface.py), [vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py), [vllm/platforms/cpu.py](../../vllm/vllm/platforms/cpu.py), [vllm/platforms/cuda.py](../../vllm/vllm/platforms/cuda.py) | Device abstraction and config mutation | `Platform`, `PlatformEnum`, `CpuArchEnum`, `resolve_current_platform_cls_qualname`, `CpuPlatform.check_and_update_config` |
|
||||
| Custom ops / IR | [vllm/model_executor/custom_op.py](../../vllm/vllm/model_executor/custom_op.py), [vllm/ir/op.py](../../vllm/vllm/ir/op.py), [vllm/kernels/vllm_c.py](../../vllm/vllm/kernels/vllm_c.py) | Per-platform forward dispatch; provider-priority op impls | `CustomOp.dispatch_forward`, `register_oot`, `IrOp.register_impl`, `IrOp.dispatch` |
|
||||
| Compilation | [vllm/compilation/backends.py](../../vllm/vllm/compilation/backends.py), [vllm/compilation/cuda_graph.py](../../vllm/vllm/compilation/cuda_graph.py), [vllm/compilation/decorators.py](../../vllm/vllm/compilation/decorators.py), [vllm/config/compilation.py](../../vllm/vllm/config/compilation.py) | Dynamo piecewise compile, Inductor passes, CUDA graph wrappers | `CompilationMode`, `CUDAGraphMode`, `CompilationConfig` |
|
||||
| Quantization | [vllm/model_executor/layers/quantization/__init__.py](../../vllm/vllm/model_executor/layers/quantization/__init__.py), [vllm/model_executor/kernels/linear/mixed_precision/cpu.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py), [vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py) | Method registry, per-kernel `can_implement` selection | `QUANTIZATION_METHODS`, `CPUWNA16LinearKernel`, `Dynamic4bitLinearKernel` |
|
||||
| Model registry | [vllm/model_executor/models/registry.py](../../vllm/vllm/model_executor/models/registry.py), [vllm/model_executor/models/interfaces.py](../../vllm/vllm/model_executor/models/interfaces.py) | Arch → module map, OOT registration, capability protocols | `_MULTIMODAL_MODELS`, `ModelRegistry.register_model`, `SupportsMultiModal`, `SupportsTranscription`, `SupportsRealtime` |
|
||||
| Multimodal | [vllm/multimodal/registry.py](../../vllm/vllm/multimodal/registry.py), [vllm/multimodal/processing/processor.py](../../vllm/vllm/multimodal/processing/processor.py), [vllm/multimodal/cache.py](../../vllm/vllm/multimodal/cache.py), [vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py) | HF-processor wrapping, prompt updates, processor caches, audio resample/split | `MultiModalRegistry`, `BaseMultiModalProcessor`, `MultiModalProcessorCache*`, `AudioResampler`, `split_audio` |
|
||||
| Sampling | [vllm/v1/sample/sampler.py](../../vllm/vllm/v1/sample/sampler.py), [vllm/v1/sample/ops/topk_topp_sampler.py](../../vllm/vllm/v1/sample/ops/topk_topp_sampler.py), [vllm/v1/sample/rejection_sampler.py](../../vllm/vllm/v1/sample/rejection_sampler.py) | Temperature/penalties/logits processors, top-k/p, spec-decode rejection | `Sampler.forward`, `TopKTopPSampler` |
|
||||
| Entrypoints | [vllm/entrypoints/llm.py](../../vllm/vllm/entrypoints/llm.py), [vllm/entrypoints/launchers/api_server/entry.py](../../vllm/vllm/entrypoints/launchers/api_server/entry.py), [vllm/entrypoints/speech_to_text/base/serving.py](../../vllm/vllm/entrypoints/speech_to_text/base/serving.py), [vllm/entrypoints/grpc_server.py](../../vllm/vllm/entrypoints/grpc_server.py) | Python API, HTTP server, ASR endpoints, gRPC | `LLM.generate`, `run_server`, `SpeechToTextBaseServing`, `OpenAIServingRealtime` |
|
||||
| Tokenizers | [vllm/tokenizers/registry.py](../../vllm/vllm/tokenizers/registry.py), [vllm/tokenizers/fastokens.py](../../vllm/vllm/tokenizers/fastokens.py) | HF / Mistral / custom tokenizer modes, optional Rust `fastokens` patch | `get_tokenizer`, `_TokenizerRegistry` |
|
||||
| C++/CUDA kernels | [csrc/libtorch_stable/torch_bindings.cpp](../../vllm/csrc/libtorch_stable/torch_bindings.cpp), [csrc/cpu/torch_bindings.cpp](../../vllm/csrc/cpu/torch_bindings.cpp), [CMakeLists.txt](../../vllm/CMakeLists.txt), [cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake) | Op registration, build targets, ISA detection | `TORCH_LIBRARY_EXPAND`, `_C`, `_C_stable_libtorch`, `_moe_C_stable_libtorch` |
|
||||
|
||||
## 2. Device layer
|
||||
|
||||
**Platform abstraction.** [Verified] Everything device-specific is behind `vllm.platforms.current_platform`, an instance of a `Platform` subclass ([vllm/platforms/interface.py](../../vllm/vllm/platforms/interface.py#L134)). `PlatformEnum` has `CUDA, ROCM, TPU, XPU, CPU, OOT, UNSPECIFIED`; `CpuArchEnum` has `X86, ARM, POWERPC, S390X, RISCV, OTHER, UNKNOWN` ([interface.py#L67](../../vllm/vllm/platforms/interface.py#L67)). The interface (~1,340 lines) covers: `get_attn_backend_cls`, `get_vit_attn_backend`, `get_device_capability/name/uuid/total_memory`, `set_device`, `inference_mode`, `check_and_update_config` (mutates `VllmConfig`, sets `worker_cls`), `get_device_communicator_cls`, `get_punica_wrapper` (LoRA), `supports_fp8/mx`, `fp8_dtype`, `use_custom_allreduce`, `get_static_graph_wrapper_cls`, `support_static_graph_mode`, `import_kernels`, `import_ir_kernels`, `get_default_ir_op_priority`, `get_cpu_architecture`, `is_pin_memory_available`, `get_global_graph_pool`, `stateless_init_device_torch_dist_pg`.
|
||||
|
||||
**Discovery/selection.** [Verified] `resolve_current_platform_cls_qualname()` ([vllm/platforms/__init__.py#L222](../../vllm/vllm/platforms/__init__.py#L222)) probes builtin detectors `{tpu, cuda, rocm, xpu, cpu}` and every `vllm.platform_plugins` entry point; exactly one may activate (two OOT or two builtin → `RuntimeError`), OOT wins over builtin, none → `UnspecifiedPlatform`. `VLLM_TARGET_DEVICE=cpu` short-circuits to CPU. Detection details:
|
||||
- CUDA: `pynvml` device count > 0 and the wheel version string does not contain `cpu`; if NVML is missing, a **Jetson** check (`/etc/nv_tegra_release` or `/sys/class/tegra-firmware`) still enables CUDA ([__init__.py#L76](../../vllm/vllm/platforms/__init__.py#L76)). `CudaPlatform` is `NvmlCudaPlatform` or `NonNvmlCudaPlatform` chosen at import time ([cuda.py#L1040](../../vllm/vllm/platforms/cuda.py#L1040)).
|
||||
- ROCm: `amdsmi` handles > 0, or WSL fallback via `torch.version.hip`.
|
||||
- XPU: `torch.xpu.is_available()`.
|
||||
- CPU: `VLLM_TARGET_DEVICE == "cpu"`, or a `+cpu` wheel version, or `sys.platform == darwin` (macOS is always CPU). AMD CPUs with AVX-512 and `zentorch` installed get `ZenCpuPlatform` ([vllm/platforms/zen_cpu.py](../../vllm/vllm/platforms/zen_cpu.py)).
|
||||
- TPU: `libtpu` importable → `vllm.platforms.tpu.TpuPlatform`, which is just `from tpu_inference.platforms import TpuPlatform` ([vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py)); i.e. TPU execution code is **not** in this tree.
|
||||
|
||||
**In-tree backend inventory.** [Verified]
|
||||
|
||||
| Platform | File | `worker_cls` | dist backend | Notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| CUDA | [vllm/platforms/cuda.py](../../vllm/vllm/platforms/cuda.py) | `vllm.v1.worker.gpu_worker.Worker` ([#L325](../../vllm/vllm/platforms/cuda.py#L325)) | nccl | dtype: bf16 needs CC≥8.0, fp16 needs CC≥6.0; `is_integrated_gpu` via `torch.cuda.get_device_properties(...).is_integrated` ([#L714](../../vllm/vllm/platforms/cuda.py#L714)) |
|
||||
| ROCm | [vllm/platforms/rocm.py](../../vllm/vllm/platforms/rocm.py) | `vllm.v1.worker.gpu_worker.Worker` ([#L916](../../vllm/vllm/platforms/rocm.py#L916)) | nccl (rccl) | shares the GPU worker; AITER/Triton attention backends |
|
||||
| XPU | [vllm/platforms/xpu.py](../../vllm/vllm/platforms/xpu.py) | `vllm.v1.worker.xpu_worker.XPUWorker` ([#L363](../../vllm/vllm/platforms/xpu.py#L363)) | xccl | own worker + `xpu_model_runner.py` |
|
||||
| CPU | [vllm/platforms/cpu.py](../../vllm/vllm/platforms/cpu.py) | `vllm.v1.worker.cpu_worker.CPUWorker` ([#L282](../../vllm/vllm/platforms/cpu.py#L282)) | gloo | see below |
|
||||
| Zen CPU | [vllm/platforms/zen_cpu.py](../../vllm/vllm/platforms/zen_cpu.py) | inherits CPU | gloo | bf16/fp32 only, zentorch GEMM |
|
||||
| TPU | [vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py) | (in `tpu_inference`) | — | shim only; `tpu_input_batch.py` is a leftover |
|
||||
| OOT | via plugin | plugin-defined | plugin-defined | e.g. Ascend, Metal, Gaudi (not in tree) |
|
||||
|
||||
**Memory allocation / pools.** [Verified] GPU worker profiles memory once (`determine_available_memory` → `model_runner.profile_run()` under `memory_profiling`, honoring `gpu_memory_utilization` or an explicit `kv_cache_memory_bytes`) ([gpu_worker.py#L525](../../vllm/vllm/v1/worker/gpu_worker.py#L525)); `EngineCore._initialize_kv_caches` then computes `KVCacheConfig` (block count, groups, layout) and calls `initialize_from_config` + `compile_or_warm_up_model` ([core.py#L254](../../vllm/vllm/v1/engine/core.py#L254)). Sleep mode / weight offload use a custom CUDA-VMM allocator (`cumem_allocator` extension, [csrc/cumem_allocator.cpp](../../vllm/csrc/cumem_allocator.cpp), [vllm/device_allocator/cumem.py](../../vllm/vllm/device_allocator/cumem.py)); XPU has an analogous `xpumem.py`. CUDA graphs share one global graph memory pool (`Platform.get_global_graph_pool`). CPU: KV space is `VLLM_CPU_KVCACHE_SPACE` (GiB) or `kv_cache_memory_bytes`, plain `torch` CPU tensors; `tcmalloc` and PyTorch's `libgomp` are force-`LD_PRELOAD`ed on Linux x86/ARM/s390x ([cpu.py#L369](../../vllm/vllm/platforms/cpu.py#L369)).
|
||||
|
||||
**Host↔device transfers and streams.** [Verified] The runner keeps `CpuGpuBuffer` pairs (pinned host + device) and copies inputs per step; async output copy uses a dedicated stream (`_get_or_create_async_output_copy_stream`, [gpu_model_runner.py#L1181](../../vllm/vllm/v1/worker/gpu_model_runner.py#L1181)). MRV2 assumes "the core model execution loop is a CUDA stream with no CPU synchronization points" ([docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md)). CPU has no streams; `_set_torch_accelerator_to_noop` stubs stream/sync calls and `VLLM_DISABLE_SHARED_EXPERTS_STREAM=1` is forced ([cpu_model_runner.py#L244](../../vllm/vllm/v1/worker/cpu_model_runner.py#L244), [cpu.py#L356](../../vllm/vllm/platforms/cpu.py#L356)). Collectives go through `DeviceCommunicatorBase` subclasses: `cuda_communicator` (NCCL, custom all-reduce, quick-reduce, symm-mem), `cpu_communicator` (gloo + shared-memory `shm_allreduce` ops from `csrc/cpu/shm.cpp`), `xpu_communicator`, `ray_communicator` ([vllm/distributed/device_communicators/](../../vllm/vllm/distributed/device_communicators/base_device_communicator.py)).
|
||||
|
||||
**CPU platform specifics (ARM edge target).** [Verified]
|
||||
- `supported_dtypes`: x86/aarch64 → bf16/fp16/fp32; macOS ARM → fp16/fp32 (+bf16 only if `sysctl hw.optional.arm.FEAT_BF16` = 1); RISC-V → bf16/fp16/fp32; POWER → bf16/fp32/fp16 ([cpu.py#L107](../../vllm/vllm/platforms/cpu.py#L107)).
|
||||
- `check_and_update_config` forces: `async_scheduling=False`, `mp` executor (OpenMP needs it), `cudagraph_capture_sizes=[]`, compilation mode `DYNAMO_TRACE_ONCE` with Inductor (`eager` only under `VLLM_CPU_CI_ENV`), compilation disabled if LoRA, `+gelu/+gelu_tanh/+gelu_and_mul` custom ops enabled on ARM, `spawn` start method, `TORCHINDUCTOR_COMPILE_THREADS=1`, MLA on non-AMX disables chunked prefill/prefix caching ([cpu.py#L198](../../vllm/vllm/platforms/cpu.py#L198)).
|
||||
- Kernel import: x86 loads `_C`, `_C_AVX512` or `_C_AVX2` depending on runtime flags; every other arch loads `_C` ([cpu.py#L553](../../vllm/vllm/platforms/cpu.py#L553)).
|
||||
- Attention ISA: `amx` (x86 AMX + bf16 + block%32) → `neon` (ARM, block%32, "NEON FMLA and BFMMLA (bf16)") → `rvv` → `vxe` → `vsx` → `vec`/`vec16` ([cpu_attn.py#L481](../../vllm/vllm/v1/attention/backends/cpu_attn.py#L481)); implementations live in [csrc/cpu/cpu_attn_neon.hpp](../../vllm/csrc/cpu/cpu_attn_neon.hpp), [csrc/cpu/cpu_attn_neon_bfmmla.hpp](../../vllm/csrc/cpu/cpu_attn_neon_bfmmla.hpp), [csrc/cpu/cpu_types_arm.hpp](../../vllm/csrc/cpu/cpu_types_arm.hpp).
|
||||
- NUMA/thread binding: `CPUWorker.__init__` binds memory nodes and OpenMP threads (`VLLM_CPU_OMP_THREADS_BIND=auto|nobind|list`, `VLLM_CPU_NUM_OF_RESERVED_CPU`, `CPU_VISIBLE_MEMORY_NODES`) ([cpu_worker.py#L33](../../vllm/vllm/v1/worker/cpu_worker.py#L33), [docs/getting_started/installation/cpu.md](../../vllm/docs/getting_started/installation/cpu.md#L150)).
|
||||
- Pipeline/tensor parallel on CPU use gloo + shm ops; docs recommend TP = number of NUMA nodes ([cpu.md](../../vllm/docs/getting_started/installation/cpu.md)).
|
||||
|
||||
CPU-related environment variables ([vllm/envs.py#L877](../../vllm/vllm/envs.py#L877), [docs/getting_started/installation/cpu.md#L150](../../vllm/docs/getting_started/installation/cpu.md#L150)) [Verified]:
|
||||
|
||||
| Variable | Default | Effect |
|
||||
| --- | --- | --- |
|
||||
| `VLLM_TARGET_DEVICE` | `cuda` | Build/select target (`cpu` forces CPU platform even on GPU hosts) |
|
||||
| `VLLM_CPU_KVCACHE_SPACE` | unset (0) | KV cache size in GiB (legacy; `kv_cache_memory_bytes` preferred) |
|
||||
| `VLLM_CPU_OMP_THREADS_BIND` | `auto` | Core list / `auto` (per-NUMA) / `nobind` for OpenMP threads |
|
||||
| `VLLM_CPU_NUM_OF_RESERVED_CPU` | unset | Cores left free for the frontend per rank |
|
||||
| `CPU_VISIBLE_MEMORY_NODES` | unset | NUMA node mask, like `CUDA_VISIBLE_DEVICES` |
|
||||
| `VLLM_CPU_ATTN_SPLIT_KV` | `1` | Split-KV attention on CPU |
|
||||
| `VLLM_CPU_INT4_W4A8` | `1` | Enable dynamic 4-bit W4A8 path |
|
||||
| `VLLM_ZENTORCH_WEIGHT_PREPACK` | `1` | Zen: prepack linear weights for ZenDNN |
|
||||
| `VLLM_CPU_CI_ENV` | `0` | Internal: eager instead of Inductor |
|
||||
| `VLLM_ENABLE_V1_MULTIPROCESSING` | `1` | `0` = in-process engine core (CPU still forces the `mp` executor) |
|
||||
| `VLLM_USE_V2_MODEL_RUNNER` | unset | Force V1/V2 model runner (auto: V2 if Triton present and no unsupported features) |
|
||||
| `VLLM_PLUGINS` | unset | Allow-list of plugin names to load |
|
||||
|
||||
**Portability matrix (what is actually in tree).** [Verified] Linux x86_64 (CUDA/ROCm/XPU/CPU), Linux aarch64 (CUDA e.g. GH200 CI job, CPU), Linux ppc64le/s390x/riscv64 (CPU), macOS arm64 (CPU only), WSL2 (CUDA/ROCm detection). [Verified] No Windows native support code; no Android/iOS bindings; no Vulkan/OpenCL/Metal/QNN/NNAPI kernels. CUDA arch list depends on nvcc: `7.5;8.0;8.6;8.7;8.9;9.0;10.0;(10.7);11.0;12.0` for CUDA ≥13.0 — `8.7` = Jetson Orin, `11.0` = Thor ([CMakeLists.txt#L112](../../vllm/CMakeLists.txt#L112)). ROCm arch list includes `gfx1150/1151/1152/1153` (RDNA3.5 APUs) ([CMakeLists.txt#L52](../../vllm/CMakeLists.txt#L52)).
|
||||
|
||||
## 3. Kernel layer
|
||||
|
||||
**Operator set and registration.** [Verified] Three C++ extension families:
|
||||
- CUDA `_C_stable_libtorch` (132 `def(` ops in [csrc/libtorch_stable/torch_bindings.cpp](../../vllm/csrc/libtorch_stable/torch_bindings.cpp), registered with `STABLE_TORCH_LIBRARY` so the wheel is ABI-stable across torch minors) plus `_moe_C_stable_libtorch`; legacy `_C` remains ROCm-only ("CUDA ops migrated to `_C_stable_libtorch`", [CMakeLists.txt#L367](../../vllm/CMakeLists.txt#L367)). Contents: activations, layernorm(+quant fusion), RoPE, fused QK-norm+RoPE, cache kernels, `merge_attn_states`, MLA helpers, sampler/top-k, MoE align/topk/permute, Marlin/Machete/CUTLASS/AWQ/GPTQ/FP4/FP8/INT8 GEMMs, custom all-reduce.
|
||||
- CPU `_C` (73 `def(` ops in [csrc/cpu/torch_bindings.cpp](../../vllm/csrc/cpu/torch_bindings.cpp#L357)): `silu_and_mul`, `gelu_*`, `rms_norm`, `fused_add_rms_norm`, `rotary_embedding`, `onednn_mm`/`onednn_scaled_mm`, `static/dynamic_scaled_int8_quant`, `fp8_scaled_mm_cpu`, `weight_packed_linear`, `fused_experts_cpu`, `cpu_fused_moe`, `causal_conv1d_*_cpu`, `decode/extend_attention_cpu`, `shm_allreduce/gather`, `dynamic_4bit_int_moe`, plus ARM-only (`ARM_I8MM_SUPPORT && ARM_BF16_SUPPORT && !__APPLE__`) `prepack_moe_weight_int8`/`cpu_fused_moe_int8` ([torch_bindings.cpp#L690](../../vllm/csrc/cpu/torch_bindings.cpp#L690)). Micro-GEMMs per ISA in [csrc/cpu/micro_gemm/](../../vllm/csrc/cpu/micro_gemm/cpu_micro_gemm_neon.hpp) (`amx`, `neon`, `int8_neon`, `rvv`, `vec`, `vsx`).
|
||||
- Externally fetched kernel projects (CUDA only): vllm-flash-attn, FlashMLA, FlashKDA, MSA/fmha_sm100, tml-fa4, qutlass, DeepGEMM, Triton kernels ([cmake/external_projects/](../../vllm/cmake/external_projects/vllm_flash_attn.cmake)). Pip-side kernel deps: `flashinfer-python`, `tilelang`, `nvidia-cutlass-dsl`, `quack-kernels`, `humming-kernels`, `tokenspeed-mla` ([requirements/cuda.txt](../../vllm/requirements/cuda.txt)).
|
||||
|
||||
Python-side Triton kernels are spread across `vllm/model_executor/layers/**` (fused MoE, Mamba, RoPE, etc.), `vllm/v1/sample/ops/topk_topp_triton.py`, `vllm/v1/worker/gpu/*`, and [vllm/kernels/triton/](../../vllm/vllm/kernels/triton/qkv_padded_fp8_quant.py); Helion and "oink" kernels exist under [vllm/kernels/](../../vllm/vllm/kernels/oink_ops.py).
|
||||
|
||||
**Dispatch mechanism.** [Verified] Two coexisting mechanisms:
|
||||
1. `CustomOp` (nn.Module subclasses such as `RMSNorm`, `SiluAndMul`, `RotaryEmbedding`): `dispatch_forward()` picks `forward_hip/cpu/tpu/xpu/oot/cuda` by `current_platform`, or `forward_native` (pure torch, optionally `torch.compile`d) when the op is disabled. Enablement is per-op via `CompilationConfig.custom_ops` (`all`/`none` + `+op`/`-op`); with Inductor the default is `none`, so Inductor generates fused Triton for them ([custom_op.py#L174](../../vllm/vllm/model_executor/custom_op.py#L174), [docs/design/custom_op.md](../../vllm/docs/design/custom_op.md)). OOT plugins replace an in-tree class by name with `@CustomOp.register_oot(name=...)` / `@PluggableLayer.register_oot` — instantiation of the in-tree class returns the OOT subclass ([custom_op.py#L103](../../vllm/vllm/model_executor/custom_op.py#L103)).
|
||||
2. `vllm.ir` (newer): `@register_op` declares a functional op (`rms_norm`, `fused_add_rms_norm` today, [vllm/ir/ops/layernorm.py](../../vllm/vllm/ir/ops/layernorm.py)); providers register implementations with `op.register_impl("vllm_c", supported=..., supports_args=...)`; `IrOp.dispatch` walks a per-op **priority list** (platform default e.g. `["vllm_c","native"]` when not compiling, `["native"]` under Inductor, `["oink", ...]` if `VLLM_USE_OINK_OPS`) and returns the first impl whose `supports_args` passes ([ir/op.py#L244](../../vllm/vllm/ir/op.py#L244), [cuda.py#L723](../../vllm/vllm/platforms/cuda.py#L723), [docs/design/vllm_ir.md](../../vllm/docs/design/vllm_ir.md)).
|
||||
3. Attention: `AttentionBackendEnum` maps names to class paths; `register_backend` can override any member (including `CUSTOM`) at runtime; the platform's `get_attn_backend_cls` ranks candidates by device capability, head size, dtype, block size, sliding window, etc. via `AttentionBackend.validate_configuration` ([registry.py](../../vllm/vllm/v1/attention/backends/registry.py), [cuda.py#L434](../../vllm/vllm/platforms/cuda.py#L434), [backend.py#L270](../../vllm/vllm/v1/attention/backend.py#L270)). In-tree backends: FLASH_ATTN(+DIFFKV), TRITON_ATTN(+DIFFKV), FLASHINFER, FLEX_ATTENTION, ROCM_ATTN/AITER variants, ~12 MLA variants, HPC_ATTN, B12X, TURBOQUANT, NO_ATTENTION, CPU_ATTN, CPU_MLA, AMX_MLA; Mamba-family backends MAMBA1/MAMBA2/SHORT_CONV/LINEAR/GDN_ATTN.
|
||||
Default attention backend candidates per platform [Verified]:
|
||||
|
||||
| Platform | Candidate order (non-MLA, decoder) | Source |
|
||||
| --- | --- | --- |
|
||||
| CUDA CC 10.x (Blackwell) | FLASHINFER → FLASH_ATTN → TRITON_ATTN → FLEX_ATTENTION | [cuda.py#L157](../../vllm/vllm/platforms/cuda.py#L157) |
|
||||
| CUDA other (≥8.0) | FLASH_ATTN → FLASHINFER → TRITON_ATTN → FLEX_ATTENTION | [cuda.py#L165](../../vllm/vllm/platforms/cuda.py#L165) |
|
||||
| CUDA MLA (CC 9.0/10.x/12.x) | FLASHINFER_MLA / FLASH_ATTN_MLA / CUTLASS_MLA / FLASHMLA variants by CC | [cuda.py#L85](../../vllm/vllm/platforms/cuda.py#L85) |
|
||||
| ROCm | ROCM_AITER_UNIFIED_ATTN / ROCM_AITER_FA / ROCM_ATTN → TRITON_ATTN → TURBOQUANT; MLA: ROCM_AITER_MLA / TRITON_MLA | [rocm.py#L463](../../vllm/vllm/platforms/rocm.py#L463) |
|
||||
| XPU | FLASH_ATTN (SYCL) / TRITON_ATTN; MLA: TRITON_MLA / XPU_MLA_SPARSE | [xpu.py#L145](../../vllm/vllm/platforms/xpu.py#L145) |
|
||||
| CPU | CPU_ATTN (ISA-dispatched); MLA: AMX_MLA (x86 AMX) or CPU_MLA | [cpu.py#L138](../../vllm/vllm/platforms/cpu.py#L138) |
|
||||
| ViT encoders | FLASH_ATTN / TRITON_ATTN / FLASHINFER / TORCH_SDPA by platform | [cuda.py#L536](../../vllm/vllm/platforms/cuda.py#L536) |
|
||||
|
||||
4. Linear/quant kernels: `choose_mp_linear_kernel` iterates `MPLinearKernel` subclasses and picks the first whose `can_implement(config)` succeeds ([compressed_tensors_w4a8_int.py#L104](../../vllm/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a8_int.py#L104)); CPU unquantized GEMM is chosen at load time by `dispatch_cpu_unquantized_gemm`: zentorch → sgl-kernel `weight_packed_linear` (AMX only, weight ≤1 MiB) → oneDNN `onednn_mm` (non-POWER) → `torch.nn.functional.linear` ([layers/utils.py#L434](../../vllm/vllm/model_executor/layers/utils.py#L434)).
|
||||
|
||||
**Quantization formats and where dequant happens.** [Verified] `QUANTIZATION_METHODS`: awq, auto_awq, fp8, fbgemm_fp8 (deprecated), fp_quant (deprecated), modelopt, modelopt_fp4, modelopt_mxfp8, modelopt_mixed, auto_gptq, gptq, gptq_marlin, awq_marlin, humming, compressed-tensors, experts_int8, quark, moe_wna16, torchao, inc, mxfp4, gpt_oss_mxfp4, deepseek_v4_fp8, online (+ online sub-modes fp8_per_tensor/per_block/per_channel, int8_per_channel_weight_only, nvfp4_per_token, mxfp8) ([quantization/__init__.py#L16](../../vllm/vllm/model_executor/layers/quantization/__init__.py#L16)). Third-party configs can be added with `register_quantization_config`. Weight-only formats are dequantized **inside fused GEMM kernels** (Marlin/Machete/CUTLASS W4A8, oneDNN int8 on CPU, KleidiAI dynamic-4-bit on ARM via `torch.ops.aten._dyn_quant_matmul_4bit`, [dynamic_4bit.py#L14](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py#L14)); activation quant (fp8/int8) is fused into RMSNorm/SiLU kernels where available (`fused_layernorm_dynamic_per_token_quant`, `fused_silu_mul_block_quant`). KV cache dtypes: `auto, float16, bfloat16, fp8, fp8_e4m3, fp8_e5m2, fp8_inc, fp8_ds_mla, nvfp4_ds_mla, turboquant_*, int4/int8/fp8_per_token_head, nvfp4, nvfp4_4over6` ([vllm/config/cache.py#L39](../../vllm/vllm/config/cache.py#L39)); on CPU FP8 KV requires x86 AVX-512/AMX ([cpu_attn.py#L500](../../vllm/vllm/v1/attention/backends/cpu_attn.py#L500)). [Verified] Repo-reported hardware/format matrix for Arm CPU: only llm-compressor INT8 W8A8 and W4A8 marked supported; AWQ/GPTQ/Marlin/GGUF marked unsupported ([docs/features/quantization/README.md#L50](../../vllm/docs/features/quantization/README.md#L50)). GGUF and bitsandbytes now require `vllm-gguf-plugin` / `vllm-bnb-plugin` ([gguf.md](../../vllm/docs/features/quantization/gguf.md), [bnb.md](../../vllm/docs/features/quantization/bnb.md)).
|
||||
|
||||
**Compilation / JIT / graph capture.** [Verified] `CompilationMode`: `NONE`, `STOCK_TORCH_COMPILE`, `DYNAMO_TRACE_ONCE`, `VLLM_COMPILE` (piecewise: model is split at attention/`splitting_ops`, each piece compiled by Inductor with guard dropping, artifacts cached under `~/.cache/vllm/torch_compile_cache`) ([config/compilation.py#L37](../../vllm/vllm/config/compilation.py#L37), [docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md)). `CUDAGraphMode`: `NONE, PIECEWISE, FULL, FULL_DECODE_ONLY, FULL_AND_PIECEWISE`; `CudagraphDispatcher.dispatch` maps a `BatchDescriptor` (padded size, uniform-decode flag, LoRA case) to a captured graph or falls back to eager ([cudagraph_dispatcher.py#L235](../../vllm/vllm/v1/cudagraph_dispatcher.py#L235)). Optimization levels `-O0..-O3` set these defaults ([docs/design/optimization_levels.md](../../vllm/docs/design/optimization_levels.md)). Warmup: `Worker.compile_or_warm_up_model` runs `_dummy_run` for compile sizes, `kernel_warmup`, `capture_model()` (largest→smallest), then a sampler dummy run; all before the first request ([gpu_worker.py#L762](../../vllm/vllm/v1/worker/gpu_worker.py#L762)). ViT encoders have their own CUDA-graph path ([vllm/v1/worker/encoder_cudagraph.py](../../vllm/vllm/v1/worker/encoder_cudagraph.py), [docs/design/cuda_graphs_multimodal.md](../../vllm/docs/design/cuda_graphs_multimodal.md)). CPU uses `DYNAMO_TRACE_ONCE` + Inductor C++ codegen (no graphs) ([cpu.py#L292](../../vllm/vllm/platforms/cpu.py#L292)).
|
||||
|
||||
**Fallback when an op is unsupported.** [Verified] `CustomOp` → `forward_native` (pure PyTorch). `vllm.ir` → the last provider in the priority list must accept all args (normally `native`). Attention → candidate list ordered by platform; if none validates, an error lists why each was rejected ([cuda.py#L434](../../vllm/vllm/platforms/cuda.py#L434)). CPU without Triton-CPU → `_postprocess_triton` monkey-patches Triton kernels (slot mapping, EAGLE/DFlash input prep, rejection sampler, Mamba memcpy) with C++/torch equivalents ([cpu_model_runner.py#L65](../../vllm/vllm/v1/worker/cpu_model_runner.py#L65)); with Triton-CPU present the patches are skipped. Missing compiled extension modules only log a warning at import (`import_kernels`) but any op call then fails.
|
||||
|
||||
**How to add a custom op / backend (extension points).** [Verified]
|
||||
- New platform: package with `entry_points={"vllm.platform_plugins": [...]}` returning a `Platform` subclass path; implement `check_and_update_config` (set `worker_cls`), `get_attn_backend_cls`, `get_device_communicator_cls`; a `WorkerBase` subclass; an `AttentionBackend`; optional OOT `CustomOp`s. Reference skeleton: [tests/plugins/vllm_add_dummy_platform/](../../vllm/tests/plugins/vllm_add_dummy_platform/setup.py) with `dummy_platform.py`, `dummy_attention_backend.py`, `dummy_custom_ops.py` ([docs/design/plugin_system.md](../../vllm/docs/design/plugin_system.md)).
|
||||
- New op impl: `@ir.ops.<op>.register_impl("provider", supported=..., supports_args=...)` or a `CustomOp.forward_<platform>`; C++ ops via `torch.ops` (see `vllm/_custom_ops.py`).
|
||||
- New attention backend: `register_backend(AttentionBackendEnum.CUSTOM, "pkg.mod.Class")`.
|
||||
- New model: `ModelRegistry.register_model("Arch", "pkg.mod:Class")` from a `vllm.general_plugins` entry point ([docs/contributing/model/registration.md](../../vllm/docs/contributing/model/registration.md)).
|
||||
- New quant method: `register_quantization_config`.
|
||||
- Plugin loading is gated by `VLLM_PLUGINS` and happens in every process ([vllm/plugins/__init__.py](../../vllm/vllm/plugins/__init__.py)). [Verified] Triton ops cannot be overridden via the OOT mechanism ("Custom way doesn't work for triton ops now", plugin doc).
|
||||
|
||||
## 4. Model runner
|
||||
|
||||
**Model loading / formats.** [Verified] Models are HF-format directories loaded by `ModelRegistry` → architecture class → `DefaultModelLoader` (safetensors/pt/npcache), or alternative loaders: `dummy`, `fastsafetensors`, `instanttensor`, `ipc_cache`, `mistral`, `modelexpress`, `runai_streamer(_sharded)`, `sharded_state`, `tensorizer` ([vllm/model_executor/model_loader/__init__.py#L33](../../vllm/vllm/model_executor/model_loader/__init__.py#L33), [vllm/config/load.py](../../vllm/vllm/config/load.py)). There is **no export/conversion pipeline**: weights are mapped in-place into vLLM's parallel layers (`AutoWeightsLoader`, `WeightsMapper`); quantized checkpoints are consumed in their vendor formats. A `transformers` fallback backend (`_TRANSFORMERS_BACKEND_MODELS`) runs unsupported HF architectures ([registry.py#L698](../../vllm/vllm/model_executor/models/registry.py#L698)). Weight reload / RL weight-update RPCs exist (`reload_weights`, `update_weights`, [gpu_worker.py#L1391](../../vllm/vllm/v1/worker/gpu_worker.py#L1391)).
|
||||
|
||||
**Execution lifecycle.** [Verified]
|
||||
1. `EngineCore.__init__` → `Executor` spawns workers → `Worker.init_device` (device select, dist init, memory snapshot, `init_workspace_manager`, choose V1 vs V2 runner) ([gpu_worker.py#L357](../../vllm/vllm/v1/worker/gpu_worker.py#L357)) → `load_model`.
|
||||
2. `_initialize_kv_caches`: `register_all_kvcache_specs`, gather per-layer `KVCacheSpec`s, resolve `KVCacheLayout`, `determine_available_memory` (profile run), `get_kv_cache_configs`, possibly auto-fit `max_model_len`, `initialize_from_config` (allocate KV tensors, bind to attention layers), `compile_or_warm_up_model` (compile + CUDA graphs + sampler warmup) ([core.py#L254](../../vllm/vllm/v1/engine/core.py#L254)).
|
||||
3. Steady state: `EngineCore.step` → `scheduler.schedule()` → `executor.execute_model(scheduler_output, non_block=True)` → grammar bitmask → `sample_tokens` → `scheduler.update_from_output` ([core.py#L608](../../vllm/vllm/v1/engine/core.py#L608)). With pipeline parallel or async scheduling `step_with_batch_queue` keeps up to N in-flight batches.
|
||||
4. Worker step (V1 runner): `_update_states` (apply scheduler diffs to the persistent batch) → `_execute_mm_encoder` (encoder-only inputs, cached by `mm_hash`) → `_prepare_inputs` (token ids, positions, slot mappings, attention metadata per KV group, spec-decode metadata) → `_determine_batch_execution_and_padding` (DP/micro-batch/CUDA-graph descriptor) → `set_forward_context` → `_model_forward` → `compute_logits` → `_sample` → `_bookkeeping_sync` (async copy to host) ([gpu_model_runner.py#L4274](../../vllm/vllm/v1/worker/gpu_model_runner.py#L4274)).
|
||||
|
||||
Lifecycle summary [Verified]:
|
||||
|
||||
| Phase | Where | What happens |
|
||||
| --- | --- | --- |
|
||||
| Config resolution | [vllm/config/vllm.py](../../vllm/vllm/config/vllm.py) + `Platform.check_and_update_config` | Optimization level → compile/cudagraph modes; async scheduling; worker class; CPU overrides |
|
||||
| Process spawn | `EngineCoreClient.make_client` → `EngineCoreProc.run_engine_core`; `Executor._init_executor` | Engine core and worker processes, ZMQ handshake (`startup_handshake`), shm queues |
|
||||
| Device init | `Worker.init_device` / `CPUWorker.init_device` | Device select, `init_worker_distributed_environment`, memory snapshot, workspace manager, runner construction |
|
||||
| Load | `Worker.load_model` → `GPUModelRunner.load_model` | Loader by `load_format`, weight mapping, `process_weights_after_loading` (quant repack, CPU GEMM dispatch) |
|
||||
| Profile | `determine_available_memory` → `profile_run` | Dummy max batch (incl. dummy MM inputs) to size KV cache |
|
||||
| KV alloc | `EngineCore._initialize_kv_caches` → `initialize_from_config` → `initialize_kv_cache` | Group specs, block count, layout, tensor allocation, bind to attention layers |
|
||||
| Warmup | `compile_or_warm_up_model` | `_dummy_run` per compile size, `kernel_warmup`, `capture_model` (CUDA graphs), sampler dummy run |
|
||||
| Serve | `run_busy_loop` → `step` / `step_with_batch_queue` | Schedule → execute → sample → update; outputs pushed to the client socket |
|
||||
| Idle / sleep | `sleep`, `wake_up`, `pause_generation` | Free weights/KV via cumem allocator, keep process alive |
|
||||
| Shutdown | `EngineCore.shutdown`, `Executor.shutdown` | Abort in-flight requests, join workers |
|
||||
|
||||
**Dynamic shapes.** [Verified] Batches are token-flattened (`num_tokens` varies) with padding to the nearest CUDA-graph capture size or `compile_sizes`/`compile_ranges`; Dynamo marks the token dimension dynamic and vLLM drops guards ([docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md)). `max_num_seqs`/`max_num_batched_tokens` bound persistent buffers. MRV2 pre-allocates `max_num_reqs` rows (1024 default) and assigns a fixed row per request ([docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md)).
|
||||
|
||||
**State & KV cache.** [Verified] Allocation: KV tensors are allocated per `KVCacheGroupSpec` at `initialize_kv_cache` ([gpu_model_runner.py#L7491](../../vllm/vllm/v1/worker/gpu_model_runner.py#L7491)); layout chosen from six `KVCacheLayout` permutations (`LBHNC`, `LBNHC`, `LHBNC`, `BLHNC`, `BLNHC`, `BHLNC`) per backend support ([kv_cache_layout.py](../../vllm/vllm/v1/kv_cache_layout.py)); CPU backend packs its own layout (`CpuPlatform.pack_kv_cache`). Paging: fixed `block_size` (backend-preferred; hybrid models align attention and Mamba page sizes, [interface.py#L609](../../vllm/vllm/platforms/interface.py#L609)); `BlockPool` holds all blocks plus `BlockHashToBlockMap` for prefix hits; `FreeKVCacheBlockQueue` is an O(1) doubly linked LRU list, ordered so the tail of a freed chain is evicted first ([kv_cache_utils.py#L230](../../vllm/vllm/v1/core/kv_cache_utils.py#L230)). Prefix caching: block hash = `hash(parent_hash, block_tokens, extra: lora_id, mm_hashes, cache_salt)`; default `sha256`, options `sha256_cbor`, `xxhash`, `xxhash_cbor`; only full blocks are cached; partial-tail caching exists (`cache_partial_block`) ([docs/design/prefix_caching.md](../../vllm/docs/design/prefix_caching.md), [block_pool.py#L225](../../vllm/vllm/v1/core/block_pool.py#L225)). Reuse across calls: `get_computed_blocks` at admission returns the longest cached prefix per group; sliding-window/chunked-local/Mamba managers implement their own `find_longest_cache_hit` and `remove_skipped_blocks` ([single_type_kv_cache_manager.py](../../vllm/vllm/v1/core/single_type_kv_cache_manager.py)). Eviction to host: `vllm/v1/kv_offload` (CPU offload with policies, Triton swap kernels) and KV connectors (NIXL, LMCache, Mooncake, MoRIIO, FlexKV, offloading/simple CPU offload) for disaggregated prefill ([vllm/v1/kv_offload/cpu/manager.py](../../vllm/vllm/v1/kv_offload/cpu/manager.py), [vllm/distributed/kv_transfer/kv_connector/v1/base.py](../../vllm/vllm/distributed/kv_transfer/kv_connector/v1/base.py#L171)). Encoder outputs have a separate `EncoderCacheManager` budget ([vllm/v1/core/encoder_cache_manager.py](../../vllm/vllm/v1/core/encoder_cache_manager.py)). Session/streaming requests keep KV alive between turns (`WAITING_FOR_STREAMING_REQ`, `_update_request_as_session`, [scheduler.py#L1545](../../vllm/vllm/v1/core/sched/scheduler.py#L1545)).
|
||||
|
||||
**Multimodal stages.** [Verified] Preprocessing runs in the API-server process (`BaseMultiModalProcessor` wraps the HF processor; `PromptInsertion/Replacement` expand placeholder tokens; outputs are hashed and cached in `MultiModalProcessorCache` variants including a shared-memory object store between API server and engine) ([processing/processor.py#L1030](../../vllm/vllm/multimodal/processing/processor.py#L1030), [cache.py](../../vllm/vllm/multimodal/cache.py)). Encoders (ViT, Whisper encoder, audio towers) run inside the worker step under an encoder token budget and are cached per `mm_hash`; embeddings are scattered into `inputs_embeds` before the LM forward (`_execute_mm_encoder`, `_gather_mm_embeddings`, [gpu_model_runner.py#L3066](../../vllm/vllm/v1/worker/gpu_model_runner.py#L3066)). Encoder–decoder (Whisper) uses `CrossAttentionSpec` KV groups and `EncDecMultiModalProcessor`. Disaggregated encoder execution via EC connectors ([docs/features/disagg_encoder.md](../../vllm/docs/features/disagg_encoder.md), [vllm/v1/worker/mm_encoder_model_runner.py](../../vllm/vllm/v1/worker/mm_encoder_model_runner.py)). [Verified] No vocoder/decoder/diffusion *output* stages exist; `DiffusionGemma` implements block-diffusion **text** decoding with a custom `DiffusionSampler` and `ModelState` ([diffusion_gemma.py#L1043](../../vllm/vllm/model_executor/models/diffusion_gemma.py#L1043)).
|
||||
|
||||
## 5. Scheduler
|
||||
|
||||
### (a) Request / token / pipeline-level scheduling
|
||||
|
||||
[Verified] `Scheduler.schedule()` ([scheduler.py#L535](../../vllm/vllm/v1/core/sched/scheduler.py#L535)) — the code comment states there is "no decoding phase nor prefill phase": each request has `num_computed_tokens` and `num_tokens_with_spec`; the loop assigns up to `token_budget = max_num_scheduled_tokens` (defaults to `max_num_batched_tokens`) new tokens per step.
|
||||
- **Running requests first**: for each, `num_new_tokens = num_tokens_with_spec - num_computed_tokens`, capped by `long_prefill_token_threshold` (chunked prefill), token budget, `max_model_len`, Mamba block alignment, encoder budget (`_try_schedule_encoder_inputs`), multi-module MTP lookahead; `kv_cache_manager.allocate_slots` reserves blocks; on failure, **preemption**: the lowest-priority (priority policy) or last (FCFS) running request is popped, its blocks freed, `num_computed_tokens=0`, and it is prepended to `waiting` (recompute, no swap-out) ([scheduler.py#L1452](../../vllm/vllm/v1/core/sched/scheduler.py#L1452)).
|
||||
- **Waiting requests** are then admitted while budget remains and `len(running) < max_num_seqs`; prefix-cache hits reduce their initial `num_new_tokens`; requests may be parked as `WAITING_FOR_REMOTE_KVS`, `WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR`, or `WAITING_FOR_STREAMING_REQ` ([vllm/v1/request.py#L364](../../vllm/vllm/v1/request.py#L364)).
|
||||
- **Batching**: continuous batching with mixed prefill/decode in one `SchedulerOutput`; `SchedulingPolicy.FCFS | PRIORITY` via `create_request_queue` ([request_queue.py](../../vllm/vllm/v1/core/sched/request_queue.py)); DP "prefill throttling" for cadence alignment (`throttle_prefills`).
|
||||
- **Speculative decoding** reserves `draft_slots` and passes `scheduled_spec_decode_tokens`; drafts come from EAGLE/EAGLE3/MTP/Medusa/DFlash/n-gram/suffix proposers ([vllm/v1/spec_decode/](../../vllm/vllm/v1/spec_decode/eagle.py)).
|
||||
- **Async scheduling**: `AsyncScheduler` overrides `_update_after_schedule`/`_update_request_with_output` to schedule step N+1 with placeholder output tokens before step N's sampled ids are known ([async_scheduler.py](../../vllm/vllm/v1/core/sched/async_scheduler.py)); default is **on** unless pooling model, non-EAGLE/n-gram spec-decode, executor lacks support, ROCm DeepEP-DBO, or CPU platform ([config/vllm.py#L1273](../../vllm/vllm/config/vllm.py#L1273), [cpu.py#L268](../../vllm/vllm/platforms/cpu.py#L268)).
|
||||
- **Pipeline/multi-stage**: PP is handled by `step_with_batch_queue` (batch queue depth = PP stages) ([core.py#L649](../../vllm/vllm/v1/engine/core.py#L649)); multi-model "stages" (e.g. thinker→talker) are **not** a scheduler concept — N/A.
|
||||
- **Cancellation/abort**: `AsyncLLM.abort` → `EngineCoreClient.abort_requests_async` → `EngineCoreProc` aborts queue → `Scheduler.finish_requests(FINISHED_ABORTED)`; blocks are freed (possibly deferred, `_drain_deferred_frees`) ([async_llm.py#L834](../../vllm/vllm/v1/engine/async_llm.py#L834), [scheduler.py#L2483](../../vllm/vllm/v1/core/sched/scheduler.py#L2483)). Client disconnects propagate through the FastAPI handlers to `abort`.
|
||||
- **Backpressure**: `AsyncLLM.check_admission` enforces max in-flight requests/tokens ([async_llm.py#L288](../../vllm/vllm/v1/engine/async_llm.py#L288)); `pause_generation/resume_generation` and `PauseState` stop scheduling (used for sleep/weight update); `max_num_seqs`, `max_num_batched_tokens`, `stream_interval` (token buffering) are the tuning knobs ([config/scheduler.py#L190](../../vllm/vllm/config/scheduler.py#L190)).
|
||||
|
||||
Scheduler knobs and where they bite [Verified]:
|
||||
|
||||
| Knob (`SchedulerConfig` / `CacheConfig`) | Effect in `Scheduler` |
|
||||
| --- | --- |
|
||||
| `max_num_seqs` | `max_num_running_reqs`: cap on RUNNING set; waiting admission stops at it |
|
||||
| `max_num_batched_tokens` / `max_num_scheduled_tokens` | Per-step `token_budget` |
|
||||
| `long_prefill_token_threshold` | Chunk size for long prompts (chunked prefill) |
|
||||
| `enable_chunked_prefill` | Disabled automatically for non-causal (prefix-LM) layers and CPU MLA without AMX |
|
||||
| `policy` = `fcfs` / `priority` | Queue type and preemption victim selection |
|
||||
| `async_scheduling` | `AsyncScheduler` subclass; auto-resolved in `VllmConfig` |
|
||||
| `stream_interval` | Token batching for streaming output |
|
||||
| `enable_prefix_caching`, `prefix_caching_hash_algo` | `KVCacheManager.get_computed_blocks`, `BlockPool` hashing |
|
||||
| `block_size`, `kv_cache_memory_bytes`, `gpu_memory_utilization` | Block geometry and pool size |
|
||||
| `max_num_encoder_input_tokens` (derived) | Encoder compute budget per step |
|
||||
| `speculative_config.max_num_new_slots_for_drafting` | `draft_slots` reserved from `input_budget` |
|
||||
|
||||
Concrete `SchedulerOutput` contents ([vllm/v1/core/sched/output.py](../../vllm/vllm/v1/core/sched/output.py)): new request descriptors (prompt ids, MM features, block ids), cached-request diffs (new tokens, new block ids, resumed flags), `num_scheduled_tokens` per request, `scheduled_spec_decode_tokens`, `scheduled_encoder_inputs`, `free_encoder_mm_hashes`, `num_common_prefix_blocks` (cascade attention), `preempted_req_ids`, structured-output flags (`has_structured_output_requests`, `pending_structured_output_tokens`), `kv_connector_metadata`, and `finished_req_ids` ([output.py#L223](../../vllm/vllm/v1/core/sched/output.py#L223)). The worker never sees request objects — only this diff, which is what makes the persistent-batch runner possible.
|
||||
|
||||
### (b) Graph / operator / thread-level scheduling
|
||||
|
||||
[Verified] There is **no operator-level scheduler**: op ordering is whatever the PyTorch module graph (eager or Dynamo-captured) produces, with Inductor fusion passes ([vllm/compilation/passes/](../../vllm/vllm/compilation/passes/pass_manager.py)). Concurrency knobs at this layer:
|
||||
- CUDA graph replay chosen per batch by `CudagraphDispatcher` ([cudagraph_dispatcher.py](../../vllm/vllm/v1/cudagraph_dispatcher.py)).
|
||||
- Micro-batching / Dual-Batch-Overlap (`enable_dbo`, `ubatch_slices`, `gpu_ubatch_wrapper.py`) splits a batch into two streams to overlap communication and compute ([vllm/v1/worker/gpu_ubatch_wrapper.py](../../vllm/vllm/v1/worker/gpu_ubatch_wrapper.py), [docs/design/dbo.md](../../vllm/docs/design/dbo.md)); disabled on CPU.
|
||||
- Separate streams: async output copy stream; shared-experts stream in MoE (disabled on CPU).
|
||||
- Thread pools: CPU backend binds one OpenMP thread per core per TP rank (`VLLM_CPU_OMP_THREADS_BIND`) and forbids `torch.set_num_threads` after binding ([cpu_worker.py#L120](../../vllm/vllm/v1/worker/cpu_worker.py#L120)); `TORCHINDUCTOR_CPP_DYNAMIC_THREADS=1`. Engine core uses dedicated input/output socket threads ([core.py#L1705](../../vllm/vllm/v1/engine/core.py#L1705)); structured-output bitmask filling is submitted to a thread pool ([vllm/v1/structured_output/__init__.py](../../vllm/vllm/v1/structured_output/__init__.py)).
|
||||
- Stream assignment per op / explicit op scheduling: **N/A**.
|
||||
|
||||
## 6. I/O layer
|
||||
|
||||
- **Text preprocessing / tokenization.** [Verified] `Renderer` classes turn chat/completion prompts into token ids and MM data (`hf`, `mistral`, `cohere`, `deepseek_v32/v4`, `kimi_k3`, `inkling`, `terratorch`) ([vllm/renderers/registry.py](../../vllm/vllm/renderers/registry.py), [vllm/renderers/base.py](../../vllm/vllm/renderers/base.py)). Tokenizer modes: HF `tokenizers` (fast, via `CachedHfTokenizer`), Mistral `mistral_common`, custom (`kimi_audio`, DeepSeek encodings) ([vllm/tokenizers/registry.py](../../vllm/vllm/tokenizers/registry.py)); `fastokens` (optional Rust package) patches HF fast tokenizers ([vllm/tokenizers/fastokens.py](../../vllm/vllm/tokenizers/fastokens.py)); `sentencepiece` and `tiktoken` are dependencies ([requirements/common.txt](../../vllm/requirements/common.txt)). Prompt types: `TextPrompt`, `TokensPrompt`, `EmbedsPrompt` (prompt embeddings), `ExplicitEncoderDecoderPrompt`, `DataPrompt` ([vllm/inputs/llm.py](../../vllm/vllm/inputs/llm.py)).
|
||||
- **Image/video/audio preprocessing.** [Verified] `vllm/multimodal/media/{image,video,audio}.py` load bytes/base64/URL; images via Pillow, video via OpenCV/PyAV/torchcodec/PyNvVideoCodec backends ([vllm/multimodal/video.py](../../vllm/vllm/multimodal/video.py)), audio via PyAV decode with resampling by `soxr`/`scipy`/PyAV and mono downmix (`AudioResampler`, `normalize_audio`) ([vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py)); optional `audio` extra = `av, scipy, soundfile, soxr, mistral_common[audio]` ([setup.py#L1515](../../vllm/setup.py#L1515)). `split_audio`/`find_split_point` chunk long audio at quiet points for ASR ([audio.py#L325](../../vllm/vllm/multimodal/audio.py#L325)).
|
||||
- **Streaming.** [Verified] Token streaming: `RequestOutputCollector` (async queue per request) → SSE in the OpenAI routers; `stream_interval` batches tokens ([output_processor.py#L48](../../vllm/vllm/v1/engine/output_processor.py#L48)). Streaming **input**: `AsyncLLM.generate` accepts an async generator of `StreamingInput` (prompt chunks + sampling params) for multi-turn sessions that keep KV; used by the realtime ASR path ([async_llm.py#L515](../../vllm/vllm/v1/engine/async_llm.py#L515), [vllm/engine/protocol.py#L30](../../vllm/vllm/engine/protocol.py#L30), [tests/v1/streaming_input/](../../vllm/tests/v1/streaming_input/test_async_llm_streaming.py)). Realtime audio: WebSocket `/v1/realtime` (`input_audio_buffer.append/commit`, `transcription.delta/done` events) → `RealtimeConnection.audio_stream_generator` → model's `buffer_realtime_audio` → streaming prompts ([speech_to_text/realtime/connection.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/connection.py), [docs/serving/online_serving/speech_to_text.md#L169](../../vllm/docs/serving/online_serving/speech_to_text.md#L169)). No audio *output* streaming exists.
|
||||
- **Postprocessing.** [Verified] Incremental detokenization (`FastIncrementalDetokenizer` using `tokenizers.DecodeStream`, slow fallback), stop-string checking, logprobs, reasoning/tool parsers ([vllm/v1/engine/detokenizer.py](../../vllm/vllm/v1/engine/detokenizer.py), [vllm/reasoning/](../../vllm/vllm/reasoning/__init__.py), [vllm/tool_parsers/](../../vllm/vllm/tool_parsers/__init__.py)); pooling outputs (embedding/classification/score) ([vllm/outputs.py](../../vllm/vllm/outputs.py)); ASR post-processing (`post_process_output`, segment timestamps, diarization) in `SupportsTranscription`. IO-processor plugins allow custom pre/post for pooling models ([vllm/plugins/io_processors/interface.py](../../vllm/vllm/plugins/io_processors/interface.py)).
|
||||
- **Application boundaries.** [Verified] Python: `LLM` (offline: `generate/chat/encode/embed/score/classify`, `apply_model`, `collective_rpc`) and `AsyncLLM`. HTTP: OpenAI-compatible FastAPI (`/v1/completions`, `/v1/chat/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/audio/transcriptions`, `/v1/audio/translations`, `/v1/realtime`, scoring, tokenize, Anthropic `/v1/messages`, Cohere), endpoint plugins ([docs/serving/online_serving/openai_compatible_server.md](../../vllm/docs/serving/online_serving/openai_compatible_server.md), [vllm/entrypoints/anthropic/](../../vllm/vllm/entrypoints/anthropic/__init__.py)). gRPC: `vllm.entrypoints.grpc_server` backed by the external `smg-grpc-servicer` package ([grpc_server.py](../../vllm/vllm/entrypoints/grpc_server.py)). CLI: `vllm serve|chat|complete|bench|run-batch|collect-env` ([vllm/entrypoints/cli/main.py](../../vllm/vllm/entrypoints/cli/main.py)). Rust: experimental drop-in frontend (`vllm-server` axum, tokenizer, chat templates) talking ZMQ to the Python engine core ([rust/README.md](../../vllm/rust/README.md)). **No C API, no mobile bindings.**
|
||||
|
||||
## 7. Execution flow
|
||||
|
||||
### (a) Text LLM decode (online, V1 runner, single GPU)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as HTTP client
|
||||
participant API as FastAPI router (api server proc)
|
||||
participant A as AsyncLLM
|
||||
participant IP as InputProcessor/Renderer
|
||||
participant CC as AsyncMPClient (ZMQ)
|
||||
participant EC as EngineCoreProc (busy loop)
|
||||
participant S as Scheduler + KVCacheManager
|
||||
participant EX as MultiprocExecutor
|
||||
participant W as Worker / GPUModelRunner
|
||||
participant OP as OutputProcessor
|
||||
C->>API: POST /v1/chat/completions (stream=true)
|
||||
API->>A: generate(prompt, SamplingParams)
|
||||
A->>IP: render chat template, tokenize, validate lens
|
||||
A->>CC: add_request_async(EngineCoreRequest)
|
||||
CC-->>EC: msgspec over ZMQ input socket
|
||||
EC->>S: add_request → waiting queue
|
||||
loop each step (run_busy_loop → step)
|
||||
EC->>S: schedule(): get_computed_blocks (prefix hit), allocate_slots, token budget
|
||||
S-->>EC: SchedulerOutput (new/cached reqs, num_scheduled_tokens, block ids)
|
||||
EC->>EX: execute_model(scheduler_output, non_block=True)
|
||||
EX-->>W: shm MessageQueue broadcast
|
||||
W->>W: _update_states, _prepare_inputs, attn metadata
|
||||
W->>W: CudagraphDispatcher.dispatch(BatchDescriptor)
|
||||
W->>W: model forward (compiled pieces / CUDA graph), compute_logits
|
||||
EC->>EX: sample_tokens(grammar_bitmask)
|
||||
W->>W: Sampler (temp, penalties, top-k/p), async copy to host
|
||||
W-->>EC: ModelRunnerOutput (sampled ids, logprobs)
|
||||
EC->>S: update_from_output → append tokens, cache_blocks, finish checks
|
||||
EC-->>CC: EngineCoreOutputs via ZMQ output socket
|
||||
CC->>OP: process_outputs → IncrementalDetokenizer, stop strings
|
||||
OP-->>A: RequestOutputCollector.put(RequestOutput delta)
|
||||
A-->>API: async iterator yields delta
|
||||
API-->>C: SSE chunk
|
||||
end
|
||||
C--xAPI: disconnect / cancel
|
||||
API->>A: abort(request_id) → EngineCore abort → scheduler frees blocks
|
||||
```
|
||||
|
||||
### (b) Realtime streaming ASR (audio in → text out) — the closest TTS/VLA-relevant path
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as WS client (mic)
|
||||
participant R as RealtimeConnection (/v1/realtime)
|
||||
participant SV as OpenAIServingRealtime
|
||||
participant M as Model cls (SupportsRealtime, e.g. VoxtralRealtimeGeneration)
|
||||
participant A as AsyncLLM (streaming input session)
|
||||
participant S as Scheduler
|
||||
participant W as GPUModelRunner
|
||||
C->>R: session.update, input_audio_buffer.append (base64 PCM) ...
|
||||
R->>R: audio_stream_generator yields np.ndarray chunks
|
||||
R->>SV: transcribe_realtime(audio_stream)
|
||||
SV->>M: buffer_realtime_audio(audio_stream, input_stream, model_config)
|
||||
M-->>A: async generator of StreamingInput prompts (audio segment + prefix tokens)
|
||||
A->>A: _add_streaming_input_request (session keeps KV between chunks)
|
||||
loop per audio segment
|
||||
A->>S: request enters WAITING_FOR_STREAMING_REQ → RUNNING when chunk arrives
|
||||
S->>W: schedule encoder input (audio encoder budget) + LM tokens
|
||||
W->>W: _execute_mm_encoder (audio tower) → embeddings → LM forward → sample (realtime_max_tokens)
|
||||
W-->>S: sampled ids
|
||||
S-->>A: EngineCoreOutputs
|
||||
A-->>SV: RequestOutput delta
|
||||
SV-->>R: transcription.delta
|
||||
R-->>C: WS event
|
||||
end
|
||||
C->>R: input_audio_buffer.commit → transcription.done
|
||||
```
|
||||
|
||||
[Verified] For OpenVLA the flow is exactly (a) with an image: `OpenVLAMultiModalProcessor` → DINOv2+SigLIP `PrismaticVisionBackbone` → `PrismaticProjector` → Llama decode of `n_action_bins` action tokens ([openvla.py#L377](../../vllm/vllm/model_executor/models/openvla.py#L377)); un-normalizing action tokens into continuous actions is left to the client (not in tree). TTS: **N/A** (no path).
|
||||
|
||||
## 8. Tests and examples
|
||||
|
||||
- [Verified] Tests: `tests/` (~60 top-level areas): `tests/v1/{core,engine,worker,e2e,attention,cudagraph,kv_connector,kv_offload,sample,spec_decode,structured_output,streaming_input,...}` ([tests/v1/core/test_scheduler.py](../../vllm/tests/v1/core/test_scheduler.py), [tests/v1/core/test_prefix_caching.py](../../vllm/tests/v1/core/test_prefix_caching.py), [tests/v1/engine/test_engine_core.py](../../vllm/tests/v1/engine/test_engine_core.py)); kernels in `tests/kernels/{attention,core,mamba,moe,quantization,ir,helion}` incl. [tests/kernels/test_onednn.py](../../vllm/tests/kernels/test_onednn.py); models in `tests/models` (e.g. [tests/models/multimodal/processing/test_openvla.py](../../vllm/tests/models/multimodal/processing/test_openvla.py)); platform test [tests/platforms/test_cpu.py](../../vllm/tests/platforms/test_cpu.py); plugin fixtures in [tests/plugins/](../../vllm/tests/plugins/vllm_add_dummy_platform/setup.py) and [tests/plugins_tests/](../../vllm/tests/plugins_tests/test_endpoint_plugins.py) (gguf/bnb plugin tests live here). Evals under `tests/evals`.
|
||||
- [Verified] How to run (per [AGENTS.md](../../vllm/AGENTS.md)): `uv venv --python 3.12`, `VLLM_USE_PRECOMPILED=1 uv pip install -e . --torch-backend=auto`, `uv pip install -r requirements/test/cuda.in`, then `.venv/bin/python -m pytest tests/path/to/test_file.py -v`; never bare `pip`. Model-specific test guidance: [docs/contributing/model/tests.md](../../vllm/docs/contributing/model/tests.md).
|
||||
- [Verified] CI: Buildkite pipelines ([.buildkite/test-pipeline.yaml](../../vllm/.buildkite/test-pipeline.yaml), hardware jobs for AMD, Ascend NPU, **CPU x86 + "Arm CPU Test Shard" via `run-cpu-test-arm.sh`**, GH200 (aarch64 + Hopper), Intel/XPU ([.buildkite/hardware_tests/cpu.yaml#L149](../../vllm/.buildkite/hardware_tests/cpu.yaml#L149), [.buildkite/hardware_tests/gh200.yaml](../../vllm/.buildkite/hardware_tests/gh200.yaml), [.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh](../../vllm/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh)); GitHub Actions macOS Apple-Silicon smoke test ([.github/workflows/macos-smoke-test.yml](../../vllm/.github/workflows/macos-smoke-test.yml)); wheel size guard 500 MB ([.buildkite/check-wheel-size.py](../../vllm/.buildkite/check-wheel-size.py)); CPU performance benchmark JSONs referenced from the CPU docs.
|
||||
- [Verified] Examples: basic offline/online ([examples/basic/offline_inference/basic.py](../../vllm/examples/basic/offline_inference/basic.py)); vision/audio ([examples/generate/multimodal/audio_language_offline.py](../../vllm/examples/generate/multimodal/audio_language_offline.py), [vision_language_offline.py](../../vllm/examples/generate/multimodal/vision_language_offline.py), [encoder_decoder_multimodal_offline.py](../../vllm/examples/generate/multimodal/encoder_decoder_multimodal_offline.py)); Qwen2.5-Omni **thinker only** ([examples/generate/multimodal/qwen2_5_omni/only_thinker.py](../../vllm/examples/generate/multimodal/qwen2_5_omni/only_thinker.py)); ASR clients ([examples/speech_to_text/openai/openai_transcription_client.py](../../vllm/examples/speech_to_text/openai/openai_transcription_client.py)); realtime microphone Gradio demo ([examples/speech_to_text/realtime/openai_realtime_microphone_client.py](../../vllm/examples/speech_to_text/realtime/openai_realtime_microphone_client.py)); disaggregated / RL / Ray serving. **No mobile, embedded, or TTS examples.**
|
||||
|
||||
## 9. TTS relevance
|
||||
|
||||
- [Verified] **Models producing audio: none.** Search for vocoders/codec decoders/audio-output endpoints returns only audio *encoder* code (`mimo_audio.py` holds MiMo's RVQ audio tokenizer/encoder used as an input tower by `MiMoV2OmniForCausalLM`; Qwen2.5/3-Omni are registered as **Thinker** halves with `"talker.": None` weight-skip; the example README is titled "Thinker Only") ([mimo_audio.py](../../vllm/vllm/model_executor/models/mimo_audio.py), [qwen2_5_omni_thinker.py#L1062](../../vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py#L1062), [examples/generate/multimodal/qwen2_5_omni/README.md](../../vllm/examples/generate/multimodal/qwen2_5_omni/README.md)).
|
||||
- [Verified] **Audio input models** (registry `_MULTIMODAL_MODELS`): AudioFlamingo3, FunAudioChat, Gemma3n, Granite Speech(+Plus), Kimi-Audio, MiDashengLM, MiMo-V2-Omni, MiniCPM-O, MOSS-Audio, MossTranscribeDiarize, Phi-4-MM, Qwen2-Audio, Qwen2.5-Omni thinker, Qwen3-Omni thinker, Qwen3-ASR, Ultravox, Voxtral, Voxtral Realtime, Whisper, VibeVoice-ASR, GLM-ASR, FireRedASR2, FunASR, Cohere-Transcribe ([registry.py#L345](../../vllm/vllm/model_executor/models/registry.py#L345), [docs/models/supported_models.md#L706](../../vllm/docs/models/supported_models.md#L706)).
|
||||
- [Verified] **Audio ops**: resampling (soxr/scipy/PyAV), mono downmix, normalization, duration, quiet-point chunking ([vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py)); Whisper-style encoder attention via `MMEncoderAttention`; sinusoidal position embeddings; RVQ codebooks (encoder side only). No STFT/iSTFT, no mel-inverse, no HiFi-GAN/BigVGAN/WavTokenizer decoder, no flow-matching/DiT audio decoder.
|
||||
- [Verified] **Streaming audio**: input-side only (`SupportsRealtime.buffer_realtime_audio`, `VoxtralRealtimeBuffer` with look-ahead/look-back ms, `realtime_max_tokens` per segment) ([voxtral_realtime.py#L127](../../vllm/vllm/model_executor/models/voxtral_realtime.py#L127), [interfaces.py#L1267](../../vllm/vllm/model_executor/models/interfaces.py#L1267)).
|
||||
- [Verified] **Interfaces to build on**: `SupportsTranscription` + `SpeechToTextBaseServing` show how an audio-modal endpoint, chunking, language detection, and streaming post-processors are wired ([docs/contributing/model/transcription.md](../../vllm/docs/contributing/model/transcription.md)). Streaming-input sessions (`StreamingInput`, `WAITING_FOR_STREAMING_REQ`) give a KV-preserving multi-turn primitive that a text→speech-token model could reuse.
|
||||
- **What is missing for TTS** [Verified unless noted]: (1) any speech-token → waveform stage and a place to run it (no "post-LM decoder stage" in the runner); (2) audio output types in `RequestOutput`/OpenAI `/v1/audio/speech`; (3) non-text sampling constraints for codec-token vocabularies (would have to be logits processors); (4) [Inferred] multi-stage pipeline scheduling (thinker→talker→vocoder) — the scheduler has one model per engine; vLLM-Omni is the project's answer, not this repo. Registering a TTS LM whose output is codec tokens is feasible as a plain text-generation model, but decoding to audio must happen outside vLLM. [Proposal] For edge TTS, use vLLM only if the LM half dominates cost and a separate lightweight vocoder can run on the same device.
|
||||
|
||||
## 10. VLA relevance
|
||||
|
||||
- [Verified] **Vision encoders**: many ViT/SigLIP/CLIP/DINOv2/InternViT/Qwen-VL towers with their own CUDA-graph and attention-backend selection (`get_vit_attn_backend`: FLASH_ATTN/TRITON/FLASHINFER/TORCH_SDPA), `SupportsEncoderCudaGraph`, `SupportsMultiModalPruning` ([interfaces.py#L1856](../../vllm/vllm/model_executor/models/interfaces.py#L1856), [docs/design/cuda_graphs_multimodal.md](../../vllm/docs/design/cuda_graphs_multimodal.md)).
|
||||
- [Verified] **Action model**: `OpenVLAForActionPrediction` — Prismatic backbone (`timm` DINOv2 + SigLIP fused), 2/3-layer MLP projector, Llama LM, `n_action_bins` from `OpenVLAConfig`, image-only modality (`get_placeholder_str` raises for non-image), 224×224 fixed patches; docs list it as supported with PP ([openvla.py](../../vllm/vllm/model_executor/models/openvla.py), [vllm/transformers_utils/configs/openvla.py](../../vllm/vllm/transformers_utils/configs/openvla.py), [docs/models/supported_models.md#L606](../../vllm/docs/models/supported_models.md#L606)).
|
||||
- [Verified] **Structured/proprioceptive inputs**: none as a modality; only `EmbedsPrompt` (raw prompt embeddings) could carry precomputed state embeddings ([vllm/inputs/llm.py#L131](../../vllm/vllm/inputs/llm.py#L131), [docs/features/prompt_embeds.md](../../vllm/docs/features/prompt_embeds.md)).
|
||||
- [Verified] **Action heads / diffusion / flow policies**: none. `DiffusionGemma` proves the runner can host a non-AR block-diffusion decoder with a custom sampler and per-request state (`DiffusionGemmaRequestStates`, `ModelState`) — a template for iterative denoising heads over the LM's KV cache, but it produces text tokens ([diffusion_gemma.py#L660](../../vllm/vllm/model_executor/models/diffusion_gemma.py#L660)).
|
||||
- [Verified] **Low-latency loop primitives**: async scheduling, full CUDA graphs for uniform decode, `FULL_DECODE_ONLY`, MRV2's sync-free stream loop, priority scheduling, streaming input sessions that keep KV (`session_id`) across turns, `stream_interval`. [Inferred] The 3-process architecture (API server ↔ engine core ↔ worker over ZMQ/shm) adds fixed IPC latency per step that a single-process control loop would avoid; `InprocClient` + `UniprocExecutor` (`VLLM_ENABLE_V1_MULTIPROCESSING=0`) reduce this to one process on GPU, but CPU forces the `mp` executor ([cpu.py#L273](../../vllm/vllm/platforms/cpu.py#L273)).
|
||||
- **Missing for VLA** [Verified]: proprioception/state tokens, continuous action heads (MLP/diffusion/flow), action chunk decoding, KV reuse across timesteps for a fixed instruction with changing images (prefix caching covers the text prefix only; image embeddings are cached by `mm_hash` so identical frames hit, new frames do not), and any robotics I/O.
|
||||
|
||||
## 11. Edge deployment profile
|
||||
|
||||
- **Build system.** [Verified] `setuptools` + CMake (`cmake>=3.26.1`, `ninja`), `setuptools-rust` and a Rust `1.95` toolchain ([rust-toolchain.toml](../../vllm/rust-toolchain.toml)) for the optional Rust frontend/`fastokens` (precompiled artifacts can be reused: `VLLM_USE_PRECOMPILED`, `VLLM_USE_PRECOMPILED_RUST`) ([setup.py#L49](../../vllm/setup.py#L49), [build_rust.sh](../../vllm/build_rust.sh)). Target selection: `VLLM_TARGET_DEVICE=cuda|rocm|xpu|cpu|empty` (macOS forced to `cpu`) ([setup.py#L85](../../vllm/setup.py#L85)). CPU build: [cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake) detects ISA from `/proc/cpuinfo`/`sysctl` (`asimd`, `bf16`, `i8mm`, AVX2/512/AMX, POWER, S390 VXE, RVV), sets `-march=armv8.2-a+bf16+dotprod+fp16` (or without bf16), builds **oneDNN** (pinned commit for aarch64) and **Arm Compute Library** via SCons (`ONEDNN_AARCH64_USE_ACL`, `VLLM_USE_ACL`; `ACL_ROOT_DIR` to reuse a prebuilt one), and C++20. Cross-compile flags: `VLLM_CPU_ARM_BF16`, `VLLM_CPU_ARM_I8MM`, `VLLM_CPU_RVV_BF16`. Apple Silicon skips oneDNN/ACL (`APPLE_SILICON_FOUND`).
|
||||
- **Dependencies (runtime).** [Verified] `torch==2.13.0` (+cpu on Linux x86/aarch64/s390x), `transformers>=5.10.4`, `tokenizers`, `safetensors`, `fastapi[standard]`, `uvicorn` (via fastapi standard), `pyzmq`, `msgspec`, `numba==0.65.0`, `opencv-python-headless`, `pillow`, `torchaudio`, `torchvision`, `torchcodec`, `xgrammar`, `llguidance`, `outlines_core`, `lm-format-enforcer`, `compressed-tensors`, `mistral_common`, `openai`, `anthropic`, `mcp`, `opentelemetry-*`, `prometheus_client`, `intel-openmp` (x86 CPU), `py-cpuinfo` ([requirements/common.txt](../../vllm/requirements/common.txt), [requirements/cpu.txt](../../vllm/requirements/cpu.txt)). [Inferred] This is a multi-GB Python environment even on CPU; there is no "minimal" profile.
|
||||
- **Binary size.** [Verified] CI fails if a wheel exceeds `VLLM_MAX_SIZE_MB` (default 500 MB) ([.buildkite/check-wheel-size.py#L12](../../vllm/.buildkite/check-wheel-size.py#L12)). [Unknown] Actual CPU/aarch64 wheel size and RSS were not measured here.
|
||||
- **Process model.** [Verified] API server + engine core + N workers (+DP coordinator), `spawn`/`forkserver` start methods, ZMQ sockets, shared-memory `MessageQueue`; single-process mode via `VLLM_ENABLE_V1_MULTIPROCESSING=0` (GPU only in practice) ([docs/design/arch_overview.md#L67](../../vllm/docs/design/arch_overview.md#L67), [docs/design/multiprocessing.md](../../vllm/docs/design/multiprocessing.md)).
|
||||
- **Supported platforms for edge** [Verified]:
|
||||
- *NVIDIA Jetson/Orin/Thor*: detection path exists (`cuda_is_jetson`, non-NVML platform), arch `8.7` and `11.0` in `CUDA_SUPPORTED_ARCHS`; docs only contain a Thor `sm_110a` Triton `ptxas` workaround (`TRITON_PTXAS_PATH`); no Jetson wheels/Docker/CI in tree; kernel deps (FlashInfer, FA, CUTLASS DSL) would need aarch64+CUDA builds — the GH200 CI job shows aarch64+CUDA is built ([__init__.py#L76](../../vllm/vllm/platforms/__init__.py#L76), [troubleshooting.md](../../vllm/docs/usage/troubleshooting.md)). `is_integrated_gpu()` exists on the platform interface (used for unified-memory awareness) ([cuda.py#L714](../../vllm/vllm/platforms/cuda.py#L714)).
|
||||
- *ARM CPU (Linux aarch64)*: first-class: NEON/BF16/I8MM kernels, oneDNN+ACL GEMM, KleidiAI W4A8 via torch, INT8 W8A8 (`onednn_scaled_mm`), int8 fused MoE, prebuilt `manylinux_2_34_aarch64` wheels since 0.11.2, nightly wheels per commit, `vllm/vllm-openai-cpu:latest-arm64` images, ARM CI shards, "tested on AWS Graviton3" ([cpu.arm.inc.md](../../vllm/docs/getting_started/installation/cpu.arm.inc.md), [docker/Dockerfile.cpu#L143](../../vllm/docker/Dockerfile.cpu#L143)). Requirements: Linux, gcc≥12.3, NEON; BF16 needs ARMv8.6 (`FEAT_BF16`). Triton-CPU is only built into the x86 Docker image (`VLLM_CPU_X86` guard), so on ARM the runner uses the C++ monkey-patches and **Model Runner V1** (`HAS_TRITON` false → V2 disabled) ([Dockerfile.cpu#L311](../../vllm/docker/Dockerfile.cpu#L311), [vllm/triton_utils/importing.py#L15](../../vllm/vllm/triton_utils/importing.py#L15), [config/vllm.py#L672](../../vllm/vllm/config/vllm.py#L672)).
|
||||
- *Apple Silicon*: in-tree **CPU only** (experimental; FP16/FP32, BF16 if `FEAT_BF16`; build from source; no wheels/images; oneDNN/ACL disabled; CI smoke test); GPU only via OOT `vllm-metal` (MLX) ([cpu.apple.inc.md](../../vllm/docs/getting_started/installation/cpu.apple.inc.md), [gpu.apple.inc.md](../../vllm/docs/getting_started/installation/gpu.apple.inc.md)). No CoreML/ANE.
|
||||
- *Android / mobile GPU-NPU (Vulkan, OpenCL, QNN, NNAPI)*: **nothing in tree**; would require an OOT platform plugin plus a PyTorch build for that target. [Inferred] Not realistic given the torch/transformers/FastAPI dependency stack.
|
||||
- *Other*: AMD RDNA3/3.5 APU gfx targets are in the ROCm arch list ([CMakeLists.txt#L52](../../vllm/CMakeLists.txt#L52)); RISC-V RVV and s390x CPU builds exist.
|
||||
- **Model conversion path and constraints.** [Verified] None required: HF checkpoints (safetensors) are loaded directly; quantized checkpoints must come pre-quantized in a supported format (AWQ/GPTQ/compressed-tensors/ModelOpt/etc.) or use the `online` quant method; on ARM CPU only INT8 W8A8 and dynamic 4-bit W4A8 (compressed-tensors `W4A8Int` with `Dynamic4bitLinearKernel`, groupwise bf16 scales, KleidiAI-supported shapes) have kernels ([dynamic_4bit.py#L26](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py#L26)). `torch.compile` cache directory can be copied to a target to avoid compile time ([docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md)). [Verified] CPU-validated model list is Intel Xeon-centric ([docs/models/hardware_supported_models/cpu.md](../../vllm/docs/models/hardware_supported_models/cpu.md)); no ARM-specific validated list.
|
||||
|
||||
## 12. Limitations and unknowns
|
||||
|
||||
- [Verified] No TTS / audio generation of any kind; no vocoder or codec-decoder ops; no `/v1/audio/speech`.
|
||||
- [Verified] Only one VLA model (OpenVLA), text-token action outputs only; no proprioceptive modality or continuous action heads.
|
||||
- [Verified] Hard dependencies on PyTorch 2.13, transformers ≥5.10, FastAPI/uvicorn, ZMQ; no C API, no mobile bindings, no Android/Vulkan/OpenCL/QNN/NNAPI/CoreML support.
|
||||
- [Verified] TPU, Ascend NPU, Apple Metal, Gaudi, GGUF, bitsandbytes all live in out-of-tree packages; compatibility of those plugins with this HEAD is outside this tree ([Unknown]).
|
||||
- [Verified] CPU platform disables async scheduling, DBO, CUDA-graph-equivalents; compilation is `DYNAMO_TRACE_ONCE` + Inductor (C++ codegen), LoRA disables compilation; MLA on non-AMX CPUs disables chunked prefill and prefix caching.
|
||||
- [Verified] Model Runner V2 requires Triton; on ARM CPU (no Triton-CPU in the ARM image) V1 runner is used, and several Triton kernels are replaced by C++ fallbacks (`_postprocess_triton`).
|
||||
- [Verified] Arm CPU quantization coverage per docs: INT8 W8A8 and W4A8 only; AWQ/GPTQ/Marlin/GGUF marked unsupported.
|
||||
- [Verified] Apple Silicon in-tree is CPU-only, experimental, source-build only, no prebuilt wheels.
|
||||
- [Verified] Jetson: detection only; no wheels, Docker, CI, or docs beyond a Thor ptxas workaround.
|
||||
- [Verified] `bitsandbytes`/`DeepSpeedFP` rows still appear in the quantization support table although BnB is now a plugin — docs lag code.
|
||||
- [Unknown] Binary/wheel size, RSS, startup time and latency on ARM/Jetson — not measured; the repo publishes no edge numbers (CPU benchmark dashboard referenced in docs is Xeon-based).
|
||||
- [Unknown] Whether `vllm-metal`, `vllm-ascend`, or other OOT platforms track this exact HEAD (platform interface has had removals: `use_v1`, `_Backend`, `seed_everything`, `validate_request(prompt)` per the plugin doc's deprecation list).
|
||||
- [Inferred] Minimum practical footprint is a server-class Python environment; trimming to an embedded profile would mean forking the executor/worker layer (e.g. `InprocClient` + `UniprocExecutor` + a slim frontend) and dropping the HTTP stack.
|
||||
- [Verified] The installed `vllm 0.28.0` wheel on this machine differs from the analyzed checkout (post-0.29.0rc4 main); behaviours described here (e.g. MRV2 default, `vllm.ir`, `_C_stable_libtorch`, streaming input sessions) may be absent or different in 0.28.0.
|
||||
|
||||
## 13. Reference index
|
||||
|
||||
| Path | Role |
|
||||
| --- | --- |
|
||||
| [AGENTS.md](../../vllm/AGENTS.md) | Repo contribution/dev workflow (uv, tests, lint) |
|
||||
| [pyproject.toml](../../vllm/pyproject.toml) | Build requirements (cmake, ninja, setuptools-rust, torch==2.13.0), Python 3.10–3.14 |
|
||||
| [setup.py](../../vllm/setup.py) | Target-device selection, CMake extension build, precompiled wheel logic, extras (`zen`, `audio`, `tensorizer`, `runai`) |
|
||||
| [CMakeLists.txt](../../vllm/CMakeLists.txt) | CUDA/HIP arch lists, extension targets (`_C_stable_libtorch`, `_moe_C_stable_libtorch`, `cumem_allocator`, `_C`, `_rocm_C`, `spinloop`, `fs_io_C`) |
|
||||
| [cmake/cpu_extension.cmake](../../vllm/cmake/cpu_extension.cmake) | CPU ISA detection, ARM flags, oneDNN + ACL build, CPU `_C` sources |
|
||||
| [cmake/external_projects/vllm_flash_attn.cmake](../../vllm/cmake/external_projects/vllm_flash_attn.cmake) | Example of FetchContent kernel dependency |
|
||||
| [requirements/common.txt](../../vllm/requirements/common.txt), [requirements/cpu.txt](../../vllm/requirements/cpu.txt), [requirements/cuda.txt](../../vllm/requirements/cuda.txt) | Runtime dependency sets |
|
||||
| [rust-toolchain.toml](../../vllm/rust-toolchain.toml), [build_rust.sh](../../vllm/build_rust.sh), [rust/README.md](../../vllm/rust/README.md) | Experimental Rust frontend |
|
||||
| [docker/Dockerfile.cpu](../../vllm/docker/Dockerfile.cpu) | x86/arm64 CPU image; Triton-CPU only on x86 |
|
||||
| [csrc/libtorch_stable/torch_bindings.cpp](../../vllm/csrc/libtorch_stable/torch_bindings.cpp) | CUDA op registrations (STABLE_TORCH_LIBRARY) |
|
||||
| [csrc/cpu/torch_bindings.cpp](../../vllm/csrc/cpu/torch_bindings.cpp) | CPU op registrations incl. ARM I8MM-guarded MoE ops |
|
||||
| [csrc/cpu/cpu_attn_neon.hpp](../../vllm/csrc/cpu/cpu_attn_neon.hpp), [csrc/cpu/cpu_attn_neon_bfmmla.hpp](../../vllm/csrc/cpu/cpu_attn_neon_bfmmla.hpp), [csrc/cpu/cpu_types_arm.hpp](../../vllm/csrc/cpu/cpu_types_arm.hpp) | ARM NEON/BFMMLA attention and vector types |
|
||||
| [csrc/cpu/micro_gemm/cpu_micro_gemm_neon.hpp](../../vllm/csrc/cpu/micro_gemm/cpu_micro_gemm_neon.hpp) | ARM micro-GEMM |
|
||||
| [csrc/cpu/dnnl_kernels.cpp](../../vllm/csrc/cpu/dnnl_kernels.cpp) | oneDNN GEMM / int8 quant kernels |
|
||||
| [csrc/cumem_allocator.cpp](../../vllm/csrc/cumem_allocator.cpp) | CUDA VMM allocator for sleep mode |
|
||||
| [vllm/platforms/__init__.py](../../vllm/vllm/platforms/__init__.py) | Platform detection and plugin resolution |
|
||||
| [vllm/platforms/interface.py](../../vllm/vllm/platforms/interface.py) | `Platform` ABC, enums |
|
||||
| [vllm/platforms/cuda.py](../../vllm/vllm/platforms/cuda.py), [vllm/platforms/rocm.py](../../vllm/vllm/platforms/rocm.py), [vllm/platforms/xpu.py](../../vllm/vllm/platforms/xpu.py), [vllm/platforms/cpu.py](../../vllm/vllm/platforms/cpu.py), [vllm/platforms/zen_cpu.py](../../vllm/vllm/platforms/zen_cpu.py), [vllm/platforms/tpu.py](../../vllm/vllm/platforms/tpu.py) | In-tree platforms |
|
||||
| [vllm/plugins/__init__.py](../../vllm/vllm/plugins/__init__.py) | Entry-point plugin groups |
|
||||
| [vllm/v1/engine/async_llm.py](../../vllm/vllm/v1/engine/async_llm.py), [vllm/v1/engine/llm_engine.py](../../vllm/vllm/v1/engine/llm_engine.py) | Engine front-ends |
|
||||
| [vllm/v1/engine/core.py](../../vllm/vllm/v1/engine/core.py), [vllm/v1/engine/core_client.py](../../vllm/vllm/v1/engine/core_client.py) | Engine core process and clients |
|
||||
| [vllm/v1/engine/input_processor.py](../../vllm/vllm/v1/engine/input_processor.py), [vllm/v1/engine/output_processor.py](../../vllm/vllm/v1/engine/output_processor.py), [vllm/v1/engine/detokenizer.py](../../vllm/vllm/v1/engine/detokenizer.py) | Input validation, output streaming, detokenization |
|
||||
| [vllm/engine/protocol.py](../../vllm/vllm/engine/protocol.py) | `EngineClient`, `StreamingInput` |
|
||||
| [vllm/v1/core/sched/scheduler.py](../../vllm/vllm/v1/core/sched/scheduler.py), [vllm/v1/core/sched/async_scheduler.py](../../vllm/vllm/v1/core/sched/async_scheduler.py), [vllm/v1/core/sched/interface.py](../../vllm/vllm/v1/core/sched/interface.py), [vllm/v1/core/sched/request_queue.py](../../vllm/vllm/v1/core/sched/request_queue.py) | Request scheduler |
|
||||
| [vllm/v1/core/kv_cache_manager.py](../../vllm/vllm/v1/core/kv_cache_manager.py), [vllm/v1/core/kv_cache_coordinator.py](../../vllm/vllm/v1/core/kv_cache_coordinator.py), [vllm/v1/core/single_type_kv_cache_manager.py](../../vllm/vllm/v1/core/single_type_kv_cache_manager.py), [vllm/v1/core/block_pool.py](../../vllm/vllm/v1/core/block_pool.py), [vllm/v1/core/kv_cache_utils.py](../../vllm/vllm/v1/core/kv_cache_utils.py), [vllm/v1/core/encoder_cache_manager.py](../../vllm/vllm/v1/core/encoder_cache_manager.py) | KV / encoder cache management |
|
||||
| [vllm/v1/kv_cache_interface.py](../../vllm/vllm/v1/kv_cache_interface.py), [vllm/v1/kv_cache_layout.py](../../vllm/vllm/v1/kv_cache_layout.py) | KV specs and layouts |
|
||||
| [vllm/v1/kv_offload/cpu/manager.py](../../vllm/vllm/v1/kv_offload/cpu/manager.py), [vllm/distributed/kv_transfer/kv_connector/v1/base.py](../../vllm/vllm/distributed/kv_transfer/kv_connector/v1/base.py) | KV offload / connectors |
|
||||
| [vllm/v1/request.py](../../vllm/vllm/v1/request.py) | `Request`, `RequestStatus` |
|
||||
| [vllm/v1/executor/abstract.py](../../vllm/vllm/v1/executor/abstract.py), [vllm/v1/executor/multiproc_executor.py](../../vllm/vllm/v1/executor/multiproc_executor.py), [vllm/v1/executor/uniproc_executor.py](../../vllm/vllm/v1/executor/uniproc_executor.py) | Executors |
|
||||
| [vllm/v1/worker/worker_base.py](../../vllm/vllm/v1/worker/worker_base.py), [vllm/v1/worker/gpu_worker.py](../../vllm/vllm/v1/worker/gpu_worker.py), [vllm/v1/worker/cpu_worker.py](../../vllm/vllm/v1/worker/cpu_worker.py), [vllm/v1/worker/xpu_worker.py](../../vllm/vllm/v1/worker/xpu_worker.py) | Workers |
|
||||
| [vllm/v1/worker/gpu_model_runner.py](../../vllm/vllm/v1/worker/gpu_model_runner.py), [vllm/v1/worker/cpu_model_runner.py](../../vllm/vllm/v1/worker/cpu_model_runner.py), [vllm/v1/worker/mm_encoder_model_runner.py](../../vllm/vllm/v1/worker/mm_encoder_model_runner.py), [vllm/v1/worker/encoder_cudagraph.py](../../vllm/vllm/v1/worker/encoder_cudagraph.py), [vllm/v1/worker/gpu_ubatch_wrapper.py](../../vllm/vllm/v1/worker/gpu_ubatch_wrapper.py) | Model runner V1 and helpers |
|
||||
| [vllm/v1/worker/gpu/model_runner.py](../../vllm/vllm/v1/worker/gpu/model_runner.py), [vllm/v1/worker/gpu/README.md](../../vllm/vllm/v1/worker/gpu/README.md) | Model Runner V2 |
|
||||
| [vllm/v1/cudagraph_dispatcher.py](../../vllm/vllm/v1/cudagraph_dispatcher.py) | CUDA graph runtime dispatch |
|
||||
| [vllm/compilation/backends.py](../../vllm/vllm/compilation/backends.py), [vllm/compilation/cuda_graph.py](../../vllm/vllm/compilation/cuda_graph.py), [vllm/compilation/decorators.py](../../vllm/vllm/compilation/decorators.py), [vllm/compilation/passes/pass_manager.py](../../vllm/vllm/compilation/passes/pass_manager.py) | torch.compile integration |
|
||||
| [vllm/config/compilation.py](../../vllm/vllm/config/compilation.py), [vllm/config/scheduler.py](../../vllm/vllm/config/scheduler.py), [vllm/config/cache.py](../../vllm/vllm/config/cache.py), [vllm/config/load.py](../../vllm/vllm/config/load.py), [vllm/config/vllm.py](../../vllm/vllm/config/vllm.py) | Config dataclasses and default resolution |
|
||||
| [vllm/v1/attention/backend.py](../../vllm/vllm/v1/attention/backend.py), [vllm/v1/attention/backends/registry.py](../../vllm/vllm/v1/attention/backends/registry.py), [vllm/v1/attention/selector.py](../../vllm/vllm/v1/attention/selector.py), [vllm/v1/attention/backends/cpu_attn.py](../../vllm/vllm/v1/attention/backends/cpu_attn.py) | Attention abstraction and CPU backend |
|
||||
| [vllm/model_executor/custom_op.py](../../vllm/vllm/model_executor/custom_op.py) | `CustomOp` / `PluggableLayer` dispatch and OOT registration |
|
||||
| [vllm/ir/op.py](../../vllm/vllm/ir/op.py), [vllm/ir/ops/layernorm.py](../../vllm/vllm/ir/ops/layernorm.py), [vllm/kernels/vllm_c.py](../../vllm/vllm/kernels/vllm_c.py), [vllm/kernels/oink_ops.py](../../vllm/vllm/kernels/oink_ops.py), [vllm/kernels/triton/qkv_padded_fp8_quant.py](../../vllm/vllm/kernels/triton/qkv_padded_fp8_quant.py) | vLLM IR ops and providers |
|
||||
| [vllm/model_executor/layers/utils.py](../../vllm/vllm/model_executor/layers/utils.py) | CPU unquantized GEMM dispatch |
|
||||
| [vllm/model_executor/layers/quantization/__init__.py](../../vllm/vllm/model_executor/layers/quantization/__init__.py) | Quantization method registry |
|
||||
| [vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a8_int.py](../../vllm/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w4a8_int.py) | W4A8 int scheme (kernel selection) |
|
||||
| [vllm/model_executor/kernels/linear/mixed_precision/cpu.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/cpu.py), [vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py](../../vllm/vllm/model_executor/kernels/linear/mixed_precision/dynamic_4bit.py) | CPU WNA16 (AMX/RISC-V) and ARM KleidiAI dynamic-4bit kernels |
|
||||
| [vllm/model_executor/model_loader/__init__.py](../../vllm/vllm/model_executor/model_loader/__init__.py) | Load formats |
|
||||
| [vllm/model_executor/models/registry.py](../../vllm/vllm/model_executor/models/registry.py), [vllm/model_executor/models/interfaces.py](../../vllm/vllm/model_executor/models/interfaces.py) | Model registry and capability protocols |
|
||||
| [vllm/model_executor/models/openvla.py](../../vllm/vllm/model_executor/models/openvla.py), [vllm/transformers_utils/configs/openvla.py](../../vllm/vllm/transformers_utils/configs/openvla.py) | OpenVLA |
|
||||
| [vllm/model_executor/models/diffusion_gemma.py](../../vllm/vllm/model_executor/models/diffusion_gemma.py) | Block-diffusion text decoder + custom sampler |
|
||||
| [vllm/model_executor/models/whisper.py](../../vllm/vllm/model_executor/models/whisper.py), [vllm/model_executor/models/voxtral_realtime.py](../../vllm/vllm/model_executor/models/voxtral_realtime.py), [vllm/model_executor/models/kimi_audio.py](../../vllm/vllm/model_executor/models/kimi_audio.py), [vllm/model_executor/models/moss_audio.py](../../vllm/vllm/model_executor/models/moss_audio.py), [vllm/model_executor/models/mimo_audio.py](../../vllm/vllm/model_executor/models/mimo_audio.py), [vllm/model_executor/models/qwen2_5_omni_thinker.py](../../vllm/vllm/model_executor/models/qwen2_5_omni_thinker.py), [vllm/model_executor/models/ultravox.py](../../vllm/vllm/model_executor/models/ultravox.py), [vllm/model_executor/models/qwen2_audio.py](../../vllm/vllm/model_executor/models/qwen2_audio.py) | Audio-input models |
|
||||
| [vllm/multimodal/registry.py](../../vllm/vllm/multimodal/registry.py), [vllm/multimodal/processing/processor.py](../../vllm/vllm/multimodal/processing/processor.py), [vllm/multimodal/cache.py](../../vllm/vllm/multimodal/cache.py), [vllm/multimodal/audio.py](../../vllm/vllm/multimodal/audio.py), [vllm/multimodal/media/audio.py](../../vllm/vllm/multimodal/media/audio.py), [vllm/multimodal/video.py](../../vllm/vllm/multimodal/video.py), [vllm/multimodal/encoder_budget.py](../../vllm/vllm/multimodal/encoder_budget.py) | Multimodal processing, caches, media I/O |
|
||||
| [vllm/v1/sample/sampler.py](../../vllm/vllm/v1/sample/sampler.py), [vllm/v1/sample/ops/topk_topp_sampler.py](../../vllm/vllm/v1/sample/ops/topk_topp_sampler.py), [vllm/v1/sample/rejection_sampler.py](../../vllm/vllm/v1/sample/rejection_sampler.py) | Sampling |
|
||||
| [vllm/v1/spec_decode/eagle.py](../../vllm/vllm/v1/spec_decode/eagle.py) | Speculative decoding proposer example |
|
||||
| [vllm/v1/structured_output/__init__.py](../../vllm/vllm/v1/structured_output/__init__.py) | Grammar-constrained decoding manager |
|
||||
| [vllm/lora/punica_wrapper/punica_cpu.py](../../vllm/vllm/lora/punica_wrapper/punica_cpu.py) | LoRA kernels on CPU |
|
||||
| [vllm/distributed/device_communicators/base_device_communicator.py](../../vllm/vllm/distributed/device_communicators/base_device_communicator.py), [vllm/distributed/device_communicators/cpu_communicator.py](../../vllm/vllm/distributed/device_communicators/cpu_communicator.py) | Device communicators |
|
||||
| [vllm/device_allocator/cumem.py](../../vllm/vllm/device_allocator/cumem.py) | Sleep-mode allocator |
|
||||
| [vllm/triton_utils/importing.py](../../vllm/vllm/triton_utils/importing.py) | `HAS_TRITON` incl. Triton-CPU detection |
|
||||
| [vllm/envs.py](../../vllm/vllm/envs.py) | Env vars (`VLLM_TARGET_DEVICE`, `VLLM_CPU_*`, `VLLM_USE_V2_MODEL_RUNNER`, `VLLM_PLUGINS`) |
|
||||
| [vllm/entrypoints/llm.py](../../vllm/vllm/entrypoints/llm.py), [vllm/entrypoints/launchers/api_server/entry.py](../../vllm/vllm/entrypoints/launchers/api_server/entry.py), [vllm/entrypoints/cli/main.py](../../vllm/vllm/entrypoints/cli/main.py), [vllm/entrypoints/grpc_server.py](../../vllm/vllm/entrypoints/grpc_server.py), [vllm/entrypoints/anthropic/__init__.py](../../vllm/vllm/entrypoints/anthropic/__init__.py) | Application boundaries |
|
||||
| [vllm/entrypoints/speech_to_text/base/serving.py](../../vllm/vllm/entrypoints/speech_to_text/base/serving.py), [vllm/entrypoints/speech_to_text/realtime/serving.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/serving.py), [vllm/entrypoints/speech_to_text/realtime/connection.py](../../vllm/vllm/entrypoints/speech_to_text/realtime/connection.py) | ASR / realtime endpoints |
|
||||
| [vllm/renderers/base.py](../../vllm/vllm/renderers/base.py), [vllm/renderers/registry.py](../../vllm/vllm/renderers/registry.py), [vllm/tokenizers/registry.py](../../vllm/vllm/tokenizers/registry.py), [vllm/tokenizers/fastokens.py](../../vllm/vllm/tokenizers/fastokens.py), [vllm/inputs/llm.py](../../vllm/vllm/inputs/llm.py), [vllm/outputs.py](../../vllm/vllm/outputs.py), [vllm/reasoning/__init__.py](../../vllm/vllm/reasoning/__init__.py), [vllm/tool_parsers/__init__.py](../../vllm/vllm/tool_parsers/__init__.py), [vllm/plugins/io_processors/interface.py](../../vllm/vllm/plugins/io_processors/interface.py) | I/O layer |
|
||||
| [docs/design/arch_overview.md](../../vllm/docs/design/arch_overview.md), [docs/design/multiprocessing.md](../../vllm/docs/design/multiprocessing.md) | Process architecture |
|
||||
| [docs/design/plugin_system.md](../../vllm/docs/design/plugin_system.md), [docs/contributing/model/registration.md](../../vllm/docs/contributing/model/registration.md), [docs/design/custom_op.md](../../vllm/docs/design/custom_op.md), [docs/design/vllm_ir.md](../../vllm/docs/design/vllm_ir.md) | Extension mechanisms |
|
||||
| [docs/design/torch_compile.md](../../vllm/docs/design/torch_compile.md), [docs/design/cuda_graphs.md](../../vllm/docs/design/cuda_graphs.md), [docs/design/cuda_graphs_multimodal.md](../../vllm/docs/design/cuda_graphs_multimodal.md), [docs/design/optimization_levels.md](../../vllm/docs/design/optimization_levels.md), [docs/design/model_runner_v2.md](../../vllm/docs/design/model_runner_v2.md), [docs/design/dbo.md](../../vllm/docs/design/dbo.md) | Execution design docs |
|
||||
| [docs/design/prefix_caching.md](../../vllm/docs/design/prefix_caching.md), [docs/features/disagg_encoder.md](../../vllm/docs/features/disagg_encoder.md), [docs/features/prompt_embeds.md](../../vllm/docs/features/prompt_embeds.md) | Caching / feature docs |
|
||||
| [docs/getting_started/installation/README.md](../../vllm/docs/getting_started/installation/README.md), [docs/getting_started/installation/cpu.md](../../vllm/docs/getting_started/installation/cpu.md), [docs/getting_started/installation/cpu.arm.inc.md](../../vllm/docs/getting_started/installation/cpu.arm.inc.md), [docs/getting_started/installation/cpu.apple.inc.md](../../vllm/docs/getting_started/installation/cpu.apple.inc.md), [docs/getting_started/installation/cpu.x86.inc.md](../../vllm/docs/getting_started/installation/cpu.x86.inc.md), [docs/getting_started/installation/gpu.apple.inc.md](../../vllm/docs/getting_started/installation/gpu.apple.inc.md), [docs/getting_started/installation/gpu.cuda.inc.md](../../vllm/docs/getting_started/installation/gpu.cuda.inc.md) | Hardware/installation docs |
|
||||
| [docs/features/quantization/README.md](../../vllm/docs/features/quantization/README.md), [docs/features/quantization/gguf.md](../../vllm/docs/features/quantization/gguf.md), [docs/features/quantization/bnb.md](../../vllm/docs/features/quantization/bnb.md) | Quantization support matrix and plugin notes |
|
||||
| [docs/models/supported_models.md](../../vllm/docs/models/supported_models.md), [docs/models/hardware_supported_models/cpu.md](../../vllm/docs/models/hardware_supported_models/cpu.md) | Model lists (audio, transcription, realtime, OpenVLA; CPU-validated list) |
|
||||
| [docs/serving/online_serving/openai_compatible_server.md](../../vllm/docs/serving/online_serving/openai_compatible_server.md), [docs/serving/online_serving/speech_to_text.md](../../vllm/docs/serving/online_serving/speech_to_text.md), [docs/contributing/model/transcription.md](../../vllm/docs/contributing/model/transcription.md), [docs/features/multimodal_inputs.md](../../vllm/docs/features/multimodal_inputs.md) | API and audio docs |
|
||||
| [docs/usage/troubleshooting.md](../../vllm/docs/usage/troubleshooting.md) | Jetson Thor `sm_110a` ptxas note |
|
||||
| [docs/contributing/model/tests.md](../../vllm/docs/contributing/model/tests.md) | Model test guidance |
|
||||
| [tests/v1/core/test_scheduler.py](../../vllm/tests/v1/core/test_scheduler.py), [tests/v1/core/test_prefix_caching.py](../../vllm/tests/v1/core/test_prefix_caching.py), [tests/v1/engine/test_engine_core.py](../../vllm/tests/v1/engine/test_engine_core.py), [tests/v1/streaming_input/test_async_llm_streaming.py](../../vllm/tests/v1/streaming_input/test_async_llm_streaming.py), [tests/kernels/test_onednn.py](../../vllm/tests/kernels/test_onednn.py), [tests/platforms/test_cpu.py](../../vllm/tests/platforms/test_cpu.py), [tests/models/multimodal/processing/test_openvla.py](../../vllm/tests/models/multimodal/processing/test_openvla.py), [tests/plugins/vllm_add_dummy_platform/setup.py](../../vllm/tests/plugins/vllm_add_dummy_platform/setup.py), [tests/plugins_tests/test_endpoint_plugins.py](../../vllm/tests/plugins_tests/test_endpoint_plugins.py) | Representative tests |
|
||||
| [.buildkite/test-pipeline.yaml](../../vllm/.buildkite/test-pipeline.yaml), [.buildkite/hardware_tests/cpu.yaml](../../vllm/.buildkite/hardware_tests/cpu.yaml), [.buildkite/hardware_tests/gh200.yaml](../../vllm/.buildkite/hardware_tests/gh200.yaml), [.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh](../../vllm/.buildkite/scripts/hardware_ci/run-cpu-test-arm.sh), [.buildkite/check-wheel-size.py](../../vllm/.buildkite/check-wheel-size.py), [.github/workflows/macos-smoke-test.yml](../../vllm/.github/workflows/macos-smoke-test.yml) | CI |
|
||||
| [examples/basic/offline_inference/basic.py](../../vllm/examples/basic/offline_inference/basic.py), [examples/generate/multimodal/audio_language_offline.py](../../vllm/examples/generate/multimodal/audio_language_offline.py), [examples/generate/multimodal/vision_language_offline.py](../../vllm/examples/generate/multimodal/vision_language_offline.py), [examples/generate/multimodal/encoder_decoder_multimodal_offline.py](../../vllm/examples/generate/multimodal/encoder_decoder_multimodal_offline.py), [examples/generate/multimodal/qwen2_5_omni/only_thinker.py](../../vllm/examples/generate/multimodal/qwen2_5_omni/only_thinker.py), [examples/generate/multimodal/qwen2_5_omni/README.md](../../vllm/examples/generate/multimodal/qwen2_5_omni/README.md), [examples/speech_to_text/openai/openai_transcription_client.py](../../vllm/examples/speech_to_text/openai/openai_transcription_client.py), [examples/speech_to_text/realtime/openai_realtime_microphone_client.py](../../vllm/examples/speech_to_text/realtime/openai_realtime_microphone_client.py) | Examples |
|
||||
@@ -0,0 +1,575 @@
|
||||
# vLLM-Omni (vllm-project/vllm-omni) — Engine Analysis
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| Local path | `/data/zhoutaichang/embedding_infer/vllm-omni` (links below are relative to `analysis/engines/`, pointing into the sibling `vllm-omni/` checkout) |
|
||||
| Remote | `https://github.com/vllm-project/vllm-omni.git` (origin) |
|
||||
| Branch | `main` |
|
||||
| HEAD | `7be014bce6374f06c95b703763bdbac4c6198f31` — 2026-09-07 "[Bugfix][NPU] Fix MiniMax-H3 INT8 quantization dispatch (#6876)" |
|
||||
| Dirty status | clean (`git status --short` empty) |
|
||||
| Submodules | none (`git submodule status` empty) |
|
||||
| Analysis date | 2026-09-08 |
|
||||
| Installed package caveat | The Python environment has vllm-omni installed **editable from a different checkout**: `/home/zhoutaichang/feature/vllm-omni-main` @ `be335a86f` (2026-08-28), version string `0.28.0rc2.dev22+gbe335a86f`, with `vllm==0.28.0`. This report analyses the newer `7be014bc` tree only; nothing was executed. [Verified] |
|
||||
| Size | 1,382 `.py` files / 508,455 LOC under `vllm_omni/`; 912 `test_*.py` files; 266 files under `docs/`; 90 deploy YAMLs. [Verified] |
|
||||
|
||||
Evidence labels: **[Verified]** read in code/docs; **[Inferred]** reasoned from structure; **[Proposal]** design suggestion; **[Unknown]** not determinable from the tree.
|
||||
|
||||
## 0. Summary
|
||||
|
||||
- vLLM-Omni is a **Python-only serving framework layered on top of vLLM (v1 engine)**; it does not ship its own kernels, allocator, or C++ runtime. Its `setup.py` builds a pure-Python wheel and only selects a platform-specific `requirements/*.txt` (`cuda|rocm|npu|xpu|musa|cpu`). [Verified] (`../../vllm-omni/setup.py`, `../../vllm-omni/requirements/common.txt`)
|
||||
- **vLLM is not pinned in packaging metadata.** No `vllm==` appears in `requirements/` or `pyproject.toml`; instead `version.py` warns at import if major.minor differ, and docs pin `vllm==0.28.0` for the 0.28.x line. [Verified] (`../../vllm-omni/vllm_omni/version.py`, `../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md`)
|
||||
- It **imports 284 distinct `vllm.*` modules** and 104 files import `vllm.v1.*` internals; `patch.py` **monkeypatches** vLLM classes at import (`EngineCoreRequest/Output(s)`, `Request`, `StreamingUpdate`, `TokensPrompt`, `MRotaryEmbedding`, `ModelConfig.is_mm_prefix_lm`, `CuMemAllocator`, NVFP4/FP8 kernels, torch inductor). Backend substitution below vLLM is therefore not a plugin exercise; it is a re-implementation of the AR runtime. [Verified] (`../../vllm-omni/vllm_omni/patch.py`)
|
||||
- Core abstraction: a model is a **frozen `PipelineConfig` of stages** (`LLM_AR`, `LLM_GENERATION`, `DIFFUSION`), and a **`DeployConfig` YAML** places stages on devices, sizes them, and wires named connectors. 60 pipelines are registered in `OMNI_PIPELINES`. [Verified] (`../../vllm-omni/vllm_omni/config/stage_config.py`, `../../vllm-omni/vllm_omni/config/pipeline_registry.py`)
|
||||
- Each AR/generation stage is a **separate vLLM `EngineCoreProc` subprocess** (spawned by `StageEngineCoreProcManager`) reached over ZMQ; diffusion stages run either in a separate `StageDiffusionProc` (ZMQ PUSH/PULL) or inline in the orchestrator process. An `Orchestrator` asyncio loop routes outputs stage→stage. [Verified]
|
||||
- **Inter-stage tensors travel host-side by default** ("D2H2D") through `OmniConnector`s: `SharedMemoryConnector` (POSIX shm + msgpack); Mooncake (TCP/RDMA) and Yuanrong (Ascend) for multi-node, Mori (RDMA, plus an intra-node AMD `xgmi` backend configured in `deploy/qwen3_omni_moe_mori_intranode.yaml`); `MooncakeTransferEngineConnector.put` and the Mori connector pass `torch.Tensor`/`ManagedBuffer` payloads without msgpack serialization and can use device memory pools (repo docs still describe all connectors as D2H2D); none of these device paths is validated for an edge target. [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/`)
|
||||
- **Streaming audio** is produced by `async_chunk`: talker decode steps accumulate codec frames (`codec_chunk_frames`, default 25, with a smaller dynamic first chunk) and push them to the code2wav stage, which decodes chunk-by-chunk while the request sits in a patched `RequestStatus.WAITING_FOR_CHUNK`. [Verified] (`../../vllm-omni/docs/design/feature/async_chunk.md`)
|
||||
- **Platforms actually in tree:** CUDA, ROCm, Ascend NPU (via `vllm_ascend`), Intel XPU, Moore Threads MUSA (via `vllm_musa`). There is **no CPU platform/worker**; `requirements/cpu.txt` exists only for docs builds and the fallback `UnspecifiedOmniPlatform` reports `device_count()==0`. [Verified] (`../../vllm-omni/vllm_omni/platforms/__init__.py`)
|
||||
- **No edge/mobile targets**: no Jetson/ARM/Metal/CoreML/Vulkan/OpenCL/QNN/NNAPI code, no ONNX/TensorRT export path (onnxruntime is used only to load a few speaker-embedding / codec helper models). [Verified]
|
||||
- **TTS is a first-class citizen**: 15 TTS model families, an OpenAI `/v1/audio/speech` endpoint with a per-model `TTSModelAdapter` registry, WebSocket streaming `/v1/audio/speech/stream`, PCM/WAV/MP3/Opus encoding via `soundfile`, and a repo-published Qwen3-TTS TTFP of 64 ms (H200, conc. 1; repo-reported, not reproduced). [Verified]
|
||||
- **VLA is present but server-side**: π0, GR00T-N1.7, InternVLA-A1 (single diffusion stage returning `actions`), DreamZero-DROID and LingBot-World (AR-diffusion world models with engine-level paged KV), Cosmos3 action modes; served over `WS /v1/realtime/robot/openpi` (msgpack). Actions are not streamed; one observation → one action chunk. [Verified]
|
||||
- Diffusion has its own runtime (`DiffusionEngine`, `RequestScheduler`/`StepScheduler`, `DiffusionWorker`, 11 attention backends, cache-DiT/TeaCache/MagCache/StepCache, CPU/layerwise/distributed offload, HSDP/Ulysses/Ring/CFG/VAE parallel) that reuses vLLM layers/quant/`KVCacheManager` as a library. [Verified]
|
||||
- Compilation: AR stages inherit vLLM CUDA graphs/`torch.compile`; diffusion uses `regionally_compile()` (`torch.compile` per repeated block) and per-model CUDA-graph wrappers (e.g., `CUDAGraphDecoderWrapper` for Qwen3-TTS code2wav). [Verified]
|
||||
- Quantization is inherited from vLLM (`quantization_config` path) with omni additions (Int8, MXFP4/MXFP8 for NPU, SVDQuant, quack FP8, bitsandbytes); non-AR TTS stages stay BF16 per docs. [Verified] (`../../vllm-omni/docs/user_guide/quantization/overview.md`)
|
||||
- Bottom line for the edge study: vLLM-Omni is the reference **serving stack** (pipeline/stage/streaming semantics, model ports, API contracts) but its **execution runtime is vLLM+CUDA-class accelerators**; an edge port must replace the AR runtime (vLLM EngineCore/scheduler/worker) and the diffusion worker while keeping `PipelineConfig`, stage input processors, connector payload schemas, and the OpenAI/OpenPI protocol layers. [Inferred]
|
||||
|
||||
## 1. Architecture overview
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Entry["Entrypoints (vllm_omni/entrypoints)"]
|
||||
CLI["vllm serve --omni<br/>cli/serve.py"]
|
||||
API["OpenAI-compatible FastAPI<br/>openai/api_server.py"]
|
||||
OFF["Omni / AsyncOmni<br/>omni.py / async_omni.py"]
|
||||
DUP["Duplex + Realtime WS<br/>duplex/*, openai/realtime_connection.py"]
|
||||
OPI["OpenPI robot WS<br/>openpi/serving.py"]
|
||||
end
|
||||
subgraph Ctrl["Control plane (vllm_omni/engine)"]
|
||||
ENG["AsyncOmniEngine<br/>async_omni_engine.py"]
|
||||
ORCH["Orchestrator<br/>orchestrator.py"]
|
||||
RT["StageRuntime / DistStageRuntime<br/>stage_runtime.py"]
|
||||
POOL["StagePool (per logical stage)<br/>stage_pool.py"]
|
||||
MS["OmniMasterServer (ZMQ ROUTER)<br/>stage_engine_startup.py"]
|
||||
end
|
||||
subgraph Cfg["Config (vllm_omni/config)"]
|
||||
PC["PipelineConfig (frozen topology)<br/>stage_config.py"]
|
||||
DC["DeployConfig YAML<br/>vllm_omni/deploy/*.yaml"]
|
||||
REG["OMNI_PIPELINES<br/>pipeline_registry.py"]
|
||||
end
|
||||
subgraph AR["AR / generation stage (subprocess)"]
|
||||
ECP["StageEngineCoreProc (vLLM EngineCoreProc)"]
|
||||
SCH["OmniARScheduler / OmniGenerationScheduler<br/>core/sched/"]
|
||||
WRK["GPUARWorker / GPUGenerationWorker<br/>worker/"]
|
||||
MR["GPUARModelRunner / GPUGenerationModelRunner"]
|
||||
MODEL["Omni model classes<br/>model_executor/models/*"]
|
||||
end
|
||||
subgraph DIFF["Diffusion stage (subprocess or inline)"]
|
||||
SDP["StageDiffusionProc (ZMQ)"]
|
||||
DE["DiffusionEngine + RequestScheduler/StepScheduler"]
|
||||
DW["DiffusionWorker / DiffusionModelRunner"]
|
||||
PIPE["Pipelines<br/>diffusion/models/*"]
|
||||
end
|
||||
CONN["OmniConnector (SHM / Mooncake / Mori / Yuanrong)<br/>distributed/omni_connectors"]
|
||||
PLAT["OmniPlatform (cuda/rocm/npu/xpu/musa)<br/>platforms/"]
|
||||
VLLM["vLLM 0.28.x (patched by vllm_omni/patch.py)"]
|
||||
|
||||
CLI --> API --> ENG
|
||||
OFF --> ENG
|
||||
DUP --> ENG
|
||||
OPI --> ENG
|
||||
ENG --> ORCH
|
||||
ENG --> RT
|
||||
RT --> POOL
|
||||
RT --> MS
|
||||
REG --> PC --> ENG
|
||||
DC --> ENG
|
||||
POOL -->|ZMQ| ECP
|
||||
POOL -->|ZMQ / inline| SDP
|
||||
ECP --> SCH --> WRK --> MR --> MODEL
|
||||
SDP --> DE --> DW --> PIPE
|
||||
MR -->|put/get chunks, KV| CONN
|
||||
DW -->|put/get| CONN
|
||||
WRK --> PLAT
|
||||
DW --> PLAT
|
||||
ECP --> VLLM
|
||||
MR --> VLLM
|
||||
```
|
||||
|
||||
Key modules:
|
||||
|
||||
| Module | Path | Responsibility | Key symbols |
|
||||
| --- | --- | --- | --- |
|
||||
| Package init / patches | `../../vllm-omni/vllm_omni/__init__.py`, `../../vllm-omni/vllm_omni/patch.py` | Version check, then monkeypatch vLLM internals at import | `warn_if_misaligned_vllm_version`, `_patched_is_mm_prefix_lm`, `_patch_cumem_free_callback_cuda` |
|
||||
| Engine args | `../../vllm-omni/vllm_omni/engine/arg_utils.py` | `OmniEngineArgs(EngineArgs)` per-stage fields; `OrchestratorArgs`; registers omni models into vLLM via entry point | `OmniEngineArgs`, `OrchestratorArgs`, `register_omni_models_to_vllm` |
|
||||
| Composition root | `../../vllm-omni/vllm_omni/engine/async_omni_engine.py` | Resolves stage configs, boots stages, hosts orchestrator loop, request submit/abort/RPC/sleep | `AsyncOmniEngine`, `_create_default_diffusion_stage_cfg`, `add_request_async`, `abort_async` |
|
||||
| Orchestrator | `../../vllm-omni/vllm_omni/engine/orchestrator.py` | Cross-stage routing, output ordering, cancellation, CFG companions, PD pair | `Orchestrator`, `_route_output`, `_forward_to_next_stage_unguarded`, `_prewarm_async_chunk_stages`, `_handle_abort` |
|
||||
| Stage runtime | `../../vllm-omni/vllm_omni/engine/stage_runtime.py`, `../../vllm-omni/vllm_omni/engine/stage_pool.py` | Expand logical stages into local/remote replicas; replica selection/affinity; metrics | `StageRuntime`, `DistStageRuntime`, `StagePool.pick`, `submit_initial` |
|
||||
| Stage process launch | `../../vllm-omni/vllm_omni/engine/stage_engine_startup.py`, `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc.py`, `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc_manager.py` | ZMQ address allocation, headless replica registration, spawn of `EngineCoreProc` subclasses | `OmniMasterServer`, `StageEngineCoreProc.run_stage_core`, `launch_stage_replica` |
|
||||
| Stage client | `../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py` | vLLM `AsyncMPClient` subclass per replica; KV sender endpoint | `StageEngineCoreClient`, `DPLBStageEngineCoreClient` |
|
||||
| Pipeline/deploy config | `../../vllm-omni/vllm_omni/config/stage_config.py`, `../../vllm-omni/vllm_omni/config/pipeline_registry.py`, `../../vllm-omni/vllm_omni/config/omni_config.py` | Frozen topology, deploy overlay, precedence, scheduler resolution | `PipelineConfig`, `StagePipelineConfig`, `StageDeployConfig`, `StageExecutionType`, `_resolve_scheduler`, `OMNI_PIPELINES`, `register_pipeline` |
|
||||
| AR scheduler | `../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py`, `../../vllm-omni/vllm_omni/core/sched/omni_scheduler_mixin.py` | vLLM `Scheduler` subclass adding chunk/KV-transfer hooks | `OmniARScheduler`, `OmniARAsyncScheduler`, `OmniSchedulerMixin` |
|
||||
| Generation scheduler | `../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py` | One-shot non-AR stage scheduling (code2wav, DiT-in-vLLM) | `OmniGenerationScheduler.schedule` |
|
||||
| Workers/runners | `../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py`, `../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py`, `../../vllm-omni/vllm_omni/worker/omni_connector_model_runner_mixin.py` | vLLM `GPUWorker`/`GPUModelRunner` subclasses; multimodal outputs, talker MTP, connector I/O | `GPUARWorker`, `GPUARModelRunner.execute_model`, `GPUGenerationModelRunner`, `OmniConnectorModelRunnerMixin.send_chunk` |
|
||||
| Model registry | `../../vllm-omni/vllm_omni/model_executor/models/registry.py`, `../../vllm-omni/vllm_omni/diffusion/registry.py` | 89 omni AR/generation archs + all vLLM archs; 66 diffusion pipelines | `OmniModelRegistry`, `DiffusionModelRegistry`, `register_diffusion_model` |
|
||||
| Stage input processors | `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/` | Model-owned functions converting stage-N output → stage-N+1 input (full payload or async chunk) | `talker2code2wav_async_chunk` (qwen3_tts.py) |
|
||||
| Diffusion engine | `../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py`, `../../vllm-omni/vllm_omni/diffusion/sched/`, `../../vllm-omni/vllm_omni/diffusion/executor/`, `../../vllm-omni/vllm_omni/diffusion/worker/` | Request/step batching, uniproc/multiproc executors, worker & runner | `DiffusionEngine`, `RequestScheduler`, `StepScheduler`, `MultiprocDiffusionExecutor`, `DiffusionWorker`, `DiffusionModelRunner` |
|
||||
| Diffusion attention | `../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py`, `../../vllm-omni/vllm_omni/diffusion/attention/selector.py`, `../../vllm-omni/vllm_omni/diffusion/attention/layer.py` | Enum registry, per-role selection, platform defaults | `DiffusionAttentionBackendEnum`, `register_diffusion_backend`, `get_attn_backend_for_role`, `Attention` |
|
||||
| Connectors | `../../vllm-omni/vllm_omni/distributed/omni_connectors/` | Transport-only put/get + chunk transfer adapter | `OmniConnectorBase`, `SharedMemoryConnector`, `OmniConnectorFactory`, `OmniChunkTransferAdapter` |
|
||||
| Platforms | `../../vllm-omni/vllm_omni/platforms/interface.py`, `../../vllm-omni/vllm_omni/platforms/__init__.py` | Backend detection, worker class selection, diffusion attention defaults | `OmniPlatform`, `resolve_current_omni_platform_cls_qualname`, `CudaOmniPlatform`, `NPUOmniPlatform` |
|
||||
| I/O types | `../../vllm-omni/vllm_omni/inputs/data.py`, `../../vllm-omni/vllm_omni/outputs/__init__.py`, `../../vllm-omni/vllm_omni/data_entry_keys.py`, `../../vllm-omni/vllm_omni/request.py` | Prompt types with embeds/payloads; `OmniRequestOutput`; typed inter-stage payload | `OmniTokensPrompt`, `OmniRequestOutput`, `OmniPayload`, `OmniRequest` |
|
||||
| Serving | `../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py`, `../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py` | HTTP/WS routes; TTS request building & audio encoding; robot policy WS | `omni_run_server`, `create_speech`, `ServingRealtimeRobotOpenPI.infer` |
|
||||
| Experimental | `../../vllm-omni/vllm_omni/experimental/ar_diffusion/engine.py`, `../../vllm-omni/vllm_omni/experimental/world_models/` | AR-diffusion (world model) engine with paged KV; session state | `ARDiffusionEngine`, `ARDiffusionModelRunner` |
|
||||
|
||||
## 2. Device layer
|
||||
|
||||
**Backends and registration.** `vllm_omni/platforms/__init__.py` defines five builtin probe functions and an entry-point group `vllm_omni.platform_plugins` for out-of-tree platforms; exactly one may activate. [Verified] (`../../vllm-omni/vllm_omni/platforms/__init__.py` — `builtin_omni_platform_plugins`, `resolve_current_omni_platform_cls_qualname`)
|
||||
|
||||
| Platform | Probe | Class (inherits vLLM/vendor platform) | AR/generation worker | Notes |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| CUDA | `pynvml.nvmlDeviceGetCount()>0` | `CudaOmniPlatform(OmniPlatform, CudaPlatformBase)` `../../vllm-omni/vllm_omni/platforms/cuda/platform.py` | `GPUARWorker`, `GPUGenerationWorker` | Default deploy dir `vllm_omni/deploy`; diffusion attention default chain (TRTLLM→cuDNN→FlashInfer→FA→SDPA) keyed on SM version [Verified] |
|
||||
| ROCm | `amdsmi` handles>0 | `RocmOmniPlatform(OmniPlatform, RocmPlatform)` `../../vllm-omni/vllm_omni/platforms/rocm/platform.py` | same GPU workers | Forces `TRITON_ATTN`/`ROCM_AITER_FA` for AR; `patch_groupnorm.py`; several deploy YAMLs flip code2wav to eager (MIOpen not capture-safe) [Verified] |
|
||||
| Ascend NPU | `torch.npu.is_available()` | `NPUOmniPlatform(OmniPlatform, vllm_ascend.NPUPlatform)` `../../vllm-omni/vllm_omni/platforms/npu/platform.py` | `NPUARWorker`, `NPUGenerationWorker` (`../../vllm-omni/vllm_omni/platforms/npu/worker/`) | HCCL; applies model patches for code2wav conv ops, 310P patches; `YuanrongTransferEngineConnector`; ACL graph wrapper [Verified] |
|
||||
| Intel XPU | `torch.xpu.is_available()` | `XPUOmniPlatform(OmniPlatform, XPUPlatform)` `../../vllm-omni/vllm_omni/platforms/xpu/platform.py` | `XPUARWorker`, `XPUGenerationWorker` | Disables XPU sampler kernel via env [Verified] |
|
||||
| MUSA | `torchada.is_musa_platform()` | `MUSAOmniPlatform(OmniPlatform, vllm_musa.MUSAPlatformBase)` `../../vllm-omni/vllm_omni/platforms/musa/platform.py` | GPU workers | requires `torchada`, `mate`, `flash_attn_3` (`../../vllm-omni/requirements/musa.txt`) [Verified] |
|
||||
| CPU | — | `UnspecifiedOmniPlatform` (`device_type="cpu"`, `get_device_count()==0`) | none | No CPU worker/runner exists; `cli/serve.py` falls back to vLLM `CpuPlatform` **only for argument parsing** [Verified] |
|
||||
|
||||
**Device discovery/selection.** Placement is declarative: each stage's `devices: "0,1"` in the deploy YAML must equal TP×DP_local×PP (×replicas); `StageRuntime._scoped_spawn_device_env` sets the platform's `device_control_env_var` (e.g., `CUDA_VISIBLE_DEVICES`) around subprocess spawn, and `GPUARWorker.init_device` maps local rank → visible device via `current_omni_platform.get_torch_device`. [Verified] (`../../vllm-omni/vllm_omni/engine/stage_runtime.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py`, `../../vllm-omni/docs/configuration/stage_configs.md`)
|
||||
|
||||
**Memory allocation/pools.**
|
||||
- AR/generation stages inherit vLLM's paged KV allocation; `OmniGPUWorkerBase.determine_available_memory` re-implements the budget as `total*gpu_memory_utilization − (weights + peak activation + non-torch)` measured by `memory_profiling` around `profile_run()`, or honors explicit `kv_cache_memory_bytes`. [Verified] (`../../vllm-omni/vllm_omni/worker/base.py`)
|
||||
- Multi-stage co-location on one GPU is guarded by `stage_admission.py` (`DeviceLedger`, `check_admission`) using per-stage `gpu_memory_utilization` plus a CUDA-graph reserve, and by SH/EX device locks for sequential profiling (`parallel_stage_init`). [Verified] (`../../vllm-omni/vllm_omni/engine/stage_admission.py`)
|
||||
- Sleep mode uses vLLM's `CuMemAllocator` tagged pools (`_maybe_get_memory_pool_context`), patched in `patch.py` to avoid a CUDA double-free on shutdown. [Verified]
|
||||
- Diffusion workers: `DiffusionWorker.init_device` calls `init_distributed_environment`/`initialize_model_parallel` from `vllm_omni.diffusion.distributed`, and only takes a `MemorySnapshot`/KV budget when `diffusion_kv_mode=paged_scheduler`. Host-side: pinned memory for offload (`pin_cpu_memory`), CUDA host registration for DLO, and the Host Weight Runtime (mmap'd safetensors shared across same-host workers). [Verified] (`../../vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py`, `../../vllm-omni/vllm_omni/diffusion/offloader/`, `../../vllm-omni/docs/design/feature/host_weight_runtime.md`)
|
||||
- NVML process-scoped accounting helpers remain for diffusion memory reporting. [Verified] (`../../vllm-omni/vllm_omni/worker/gpu_memory_utils.py`)
|
||||
|
||||
**Host↔device transfers.** Inter-stage data is always staged to host: connector `put()` serializes CPU tensors (msgpack, `OmniMsgpackEncoder`, "zero-copy not implemented yet") into a POSIX shm segment; the receiving stage deserializes and re-uploads. `GPUARModelRunner` snapshots multimodal outputs to CPU on a dedicated copy stream (`_get_or_create_omni_payload_copy_stream`, `_snapshot_tensor_payload_to_cpu_async`) so D2H does not block decode ("async omni output materialization"). [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py`, `../../vllm-omni/docs/design/feature/omni_async_output_materialization.md`)
|
||||
|
||||
**Synchronization/streams.** `OmniPlatform.record_device_event()` marks tensor readiness for async diffusion output (NPU overrides to sync the default stream first, since HCCL uses hidden streams). Prefix cache writes record a CUDA event on the copy stream (`_PendingAsyncWrite`). Ruff bans `torch.cuda.*` in favor of `torch.accelerator.*` to keep code device-neutral. [Verified] (`../../vllm-omni/vllm_omni/platforms/interface.py`, `../../vllm-omni/vllm_omni/core/prefix_cache.py`, `../../vllm-omni/pyproject.toml`)
|
||||
|
||||
**Portability.** Linux only (`docs/getting_started/quickstart.md`: "OS: Linux"); Python 3.10–3.13; Dockerfiles for cuda/rocm/xpu/npu (A2/A3). No macOS/Windows/ARM CPU/mobile code paths were found. [Verified] (`../../vllm-omni/docker/Dockerfile.cuda`, `../../vllm-omni/docker/Dockerfile.npu`)
|
||||
|
||||
## 3. Kernel layer
|
||||
|
||||
**Operator set & registration.** vLLM-Omni has no compiled extension of its own; operators come from (a) vLLM's layers (`vllm.model_executor.layers.{linear,layernorm,activation,rotary_embedding,fused_moe,quantization}` — 62/25/15/9/5/17 import sites respectively) and vLLM's custom-op/IR mechanism, (b) PyTorch/Triton code inside model files (e.g., `attention/fish_kvcache_triton.py`, `diffusion/layers/mot/ops/mot_gemm.py`, `diffusion/layers/fused_qk_norm_rope.py`), and (c) optional external kernel packages (flash-attn 2/3/4, FlashInfer, SageAttention 2/3, cuDNN SDPA, HF `kernels` hub, quack FP8, TRT-LLM gen). [Verified]
|
||||
|
||||
**Dispatch mechanisms.**
|
||||
- AR stages: attention/backend dispatch is vLLM's (`attention_backend` engine arg, forced to `TRITON_ATTN`/`ROCM_AITER_FA` on ROCm by `stage_init_utils.py`). Omni adds a Fish-Speech-specific KV-cache attention Triton path selected by model arch (`is_fish_kvcache_attn_active_for_model`) with fallback counters. [Verified] (`../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py`)
|
||||
- Diffusion attention: `Attention` layer → `get_attn_backend_for_role(role, head_size, attention_config)` with precedence per-role config → role category → global default → `current_omni_platform.get_diffusion_attn_backend_cls()`; classes resolved from `DiffusionAttentionBackendEnum` (11 members) with runtime overrides via `register_diffusion_backend`. Each backend declares `supported_platforms`, `validate_available()`, head-size sets, mask/paged-KV capabilities. [Verified] (`../../vllm-omni/vllm_omni/diffusion/attention/selector.py`, `../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py`, `../../vllm-omni/vllm_omni/diffusion/attention/backends/abstract.py`)
|
||||
- Diffusion generic ops: `CustomOp.dispatch_forward()` picks `forward_cuda|hip|npu|xpu|musa|native` from `current_omni_platform`; HIP/MUSA default to the CUDA path, XPU to native. [Verified] (`../../vllm-omni/vllm_omni/diffusion/layers/custom_op.py`)
|
||||
- IR op priority: CUDA platform prefers `["vllm_c","native"]` (optionally `oink` for rms_norm); per-pipeline `get_*_ir_op_priority_func` (Cosmos3) can merge overrides. [Verified] (`../../vllm-omni/vllm_omni/platforms/cuda/platform.py`, `../../vllm-omni/vllm_omni/diffusion/registry.py`)
|
||||
|
||||
**Quantization formats & where dequant happens.** Unified `quantization_config` path (`../../vllm-omni/vllm_omni/quantization/factory.py`); vLLM quant methods are reused (ModelOpt FP8/NVFP4, AutoRound W4A16, TorchAO, compressed-tensors) and omni adds `int8_config.py`, `mxfp4_config.py`, `mxfp8_config.py` (Ascend), `svdquant_config.py`, `bitsandbytes_config.py`, `inc_config.py`, `quack_fp8.py` (Blackwell fused-bias FP8 GEMM). Dequant/scaled-GEMM occurs inside vLLM's `LinearMethod.apply`/`ScaledMMLinearKernel` (patched in `patch.py` for non-contiguous batched diffusion activations and FlashInfer output shape). GGUF loading for diffusion via `gguf_adapters` (`requirements/common.txt`). Runtime FP8 attention quant for diffusion FA (`AttnQuantSpec`). Docs state omni/TTS quantization is scoped to the AR LM stage; non-AR stages stay BF16. [Verified] (`../../vllm-omni/docs/user_guide/quantization/overview.md`, `../../vllm-omni/vllm_omni/quantization/`)
|
||||
|
||||
**Compilation / JIT / graph capture.**
|
||||
- AR stages: vLLM `compilation_config` (`cudagraph_mode: PIECEWISE|FULL`, `custom_ops`) is settable per stage in deploy YAML (see `platforms.npu.stages[*].compilation_config` in `../../vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml`). `GPUARModelRunner._capture_talker_mtp_graphs` captures a dedicated CUDA graph for the talker MTP/code-predictor path; `supports_talker_mtp_graph_capture()` is a platform hook. NPU uses `ACLGraphWrapper` via `get_graph_wrapper_cls()`. [Verified]
|
||||
- Generation stages (code2wav): model-owned wrappers, e.g., `CUDAGraphDecoderWrapper` with capture sizes derived from chunk ramp/left context (`decode_cudagraph_capture_sizes`, `decode_cudagraph_batch_sizes` connector extras). [Verified] (`../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py`, `../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml`)
|
||||
- Diffusion: `regionally_compile()` applies `torch.compile` to each repeated block (`_repeated_blocks`), keeping offload hook wrappers outside the graph; `diffusion_compile_granularity="regional"`, `diffusion_compile_dynamic=True` defaults; `enforce_eager` disables. `patch.py` also patches torch inductor's `statically_known_multiple_of` for FLUX fp8 dynamic shapes. [Verified] (`../../vllm-omni/vllm_omni/diffusion/compile.py`, `../../vllm-omni/docs/user_guide/diffusion/regional_compilation.md`)
|
||||
|
||||
**Fallback when an op is unsupported.** Diffusion attention: explicit selection of an unavailable backend raises at platform resolution (fail-fast); automatic selection degrades to `TORCH_SDPA`. `CustomOp.forward_native` is the last resort. Fish KV attention records fallback reasons and uses vLLM attention otherwise. Platform-specific model patches (NPU code2wav conv2d, ROCm groupnorm) replace individual ops. [Verified]
|
||||
|
||||
**Extension points.**
|
||||
- Out-of-tree platform: entry point `vllm_omni.platform_plugins` returning a class path; general plugins `vllm_omni.general_plugins`. [Verified] (`../../vllm-omni/vllm_omni/plugins/__init__.py`)
|
||||
- Attention backend: `register_diffusion_backend(enum, "pkg.Class")`. [Verified]
|
||||
- Model: `register_diffusion_model(arch, module, cls, pre/post-process funcs)`; AR models via `OmniModelRegistry` (vLLM `_ModelRegistry`); pipelines via `register_pipeline`. [Verified]
|
||||
- Connector: `OmniConnectorFactory.register_connector(name, ctor)`. [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py`)
|
||||
- Custom diffusion pipeline without registry edits: `diffusion_load_format="dummy"` + `custom_pipeline_args={"pipeline_class": ...}` + `CustomPipelineWorkerExtension`. [Verified] (`../../vllm-omni/docs/features/custom_pipeline.md`)
|
||||
- TTS serving adapter: `@register_tts_adapter` subclasses of `TTSModelAdapter` (`ARTTSAdapter`, `DiffusionTTSAdapter`). [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/__init__.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/base.py`)
|
||||
- Contributor guides: `../../vllm-omni/docs/contributing/model/adding_omni_model.md`, `../../vllm-omni/docs/contributing/model/adding_tts_model.md`, `../../vllm-omni/docs/contributing/model/adding_diffusion_model.md`. [Verified]
|
||||
|
||||
## 4. Model runner
|
||||
|
||||
**Model loading / formats.**
|
||||
- AR/generation stages: standard vLLM loader path (HF `config.json` + safetensors; `trust_remote_code`); the omni arch is looked up in `OmniModelRegistry` (vLLM archs + `_OMNI_MODELS`), and `register_omni_models_to_vllm` is installed as a `vllm.general_plugins` entry point so vLLM worker subprocesses that only import `vllm` still see omni archs. Per-stage `hf_config_name`, `model_subdir`, `tokenizer_subdir`, `model_path_resolver` select sub-configs inside multi-component repos. [Verified] (`../../vllm-omni/pyproject.toml`, `../../vllm-omni/vllm_omni/config/stage_config.py`)
|
||||
- Diffusion stages: `DiffusersPipelineLoader` reads Diffusers-style `model_index.json` repos (safetensors index or unindexed shards, GGUF via `download_gguf`, ModelOpt/direct-mmap checkpoint adapters, multithreaded weight load, HWR host-weight plan); `diffusion_load_format ∈ {default, dummy, diffusers}` where `diffusers` runs the stock `diffusers` pipeline through `DiffusersAdapterPipeline`. Some VLA pipelines self-load (π0 loads `model.safetensors` in `__init__`; GR00T uses `AutoModel`/`AutoProcessor`). [Verified] (`../../vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py`, `../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py`, `../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py`)
|
||||
- No export/conversion pipeline exists (no ONNX/TensorRT/GGUF-export tooling; GGUF is consumed, not produced). [Verified]
|
||||
- Pipeline auto-detection: `StageConfigFactory.create_from_model()` maps HF `model_type`/`architectures`/`_class_name` to `OMNI_PIPELINES`; single-stage diffusion models without a registered pipeline use `_create_default_diffusion_stage_cfg`. [Verified] (`../../vllm-omni/vllm_omni/config/config_factory.py`, `../../vllm-omni/vllm_omni/engine/async_omni_engine.py`)
|
||||
|
||||
**Execution lifecycle (init → warmup → step).**
|
||||
1. `AsyncOmniEngine.__init__` → `_resolve_stage_configs` → `StageRuntime.initialize()` builds `LogicalStageInitPlan`/`ReplicaInitPlan`, runs admission, spawns replicas (LLM via `launch_stage_replica` → `StageEngineCoreProcManager` → `StageEngineCoreProc.run_stage_core`; diffusion via `StageDiffusionProcManager.launch_headless` or `InlineStageDiffusionClient`). [Verified]
|
||||
2. Inside an LLM stage, vLLM's `EngineCore` init runs `init_device` → `load_model` → `determine_available_memory` (profile run) → `initialize_kv_cache` → `compile_or_warm_up_model` (CUDA graph capture); omni overrides `load_model` (`_init_talker_mtp`, `_prewarm_attention_capture_workspaces`) and `capture_model` (`_capture_talker_mtp_graphs`). [Verified] (`../../vllm-omni/vllm_omni/worker/gpu_model_runner.py`, `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py`)
|
||||
3. Diffusion: `DiffusionEngine.make_engine()` → executor init → `load_model` → `run_startup_warmup()` (`_dummy_run`, KV profile requests when paged) → `_busy_loop` thread performing `schedule → execute_batch/execute_step → update_from_output → per-request asyncio queue`. [Verified] (`../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py`, `../../vllm-omni/docs/design/feature/diffusion_continuous_batching.md`)
|
||||
4. Orchestrator `_prewarm_async_chunk_stages` pre-submits downstream stage requests when `async_chunk` is on, so they can park in `WAITING_FOR_CHUNK`. [Verified]
|
||||
5. Step: AR stage `GPUARModelRunner.execute_model` → `_preprocess` (model-specific `preprocess` builds `inputs_embeds`; `_build_model_kwargs_extra`) → `_model_forward` → `sample_tokens` (may run `_run_post_sample_talker_mtp`) → `_build_omni_step_outputs` → `OmniModelRunnerOutput` with `multimodal_outputs`/`pooler_output`. [Verified]
|
||||
|
||||
**Dynamic shapes.** AR stages rely on vLLM's padded batch buckets (`max_cudagraph_capture_size`, `cudagraph_mode`); talker MTP graphs are captured per batch bucket. Code2wav graph shapes are derived from the chunk-size schedule (`initial_codec_chunk_frames`, `codec_chunk_frames`, `codec_left_context_frames`, optional `codec_chunk_ramp`/adaptive ramp) and batch buckets; eager fallback on unmatched shapes (`_get_padded_size` returns None). Diffusion uses `diffusion_compile_dynamic=True` and per-request `DiffusionRequestBatch` slicing. [Verified]
|
||||
|
||||
**State & KV cache.**
|
||||
- AR/generation: inherits vLLM paged KV (`KVCacheManager`, block tables, prefix caching, chunked prefill, preemption). Omni additions: `OmniTensorPrefixCache` stores hidden states and multimodal tensors aligned to vLLM block slot mappings so prefix hits reuse stage outputs (`../../vllm-omni/vllm_omni/core/prefix_cache.py`, `../../vllm-omni/docs/design/feature/prefix_caching.md`); `retains_state_across_chunks` keeps a request resident while awaiting chunks; KV transfer between stages (`omni_kv_config`, `send_kv_cache`/`recv_kv_cache`, rank-aware keys) for PD disaggregation and thinker→talker reuse. [Verified]
|
||||
- Across calls (sessions): streaming input sessions (`OmniStreamingUpdate`, `resumable` requests, `session_id`) and duplex sessions retain KV between segments; `session_mode: duplex` in deploy YAML. [Verified] (`../../vllm-omni/vllm_omni/request.py`, `../../vllm-omni/docs/design/fullduplex.md`)
|
||||
- Diffusion: `DiffusionKVCacheMode ∈ {dense_legacy, paged_scheduler}`; `DiffusionKVCacheManager` wraps vLLM `KVCacheManager` (caching disabled) for scheduler-owned paged KV in AR-DiT/world models ("unified AR/DiT paged KV cache runtime", 0.28.0). Experimental `ARDiffusionEngine` keeps runner-owned paged KV per session with sliding window + sink frames (`ARDiffusionKVCacheSpec`). Prompt-embedding cache (`enable_prompt_embed_cache`). Eviction: LRU-style session accounting in `experimental/world_models/session_state`. [Verified] (`../../vllm-omni/vllm_omni/diffusion/diffusion_kv/manager.py`, `../../vllm-omni/vllm_omni/experimental/ar_diffusion/capability.py`)
|
||||
|
||||
**Multimodal stages.**
|
||||
- Encoders (audio/vision) live inside the thinker model class (vLLM multimodal processors, `mm_features`); `requires_multimodal_data` lets a downstream stage receive raw media. [Verified]
|
||||
- Projectors/talkers: `LLM_AR` stage with `engine_output_type="latent"` emitting codec codes via `multimodal_outputs`. [Verified]
|
||||
- Decoders/vocoders: `LLM_GENERATION` stage (`GPUGenerationModelRunner`: no logits/sampling, returns tensors via `pooler_output`) e.g. `Qwen3TTSCode2Wav`, `Qwen3OmniMoeCode2Wav`, `MossTTSCodecDecoder`, `FishSpeechDACDecoder`, `IndexTTS2S2MelDecoder`; or a DIFFUSION stage for flow-matching decoders (`CosyVoice3` DiT, `MiniMaxMusic3Acoustic`, `OmniVoicePipeline`). [Verified] (`../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py`, `../../vllm-omni/vllm_omni/model_executor/models/registry.py`)
|
||||
- Diffusion (image/video/audio/action) as DIFFUSION stage pipelines. [Verified]
|
||||
|
||||
## 5. Scheduler
|
||||
|
||||
### (a) Request / token / pipeline-level scheduling
|
||||
|
||||
- **Per-stage continuous batching** is inherited from vLLM: each LLM stage runs its own `Scheduler` subclass chosen by `_resolve_scheduler(execution_type, async_scheduling)`: `OmniARScheduler` (sync), `OmniARAsyncScheduler` (vLLM `AsyncScheduler`), `OmniGenerationScheduler`; diffusion stages return `None` (own scheduler). Batching knobs are per stage (`max_num_seqs`, `max_num_batched_tokens`, `enable_chunked_prefill`, `enable_prefix_caching`, `async_scheduling`). [Verified] (`../../vllm-omni/vllm_omni/config/stage_config.py`)
|
||||
- **AR schedule hooks**: `OmniARScheduler.schedule` drops aborted requests, `_process_pending_omni_inputs`, optionally defers waiting admission, calls `super().schedule()`, restores omni wait queues, post-processes with cached payloads, and wraps output with `finished_requests_needing_kv_transfer`. `update_from_output` triggers `chunk_transfer_adapter.save_async` per decode step and handles KV-transfer readiness. [Verified] (`../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py`)
|
||||
- **Generation schedule** (`OmniGenerationScheduler.schedule`): one-shot fast path feeding all input tokens at once (placeholder token if 0), chunk-aware: skips placeholder tokens when no new chunk arrived, finishes when `is_done_receiving_chunks`. [Verified]
|
||||
- **Chunk lifecycle** (`OmniChunkTransferAdapter`): keys `{req_id}_{stage_id}_{chunk_id}`; background `recv_loop`/`save_loop` threads; `process_pending_chunks` parks requests in `WAITING_FOR_CHUNK` (enum value −1 added by `patch.py`), `restore_queues` re-admits when ready, `_preempt_non_active_running` with `active_stream_window`. Full-payload (non-async) handoff uses `WAITING_FOR_INPUT` (−2) and `OmniSchedulingCoordinator`. [Verified] (`../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py`, `../../vllm-omni/vllm_omni/core/sched/omni_scheduling_coordinator.py`)
|
||||
- **Pipeline-level**: `Orchestrator._orchestration_loop` (poll) or `_orchestration_loop_event_driven` (`VLLM_OMNI_EVENT_DRIVEN_ORCH=1`) reads each replica's `EngineCoreOutputs`, runs `MultimodalOutputProcessor`, then `_route_output`: final-output stages → `OutputMessage` to client queue; non-final → `_forward_to_next_stage` (unless the next stage receives async chunks, in which case data flows via connectors and the stage was pre-submitted). Diffusion next stages get inputs via `custom_process_input_func(source_outputs, prompt, requires_multimodal_data)`; CFG companions are bundled (`CfgCompanionTracker`); PD pairs forward original prompt + `kv_transfer_params`. [Verified] (`../../vllm-omni/vllm_omni/engine/orchestrator.py`, `../../vllm-omni/vllm_omni/engine/cfg_companion_tracker.py`)
|
||||
- **Replica selection / load balancing**: `StagePool.pick`/`select_replica_id` with `LoadBalancer` policies (`--omni-lb-policy`, default `random`), request→replica binding for affinity; `OmniCoordinator` tracks heartbeats for distributed replicas. [Verified] (`../../vllm-omni/vllm_omni/engine/stage_pool.py`, `../../vllm-omni/vllm_omni/distributed/omni_coordinator/omni_coordinator.py`)
|
||||
- **Diffusion scheduling**: `REQUEST_BATCH` (`RequestScheduler`: fuse compatible requests by `RequestBatchSamplingParamsKey`, optional `request_batch_max_wait_ms` admission window) or `STEP_BATCH` (`StepScheduler`: continuous batching at denoise-step granularity, `max_num_seqs>1`), plus `streaming_output` for chunked video and `submit_interaction` for mid-stream prompt events. [Verified] (`../../vllm-omni/vllm_omni/diffusion/sched/request_scheduler.py`, `../../vllm-omni/vllm_omni/diffusion/sched/step_scheduler.py`, `../../vllm-omni/docs/user_guide/diffusion/execution_modes.md`)
|
||||
- **Cancellation/abort**: `AsyncOmni.abort` → `AsyncOmniEngine.abort_async` → `AbortRequestMessage` → `Orchestrator._handle_abort` → `StagePool.abort_requests` on every stage holding the request; AR final-stage abort yields a terminal output with `finish_reason="abort"` and the generated prefix; diffusion abort is whole-sample (`DiffusionEngine.abort` → `_process_aborts_queue`); connector `cleanup(request_id)` unlinks shm segments; `abort_requests_collecting_outputs` in the output processor; `FINISHED_ABORTED` requests are dropped before vLLM sees them. [Verified] (`../../vllm-omni/docs/design/module/engine_orchestration.md`)
|
||||
- **Backpressure**: generation admission gate in `AsyncOmni._submit_with_admission`; `duplex_max_sessions` engine-owned session capacity; `max_num_seqs` per stage; `collect_timed_out_request_ids` for chunk/input timeouts (`connector_get_max_wait*`); `active_stream_window` limits concurrently fed downstream streams. [Verified]
|
||||
|
||||
### (b) Graph / operator / thread-level scheduling
|
||||
|
||||
- Within a stage step the model forward is a single torch call sequence (eager, piecewise or full CUDA graph) — there is **no omni-level operator DAG scheduler**; ordering is the model's Python. [Verified]
|
||||
- Thread pools: chunk transfer adapter `recv_loop`/`save_loop` threads; async omni output builder thread (`OmniAsyncGPUModelRunnerOutput._build_output_in_background`); `DiffusionEngine._busy_loop` thread; multiproc diffusion `_result_pump` thread; `orchestrator_monitor`. [Verified]
|
||||
- Stream assignment: default compute stream plus a dedicated payload copy stream for D2H snapshots and prefix-cache writes; NPU synchronizes default stream before events. Diffusion sequence-parallel comms use `GroupCoordinator` in `vllm_omni.diffusion.distributed`. [Verified]
|
||||
- Diffusion regional compilation is the only compile-graph scheduling; N/A otherwise. [Verified]
|
||||
|
||||
## 6. I/O layer
|
||||
|
||||
- **Preprocessing.** Media parsing reuses `vllm.multimodal` (audio via `vllm.multimodal.audio` — `librosa` is banned by ruff in favor of vLLM helpers/torchaudio); `utils/audio.py` provides mel filterbanks; per-model processors under `transformers_utils/processors`; TTS reference audio encoding happens in serving adapters (`_encode_moss_references`, `_build_qwen3_tts_request`); diffusion image preprocessing via `get_*_pre_process_func`. [Verified] (`../../vllm-omni/vllm_omni/utils/audio.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py`)
|
||||
- **Tokenization.** HF `transformers` tokenizers through vLLM's tokenizer layer (`vllm.tokenizers` imported in 20 files); `owns_tokenizer` marks the stage whose tokenizer the pipeline exposes; custom `MammothModa2Tokenizer`; speech tokenizers/codecs per model (e.g., `Qwen3TTSTokenizer` 12 Hz/25 Hz, `s3tokenizer`, `whisper_vq`). [Verified] (`../../vllm-omni/vllm_omni/tokenizers/mammoth_moda2_tokenizer.py`, `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py`)
|
||||
- **Streaming.** Text deltas via vLLM `RequestOutputKind.DELTA`; audio via async chunk `OmniEngineCoreOutput.multimodal_output` per chunk, accumulated by `OmniRequestState.add_multimodal_tensor` / `multimodal_accumulation.py`; HTTP TTS streams PCM through `_generate_pcm_chunks`/`_iter_pcm_audio_bytes` or SSE (`_generate_audio_sse_events`); WebSocket `OmniStreamingSpeechHandler` (`/v1/audio/speech/stream`, optional word alignments via forced aligner); `StreamingAudioResampler` for integer-ratio downsampling (24 kHz→8 kHz). Video streaming output via `/v1/video/chat/stream`, `/v1/realtime/video`. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py`, `../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py`, `../../vllm-omni/vllm_omni/outputs/output_processor.py`)
|
||||
- **Buffering.** Talker→code2wav frame accumulation (`code_prompt_token_ids` deque per request; chunk = 25 frames, left context 25–72 frames, first chunk 1–4 or dynamic); shm segments per chunk; per-request asyncio queues for diffusion; duplex `realtime_input.py` audio buffers with Silero VAD (`SileroStreamingVAD`, 16 kHz). [Verified] (`../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py`, `../../vllm-omni/vllm_omni/entrypoints/duplex/vad.py`)
|
||||
- **Postprocessing.** Detokenize via vLLM output processor (`sampling_constraints={"detokenize": False}` on talkers); vocoders as generation/diffusion stages; diffusion `get_*_post_process_func` converts latents → PIL/video/actions; audio encoding with `soundfile` (`wav`, `pcm`, `mp3`, `opus`), speed change via torchaudio phase-vocoder. [Verified]
|
||||
- **Application boundaries.**
|
||||
- Python: `Omni.generate(prompts, sampling_params_list, py_generator=...)`, `AsyncOmni` (implements vLLM `EngineClient`). [Verified] (`../../vllm-omni/vllm_omni/entrypoints/omni.py`, `../../vllm-omni/vllm_omni/entrypoints/async_omni.py`)
|
||||
- CLI: `vllm serve <model> --omni` (vLLM subcommand plugin) and `vllm-omni` console script; stage-based launch `--stage-id N --headless --omni-master-address/-port`; `vllm-omni bench serve`. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/cli/main.py`, `../../vllm-omni/vllm_omni/entrypoints/cli/serve.py`)
|
||||
- HTTP (FastAPI): `/v1/chat/completions` (+`/batch`), `/v1/audio/speech` (+`/batch`), `/v1/audio/generate`, `/v1/audio/voices` (GET/POST/DELETE), `/v1/images/generations`, `/v1/images/edits`, `/v1/videos` (+`/sync`, list/get/delete/content), `/v1/models`, `/health`, `/v1/omni/sleep|wakeup`, `/start_profile|/stop_profile`. WebSocket: `/v1/audio/speech/stream`, `/v1/video/chat/stream`, `/v1/realtime/video`, `/v1/realtime` (turn-based PCM), `/v1/realtime/robot/openpi`, `/v1/duplex`. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py`)
|
||||
- No C API, no mobile bindings, no gRPC. [Verified]
|
||||
|
||||
## 7. Execution flow
|
||||
|
||||
### (a) Text LLM decode (thinker-only pipeline, e.g. `qwen3_omni_moe_thinker_only`)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client (HTTP)
|
||||
participant S as api_server / OmniOpenAIServingChat
|
||||
participant A as AsyncOmni.generate
|
||||
participant E as AsyncOmniEngine
|
||||
participant O as Orchestrator (asyncio)
|
||||
participant P as StagePool[0] / StageEngineCoreClient (ZMQ)
|
||||
participant EC as StageEngineCoreProc (vLLM EngineCore)
|
||||
participant SC as OmniARAsyncScheduler
|
||||
participant W as GPUARWorker / GPUARModelRunner
|
||||
C->>S: POST /v1/chat/completions (stream)
|
||||
S->>A: generate(prompt, sampling_params_list)
|
||||
A->>E: add_request_async (StageSubmissionMessage)
|
||||
E->>O: request queue
|
||||
O->>P: submit_initial(req) -> OmniEngineCoreRequest
|
||||
P->>EC: ZMQ add_request
|
||||
loop each engine step
|
||||
EC->>SC: schedule()
|
||||
SC-->>EC: SchedulerOutput (prefill/decode batch)
|
||||
EC->>W: execute_model -> sample_tokens
|
||||
W-->>EC: OmniModelRunnerOutput (token ids, hidden states opt.)
|
||||
EC->>SC: update_from_output
|
||||
EC-->>P: OmniEngineCoreOutputs (ZMQ)
|
||||
P-->>O: poll_llm_raw_output -> MultimodalOutputProcessor
|
||||
O-->>E: OutputMessage(final_output=True, DELTA)
|
||||
E-->>A: _process_orchestrator_results
|
||||
A-->>S: OmniRequestOutput
|
||||
S-->>C: SSE chunk
|
||||
end
|
||||
C-->>S: disconnect / cancel
|
||||
S->>A: abort(request_id)
|
||||
A->>O: AbortRequestMessage
|
||||
O->>P: abort_requests -> EC finish_requests(FINISHED_ABORTED)
|
||||
```
|
||||
[Verified] for the participants and call names (`../../vllm-omni/vllm_omni/entrypoints/async_omni.py`, `../../vllm-omni/vllm_omni/engine/orchestrator.py`, `../../vllm-omni/vllm_omni/engine/stage_pool.py`); exact vLLM-internal ZMQ framing inherited from `AsyncMPClient` [Inferred].
|
||||
|
||||
### (b) TTS with async chunk streaming (Qwen3-TTS: talker → code2wav)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant SP as OmniOpenAIServingSpeech (+Qwen3TTS adapter)
|
||||
participant O as Orchestrator
|
||||
participant T as Stage0 talker (OmniARScheduler + GPUARModelRunner)
|
||||
participant AD as OmniChunkTransferAdapter (stage0 sched proc)
|
||||
participant SHM as SharedMemoryConnector (/dev/shm)
|
||||
participant G as Stage1 code2wav (OmniGenerationScheduler + GPUGenerationModelRunner)
|
||||
participant M as Qwen3TTSCode2Wav (+CUDAGraphDecoderWrapper)
|
||||
C->>SP: POST /v1/audio/speech {input, voice, stream}
|
||||
SP->>SP: build prompt (prompt_embeds/additional_information, ref audio codes)
|
||||
SP->>O: AsyncOmni.generate (2 sampling params)
|
||||
O->>T: submit stage0 request
|
||||
O->>G: _prewarm_async_chunk_stages: pre-submit stage1 request (WAITING_FOR_CHUNK)
|
||||
loop each talker decode step
|
||||
T->>T: execute_model -> codec frame in multimodal_outputs
|
||||
T->>AD: update_from_output -> save_async(pooler_output, request)
|
||||
AD->>AD: talker2code2wav_async_chunk: accumulate frames; emit when >= initial/steady chunk (1..25 frames + left ctx)
|
||||
AD->>SHM: put(key=req_stage_chunk, OmniPayloadStruct{codes,meta})
|
||||
end
|
||||
G->>SHM: recv_loop get(key) -> chunk ready
|
||||
G->>G: restore_queues -> schedule (append chunk tokens)
|
||||
G->>M: execute_model -> chunked_decode_with_cudagraph
|
||||
M-->>G: waveform chunk (per request via ubatch_slices)
|
||||
G-->>O: OmniEngineCoreOutput.multimodal_output {audio}
|
||||
O-->>SP: OutputMessage(stage1 final_output=True)
|
||||
SP->>SP: soundfile encode (pcm/wav/mp3/opus), optional resample
|
||||
SP-->>C: chunked HTTP / SSE / WS audio
|
||||
T-->>AD: finished -> meta.finished chunk
|
||||
G-->>O: last chunk, finished=True -> cleanup shm segments
|
||||
```
|
||||
[Verified] (`../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py`, `../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/shm_connector.py`, `../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py`, `../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml`, `../../vllm-omni/docs/design/feature/async_chunk.md`).
|
||||
|
||||
VLA path (π0 / GR00T): `WS /v1/realtime/robot/openpi` → `ServingRealtimeRobotOpenPI.infer(obs, session_id, reset)` → `_build_request` puts the raw observation into `sampling_params.extra_args["robot_obs"]` → `AsyncOmni.generate` → single DIFFUSION stage (`DiffusionEngine.step_streaming`) → `Pi0Pipeline.forward(req)` (preprocess + flow-matching denoise) → `DiffusionOutput(output={"actions": ndarray})` → `multimodal_output["actions"]` → msgpack reply. [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py`, `../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py`)
|
||||
|
||||
## 8. Tests and examples
|
||||
|
||||
- **Layout**: `../../vllm-omni/tests/` (912 test files) split by module: `engine/` (48), `core/` (21), `worker/` (14), `diffusion/` (311), `entrypoints/` (66), `e2e/` (180: `offline_inference/`, `online_serving/`, `accuracy/`, `features/`), `model_executor/` (136), `distributed/` (18), `platforms/` (9), `examples/` (17), plus `dfx/` (perf/reliability/stability). Root `conftest.py` is thin; fixtures in `tests/helpers/fixtures/`. [Verified] (`../../vllm-omni/tests/conftest.py`)
|
||||
- **Markers / levels** (pyproject): `core_model` (L1/L2 per PR), `advanced_model` (L3 per merge), `full_model` (L4 nightly), `local_model`; module markers `diffusion|omni|tts|cache|parallel|sp|example`; platform markers `cpu|gpu|cuda|rocm|xpu|npu|musa`; SKU markers `H100|H800|H200|L4|B200|MI325|B60|S5000|A2|A3`; `cards_1..8`. [Verified] (`../../vllm-omni/pyproject.toml`)
|
||||
- **How to run** (`../../vllm-omni/docs/contributing/ci/test_execution_guide.md`): `uv pip install ".[dev]"; apt-get install espeak-ng jq`; L1: `cd tests && pytest -s -v -m "core_model and cpu"`; L2/L3/L4 via `bash tools/run_ready_jobs.sh|run_merge_jobs.sh|tools/nightly/run_nightly_jobs.sh` which read `.buildkite/cuda/test-{ready,merge,nightly}.yml`; ad hoc `pytest -s -v test_x.py --run-level=core_model`. [Verified]
|
||||
- **CI**: Buildkite pipelines for cuda (`../../vllm-omni/.buildkite/cuda/test-ready.yml`, `test-merge.yml`, `test-nightly.yml`, `test-weekly.yml`), amd, intel (XPU, `VLLM_VERSION: v0.28.0`), npu (A2/A3); GitHub Actions only `pre-commit.yml` and `build_wheel.yml`. Nightly docker `vllm/vllm-omni:nightly`. [Verified]
|
||||
- **Relevant examples**: TTS `../../vllm-omni/examples/offline_inference/text_to_speech/` (qwen3_tts, cosyvoice3, fish_speech, gepard, glm_tts, higgs_audio_v2/v3, indextts2, ming_tts, moss_tts(_nano), omnivoice, voxcpm2, voxtral_tts; `qwen3_tts/word_timestamps.py`) and `../../vllm-omni/examples/online_serving/text_to_speech/`; omni `../../vllm-omni/examples/offline_inference/qwen3_omni/end2end_async_chunk.py`, `../../vllm-omni/examples/online_serving/qwen3_omni/openai_realtime_client.py`; VLA `../../vllm-omni/examples/online_serving/pi0/openpi_client.py`, `../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py`, `../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py`, `../../vllm-omni/examples/offline_inference/dreamzero/client_schedule.py`; custom pipeline `../../vllm-omni/examples/offline_inference/custom_pipeline/image_to_image/custom_pipeline.py`; apps `../../vllm-omni/apps/ComfyUI-vLLM-Omni`. No mobile examples. [Verified]
|
||||
- **Relevant e2e tests**: `../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py`, `test_qwen3_omni_colocate_async.py`, `test_lingbot_world_v2_stepwise.py`; `../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py`, `test_gr00t_openpi_expansion.py`, `test_dreamzero_expansion.py`, `test_minicpmo_4_5_duplex.py`. [Verified]
|
||||
- **Benchmarks**: `../../vllm-omni/benchmarks/tts/bench_tts.py` (+`model_configs.yaml`), `benchmarks/diffusion`, `benchmarks/kernels`, `vllm-omni bench serve` (TTFP/RTF metrics in `vllm_omni/metrics`). [Verified]
|
||||
|
||||
## 9. TTS relevance
|
||||
|
||||
**Pipeline-level facts.**
|
||||
- TTS models are two-stage `LLM_AR` (talker → codec tokens) + `LLM_GENERATION` (code2wav) or + `DIFFUSION` (flow-matching acoustic decoder) pipelines; single-stage native-AR TTS also exists (Gepard). `sampling_constraints={"detokenize": False, "stop_token_ids": [...]}` keep vLLM from detokenizing codec ids. [Verified] (`../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py`)
|
||||
- Streaming audio is produced by `async_chunk: true` with connector extras (`codec_streaming`, `codec_chunk_frames`, `codec_left_context_frames`, `initial_codec_chunk_frames`, `codec_chunk_ramp`, `codec_chunk_adaptive`, `decode_batch_max_size`, `decode_cudagraph_*`). Dynamic initial chunk sizing (`compute_dynamic_initial_chunk_size`) trades TTFP vs load. [Verified] (`../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml`, `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py`)
|
||||
- Code2wav batching across requests uses `ubatch_slices` to split batched waveforms; CUDA graphs captured for the stateful streaming decoder shapes. [Verified] (`../../vllm-omni/docs/design/feature/async_chunk.md`, `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py`)
|
||||
- Serving: `/v1/audio/speech` with per-model adapters (`qwen3_tts`, `cosyvoice3`, `fish_speech`, `voxtral`, `moss_tts`, `higgs_audio_v2/v3`, `indextts2`, `ming_tts`, `ming_flash_omni_tts`, `glm_tts`, `dots_tts`, `voxcpm2`, `omnivoice`, `step_audio2`, `covo_audio`, `audex`, `audex_tta`, `minimax_music3`); voices registry endpoints (`/v1/audio/voices`, `custom_voice_dir`), speaker cache (`utils/speaker_cache.py`), forced aligner for word timestamps (`utils/forced_aligner.py`, `--forced-aligner`), speech usage accounting (`speech_usage.py`). [Verified] (`../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/`)
|
||||
- Realtime speech-to-speech: `/v1/realtime` (turn-based) and full-duplex `/v1/duplex` / `/v1/realtime?duplex=1` with server VAD (Silero), used by MiniCPM-o 4.5, PersonaPlex (Moshi finetune), Nemotron VoiceChat. [Verified] (`../../vllm-omni/docs/serving/realtime_duplex_api.md`, `../../vllm-omni/docs/serving/full_duplex_api.md`)
|
||||
- Repo-reported performance (not reproduced): Qwen3-TTS on H200, concurrency 1: E2E 941 ms, TTFP 64 ms, RTF 0.16 vs HF 15,513 ms / 2.64; Qwen3-Omni on A100: TTFP 0.934 s, RTF 0.32. [Verified as repo claims] (`../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md`)
|
||||
|
||||
**TTS/audio model families in tree** (registry keys in `../../vllm-omni/vllm_omni/model_executor/models/registry.py`, dirs under `../../vllm-omni/vllm_omni/model_executor/models/`):
|
||||
|
||||
| Family | Dir | Stages | Vocoder/codec |
|
||||
| --- | --- | --- | --- |
|
||||
| Qwen3-TTS (CustomVoice/VoiceDesign/Base) | `qwen3_tts` | Talker (AR) → `Qwen3TTSCode2Wav` (gen) | 12 Hz/25 Hz speech tokenizer, ConvNet code2wav, x-vector speaker enc |
|
||||
| CosyVoice3 | `cosyvoice3` (+ `diffusion/models/cosyvoice3_audio`) | Talker → flow-matching DiT code2wav | s3tokenizer; ONNX speaker embedding |
|
||||
| Fish Speech S2 Pro | `fish_speech` | Slow AR (+fast AR) → `FishSpeechDACDecoder` | DAC; custom KV-cache Triton attention |
|
||||
| Gepard-1.0 | `gepard` | single-stage native AR | FSQ/NanoCodec |
|
||||
| VoxCPM2 | `voxcpm2` | Talker (+ diffusion head) | — |
|
||||
| dots.tts | `dots_tts` | Talker | — |
|
||||
| Voxtral TTS | `voxtral_tts` | AR → `VoxtralTTSAudioGeneration` (flow matching) + audio tokenizer | — |
|
||||
| MOSS-TTS family (Delay/Realtime/Local, TTSD, SoundEffect, VoiceGenerator) | `moss_tts`, `moss_tts_nano` | Talker → `MossTTSCodecDecoder` | MOSS codec |
|
||||
| IndexTTS-2 / 2.5 | `indextts2` | Talker → `IndexTTS2S2MelDecoder` | s2mel + vocoder |
|
||||
| GLM-TTS | `glm_tts` | Talker (+ voice clone helper, ONNX) | — |
|
||||
| Higgs-Audio v2 / v3 | `higgs_audio_v2`, `higgs_audio_v3` | Talker → Code2Wav | — |
|
||||
| OmniVoice | `omnivoice` (+ `diffusion/models/omnivoice`) | diffusion pipeline | — |
|
||||
| Ming-TTS (dense/MoE), Ming-flash-omni TTS | `ming_tts`, `ming_flash_omni` | LLM → `MingAudioVAEModel` | audio VAE |
|
||||
| Audex (Nemotron) TTS/TTA/S2S | `audex` | Thinker → `AudexCode2Wav` / `AudexXCodec1` | XCodec |
|
||||
| Step-Audio2, MiMo-Audio, Covo-Audio, MiniCPM-o 4.5, Qwen2.5/3-Omni, Nemotron VoiceChat, PersonaPlex, Dynin-Omni, Aura-Omni | respective dirs | omni thinker → talker → token2wav | model codecs (Token2Wav / CosyVoice2 flow) |
|
||||
| MiniMax-Music3 (text-to-music) | `minimax_music3` | Talker → acoustic flow decoder | — |
|
||||
| Stable-Audio-Open | `diffusion/models/stable_audio` | diffusion | — |
|
||||
|
||||
**What's missing for edge TTS.** No CPU or mobile execution path for talker or vocoder; vocoder stages assume CUDA graphs or Ascend/ROCm patches; audio chunks cross a POSIX-shm + msgpack boundary (fine on a server, wasteful on a single-process edge device); no low-latency in-process single-stage mode (a stage is always a vLLM `EngineCore` subprocess with its own scheduler); no quantization validated for talker/code2wav stages (docs: "not validated"); no INT8/INT4 vocoder path; `Omni` offline API is batch-oriented (`py_generator=True` streams finished requests, not audio chunks — chunk streaming is only via `AsyncOmni`/HTTP). [Verified/Inferred]
|
||||
|
||||
## 10. VLA relevance
|
||||
|
||||
**Pipeline-level facts.**
|
||||
- VLA policies are **single `DIFFUSION` stages** with `final_output_type="action"|"actions"`: `PI0_PIPELINE` (`../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py`), `GR00T_N1D7_PIPELINE` (`../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py`), InternVLA-A1 (`InternVLAA1Pipeline`, default single-stage fallback). Observations arrive in `sampling_params.extra_args["robot_obs"]`; the pipeline owns all preprocessing (image resize, tokenization of the language instruction, state normalization) and returns `DiffusionOutput(output={"actions": ...})`. [Verified]
|
||||
- Vision encoders: π0 uses PaliGemma (SigLIP + Gemma-2B prefix, `paligemma_variant`, `action_expert_variant: gemma_300m`, 224×224, up to 3 cameras); GR00T N1.7 uses its own VLM backbone through `Gr00tN1d7`/`Gr00tN1d7Processor` with embodiment-conditioned MLP and DiT action head (`modeling/modules/dit.py`, `embodiment_conditioned_mlp.py`); InternVLA-A1 uses a Qwen3-VL adapter (`adapter_qwen3_vl.py`) with a Cosmos-based world/action model (`model_cosmos.py`). [Verified] (`../../vllm-omni/vllm_omni/deploy/pi0.yaml`, `../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py`, `../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py`)
|
||||
- Structured/proprioceptive inputs: π0 `max_state_dim: 32`, `max_action_dim: 32`, `chunk_size: 50`, `num_inference_steps: 10`; GR00T `dataio/state_action/{state_action_processor,action_chunking,pose}.py` and `embodiment_tags.py` (DROID, G1, R1 Pro, LIBERO, SimplerEnv …); DreamZero `transform/droid.py`, `roboarena.py`, `action_encoder.py`. [Verified]
|
||||
- Action heads / diffusion-flow policies: flow matching (π0 `sample_actions` loop, Euler; GR00T DiT; InternVLA Cosmos CI); Cosmos3 action modes `policy|forward_dynamics|inverse_dynamics` with embodiment→domain id map (`../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py`). [Verified]
|
||||
- World models: DreamZero-DROID (`DreamZeroPipeline`, `engine_backend: ARDiffusionEngine`, `step_cache` DiT cache, `dreamzero_tp1_cfg2.yaml` CFG-parallel) and LingBot-World 2.0 (`LingBotWorldCausalDMDPipeline`, stepwise deploy `lingbot_world_v2_stepwise.yaml`) run on the experimental AR-diffusion engine: per-session paged KV (`ARDiffusionKVCacheSpec`: tokens_per_frame, window/sink frames), tick protocol with `session_id/event_id/chunk_index/request_id`, one latent block per request. SANA-WM (`sana_wm`) and Cosmos3 world generation also exist. [Verified] (`../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md`, `../../vllm-omni/vllm_omni/deploy/dreamzero.yaml`)
|
||||
- Low-latency loop: OpenPI WebSocket (`/v1/realtime/robot/openpi`, msgpack + numpy ext, 64 MiB max frame, 30 s idle close, `session_id`/`reset` semantics, first frame = `policy_server_config`); request→response per observation; `max_num_seqs: 1`, `enforce_eager: true`, float32 for π0 by default. Session state manager (`enable_session_state_manager`, `experimental/world_models/session_state`) keeps model state across observations. [Verified] (`../../vllm-omni/docs/serving/openpi_api.md`, `../../vllm-omni/docs/features/session_state_manager.md`)
|
||||
- Deploy YAMLs: `../../vllm-omni/vllm_omni/deploy/pi0.yaml`, `../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml`, `../../vllm-omni/vllm_omni/deploy/dreamzero.yaml`, `../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml`, `../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml`. [Verified]
|
||||
|
||||
**What's missing for edge VLA.** No action-token streaming (one reply per observation); no explicit control-loop latency budget/deadline scheduling; VLA pipelines are eager PyTorch modules (no CUDA graph / compile path exercised for π0/GR00T by default; GR00T loads via HF `AutoModel`); no CPU/Jetson-specific path (π0 picks `cuda` if available else falls to CPU only implicitly via `torch.cuda.is_available()` in GR00T — untested); KV/state reuse across observations is only implemented for world models (AR-diffusion), not for VLM-prefix caching in π0/GR00T; no quantized VLA recipes. [Verified/Inferred]
|
||||
|
||||
## 11. Edge deployment profile
|
||||
|
||||
- **Binary size / dependencies.** Pure-Python wheel; runtime deps (`../../vllm-omni/requirements/common.txt`): `transformers>=5.10.1,<5.15`, `diffusers==0.40.0`, `kernels==0.15.2`, `accelerate`, `cache-dit`, `torchsde`, `openai-whisper`, `imageio[ffmpeg]`, `x-transformers`, `einops`, `pyzmq`, `janus`, `msgpack`, `pydantic`, `gguf`, `cosmos-guardrail`, `soundfile`, `av`; platform files add `onnxruntime` (+`fa3-fwd` on CUDA, `onnxruntime-cann`+`torchaudio` on NPU, `torchada/mate/flash_attn_3` on MUSA, `auto-round-lib` on XPU). Plus vLLM 0.28.x itself (compiled CUDA kernels) and PyTorch. Docker images derive from `vllm/vllm-openai:v0.28.0`. Footprint is multi-GB; no size-optimized build exists. [Verified]
|
||||
- **Build system.** `setuptools` + `setuptools-scm`; `setup.py` picks the requirements file from `VLLM_OMNI_TARGET_DEVICE` or torch introspection; version suffix `+rocm|+npu|+xpu|+musa|+cpu`. No CMake, no native code. [Verified] (`../../vllm-omni/setup.py`)
|
||||
- **Supported platforms** (docs `../../vllm-omni/docs/getting_started/installation/README.md`): NVIDIA CUDA (CC ≥ 7.0 per docs; diffusion FA needs ≥ 8.0; Blackwell needs FA4/cuDNN ≥ 9.5), AMD ROCm, Intel XPU, MThreads MUSA, Ascend NPU (Atlas A2/A3, 310P patches). Recipes mention consumer GPUs (RTX 4090/5090, RTX PRO, DGX Spark GB10 for MiniMax-H3). [Verified] (`../../vllm-omni/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md`)
|
||||
- **Embedded NVIDIA / Jetson**: no Jetson/aarch64-specific code, docs, or CI; the DGX Spark (GB10, aarch64+Blackwell) recipe is the closest evidence that an aarch64 CUDA host can run it, contingent on a vLLM aarch64 wheel. [Verified: absence] [Inferred: GB10 feasibility]
|
||||
- **ARM CPU, Apple Silicon/Metal/CoreML, Android GPU/NPU (Vulkan/OpenCL/QNN/NNAPI)**: none. `grep` for `metal|coreml|vulkan|opencl|qnn|nnapi|jetson` over `vllm_omni/` and `docs/` found no implementation. [Verified: absence]
|
||||
- **Model conversion path & constraints.** Models are consumed directly as HF/Diffusers checkpoints (safetensors; GGUF for some diffusion models; ModelOpt/AutoRound/TorchAO pre-quantized checkpoints). There is no exporter to ONNX/TensorRT/CoreML/TFLite; the only "conversion" is loader-side (weight renaming, TP slicing, online quantization, Host Weight Runtime artifacts). Constraints: vLLM's model-class contract (`forward(input_ids, positions, intermediate_tensors, inputs_embeds)`), vLLM paged attention layout, and `OmniPayload` schemas between stages. [Verified]
|
||||
- **Process model on the edge.** Minimum footprint for a 2-stage TTS is: API server process + orchestrator thread + 2 vLLM `EngineCoreProc` subprocesses (each with scheduler + worker) + shm connector; single-stage diffusion can run inline (`inline_diffusion`/`uni` executor). [Verified] (`../../vllm-omni/docs/configuration/stage_configs.md`)
|
||||
- [Proposal] For an edge port, retain: `PipelineConfig`/`StagePipelineConfig` topology descriptions, `stage_input_processors` (chunking math in `chunk_size_utils.py`), `OmniPayload` typed schemas, TTS adapters' request building, OpenPI/realtime protocol handlers; replace: `EngineCoreProc` stages with in-process executors, `SharedMemoryConnector` with in-memory queues, vLLM model classes with the target runtime's graphs.
|
||||
|
||||
## 12. Limitations and unknowns
|
||||
|
||||
- [Verified] Hard dependency on vLLM 0.28.x internals (284 modules, v1 engine, monkeypatched classes); minor-version drift is expected to break (`version.py` warning, `patch.py` FRAGILITY notes).
|
||||
- [Verified] No CPU/edge/mobile execution backend; `requirements/cpu.txt` is for docs builds only.
|
||||
- [Verified] The default single-node transport is host-staged (D2H2D) even when stages share a GPU (`docs/design/feature/disaggregated_inference.md`); zero-copy is a TODO in `OmniMsgpackEncoder`. Mooncake TE and Mori have device-memory fast paths, including Mori's intra-node XGMI backend; their applicability to an edge target is unvalidated.
|
||||
- [Verified] Each LLM stage is a separate subprocess with its own memory budget; co-location relies on `gpu_memory_utilization` partitioning and admission checks.
|
||||
- [Verified] Quantization for TTS/omni non-AR stages is documented as not validated; MXFP4/MXFP8 are Ascend-only.
|
||||
- [Verified] Many design docs are `status: draft` with "candidate invariants" (engine orchestration, stage runtime, connectors, platforms); the stage-client/process refactor #5441 is in flight.
|
||||
- [Verified] Experimental subsystems: AR-diffusion engine/world models (`vllm_omni/experimental/ar_diffusion`), full-duplex JoyVL/Mage-VL (`experimental/fullduplex`), `PAGED_WORKER_LOCAL` KV mode reserved-but-rejected.
|
||||
- [Verified] VLA serving returns whole action chunks; no intermediate streaming; `session_id`/`reset` ignored by π0 (stateless).
|
||||
- [Inferred] Latency floor for TTS on a single device includes subprocess ZMQ hops, shm serialization, and scheduler polling (`connector_get_sleep_s: 0.01`), which matter more on edge SoCs than on H100.
|
||||
- [Unknown] Actual runtime behavior of the `7be014bc` tree was not executed here (installed editable is `be335a86f`); no benchmark numbers were reproduced.
|
||||
- [Unknown] Whether vLLM's aarch64/Jetson wheels satisfy the kernel packages omni's CUDA path probes (`fa3-fwd`, FlashInfer, FA4) — not determinable from this tree.
|
||||
- [Unknown] Memory footprint of the smallest supported TTS deployment (e.g., Qwen3-TTS 1.7B two-stage) — deploy YAMLs use `gpu_memory_utilization: 0.3` per stage on H100 but no absolute numbers are published in-tree beyond recipes.
|
||||
|
||||
## 13. Reference index
|
||||
|
||||
| Path | Role |
|
||||
| --- | --- |
|
||||
| `../../vllm-omni/pyproject.toml` | Package metadata, extras, pytest markers, `vllm.general_plugins` entry point |
|
||||
| `../../vllm-omni/setup.py` | Platform-aware dependency routing, version suffix |
|
||||
| `../../vllm-omni/requirements/common.txt` | Common runtime deps (no vllm pin) |
|
||||
| `../../vllm-omni/requirements/cpu.txt` | CPU requirements (docs builds) |
|
||||
| `../../vllm-omni/requirements/musa.txt` | MUSA deps |
|
||||
| `../../vllm-omni/README.md` | Project overview, model classes |
|
||||
| `../../vllm-omni/docker/Dockerfile.cuda` | CUDA image from `vllm/vllm-openai:v0.28.0` |
|
||||
| `../../vllm-omni/docker/Dockerfile.npu` | Ascend image |
|
||||
| `../../vllm-omni/vllm_omni/__init__.py` | Import-time version check + patch |
|
||||
| `../../vllm-omni/vllm_omni/version.py` | vLLM major.minor alignment warning |
|
||||
| `../../vllm-omni/vllm_omni/patch.py` | Monkeypatches of vLLM/torch/transformers |
|
||||
| `../../vllm-omni/vllm_omni/plugins/__init__.py` | Omni plugin groups |
|
||||
| `../../vllm-omni/vllm_omni/platforms/__init__.py` | Platform probes and resolution |
|
||||
| `../../vllm-omni/vllm_omni/platforms/interface.py` | `OmniPlatform` abstract hooks, `UnspecifiedOmniPlatform` |
|
||||
| `../../vllm-omni/vllm_omni/platforms/cuda/platform.py` | CUDA platform, diffusion attention default chain |
|
||||
| `../../vllm-omni/vllm_omni/platforms/rocm/platform.py` | ROCm platform |
|
||||
| `../../vllm-omni/vllm_omni/platforms/npu/platform.py` | Ascend platform (vllm_ascend) |
|
||||
| `../../vllm-omni/vllm_omni/platforms/npu/worker/` | NPU AR/generation workers and runners |
|
||||
| `../../vllm-omni/vllm_omni/platforms/xpu/platform.py` | Intel XPU platform |
|
||||
| `../../vllm-omni/vllm_omni/platforms/musa/platform.py` | MUSA platform |
|
||||
| `../../vllm-omni/vllm_omni/engine/arg_utils.py` | `OmniEngineArgs`, `OrchestratorArgs`, model registration |
|
||||
| `../../vllm-omni/vllm_omni/engine/async_omni_engine.py` | `AsyncOmniEngine` composition root |
|
||||
| `../../vllm-omni/vllm_omni/engine/orchestrator.py` | Cross-stage routing, abort, CFG, PD |
|
||||
| `../../vllm-omni/vllm_omni/engine/cfg_companion_tracker.py` | CFG companion bookkeeping |
|
||||
| `../../vllm-omni/vllm_omni/engine/stage_runtime.py` | Local/distributed replica lifecycle |
|
||||
| `../../vllm-omni/vllm_omni/engine/stage_pool.py` | Per-stage replica pool, LB, metrics |
|
||||
| `../../vllm-omni/vllm_omni/engine/stage_engine_startup.py` | ZMQ master server, replica launch |
|
||||
| `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc.py` | `StageEngineCoreProc` |
|
||||
| `../../vllm-omni/vllm_omni/engine/stage_engine_core_proc_manager.py` | Subprocess spawning per replica |
|
||||
| `../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py` | ZMQ client per replica |
|
||||
| `../../vllm-omni/vllm_omni/engine/stage_admission.py` | Device memory admission |
|
||||
| `../../vllm-omni/vllm_omni/engine/__init__.py` | `OmniEngineCoreRequest/Output(s)` |
|
||||
| `../../vllm-omni/vllm_omni/config/stage_config.py` | `PipelineConfig`, `StageDeployConfig`, scheduler resolution |
|
||||
| `../../vllm-omni/vllm_omni/config/pipeline_registry.py` | `OMNI_PIPELINES` |
|
||||
| `../../vllm-omni/vllm_omni/config/config_factory.py` | Pipeline auto-detection from HF config |
|
||||
| `../../vllm-omni/vllm_omni/config/omni_config.py` | Typed control-plane config |
|
||||
| `../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml` | Qwen3-TTS deploy (chunk streaming knobs) |
|
||||
| `../../vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml` | Qwen3-Omni 3-stage deploy with platform overrides |
|
||||
| `../../vllm-omni/vllm_omni/deploy/pi0.yaml` | π0 deploy |
|
||||
| `../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml` | GR00T deploy |
|
||||
| `../../vllm-omni/vllm_omni/deploy/dreamzero.yaml` | DreamZero deploy (AR-diffusion engine) |
|
||||
| `../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml` | DreamZero CFG-parallel deploy |
|
||||
| `../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml` | LingBot-World stepwise deploy |
|
||||
| `../../vllm-omni/vllm_omni/core/sched/omni_ar_scheduler.py` | AR scheduler |
|
||||
| `../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py` | Generation scheduler |
|
||||
| `../../vllm-omni/vllm_omni/core/sched/omni_scheduler_mixin.py` | Shared omni scheduling state |
|
||||
| `../../vllm-omni/vllm_omni/core/sched/omni_scheduling_coordinator.py` | Full-payload input coordination |
|
||||
| `../../vllm-omni/vllm_omni/core/prefix_cache.py` | Hidden-state/multimodal prefix cache |
|
||||
| `../../vllm-omni/vllm_omni/worker/base.py` | Worker base, memory profiling, sleep |
|
||||
| `../../vllm-omni/vllm_omni/worker/gpu_ar_worker.py` | AR worker |
|
||||
| `../../vllm-omni/vllm_omni/worker/gpu_model_runner.py` | Shared omni runner (MRoPE, talker MTP init, preprocess) |
|
||||
| `../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py` | AR runner execute/sample/async output |
|
||||
| `../../vllm-omni/vllm_omni/worker/gpu_generation_model_runner.py` | Non-AR generation runner |
|
||||
| `../../vllm-omni/vllm_omni/worker/omni_connector_model_runner_mixin.py` | Runner-side connector I/O (chunks, payloads, KV) |
|
||||
| `../../vllm-omni/vllm_omni/worker/gpu_memory_utils.py` | NVML memory helpers |
|
||||
| `../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py` | Fish Speech Triton KV attention |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/registry.py` | Omni AR/generation model registry (89 archs) |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/output_templates.py` | `OmniOutput` |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py` | Qwen3-TTS topology |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py` | Code2wav CUDA graph capture |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_tokenizer.py` | Qwen3-TTS speech tokenizer |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py` | GR00T topology |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/dreamzero/pipeline.py` | DreamZero topology |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/models/lingbot_world/pipeline.py` | LingBot-World topology |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py` | Talker→code2wav chunk/full-payload processors |
|
||||
| `../../vllm-omni/vllm_omni/model_executor/stage_input_processors/chunk_size_utils.py` | Chunk ramp / dynamic IC math |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/registry.py` | Diffusion pipeline registry (66) |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/diffusion_engine.py` | `DiffusionEngine` |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/sched/request_scheduler.py` | Request-level batching |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/sched/step_scheduler.py` | Step-level continuous batching |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/executor/` | Uniproc/multiproc executors |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/worker/diffusion_worker.py` | Diffusion worker, device init, sleep |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/worker/request_batch.py` | `DiffusionRequestBatch` |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/compile.py` | Regional `torch.compile` |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/attention/backends/registry.py` | Attention backend enum/overrides |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/attention/backends/abstract.py` | Backend capability contract |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/attention/selector.py` | Per-role backend resolution |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/attention/layer.py` | Diffusion `Attention` layer |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/layers/custom_op.py` | Platform op dispatch |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/diffusion_kv/manager.py` | Scheduler-owned paged KV for diffusion |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/model_loader/diffusers_loader.py` | Diffusers-style checkpoint loader |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/offloader/` | CPU/layerwise/distributed offload |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py` | π0 topology |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py` | π0 pipeline |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py` | GR00T pipeline |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py` | GR00T policy wrapper |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py` | GR00T model |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/pipeline_internvla_a1.py` | InternVLA-A1 pipeline |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py` | InternVLA-A1 policy |
|
||||
| `../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py` | Cosmos3 action modes |
|
||||
| `../../vllm-omni/vllm_omni/experimental/ar_diffusion/engine.py` | `ARDiffusionEngine` |
|
||||
| `../../vllm-omni/vllm_omni/experimental/ar_diffusion/capability.py` | AR-diffusion KV spec contract |
|
||||
| `../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/base.py` | `OmniConnectorBase` |
|
||||
| `../../vllm-omni/vllm_omni/distributed/omni_connectors/connectors/shm_connector.py` | Shared-memory connector |
|
||||
| `../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py` | Connector factory/registry |
|
||||
| `../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py` | Chunk lifecycle adapter |
|
||||
| `../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py` | msgpack encoder for tensors |
|
||||
| `../../vllm-omni/vllm_omni/distributed/omni_coordinator/omni_coordinator.py` | Replica membership/heartbeats |
|
||||
| `../../vllm-omni/vllm_omni/inputs/data.py` | Omni prompt types |
|
||||
| `../../vllm-omni/vllm_omni/outputs/__init__.py` | `OmniRequestOutput` |
|
||||
| `../../vllm-omni/vllm_omni/outputs/output_processor.py` | Multimodal output processor |
|
||||
| `../../vllm-omni/vllm_omni/data_entry_keys.py` | `OmniPayload` schema |
|
||||
| `../../vllm-omni/vllm_omni/request.py` | `OmniRequest`, `OmniStreamingUpdate` |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/omni.py` | Offline `Omni` |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/async_omni.py` | `AsyncOmni` EngineClient |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/cli/main.py` | CLI interception of `--omni` |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/cli/serve.py` | `vllm serve --omni` subcommand |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py` | HTTP/WS routes |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py` | TTS serving |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py` | TTS WebSocket streaming |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py` | Audio encode/resample |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/__init__.py` | TTS adapter registry |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters/base.py` | `TTSModelAdapter` |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py` | OpenPI robot serving |
|
||||
| `../../vllm-omni/vllm_omni/entrypoints/duplex/vad.py` | Silero VAD |
|
||||
| `../../vllm-omni/vllm_omni/quantization/factory.py` | Quantization config factory |
|
||||
| `../../vllm-omni/vllm_omni/quantization/` | Omni quant configs |
|
||||
| `../../vllm-omni/vllm_omni/utils/audio.py` | Mel filterbank utilities |
|
||||
| `../../vllm-omni/vllm_omni/tokenizers/mammoth_moda2_tokenizer.py` | Custom tokenizer |
|
||||
| `../../vllm-omni/docs/design/architecture_overview.md` | Architecture overview (mermaid, layers) |
|
||||
| `../../vllm-omni/docs/design/module/engine_orchestration.md` | Orchestration contract |
|
||||
| `../../vllm-omni/docs/design/module/stage_runtime.md` | Stage runtime contract |
|
||||
| `../../vllm-omni/docs/design/module/omni_connector.md` | Connector contract |
|
||||
| `../../vllm-omni/docs/design/module/execution_platforms.md` | Platform contract |
|
||||
| `../../vllm-omni/docs/design/module/ar_runtime.md` | AR runtime contract |
|
||||
| `../../vllm-omni/docs/design/module/input_output_modality_contracts.md` | I/O contracts |
|
||||
| `../../vllm-omni/docs/design/feature/async_chunk.md` | Async chunk design + perf table |
|
||||
| `../../vllm-omni/docs/design/feature/disaggregated_inference.md` | Connectors/D2H2D |
|
||||
| `../../vllm-omni/docs/design/feature/diffusion_continuous_batching.md` | Diffusion batching modes |
|
||||
| `../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md` | AR-diffusion sessions |
|
||||
| `../../vllm-omni/docs/design/feature/host_weight_runtime.md` | Host weight runtime |
|
||||
| `../../vllm-omni/docs/design/feature/prefix_caching.md` | Omni prefix caching |
|
||||
| `../../vllm-omni/docs/design/feature/omni_async_output_materialization.md` | Async output materialization |
|
||||
| `../../vllm-omni/docs/design/fullduplex.md` | Full-duplex runtime |
|
||||
| `../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md` | Repo-reported TTS perf |
|
||||
| `../../vllm-omni/docs/configuration/stage_configs.md` | Pipeline/deploy schema, precedence, stage CLI |
|
||||
| `../../vllm-omni/docs/getting_started/installation/README.md` | Platform list, vLLM alignment |
|
||||
| `../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md` | `vllm==0.28.0` pin, CC≥7.0 |
|
||||
| `../../vllm-omni/docs/getting_started/quickstart.md` | Linux/Python 3.12, offline API |
|
||||
| `../../vllm-omni/docs/user_guide/quantization/overview.md` | Quant modes × hardware matrix |
|
||||
| `../../vllm-omni/docs/user_guide/diffusion/execution_modes.md` | Diffusion execution modes |
|
||||
| `../../vllm-omni/docs/user_guide/diffusion/cpu_offload.md` | Offload strategies |
|
||||
| `../../vllm-omni/docs/user_guide/diffusion/regional_compilation.md` | Regional compile |
|
||||
| `../../vllm-omni/docs/features/custom_pipeline.md` | Custom pipeline extension |
|
||||
| `../../vllm-omni/docs/features/session_state_manager.md` | Session state manager |
|
||||
| `../../vllm-omni/docs/serving/openpi_api.md` | OpenPI WS protocol |
|
||||
| `../../vllm-omni/docs/serving/speech_api.md` | TTS HTTP API |
|
||||
| `../../vllm-omni/docs/serving/realtime_api.md` | Realtime audio WS |
|
||||
| `../../vllm-omni/docs/serving/realtime_duplex_api.md` | Duplex realtime |
|
||||
| `../../vllm-omni/docs/serving/full_duplex_api.md` | Full-duplex API |
|
||||
| `../../vllm-omni/docs/models/supported_models.md` | Model × hardware table |
|
||||
| `../../vllm-omni/docs/contributing/ci/test_execution_guide.md` | How to run tests |
|
||||
| `../../vllm-omni/docs/contributing/model/adding_tts_model.md` | TTS model guide |
|
||||
| `../../vllm-omni/docs/contributing/model/adding_omni_model.md` | Omni model guide |
|
||||
| `../../vllm-omni/docs/contributing/model/adding_diffusion_model.md` | Diffusion model guide |
|
||||
| `../../vllm-omni/tests/conftest.py` | Test bootstrap |
|
||||
| `../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py` | TTS e2e |
|
||||
| `../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py` | π0 e2e |
|
||||
| `../../vllm-omni/tests/e2e/online_serving/test_gr00t_openpi_expansion.py` | GR00T e2e |
|
||||
| `../../vllm-omni/.buildkite/cuda/test-ready.yml` | L1/L2 CI |
|
||||
| `../../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts/end2end.py` | Offline TTS example |
|
||||
| `../../vllm-omni/examples/online_serving/pi0/openpi_client.py` | π0 client |
|
||||
| `../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py` | DreamZero client |
|
||||
| `../../vllm-omni/examples/offline_inference/qwen3_omni/end2end_async_chunk.py` | Async chunk example |
|
||||
| `../../vllm-omni/examples/online_serving/qwen3_omni/openai_realtime_client.py` | Realtime client |
|
||||
| `../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py` | InternVLA example |
|
||||
| `../../vllm-omni/examples/offline_inference/custom_pipeline/image_to_image/custom_pipeline.py` | Custom pipeline example |
|
||||
| `../../vllm-omni/benchmarks/tts/bench_tts.py` | TTS benchmark |
|
||||
| `../../vllm-omni/recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md` | DGX Spark recipe (aarch64 hint) |
|
||||
| `../../vllm-omni/apps/ComfyUI-vLLM-Omni` | ComfyUI app |
|
||||
@@ -0,0 +1,512 @@
|
||||
# vLLM-Omni — TTS and VLA Model-Level Analysis
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Local path | `/data/zhoutaichang/embedding_infer/vllm-omni` |
|
||||
| Remote | `https://github.com/vllm-project/vllm-omni.git` (origin) |
|
||||
| Branch | `main` |
|
||||
| HEAD | `7be014bce6374f06c95b703763bdbac4c6198f31` (expected `7be014bc` — matches) |
|
||||
| Dirty status | clean (`git status --short` empty) |
|
||||
| Submodules | none (`git submodule status` empty, no `.gitmodules`) |
|
||||
| Repo AGENTS.md/CLAUDE.md | none at repo root; `.claude/skills/` contains contributor skills only |
|
||||
| Analysis date | 2026-09-07 (UTC) |
|
||||
| Scope | Model-level deep dive of TTS and VLA/world-model implementations, feeding `tts_edge_design.md` / `vla_edge_design.md` |
|
||||
|
||||
Evidence labels: **[Verified]** = read in the cited source; **[Inferred]** = reasoned from structure; **[Proposal]** = design suggestion; **[Unknown]** = could not determine. Line numbers (`#Lnnn`) are from the cited checkout. Delegated read-outs by subagents were spot-checked against the tree; link existence was verified mechanically (see §8).
|
||||
|
||||
---
|
||||
|
||||
## 0. Summary
|
||||
|
||||
- vLLM-Omni is a **serving framework**, not an execution runtime: it composes vLLM's AR runtime (paged KV, continuous batching, CUDA graphs) with two additional stage types — `LLM_GENERATION` (non-AR "generation worker", used for codec decoders/vocoders) and `DIFFUSION` (its own diffusion worker) — orchestrated as multi-stage pipelines ([`stage_config.py#L194`](../../vllm-omni/vllm_omni/config/stage_config.py#L194), [`architecture_overview.md`](../../vllm-omni/docs/design/architecture_overview.md)). [Verified]
|
||||
- Model topology is code (`PipelineConfig` in each `models/<family>/pipeline.py`, registered in [`pipeline_registry.py`](../../vllm-omni/vllm_omni/config/pipeline_registry.py)); placement/resources are YAML in [`vllm_omni/deploy/`](../../vllm-omni/vllm_omni/deploy) (90 files). There are **no** stage YAMLs under `vllm_omni/config/`. [Verified]
|
||||
- 40 directories under `model_executor/models` (39 model families, every one shipping a `pipeline.py`, plus `common/` for shared kernels/modules; `output_templates.py` defines `OmniOutput`); by primary use: 15 TTS (+ Ming-flash-omni's TTS variant), 12 speech-to-speech/omni, 1 music, 5 image, 4 video/world, 2 VLA (+ `pi0` and `internvla_a1` living under `diffusion/models`). See §1. [Verified]
|
||||
- **Dominant TTS pattern**: stage 0 = AR talker on a vLLM `Qwen2/Qwen3Model` backbone producing multi-codebook codec tokens (often with a per-frame sub-AR "code predictor"/"depth transformer"), stage 1 = codec decoder on the generation worker, connected by a `SharedMemoryConnector` carrying msgpack-serialised **CPU** tensors. [Verified]
|
||||
- Streaming ("async chunk") is real and per-family: Qwen3-TTS emits the first chunk after **1 frame (80 ms)**, Fish after 4 frames, CosyVoice3 after 15+3 tokens with **cumulative re-decode**, MOSS-Delay variants effectively **not streaming** (`chunk_frames = 1<<30`), MOSS-Local after 1 frame. [Verified]
|
||||
- Codec state across chunks is explicit and model-owned: Qwen3-TTS keeps 2-frame/12-frame conv contexts + a 72-frame sliding-window KV; MOSS keeps ring-KV slots; CosyVoice3 keeps cumulative mel; Fish keeps nothing (25-frame overlap re-decode). [Verified]
|
||||
- **VLA models bypass vLLM entirely**: GR00T N1.7 and π0 are single `DIFFUSION`-stage wrappers around plain HF/torch models; observations arrive as raw numpy dicts in `OmniDiffusionSamplingParams.extra_args["robot_obs"]` over an OpenPI-compatible msgpack WebSocket (`/v1/realtime/robot/openpi`), actions return as `multimodal_output["actions"]`. No paged attention, no compile, `enforce_eager: true`, batch = 1. [Verified]
|
||||
- DreamZero and LingBot-World are **AR-diffusion world models** with an experimental engine (`ARDiffusionEngine`) owning per-session paged KV; DreamZero = Wan-style 14B causal DiT + UMT5-XXL + CLIP ViT-H + Wan VAE, emitting a 24×8 action chunk per tick. Far outside edge budgets. [Verified]
|
||||
- Edge-relevant small models in tree (label sources in §1): MOSS-TTS-Nano 0.1B, Gepard ~0.5B, CosyVoice3 0.5B, Ming-TTS 0.5B, Qwen3-TTS 0.6B/1.7B, MOSS-TTS-Realtime 1.7B, VoxCPM2 2B (L4-verified), Audex 2B, GR00T-N1.7 3B, π0 (~2B+0.3B+SigLIP). [Verified labels, sizes not computed]
|
||||
- Portable sub-modules (plain torch, exportable in principle): Qwen3-TTS 12 Hz tokenizer decoder, Fish DAC decoder, CosyVoice3 CFM + HiFT, MOSS codec (transformer-only), GR00T policy + DiT head, π0 model. Non-portable: every AR talker (vLLM `Qwen3Model`/paged attention), CUDA-graph wrappers, TensorRT paths, Triton Snake kernels (with torch fallback). See §5. [Verified/Inferred]
|
||||
- Platforms actually in tree: CUDA, ROCm, Ascend NPU, Intel XPU, MUSA ([`platforms/`](../../vllm-omni/vllm_omni/platforms)); **no CPU platform**, no Apple/Android/Jetson-specific code; `requirements/cpu.txt` exists only for docs builds. [Verified]
|
||||
- Dependency floor: `vllm==0.28.0`, `transformers>=5.10.1,<5.15`, `diffusers==0.40.0`, `torchaudio` only pinned for NPU, `soundfile`, `onnxruntime`, `openai-whisper`; optional `flash-attn-4`, `tensorrt` (CosyVoice3/Step-Audio2), NeMo (Gepard/IndexTTS2), `fish-speech` pip package. [Verified]
|
||||
- Repo-reported (not reproduced) TTS numbers: Qwen3-TTS TTFP 64 ms / RTF 0.16 on H200 at c=1 ([`qwen3_omni_tts_performance_optimization.md`](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md)); async-chunk cuts Qwen3-Omni TTFP 6.5 s→0.52 s on H800 ([`async_chunk.md`](../../vllm-omni/docs/design/feature/async_chunk.md)).
|
||||
- Key gaps for an edge port: no CPU/mobile backend, model code entangled with vLLM layer classes for all talkers, per-family bespoke stage processors, control loops with no KV reuse across steps (GR00T/π0 recompute full VLM prefill every tick), and several documented-but-unimplemented knobs (§7).
|
||||
|
||||
---
|
||||
|
||||
## 1. Model families inventory
|
||||
|
||||
Every directory under [`vllm_omni/model_executor/models`](../../vllm-omni/vllm_omni/model_executor/models). Stage notation: `id:model_stage(exec, arch, engine_out→final_out)`. Pipeline symbols are in each family's `pipeline.py`; registry keys from [`pipeline_registry.py#L130`](../../vllm-omni/vllm_omni/config/pipeline_registry.py#L130). Async flag from the family's default deploy YAML. Parameter counts are **labels** (HF id / comment / recipe), never computed. [Verified unless marked]
|
||||
|
||||
### 1a. TTS (text → speech)
|
||||
|
||||
| Family | in → out | Architecture type | Stages (async) | Example | Key files | Codec / vocoder | Params (source) |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| [`qwen3_tts`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts) | text+instruct+ref-audio/speaker → 24 kHz | AR (Qwen3) + code predictor + codec | `0:qwen3_tts(LLM_AR, Qwen3TTSTalkerForConditionalGeneration, latent)`, `1:code2wav(LLM_GENERATION, Qwen3TTSCode2Wav, audio→audio)` (async **true**) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/qwen3_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/qwen3_tts) | [`qwen3_tts_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py), [`qwen3_tts_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code2wav.py), [`prompt_embeds_builder.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/prompt_embeds_builder.py) | 12 Hz Mimi-style RVQ (16×2048) + causal conv/Snake decoder | 0.6B / 1.7B (HF ids, [recipe](../../vllm-omni/recipes/Qwen/Qwen3-TTS.md)) |
|
||||
| [`cosyvoice3`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3) | text+ref-audio+ref-text → 24 kHz | AR (Qwen2-0.5B) → CFM DiT → HiFT | `0:cosyvoice3_talker(LLM_AR, CosyVoice3Model, latent)`, `1:cosyvoice3_code2wav(LLM_GENERATION, latent→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/cosyvoice3), [online](../../vllm-omni/examples/online_serving/text_to_speech/cosyvoice3) | [`cosyvoice3.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py), [`cfm.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py), [`hifigan.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py) | 25 Hz FSQ tokens → 50 Hz mel (CFM, 10 Euler steps) → HiFT ISTFT | 0.5B (HF id) |
|
||||
| [`fish_speech`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech) | text+ref-audio+ref-text → 44.1 kHz | dual-AR (Qwen3 slow + 4-layer fast) + DAC | `0:fish_speech_slow_ar(LLM_AR, FishSpeechSlowARForConditionalGeneration, latent)`, `1:dac_decoder(LLM_GENERATION, FishSpeechDACDecoder, audio→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/fish_speech), [online](../../vllm-omni/examples/online_serving/text_to_speech/fish_speech) | [`fish_speech_slow_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_slow_ar.py), [`fish_speech_fast_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_fast_ar.py), [`fish_speech_dac_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_dac_decoder.py) | DAC (1 semantic 4096 + 9 residual 1024), hop 2048 ≈ 21.5 Hz | "4B" ([`speech_api.md`](../../vllm-omni/docs/serving/speech_api.md)) |
|
||||
| [`moss_tts`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts) | text+ref-audio/instruction → 24 kHz (Local: 48 kHz stereo) | AR (Qwen3 + n_vq+1 heads, delay pattern; Realtime/Local: depth transformer) + transformer codec | `0:moss_tts(LLM_AR, MossTTSDelayModel\|MossTTSRealtime\|MossTTSLocalModel, latent)`, `1:moss_tts_codec(LLM_GENERATION, MossTTSCodecDecoder, audio→audio, retains_state_across_chunks)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/moss_tts) | [`modeling_moss_tts_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py), [`modeling_moss_tts_codec.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py), [`audio_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py) | MOSS Audio Tokenizer (RLFQ, 12.5 Hz, transformer-only) | 8B / 1.7B ([`registry.py#L364`](../../vllm-omni/vllm_omni/model_executor/models/registry.py#L364), YAML headers) |
|
||||
| [`moss_tts_nano`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano) | text(+ref) → speech | single-stage AR (HF remote code) | `0:moss_tts_nano(LLM_AR, MossTTSNanoForCausalLM, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/moss_tts_nano), [online](../../vllm-omni/examples/online_serving/text_to_speech/moss_tts_nano) | [`modeling_moss_tts_nano.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano/modeling_moss_tts_nano.py) | MOSS-Audio-Tokenizer-Nano | 0.1B ([`pipeline.py#L5`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano/pipeline.py#L5)) |
|
||||
| [`dots_tts`](../../vllm-omni/vllm_omni/model_executor/models/dots_tts) | text+x-vector → 48 kHz | single-stage AR (Qwen2.5-1.5B) + DiTAR/MeanFlow + AudioVAE + BigVGAN | `0:latent_generator(LLM_AR, DotsTTSForConditionalGeneration, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/dots_tts) | [`dots_tts_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/dots_tts/dots_tts_talker.py), [`dots_tts_vocoder.py`](../../vllm-omni/vllm_omni/model_executor/models/dots_tts/dots_tts_vocoder.py) | AudioVAE → BigVGAN 48 kHz | ~1.7B ([recipe](../../vllm-omni/recipes/rednote-hilab/dots.tts.md)) |
|
||||
| [`gepard`](../../vllm-omni/vllm_omni/model_executor/models/gepard) | text → 22 kHz | single-stage AR (Qwen3.5 + 32 FSQ heads) | `0:gepard(LLM_AR, GepardTalkerForConditionalGeneration, audio→audio)` (async false, eager) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/gepard) | [`gepard_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/gepard/gepard_talker.py), [`nanocodec.py`](../../vllm-omni/vllm_omni/model_executor/models/gepard/nanocodec.py) | NeMo NanoCodec (external dep) | ~0.5B ([`gepard.yaml#L1`](../../vllm-omni/vllm_omni/deploy/gepard.yaml#L1)) |
|
||||
| [`glm_tts`](../../vllm-omni/vllm_omni/model_executor/models/glm_tts) | text+ref-audio (WhisperVQ) → 24/32 kHz | AR → CFM DiT → HiFT/Vocos | `0:glm_tts(LLM_AR, GLMTTSForConditionalGeneration, latent)`, `1:glm_tts_dit(LLM_GENERATION, latent→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/glm_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/glm_tts) | [`glm_tts.py`](../../vllm-omni/vllm_omni/model_executor/models/glm_tts/glm_tts.py), [`glm_tts_dit.py`](../../vllm-omni/vllm_omni/model_executor/models/glm_tts/glm_tts_dit.py), [`whisper_vq.py`](../../vllm-omni/vllm_omni/model_executor/models/common/whisper_vq.py) | HiFT (reused from CosyVoice3) or Vocos2D TorchScript | Unknown |
|
||||
| [`higgs_audio_v2`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v2) | text → 24 kHz | AR (Llama-3.2-3B, 8 codebooks, DualFFN) + RVQ/DAC | `0:higgs_audio_v2(LLM_AR, latent)`, `1:code2wav(LLM_GENERATION, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/higgs_audio_v2) | [`higgs_audio_v2_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v2/higgs_audio_v2_talker.py), [`higgs_audio_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v2/higgs_audio_decoder.py) | RVQ 8×1024 + DAC decoder, 25 fps | 3B (HF id) |
|
||||
| [`higgs_audio_v3`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v3) | text+ref → 24 kHz | AR (Qwen3 ~4B, delay pattern) + RVQ/DAC | same shape, key `higgs_multimodal_qwen3` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/higgs_audio_v3), [online](../../vllm-omni/examples/online_serving/text_to_speech/higgs_audio_v3) | [`higgs_audio_v3_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/higgs_audio_v3/higgs_audio_v3_talker.py) | v2 decoder | ~4B (talker docstring) |
|
||||
| [`indextts2`](../../vllm-omni/vllm_omni/model_executor/models/indextts2) | text+ref(+emotion ref) → speech | GPT2-style AR → S2Mel CFM (25 steps) + BigVGAN | `0:indextts2_talker(LLM_AR, latent)`, `1:indextts2_s2mel_decoder(LLM_GENERATION, audio→audio)` (async **false**, non-streaming) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/indextts2), [online](../../vllm-omni/examples/online_serving/text_to_speech/indextts2) | [`indextts2_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/indextts2/indextts2_talker.py), [`indextts2_s2mel_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/indextts2/indextts2_s2mel_decoder.py) | S2Mel CFM → BigVGAN; w2v-bert (~600M) semantic front-end | Unknown |
|
||||
| [`ming_tts`](../../vllm-omni/vllm_omni/model_executor/models/ming_tts) | text+ref (CAM++) → 44.1 kHz | AR LM + per-step CFM head (DiTAR) → AudioVAE | `0:llm(LLM_AR, MingTTSForConditionalGeneration, latent)`, `1:audio_vae(LLM_GENERATION, audio→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/ming_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/ming_tts) | [`ming_tts.py`](../../vllm-omni/vllm_omni/model_executor/models/ming_tts/ming_tts.py), [`common/ming/cfm_dit.py`](../../vllm-omni/vllm_omni/model_executor/models/common/ming/cfm_dit.py) | AudioVAE (Qwen2 decoder + ISTFT) | 0.5B dense / 16.8B-A3B MoE ([`pipeline.py#L18`](../../vllm-omni/vllm_omni/model_executor/models/ming_tts/pipeline.py#L18)) |
|
||||
| [`omnivoice`](../../vllm-omni/vllm_omni/model_executor/models/omnivoice) | text+ref → 24 kHz | non-AR masked-generative (Qwen3 backbone) → DAC | `0:dit(DIFFUSION, OmniVoicePipeline, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/omnivoice), [online](../../vllm-omni/examples/online_serving/text_to_speech/omnivoice) | [`omnivoice_generator.py`](../../vllm-omni/vllm_omni/model_executor/models/omnivoice/omnivoice_generator.py), [`fused_qkv_rope.py`](../../vllm-omni/vllm_omni/model_executor/models/omnivoice/fused_qkv_rope.py) (Triton) | RVQ + `transformers` DacModel | Unknown |
|
||||
| [`voxcpm2`](../../vllm-omni/vllm_omni/model_executor/models/voxcpm2) | text+ref → 48 kHz | single-stage AR (MiniCPM4 paged) + LocDiT CFM + AudioVAE | `0:latent_generator(LLM_AR, VoxCPM2TalkerForConditionalGeneration, audio→audio)`, custom scheduler (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/voxcpm2), [online](../../vllm-omni/examples/online_serving/text_to_speech/voxcpm2) | [`voxcpm2_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/voxcpm2/voxcpm2_talker.py), [`scheduler.py`](../../vllm-omni/vllm_omni/model_executor/models/voxcpm2/scheduler.py) | LocDiT CFM → AudioVAE | 2B ([recipe](../../vllm-omni/recipes/OpenBMB/VoxCPM2.md)); L4-24GB target ([`voxcpm2.yaml#L2`](../../vllm-omni/vllm_omni/deploy/voxcpm2.yaml#L2)) |
|
||||
| [`voxtral_tts`](../../vllm-omni/vllm_omni/model_executor/models/voxtral_tts) | text+voice → 24 kHz | AR LM → flow-matching acoustic transformer + codec | `0:audio_generation(LLM_AR, latent)`, `1:audio_tokenizer(LLM_GENERATION, audio→audio)` (async true) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/voxtral_tts), [online](../../vllm-omni/examples/online_serving/text_to_speech/voxtral_tts) | [`voxtral_tts_audio_generation.py`](../../vllm-omni/vllm_omni/model_executor/models/voxtral_tts/voxtral_tts_audio_generation.py) | Mistral audio tokenizer (Euler ODE + CFG) | 4B (HF id) |
|
||||
| [`ming_flash_omni`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni) (TTS variant `ming_flash_omni_tts`) | text → speech | single `LLM_GENERATION` talker (Qwen2 + CFM + AudioVAE) | `0:ming_tts(LLM_GENERATION, MingFlashOmniTalkerForConditionalGeneration, audio→audio)` (async false) | [offline](../../vllm-omni/examples/offline_inference/text_to_speech/ming_flash_omni_tts) | [`ming_flash_omni_talker.py`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni/ming_flash_omni_talker.py) | CFM DiT + AudioVAE | Unknown |
|
||||
|
||||
### 1b. Speech-to-speech / omni (multimodal in → text + speech)
|
||||
|
||||
| Family | in → out | Architecture | Stages (async) | Example | Key files | Vocoder | Params |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| [`qwen2_5_omni`](../../vllm-omni/vllm_omni/model_executor/models/qwen2_5_omni) | text/image/audio/video → text+speech | thinker AR → talker AR → DiT+BigVGAN | `0:thinker(LLM_AR)`, `1:talker(LLM_AR)`, `2:code2wav(LLM_GENERATION)` (async false) | [offline](../../vllm-omni/examples/offline_inference/qwen2_5_omni) | [`qwen2_5_omni_token2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_token2wav.py) | DiT (50 Hz) + BigVGAN | 7B / 3B |
|
||||
| [`qwen3_omni`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_omni) | same → text+16 kHz speech | MoE thinker → MoE talker (+code predictor) → ConvNeXt Code2Wav | 3 stages, async **true** (reference streaming impl) | [offline](../../vllm-omni/examples/offline_inference/qwen3_omni), [online](../../vllm-omni/examples/online_serving/qwen3_omni) | [`qwen3_omni_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_omni/qwen3_omni_code2wav.py) | ~200M ConvNet ([doc](../../vllm-omni/docs/design/architecture_overview.md)) | 30B-A3B |
|
||||
| [`minicpmo_4_5`](../../vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5) | omni → text+24 kHz; **full-duplex** | thinker AR → TTS AR → CFM+HiFT | 3 stages, async true, `duplex_control_enabled` | [offline](../../vllm-omni/examples/offline_inference/minicpmo), [online](../../vllm-omni/examples/online_serving/minicpmo) | [`minicpmo_4_5_omni_llm.py`](../../vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/minicpmo_4_5_omni_llm.py), [`duplex/stage0.py`](../../vllm-omni/vllm_omni/model_executor/models/minicpmo_4_5/duplex/stage0.py) | CosyVoice-style CFM + HiFT | Unknown |
|
||||
| [`step_audio2`](../../vllm-omni/vllm_omni/model_executor/models/step_audio2) | audio+text → text+speech | AR thinker → CosyVoice2 flow + HiFT | 2 stages (async false default; `_async_chunk.yaml` true) | [offline](../../vllm-omni/examples/offline_inference/step_audio2) | [`step_audio2_token2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/step_audio2/step_audio2_token2wav.py), [`step_audio2_dit_trt.py`](../../vllm-omni/vllm_omni/model_executor/models/step_audio2/step_audio2_dit_trt.py) | CosyVoice2 DiT (+TRT) + HiFT | Unknown |
|
||||
| [`mimo_audio`](../../vllm-omni/vllm_omni/model_executor/models/mimo_audio) | audio+text → text+24 kHz | fused thinker/talker AR + patch decoder → Vocos | 2 stages, async true; RTX 5090 profile | [offline](../../vllm-omni/examples/offline_inference/mimo_audio) | [`mimo_audio_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/mimo_audio/mimo_audio_code2wav.py) | VQ → Vocos ISTFT | 7B |
|
||||
| [`covo_audio`](../../vllm-omni/vllm_omni/model_executor/models/covo_audio) | audio+text → text+24 kHz | fused AR → CFM + BigVGAN | 2 stages, async false | [offline](../../vllm-omni/examples/offline_inference/covo_audio) | [`token2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/covo_audio/token2wav.py) | CFM + BigVGAN | 7B |
|
||||
| [`dynin_omni`](../../vllm-omni/vllm_omni/model_executor/models/dynin_omni) | text → text+image+audio | 3× `LLM_GENERATION` chain | async false; L4-sized CI profile | [offline](../../vllm-omni/examples/offline_inference/dynin_omni) | [`dynin_omni_common.py`](../../vllm-omni/vllm_omni/model_executor/models/dynin_omni/dynin_omni_common.py) | external tokenizer | Unknown |
|
||||
| [`ming_flash_omni`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni) | omni → text/speech/image | BailingMoE-v2 AR (TP4) → talker / DiT | async false | [offline](../../vllm-omni/examples/offline_inference/ming_flash_omni) | [`modeling_bailing_moe_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/ming_flash_omni/modeling_bailing_moe_v2.py) | CFM+AudioVAE | Unknown (MoE) |
|
||||
| [`aura_omni`](../../vllm-omni/vllm_omni/model_executor/models/aura_omni) | mic audio+video → text+speech | 4-stage cascade ASR→VLM→Qwen3-TTS→Code2Wav | async false, single GPU | [offline](../../vllm-omni/examples/offline_inference/aura_omni) | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/aura_omni/pipeline.py) | Qwen3-TTS Code2Wav | 1.7B+?+1.7B |
|
||||
| [`audex`](../../vllm-omni/vllm_omni/model_executor/models/audex) | text→16 kHz (TTS), audio→text/speech (S2S), caption→audio (TTA) | Nemotron AR + streaming causal speech decoder / XCodec1 | 4 pipelines; TTS/S2S async true | [offline](../../vllm-omni/examples/offline_inference/audex) | [`speech_decoder/modeling_audex_causal_speech_decoder.py`](../../vllm-omni/vllm_omni/model_executor/models/audex/speech_decoder/modeling_audex_causal_speech_decoder.py) | Audex causal decoder (4-frame lookahead) | 2B / 30B-A3B ([recipe](../../vllm-omni/recipes/NVIDIA/Nemotron-Labs-Audex.md)) |
|
||||
| [`nemotron_voicechat`](../../vllm-omni/vllm_omni/model_executor/models/nemotron_voicechat) | speech → text+22.05 kHz; full-duplex | FastConformer + NemotronH AR → EAR-TTS AR → RVQ-VAE | 3 stages; `_duplex.yaml` | [offline](../../vllm-omni/examples/offline_inference/nemotron_voicechat) | [`nemo_vendored/ear_tts_model.py`](../../vllm-omni/vllm_omni/model_executor/models/nemotron_voicechat/nemo_vendored/ear_tts_model.py) | NeMo RVQ-VAE (fp32) | 11B |
|
||||
| [`personaplex`](../../vllm-omni/vllm_omni/model_executor/models/personaplex) | audio → audio (Moshi finetune, full-duplex) | Helium temporal AR + depformer + Mimi | 2 stages, async true | [online](../../vllm-omni/examples/online_serving/personaplex) | [`personaplex_depformer.py`](../../vllm-omni/vllm_omni/model_executor/models/personaplex/personaplex_depformer.py), [`personaplex_mimi.py`](../../vllm-omni/vllm_omni/model_executor/models/personaplex/personaplex_mimi.py) | Mimi 24 kHz | Unknown |
|
||||
|
||||
### 1c. Music, image, video / world
|
||||
|
||||
| Family | in → out | Architecture | Stages | Key files | Params |
|
||||
|---|---|---|---|---|---|
|
||||
| [`minimax_music3`](../../vllm-omni/vllm_omni/model_executor/models/minimax_music3) | caption+lyrics → 32 kHz stereo music | AR (Qwen3 + depth decoder, 8 RVQ, mandatory CFG) → CFM DiT + DAV | 2 stages, async true, 140 GB card | [`talker.py`](../../vllm-omni/vllm_omni/model_executor/models/minimax_music3/talker.py), [`dit.py`](../../vllm-omni/vllm_omni/model_executor/models/minimax_music3/dit.py) | Unknown |
|
||||
| [`bagel`](../../vllm-omni/vllm_omni/model_executor/models/bagel) | text+image → text/image | AR thinker (KV transfer) → DiT; single-stage variant | 3 pipelines | [`bagel.py`](../../vllm-omni/vllm_omni/model_executor/models/bagel/bagel.py) | 7B |
|
||||
| [`glm_image`](../../vllm-omni/vllm_omni/model_executor/models/glm_image) | text → image | AR VQ tokens → DiT | 2 stages, 2 GPUs | [`glm_image_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/glm_image/glm_image_ar.py) | Unknown |
|
||||
| [`hunyuan_image3`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_image3) | text+image → text+image | MoE AR → DiT (KV reuse) | 3 pipelines, TP2–4 | [`hunyuan_image3.py`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_image3/hunyuan_image3.py) | 80B (comment) |
|
||||
| [`mammoth_moda2`](../../vllm-omni/vllm_omni/model_executor/models/mammoth_moda2) | text+image → image | AR (Qwen2.5-VL) → DiT on generation worker | 2 stages | [`mammoth_moda2.py`](../../vllm-omni/vllm_omni/model_executor/models/mammoth_moda2/mammoth_moda2.py) | Unknown |
|
||||
| [`lance`](../../vllm-omni/vllm_omni/model_executor/models/lance) | text+image → image/text/video | single DiT stage (embedded Qwen2-MoT + Wan VAE) | 1 stage | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/lance/pipeline.py) | 3B (recipe) |
|
||||
| [`wan2_2`](../../vllm-omni/vllm_omni/model_executor/models/wan2_2) | text(+image) → video | diffusion DiT | 1 stage | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/wan2_2/pipeline.py) | 5B (YAML) |
|
||||
| [`hunyuan_video`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_video) | text/image → video | diffusion DiT | 1 stage | [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/hunyuan_video/pipeline.py) | Unknown |
|
||||
| [`minimax_h3`](../../vllm-omni/vllm_omni/model_executor/models/minimax_h3) | text+refs → video+audio | Qwen3-VL text encoder (LLM_AR) → inline DiT | 2 stages, 6 GPUs default | [`text_encoder.py`](../../vllm-omni/vllm_omni/model_executor/models/minimax_h3/text_encoder.py) | Unknown |
|
||||
| [`lingbot_world`](../../vllm-omni/vllm_omni/model_executor/models/lingbot_world) | image+text+camera controls → streaming video | causal AR-diffusion DiT, paged KV (world model) | 1 DIFFUSION stage, `ARDiffusionEngine` | [`diffusion/models/lingbot_world/pipeline.py`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/pipeline.py) | 14B (HF id) |
|
||||
|
||||
### 1d. VLA / robot policy
|
||||
|
||||
| Family | in → out | Architecture | Stages | Example | Key files | Params |
|
||||
|---|---|---|---|---|---|---|
|
||||
| [`gr00t`](../../vllm-omni/vllm_omni/model_executor/models/gr00t) | images+state+instruction → action chunk | Qwen3-VL (Cosmos-Reason2-2B, 12 layers) + flow-matching DiT head | `0:diffusion(DIFFUSION, Gr00tN1d7Pipeline, →actions)`; `max_num_seqs 1`, eager | none in `examples/` (recipe only) | [`pipeline_gr00t.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py), [`policy.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py), [`gr00t_n1d7.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py) | 3B (HF id) |
|
||||
| [`dreamzero`](../../vllm-omni/vllm_omni/model_executor/models/dreamzero) | images+state+text → actions (+video latents) | Wan-style causal 14B DiT world model + UMT5-XXL + CLIP ViT-H + Wan VAE | `0:diffusion(DIFFUSION, DreamZeroPipeline, →image)`; `ARDiffusionEngine`; CFG-parallel 2 | [offline](../../vllm-omni/examples/offline_inference/dreamzero), [online](../../vllm-omni/examples/online_serving/dreamzero) | [`pipeline_dreamzero.py`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/pipeline_dreamzero.py), [`causal_wan_model.py`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/causal_wan_model.py) | 14B (comment [`causal_wan_model.py#L743`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/causal_wan_model.py#L743)) |
|
||||
| `pi0` (under `diffusion/models`) | 3 images+state+prompt → 50×32 actions | PaliGemma (SigLIP+Gemma-2B) + Gemma-300M action expert, flow matching | `0:diffusion(DIFFUSION, Pi0Pipeline, →action)` ([`pi0_pipeline_config.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py)) | [online](../../vllm-omni/examples/online_serving/pi0) | [`pipeline_pi0.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0/pipeline_pi0.py), [`modeling_pi0.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0/modeling_pi0.py) | "~13 GB checkpoint" ([recipe](../../vllm-omni/recipes/lerobot/Pi0.md)) |
|
||||
| `internvla_a1` (diffusion only, no PipelineConfig) | 3 cams×2 frames+state+task → 50×32 actions | Qwen3-VL und/gen/act three-expert + Cosmos CI8x8 tokenizer, flow matching | offline-only via `pipeline.forward` | [offline](../../vllm-omni/examples/offline_inference/internvla_a1) | [`pipeline_internvla_a1.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/pipeline_internvla_a1.py), [`model_internvla_a1.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py) | 3B (HF id) |
|
||||
|
||||
### 1e. Shared `common/` modules [Verified]
|
||||
|
||||
[`common/`](../../vllm-omni/vllm_omni/model_executor/models/common): [`snake_activation.py`](../../vllm-omni/vllm_omni/model_executor/models/common/snake_activation.py) (Snake/SnakeBeta, lazy `@triton.jit` on CUDA with torch fallback; used by Qwen3-TTS, Qwen3-Omni, Qwen2.5-Omni, CoVo, CosyVoice3); [`alias_free_activation.py`](../../vllm-omni/vllm_omni/model_executor/models/common/alias_free_activation.py) (plain torch); [`qwen3_code_predictor.py`](../../vllm-omni/vllm_omni/model_executor/models/common/qwen3_code_predictor.py) (shared code predictor, SDPA + `torch.compile` + optional manual CUDA graphs); [`nucleus_ras_sampling.py`](../../vllm-omni/vllm_omni/model_executor/models/common/nucleus_ras_sampling.py) (used by GLM-TTS only; CosyVoice3 has its own copy); [`whisper_vq.py`](../../vllm-omni/vllm_omni/model_executor/models/common/whisper_vq.py); [`cfg_pairing.py`](../../vllm-omni/vllm_omni/model_executor/models/common/cfg_pairing.py) (CFG request pairing for the vLLM scheduler); [`ops/fused_group_norm_silu.py`](../../vllm-omni/vllm_omni/model_executor/models/common/ops/fused_group_norm_silu.py) and [`ops/fused_adaptive_group_norm_silu.py`](../../vllm-omni/vllm_omni/model_executor/models/common/ops/fused_adaptive_group_norm_silu.py) (Triton `@triton.autotune`/`@triton.jit`, torch fallback when `HAS_TRITON` false); [`ming/`](../../vllm-omni/vllm_omni/model_executor/models/common/ming) (CFM DiT, AudioVAE, batch-1 CUDA-graph executor `cfm_cudagraph.py`, CAM++ via onnxruntime).
|
||||
|
||||
---
|
||||
|
||||
## 2. TTS deep dive
|
||||
|
||||
Four families chosen for documentation depth and architectural spread: **Qwen3-TTS** (AR + code predictor + neural codec, streaming reference), **CosyVoice3** (AR + flow-matching + HiFT), **Fish Speech S2 Pro** (dual-AR + DAC), **MOSS-TTS** (delay-pattern / depth-transformer + transformer codec). Cross-family mechanics are in §4.
|
||||
|
||||
### 2.1 Qwen3-TTS (`qwen3_tts`)
|
||||
|
||||
**Pipeline** [Verified] — [`pipeline.py#L16`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/pipeline.py#L16): stage 0 `qwen3_tts` (`LLM_AR`, `Qwen3TTSTalkerForConditionalGeneration`, `engine_output_type="latent"`, `owns_tokenizer`, `sampling_constraints={"detokenize": False, "stop_token_ids": [2150]}`, async func `talker2code2wav_async_chunk`, full-payload func `talker2code2wav_full_payload`); stage 1 `code2wav` (`LLM_GENERATION`, `Qwen3TTSCode2Wav`, `final_output_type="audio"`, `sync_process_input_func=talker2code2wav_token_only`, `requires_full_payload_input`, `extras.tts_args.max_instructions_length=500`). Deploy [`qwen3_tts.yaml`](../../vllm-omni/vllm_omni/deploy/qwen3_tts.yaml): `async_chunk: true`; connector extras `codec_chunk_frames: 25`, `codec_left_context_frames: 72` ("must match the decoder sliding attention window"), `initial_codec_chunk_frames: 1`, `decode_batch_max_size: 4`; stage 0 `max_num_seqs 64`, `gpu_memory_utilization 0.3`, `max_model_len 4096`, sampling `temperature 0.9, top_k 50, repetition_penalty 1.05, min_tokens 2`, `subtalker_sampling_params {do_sample, 0.9, top_k 50}`; stage 1 `max_num_seqs 64`, `enforce_eager: false`, `max_model_len 65536` ("prefill length is Q × num_frames"); `platforms.rocm` forces stage 1 eager (MIOpen Conv1d inside graph capture), `platforms.npu` uses `cudagraph_mode: PIECEWISE`. High-concurrency 2-GPU profile: [`qwen3_tts_high_concurrency.yaml`](../../vllm-omni/vllm_omni/deploy/qwen3_tts_high_concurrency.yaml).
|
||||
|
||||
**Conditioning** [Verified] — [`prompt_embeds_builder.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/prompt_embeds_builder.py) `Qwen3TTSPromptEmbedsBuilder.build_prompt_embeds()` (L918-1455): text via HF `AutoTokenizer` in `<|im_start|>assistant\n{text}<|im_end|>` template, embedded through a separate `text_embedding` table + `text_projection` MLP ([`qwen3_tts_talker.py#L371`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L371)); `instruct` prepended as a user turn; `language` inserted as codec "think" tokens from `talker_config.codec_language_id`; **CustomVoice** = speaker id embedded via codec table; **VoiceDesign** = tags + text only; **Base** (clone) = ECAPA-TDNN speaker encoder `Qwen3TTSSpeakerEncoder` ([`qwen3_tts_talker.py#L202`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L202)) over a 128-mel spectrogram (24 kHz, n_fft 1024, hop 256), optionally plus ICL: reference audio encoded to codec frames by `Qwen3TTSTokenizerV2Encoder` (a `MimiModel` subclass, [`modeling_qwen3_tts_tokenizer_v2.py#L1587`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py#L1587)) in `_encode_ref_audio_batch` ([`#L1039`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L1039)) — 16 codebooks kept, cached by SHA1 of waveform (LRU 256). Audio loading: `soundfile` for base64, vLLM `MediaConnector`/`load_audio` + `AudioResampler` otherwise (no direct librosa/torchaudio import in this dir). Streaming vs non-streaming prompt mode: non-streaming (default for CustomVoice/VoiceDesign) puts full text in prefill; streaming (default for Base) feeds one text-embedding row per decode step from a `trailing_text_hidden` queue ([`#L833`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L833)).
|
||||
|
||||
**Generation** [Verified] — AR over codec **frames**: vLLM samples codebook-0 per step from `lm_head` (logits masked to `[1, codebook_vocab) ∪ {eos}`, [`#L414`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L414)); residual codebooks 1..Q-1 predicted by the **code predictor** ([`common/qwen3_code_predictor.py`](../../vllm-omni/vllm_omni/model_executor/models/common/qwen3_code_predictor.py) `CodePredictorWrapper.forward` L996-1138): a 5-layer / hidden-1024 / 16-head decoder (config defaults [`configuration_qwen3_tts.py#L191`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/configuration_qwen3_tts.py#L191)) that **re-prefills** `[proj(hidden), proj(embed(c0)), …]` per codebook with SDPA and **no KV cache**, Gumbel-max top-k sampling; on CUDA it is `torch.compile(dynamic=False)` with power-of-2 batch buckets, manual CUDA graphs only on NPU ([`qwen3_tts_code_predictor_vllm.py#L38`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code_predictor_vllm.py#L38)). Runner hook `talker_mtp()` ([`#L1245`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L1245)) returns `(inputs_embeds [B,H], audio_codes [B,Q])`; next-step input embedding = Σ codebook embeddings + one text-step vector. Q = 16 codebooks × 2048 for the 12 Hz family ([`stage_input_processors/qwen3_tts.py#L390`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py#L390)); frame = 1920 samples @ 24 kHz = **80 ms = 12.5 Hz** ([`configuration_qwen3_tts_tokenizer_v2.py#L147`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/configuration_qwen3_tts_tokenizer_v2.py#L147)). The 25 Hz tokenizer ([`tokenizer_25hz/`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_25hz), Whisper-VQ + DiT + BigVGAN, imports onnxruntime/torchaudio-kaldi/flash-attn) is **not** used by the vLLM stages.
|
||||
|
||||
**Codec / vocoder** [Verified] — [`qwen3_tts_code2wav.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_code2wav.py) `Qwen3TTSCode2Wav` wraps `Qwen3TTSTokenizerV2Decoder` ([`modeling_qwen3_tts_tokenizer_v2.py#L807`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py#L807)) in **float32**: `SplitResidualVectorQuantizer.decode` (1 semantic + 15 acoustic) → causal `pre_conv(k=3)` → 8-layer **sliding-window-72 causal transformer** → two ×2 causal ConvTranspose+ConvNeXt upsamples → BigVGAN/DAC-style stack (SnakeBeta, rates 8,5,4,3, dilations 1,3,9) → clamp. Runs on the generation worker (stage 1, `requires_raw_input_tokens`, `requires_request_ids`); input codes arrive flat codebook-major `[Q*F]` in `runtime_additional_information["codes"]["audio"]`; **batched** across requests (`[B,Q,max_F]` + lengths, `decode_batch_max_size`), chunked via `batched_chunked_decode(chunk 300, left_context 25)` in non-async mode, or **stateful per-request** decode in async mode. Output `{"model_outputs": [float32 wav], "sr": [24000]}`. CUDA graphs: [`segmented_graph_wrapper.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/segmented_graph_wrapper.py) captures stateful ICL-prefix / xvec-prefix graphs per batch bucket and "combined suffix" graphs for every reachable chunk length derived from `initial→+25→…→72→72+25` (`_derive_suffix_transitions` L113); the older stateless [`cuda_graph_decoder_wrapper.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py) survives only for its `_batched_chunked_decode` helper.
|
||||
|
||||
**Streaming / first-audio structure** [Verified] — `talker2code2wav_async_chunk` ([`stage_input_processors/qwen3_tts.py#L77`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py#L77)) is called every talker step by `OmniChunkTransferAdapter._send_single_request_for_generation` ([`chunk_transfer_adapter.py#L654`](../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py#L654)); emits when `len == initial_chunk` or `(len-initial) % 25 == 0` or finished; payload = only the **new** frames (`left_context_size=0`, because Code2Wav keeps context per request), first chunk of an ICL request prepends ≤72 reference frames. Dynamic initial chunk from server load (`compute_dynamic_initial_chunk_size`), optional `codec_chunk_ramp [4,4,8,16,25]`. **Before first audio**: text prefill → **1 talker step** (+15 code-predictor sub-steps) → SHM put → Code2Wav first-chunk path (`_decode_xvec_first_chunk` / `_decode_icl_first_chunk`) → 1920 samples. Repo-reported TTFP 64 ms at c=1 on H200 ([doc](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md), not reproduced).
|
||||
|
||||
**State** [Verified] — Talker KV: vLLM `Qwen3Model` paged attention ([`#L355`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L355)); per-request GPU-resident buffers `hidden_states.last`, `hidden_states.trailing_text`, `codes.audio`, `codes.ref`. Code predictor: stateless. Code2Wav: `_decoder_state_cache[req_id]` holding `ref_hidden`, `suffix_quantized` (≤72 frames), `past_key_values` (`DynamicCache` of the ref prefix; suffix KV recomputed per chunk), and explicit conv contexts `_CONV_CONTEXT_FRAME=2` (pre_conv) and `_DOWNSTREAM_CONTEXT_FRAME=12` (upsample+decoder stack) ([`modeling_qwen3_tts_tokenizer_v2.py#L803`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py#L803)); tests assert these are the minimal exact contexts ([`test_qwen3_tts_incremental_decode.py`](../../vllm-omni/tests/model_executor/models/qwen3_tts/test_qwen3_tts_incremental_decode.py)).
|
||||
|
||||
**Inter-stage tensors** [Verified] — stage 0 `make_omni_output` ([`#L635`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/qwen3_tts_talker.py#L635)) → `OmniPayload{"codes": {"audio": long [span,Q], "ref": [T_ref,Q]}, "meta": {"ref_code_len", "codec_streaming"}}`, hidden states **not** shipped (`omni_pooler_payload_include_hidden=False`). Wire: `OmniPayloadStruct(codes.audio flat long [Q*F], meta.{finished,left_context_size,ref_context_*})` from [`data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py). Stage 1 → `{"model_outputs": [float32 [n]], "sr": [int32]}`.
|
||||
|
||||
**Forced aligner** [Verified] — [`qwen3_tts_forced_aligner.yaml`](../../vllm-omni/vllm_omni/deploy/qwen3_tts_forced_aligner.yaml) + [`utils/forced_aligner.py#L85`](../../vllm-omni/vllm_omni/utils/forced_aligner.py#L85) `inject_forced_aligner_stage` appends a third `LLM_AR` pooling stage (`Qwen3-ForcedAligner-0.6B`) consuming the waveform + text and emitting per-word `{word,start_ms,end_ms}` for the `word_timestamps` API option.
|
||||
|
||||
### 2.2 CosyVoice3 (`cosyvoice3`)
|
||||
|
||||
**Pipeline** [Verified] — [`pipeline.py#L23`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/pipeline.py#L23): stage 0 `cosyvoice3_talker` (`LLM_AR`, `CosyVoice3Model`, stop `[6562]` = logsumexp-merged 200 stop logits, async `talker2code2wav_async_chunk`, full-payload `text2flow_full_payload`); stage 1 `cosyvoice3_code2wav` (`LLM_GENERATION`, `final_output_type="audio"`, sync `text2flow_token_only`, `requires_full_payload_input`). Deploy [`cosyvoice3.yaml`](../../vllm-omni/vllm_omni/deploy/cosyvoice3.yaml): `async_chunk: true`; connector `codec_chunk_frames: 15`, `codec_left_context_frames: 25` (unused by this family's processor), `codec_vocab_size: 6561`; stage 0 `max_num_seqs 8`, `gpu 0.4`, `enforce_eager: false`, sampling `top_k 25, top_p 0.8, repetition_penalty 1.0001` (near-identity so vLLM tracks output ids for RAS); stage 1 `enforce_eager: true` ("CUDA graphs don't work with dynamic conv shapes"), `gpu 0.2`; both on GPU 0. No `platforms:` section.
|
||||
|
||||
**Conditioning** [Verified] — `CosyVoice3MultiModalProcessor._call_hf_processor` ([`cosyvoice3.py#L248`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L248)): inputs `prompt`, `mm_data["audio"]=(wav,sr)`, `mm_kwargs["prompt_text"]` (system prompt + transcript, e.g. `"You are a helpful assistant.<|endofprompt|>…"`), optional `voice_name` for the speaker cache. No explicit instruction/language field. Prompt speech tokens via `s3tokenizer` (`speech_tokenizer_v3_25hz`, GPU; ONNX path unused, [`#L221`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L221)); speaker embedding = **CAM++** 192-d from kaldi fbank (`torchaudio.compliance.kaldi.fbank`), TensorRT `CampplusTRT` ([`speaker_embedding_trt.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/speaker_embedding_trt.py)) or onnxruntime CPU fallback; prompt mel 80-bin/24 kHz/hop 480 ([`utils.py#L49`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/utils.py#L49)), trimmed to 2:1 mel:token. Audio loading `torchaudio.load(backend="soundfile")` + `torchaudio.transforms.Resample` ([`utils.py#L89`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/utils.py#L89)). Tokenizer: `AutoTokenizer` from `CosyVoice-BlankEN` + special tokens ([`tokenizer.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/tokenizer.py)).
|
||||
|
||||
**Generation** [Verified] — AR over semantic speech tokens, vocab 6561 (=3⁸, FSQ [Inferred]) + 200 stop ids ([`cosyvoice3_talker.py#L125`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3_talker.py#L125)), **25 Hz**, `token_mel_ratio=2`. Backbone `VLLMQwen2Encoder` wrapping vLLM `Qwen2Model` (hidden 896, 24 layers, 14/2 heads → Qwen2.5-0.5B class [Inferred]) with paged attention. Sampling: model-owned `prefer_model_sampler=True` implementing nucleus + **RAS** (repetition-aware: mask `top_id` if it appears ≥ `win_size*tau_r` in last 10 outputs; [`#L610`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L610)); bypassed to vLLM `Sampler` when logprobs/penalties requested.
|
||||
|
||||
**Code2wav** [Verified] — [`cosyvoice3_code2wav.py#L43`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3_code2wav.py#L43): `PreLookaheadLayer` (3-frame right context) → `CausalConditionalCFM` ([`cfm.py#L190`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py#L190)) with estimator `DiT` ([`diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py#L340`](../../vllm-omni/vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py#L340): dim 1024, depth 22, 16 heads, uses `vllm_omni.diffusion.attention.layer.Attention`, diffusers `AdaLayerNormZero`, `x_transformers` rotary) — **10 Euler steps**, cosine t-schedule, CFG 0.7 as batch-2 → 80-bin mel at 50 Hz → `CausalHiFTGenerator` ([`hifigan.py#L606`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py#L606)): NSF source (f0 predictor pinned to **CPU** for precision, L757) + ISTFTNet (upsample 8·5·3, n_fft 16/hop 4 → 480 samples/frame), Snake activations, forced fp32, 24 kHz. **TensorRT**: `COSYVOICE3_TRT` env (default on) swaps the DiT estimator for a TRT engine built from `flow.decoder.estimator.autocast_fp16.onnx` (downloaded from `yuekai/Fun-CosyVoice3-0.5B-2512-FP16-ONNX` if absent, [`#L901`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L901)) via [`flow_estimator_trt.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/flow_estimator_trt.py); requires the `tensorrt` python package. **Batching**: stage 1 is scheduled with `max_num_seqs 8` but `CosyVoice3Model.forward` loops requests **sequentially** at batch 1 (`inference` asserts `token.shape[0]==1`, [`cfm.py#L302`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py#L302)); only CFG doubles the DiT batch.
|
||||
|
||||
**Streaming** [Verified] — `talker2code2wav_async_chunk` ([`stage_input_processors/cosyvoice3.py#L96`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/cosyvoice3.py#L96)): first chunk after `chunk(15) + prompt_pad + 3 lookahead` tokens, subsequent hops double (15, 30, 60, 60 …; ≈0.6 s, 1.2 s, 2.4 s of audio [Inferred]); payload `codes.audio` = **cumulative prefix** of all tokens so far with `meta.left_context_size` = already-emitted length. Consumer `forward_streaming` ([`cosyvoice3_code2wav.py#L196`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3_code2wav.py#L196)) reruns the **full CFM over prompt + all tokens** and HiFT over cumulative mel, then trims — per-chunk cost grows with history (O(n²) overall) [Inferred]. No waveform cross-fade (`_stitch_stream_audio` is pass-through); `mel_window/speech_window` attributes are dead. First audio requires: mm preprocessing (s3tokenizer GPU + CAM++ + mel) → talker prefill + ≥18+pad decode steps → SHM → CFM (10 steps × batch 2) → HiFT.
|
||||
|
||||
**State** [Verified] — talker: vLLM paged KV only. Code2wav: `_stream_vocoder_cache_by_req[req] = {"mel": cumulative CPU mel, "speech_offset"}` cleared on `stream_finished`; no HiFT conv cache, no CFM noise cache.
|
||||
|
||||
**Inter-stage tensors** [Verified] — prefill emits `embed.{speech_token [1,T] int32, speech_feat [1,2T,80], embedding [1,192], speech_token_len}` once ([`#L992`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/cosyvoice3.py#L992)); async chunks `codes.audio` int64 `[prefix_len]` + `meta`; stage-1 output `{"audio": [float32 1-D], "sr": [24000]}`.
|
||||
|
||||
### 2.3 Fish Speech S2 Pro (`fish_qwen3_omni`)
|
||||
|
||||
**Pipeline** [Verified] — [`pipeline.py#L21`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/pipeline.py#L21): stage 0 `fish_speech_slow_ar` (`LLM_AR`, `FishSpeechSlowARForConditionalGeneration`, stop `[151645]`, async `slow_ar_to_dac_decoder_async_chunk` — **no sync full-payload func**); stage 1 `dac_decoder` (`LLM_GENERATION`, `FishSpeechDACDecoder`). Deploy [`fish_qwen3_omni.yaml`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni.yaml): connector `codec_chunk_frames: 25` (≈1.16 s at ~21 Hz), `codec_left_context_frames: 25`, `initial_codec_chunk_frames: 4`; stage 0 `max_num_seqs 4`, `gpu 0.6`, `enable_chunked_prefill: false` ("SlowAR decode loop isn't chunked-prefill-safe"), sampling `0.8/top_k 30/top_p 0.9`; stage 1 `enforce_eager: true`, `gpu 0.1`; `platforms.xpu` override (Intel Arc B60). Variants [`_2gpu`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni_2gpu.yaml) (64 seqs each GPU), [`_high_concurrency_dual_gpu`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni_high_concurrency_dual_gpu.yaml) / [`_single_gpu`](../../vllm-omni/vllm_omni/deploy/fish_qwen3_omni_high_concurrency_single_gpu.yaml) add `fish_speech_tensor_codes`, `fish_speech_single_initial_chunk`, `fish_speech_backlog_codec_chunk_frames: 50`, `fish_speech_dac_dtype: fp16`, `fish_speech_dac_max_padded_frames: 256`.
|
||||
|
||||
**Conditioning** [Verified] — chat-template prompt in [`prompt_utils.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/prompt_utils.py) (`<|im_start|>system … <|voice|>`); voice clone = system prompt with `ref_text` + `<|audio_start|>semantic_ids<|audio_end|>`; no separate instruction field. **DAC encoding of reference audio runs inside stage 0's model on the stage-0 GPU during prefill** (`_build_structured_voice_clone_prefill_embeds` → `encode_reference_audio_codes`, [`dac_encoder.py#L118`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/dac_encoder.py#L118); `torchaudio.transforms.Resample` to 44.1 kHz; the `dac_utils.py` docstring saying "CPU" is stale). Injection: sum of per-codebook embeddings / `sqrt(num_codebooks+1)` at the audio span ([`fish_speech_slow_ar.py#L641`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_slow_ar.py#L641)). Serving fetches audio through vLLM `MediaConnector`; offline example uses `soundfile`. Speaker cache 512 MiB LRU keyed by voice name ([`utils/speaker_cache.py`](../../vllm-omni/vllm_omni/utils/speaker_cache.py)).
|
||||
|
||||
**Generation (dual-AR)** [Verified] — **Slow AR** = vLLM `Qwen3Model` (dim 2560, 36 layers, 32/8 heads, vocab 155776; RoPE rebuilt GPT-J interleaved; logits masked to semantic range `[151678..155773]` + `<|im_end|>`; [`configuration_fish_speech.py#L14`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/configuration_fish_speech.py#L14)). **Fast AR** ([`fish_speech_fast_ar.py#L338`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_fast_ar.py#L338)) = 4-layer / dim-2560 transformer with `max_seq_len=11` predicting 9 residual codebooks (1024 entries) per frame: position 0 = projected slow hidden, position 1 = semantic-code embedding, then a Python loop of 9 single-token steps over a per-call scratch KV `[L,B,kv_heads,11,D]` with plain `F.scaled_dot_product_attention` (**no paged attention**); inline top-k/top-p/multinomial. Invoked through the runner-owned `talker_mtp` hook wrapped in a **full CUDA graph** per batch size when `talker_mtp_graph_safe=True` ([`gpu_ar_model_runner.py#L534`](../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py#L534)); Fast AR for step t uses hidden_{t-1} and code_{t-1} (matches reference flow). Frame rate: hop 2048 @ 44.1 kHz = **21.5 Hz** ([`dac_utils.py#L12`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/dac_utils.py#L12)).
|
||||
|
||||
**DAC decoder** [Verified] — [`fish_speech_dac_decoder.py#L72`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_dac_decoder.py#L72): Descript-style conv+transformer hybrid from the external **`fish-speech` pip package** (`fish_speech.models.dac.modded_dac`; `build_dac_codec` in [`dac_utils.py#L16`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/dac_utils.py#L16): decoder_dim 1536, rates [8,8,4,2], causal, 4 transformer layers in the first decoder block, RVQ 9×1024 + window-limited transformers); weight-norm baked, encoder pruned. Stage 1, eager. Input `[10, frames]` (tensor path) or flat `10*frames` ids; **batched** (`_decode_group` pads `[B,10,T]`, groups bounded by `fish_speech_dac_max_padded_frames`), decode under `autocast(enabled=False)`. Chunked decode has **no cross-chunk conv state**: each chunk is decoded with 25 frames of left overlap then trimmed proportionally (`cut = ctx/total * len(wav)`, L454-470). Output `{"model_outputs": [wav], "sr": [44100]}`.
|
||||
|
||||
**Streaming** [Verified] — `slow_ar_to_dac_decoder_async_chunk` ([`stage_input_processors/fish_speech.py#L117`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/fish_speech.py#L117)): first chunk after `initial_codec_chunk_frames=4` (≈186 ms audio), steady 25-frame chunks with 25 frames of left context, backlog mode grows to 50 frames under load. First audio needs: prefill (+DAC encode for clone) + 4 slow steps (each = Qwen3 decode + 9 fast-AR micro-steps) + SHM poll + one DAC decode of 4(+ctx) frames [Inferred structure].
|
||||
|
||||
**State** [Verified] — Slow AR KV: vLLM paged attention, optionally with a Triton decode-only kernel ([`attention/fish_kvcache_backend.py`](../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py), env `VLLM_OMNI_FISH_KVCACHE_ATTN`, requires head_dim 128 / block 16). Fast AR: scratch KV reset every step. Cross-step: `hidden_states.last` GPU-resident. Stage 1 stateless.
|
||||
|
||||
**Inter-stage tensors** [Verified] — `OmniPayloadStruct(codes.audio: long [10,F] or flat [10*F], meta.{left_context_size, finished})`; intra-stage `codes.audio [B,10]` per step from `talker_mtp_output_key=("codes","audio")`.
|
||||
|
||||
### 2.4 MOSS-TTS family (`moss_tts_delay` / `moss_tts_realtime` / `moss_tts_local`, plus `moss_tts_nano`)
|
||||
|
||||
**Pipelines** [Verified] — [`pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/pipeline.py): three 2-stage pipelines (L23, L56, L87) sharing stage 1 `moss_tts_codec` (`LLM_GENERATION`, `MossTTSCodecDecoder`, **`retains_state_across_chunks=True`** — counted by the generation scheduler as parked streaming requests, [`omni_generation_scheduler.py#L172`](../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py#L172)). Delay uses async processor `talker2codec_delay_async_chunk`; Realtime/Local use `talker2codec_raw_async_chunk`. Nano ([`moss_tts_nano/pipeline.py#L19`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano/pipeline.py#L19)) is single-stage (`final_output=True`, `engine_output_type="audio"`), wrapping HF remote code. Deploy variants: [`moss_tts.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts.yaml) (8B, n_vq 32, stage 1 `max_model_len 196608` = 4096 frames × 32 codes + margin, `decode_cudagraph_capture_sizes`), [`moss_tts_realtime.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_realtime.yaml) (1.7B, n_vq 16, `codec_chunk_frames 15`, claimed "TTFB ~180 ms" — not measured in repo), [`moss_tts_local.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_local.yaml) (v1.5, n_vq 12, 48 kHz stereo, `initial_codec_chunk_frames: 1`, stage 1 `cudagraph_capture_sizes [1..64]`), plus [`moss_ttsd.yaml`](../../vllm-omni/vllm_omni/deploy/moss_ttsd.yaml), [`moss_sound_effect.yaml`](../../vllm-omni/vllm_omni/deploy/moss_sound_effect.yaml), [`moss_voice_generator.yaml`](../../vllm-omni/vllm_omni/deploy/moss_voice_generator.yaml) (notes the Delay talker is "not CUDA-graph-safe"), [`moss_tts_80gb.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_80gb.yaml), [`moss_tts_nano.yaml`](../../vllm-omni/vllm_omni/deploy/moss_tts_nano.yaml) (0.1B, "verified on 1× L4 24 GB", eager).
|
||||
|
||||
**Conditioning** [Verified] — prompts are built by the upstream HF `AutoProcessor` (`build_user_message(text, reference, instruction, tokens, quality, sound_event, ambient_sound, language)`) into an `(L, 1+n_vq)` grid: column 0 → `prompt_token_ids`, columns 1.. → `additional_information["codes"]["ref"]` ([`end2end.py#L44`](../../vllm-omni/examples/offline_inference/text_to_speech/moss_tts/end2end.py#L44); serving `_build_moss_tts_params` [`serving_speech.py#L1494`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py#L1494)). **Reference audio is encoded outside both stages** (API-server process, `encode_audios_from_wav` on CPU via [`reference_encoder.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/reference_encoder.py) with content-addressed cache, micro-batching 10 ms / 8). The talker adds `codes.ref` embeddings at prefill. Realtime feeds remaining text ids one per decode step via forced one-hot logits (`ids.all`). Loading: `soundfile` + `torchaudio.functional.resample`.
|
||||
|
||||
**Generation** [Verified] — all on vLLM `Qwen3Model` with additive fusion `text_embed + Σ audio_embed_i`. **Delay** ([`modeling_moss_tts_talker.py#L97`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_talker.py#L97)): n_vq+1 parallel heads on the final hidden; text head drives vLLM's sampler, audio heads sampled in `make_omni_output` via one stacked matmul with hard-coded `top_k 25 / temp 1.7`; codebook i activates i steps after the delay slot; de-delay in the input processor. **Realtime**: no text head; per-step frame from a 4-layer `MossTTSRealtimeLocalTransformer` re-prefilled over codebooks (no KV) ([`modeling_moss_tts_local.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local.py)). **Local v1.5**: 1-layer GPT2-style depth transformer ([`modeling_moss_tts_local_depth.py#L35`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_local_depth.py#L35)) with a binary continue/stop head, run through the runner's `talker_mtp` hook **before** the backbone (FULL-graph wrapped when graph-safe), `torch.compile` on the depth prefix. Frame rate **12.5 Hz / 80 ms**; codec sample rate 24 kHz (v1, `downsample_rate 1920`) or 48 kHz stereo channel-interleaved (v2, 3840) ([`audio_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py)).
|
||||
|
||||
**Codec** [Verified] — MOSS Audio Tokenizer: **transformer-only** (patched reshape up/down-sampling + causal RoPE transformer blocks, no strided convs; residual LFQ quantizer 32×1024 default; v2 folds decode projections into a LUT). Stage 1 [`modeling_moss_tts_codec.py#L207`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/modeling_moss_tts_codec.py#L207): non-streaming path decodes one request at a time (optional B=1 CUDA-graph wrapper [`moss_codec_cudagraph.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/moss_codec_cudagraph.py) per padded-T bucket); streaming path groups requests with identical T into `(NQ,B,T)` batches through `_MossCodecStreamSession` with **per-slot ring KV caches** (`RingKVCache`, capacity = 10 s context) and no overlap (`codec_left_context_frames: 0` everywhere) — graphs keyed `(B_bucket, exact_T)` in [`cuda_graph_streaming_decoder_wrapper.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/cuda_graph_streaming_decoder_wrapper.py). Decoder cast bf16 on non-CPU.
|
||||
|
||||
**Streaming / first audio** [Verified] — **Delay variants do not stream**: `chunk_frames = 1 << 30` in `talker2codec_delay_async_chunk` ([`stage_input_processors/moss_tts.py#L152`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/moss_tts.py#L152)) so codes are emitted only on finish, making `codec_streaming: true / codec_chunk_frames: 25` in five YAMLs inert; stage 1 then splits internally. **Realtime**: raw rows buffered to 15 frames (1.2 s) → first audio after ~15 talker steps + one codec step. **Local**: first codec call after **1 frame** (80 ms), then 15-frame steps. **Nano**: one waveform chunk per upstream `inference_stream()` yield.
|
||||
|
||||
**State** [Verified] — backbone paged KV; per-request `audio_state` dicts + GPU-resident `audio_codes.current/accumulated`, `hidden_states.last`; depth/local transformers stateless; codec ring-KV slots leased per request and released on terminal payload. **Inter-stage**: Delay → `codes.audio` python list (codebook-major); raw → 1-D long `[NQ*T]` + `meta.{req_id, codec_chunk_frames, code_flat_numel, stream_finished, finished}`; talker inputs `codes.ref (L,n_vq)`, `ids.all`, `max_new_frames`.
|
||||
|
||||
### 2.5 TTS cross-family comparison
|
||||
|
||||
| | Qwen3-TTS | CosyVoice3 | Fish S2 Pro | MOSS (Delay / Realtime / Local) |
|
||||
|---|---|---|---|---|
|
||||
| Talker backbone | vLLM `Qwen3Model` | vLLM `Qwen2Model` 0.5B | vLLM `Qwen3Model` 36L | vLLM `Qwen3Model` |
|
||||
| Per-frame sub-AR | 5L code predictor (15 sub-steps, re-prefill) | none (single FSQ token) | 4L fast AR (9 steps, scratch KV) | none / 4L local / 1L depth |
|
||||
| Codec tokens | 16×2048 @ 12.5 Hz | 6561 @ 25 Hz | 10 cb @ 21.5 Hz | 32/16/12 cb @ 12.5 Hz |
|
||||
| Vocoder type | conv+sliding-window transformer (fp32) | CFM DiT (10 steps, CFG) + HiFT ISTFT | DAC conv+transformer | transformer-only codec |
|
||||
| First audio after | 1 frame (80 ms) | ≥18 tokens (~0.7 s) + 10-step CFM | 4 frames (~186 ms) | full utterance / 15 frames / 1 frame |
|
||||
| Cross-chunk vocoder state | conv ctx 2+12, ref KV, 72-frame window | cumulative mel (re-decode all) | none (25-frame overlap) | ring KV per slot |
|
||||
| Stage-1 batching | yes (padded, ≤4) | no (sequential loop) | yes (padded groups) | streaming: same-T grouping |
|
||||
| Stage-1 graphs | stateful CUDA graphs per chunk length | eager (TRT estimator optional) | eager | CUDA graphs `(B,T)` buckets |
|
||||
| Output SR | 24 kHz | 24 kHz | 44.1 kHz | 24 kHz / 48 kHz stereo |
|
||||
|
||||
### 2.6 Inter-stage tensor contracts (TTS)
|
||||
|
||||
What actually crosses the stage boundary, per family. All wire tensors are serialized to **CPU bytes** by the connector ([`serialization.py#L89`](../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py#L89)) and re-materialized on the consumer's device; the schema is enforced by `OmniPayloadStruct` (`forbid_unknown_fields`) in [`data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py). [Verified]
|
||||
|
||||
| Family | Stage-0 per-step `multimodal_outputs` | Wire payload (async chunk) | Stage-1 consumes as | Stage-1 emits |
|
||||
|---|---|---|---|---|
|
||||
| Qwen3-TTS | `codes.audio` long `[span, 16]`, `codes.ref` `[T_ref, 16]`, `meta.ref_code_len`, `meta.codec_streaming`; hidden states **not** shipped | `codes.audio` flat long `[16·F_new]` (codebook-major), `meta.{finished, left_context_size(=0 or ref ctx), ref_context_size, ref_context_request_id, ref_context_included}`, `speaker`, `language` | placeholder `prompt_token_ids` growing per chunk + `runtime_additional_information["codes"]["audio"]` | `{"model_outputs": [float32 [n]], "sr": [int32 24000]}` |
|
||||
| CosyVoice3 | prefill only: `embed.{speech_token [1,T] int32, speech_feat [1,2T,80], embedding [1,192], speech_token_len}`; `text_hidden_states [tokens, 896]` | `codes.audio` int64 `[prefix_len]` (**cumulative**), `meta.{left_context_size=emitted_len, stream_finished, req_id}`, `embed` on first chunk | flat codec ids as `input_ids` (embedded as zeros `[N,896]`), conditioning via `model_intermediate_buffer` | `{"audio": [float32 1-D], "sr": [24000]}` |
|
||||
| Fish S2 Pro | `codes.audio` `[B, 10]` per step (`talker_mtp_output_key`), `hidden_states.last [2560]` bf16, `embed.prefill` (CPU pinned) | `codes.audio` long `[10, F]` (tensor mode) or flat `[10·F]`, `meta.{left_context_size=25, finished}` | `[10, F]` tensor or flat ids reshaped codebook-major | `{"model_outputs": [wav], "sr": [44100]}` |
|
||||
| MOSS Delay | `codes.audio` list of `(T_acc, n_vq)` int64 accumulated snapshots | python list (codebook-major, de-delayed) on **finish only**, `meta.{left_context_size=0, finished}` | flat ids split by `seq_token_counts` → `(n_vq, T)` | `{"model_outputs": [float32 (T_wav,) or (C, T_wav)], "sr"}` |
|
||||
| MOSS Realtime / Local | `codes.audio` list of `(1, n_vq)` rows | 1-D long `[n_vq·T_chunk]`, `meta.{req_id, codec_chunk_frames, codec_left_context_frames=0, code_flat_numel, stream_finished, finished}` | same, with `retains_state_across_chunks` slot state | same |
|
||||
|
||||
Consequence for an edge port [Inferred]: the only information a codec stage needs from the talker is the integer code stream plus a small metadata record (reference-context length, chunk boundary, finished flag). Speaker conditioning for the vocoder (CosyVoice3's `embed.*`, Qwen3-TTS's `codes.ref`) is sent once at the first chunk, so a single-process edge pipeline can keep it resident and skip the transport entirely.
|
||||
|
||||
### 2.7 First-audio latency decomposition
|
||||
|
||||
Sequential components that must complete before the first PCM sample leaves the pipeline, from the cited code paths (no timings measured here; repo-reported numbers are labelled). [Verified structure / Inferred ordering]
|
||||
|
||||
| Family | 1. Front-end (CPU/GPU) | 2. Talker prefill | 3. Talker decode steps before first chunk | 4. Transport | 5. First vocoder call | Repo-reported |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Qwen3-TTS (12 Hz) | tokenizer + text_projection; Base: ECAPA mel + Mimi encoder over ref audio (cached by SHA1) | one vLLM prefill (prompt ≈ instruct + codec tags + text or first text token) | **1 step** = 1 Qwen3 decode + 15 code-predictor sub-prefills (`initial_codec_chunk_frames: 1`; `min_tokens 2` only blocks EOS) | SHM put/get, poll 10 ms, first-chunk wait ≤3 s | stateful first-chunk graph (`_decode_icl_first_chunk` incl. ≤72 ref frames, or xvec path) → 1920 samples (80 ms) | TTFP 64 ms @ c=1, H200 ([doc](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md)) |
|
||||
| CosyVoice3 (25 Hz) | s3tokenizer (GPU) + CAM++ (TRT/ORT) + prompt mel; text tokens | prefill of `[SOS, text, TASK, prompt speech tokens]` | `15 + prompt_pad + 3` steps (~0.72 s of speech) | SHM, poll 10 ms | CFM 10 Euler steps × batch 2 (CFG) over prompt+chunk (TRT estimator optional) + HiFT on cumulative mel | none for TTFP; continuity notes only |
|
||||
| Fish S2 Pro (21.5 Hz) | chat template; clone: DAC **encoder on stage-0 GPU** during prefill | prefill incl. `<\|audio_start\|>…` ref span | **4 steps** (`initial_codec_chunk_frames: 4`, ≈186 ms audio), each = Qwen3 decode + 9 fast-AR micro-steps (CUDA-graphed) | SHM, poll 10 ms (1 ms in HC profiles) | DAC decode of 4 (+ctx) frames, eager, batched | none |
|
||||
| MOSS Delay (12.5 Hz) | HF `AutoProcessor` grid; ref audio encoded in the API-server process (CPU, cached) | prefill with `codes.ref` added | **entire utterance** (chunking disabled) | one put on finish | full-utterance decode split into 25-frame steps | none |
|
||||
| MOSS Realtime | grid + `ids.all` text queue | prefill | **15 steps** (1.2 s audio), each with a 4-layer local-transformer re-prefill over 16 codebooks | SHM, poll 5 ms, first wait 1 s | one `(NQ,B,15)` ring-KV decode | "TTFB ~180 ms" (YAML header, unmeasured) |
|
||||
| MOSS Local v1.5 | grid | prefill | **1 step** (`initial_codec_chunk_frames: 1`), depth transformer runs before the backbone (`talker_mtp`) | SHM, poll 5 ms | `(B,1)` CUDA-graph bucket decode, 48 kHz stereo | none |
|
||||
|
||||
Design reading [Proposal]: for an on-device TTS the components that dominate are (3) and (5). Qwen3-TTS and MOSS-Local show the pattern to copy — emit after a single frame and keep vocoder context resident — while CosyVoice3's flow-matching vocoder needs ≥10 DiT evaluations per chunk and re-decodes history, which is the wrong shape for edge streaming unless the CFM is replaced by an incremental variant.
|
||||
|
||||
---
|
||||
|
||||
## 3. VLA / world-model deep dive
|
||||
|
||||
Shared request plumbing first (applies to all): `OmniDiffusionRequest` ([`diffusion/request.py#L39`](../../vllm-omni/vllm_omni/diffusion/request.py#L39)) carries `prompt`, `sampling_params: OmniDiffusionSamplingParams`, whose `extra_args: dict[str, Any]` ([`inputs/data.py#L319`](../../vllm-omni/vllm_omni/inputs/data.py#L319)) is **the** pathway for non-text tensors (robot obs, state, noise, camera scripts). Pipelines return `DiffusionOutput(output=...)` ([`diffusion/data.py#L1565`](../../vllm-omni/vllm_omni/diffusion/data.py#L1565)); [`output_formatter.py#L254`](../../vllm-omni/vllm_omni/diffusion/output_formatter.py#L254) copies only keys in `{"audio","actions","trajectory"}` into `OmniRequestOutput.multimodal_output`. Robot policies are served on `WS /v1/realtime/robot/openpi` ([`api_server.py#L1869`](../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py#L1869) → [`entrypoints/openpi/connection.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/connection.py) / [`serving.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py)): binary msgpack with numpy markers, server first sends `policy_server_config` from the deploy YAML, then request/response `infer` (one action reply per message, no streaming) and `reset` ([`openpi_api.md`](../../vllm-omni/docs/serving/openpi_api.md)); 64 MiB payload cap, 30 s idle timeout. Enabled only when the deploy YAML declares `model_config.policy_server_config`. [Verified]
|
||||
|
||||
### 3.1 GR00T N1.7 (`Gr00tN1d7`)
|
||||
|
||||
- **Pipeline** [Verified]: [`gr00t/pipeline.py#L11`](../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py#L11) single `DIFFUSION` stage, `final_output_type="actions"`, no custom funcs. Deploy [`Gr00tN1d7.yaml`](../../vllm-omni/vllm_omni/deploy/Gr00tN1d7.yaml): `async_chunk: false` (validator requires it for single stage), `max_num_seqs 1`, `enforce_eager: true`, `model_config.embodiment_tag: OXE_DROID_RELATIVE_EEF_RELATIVE_JOINT`, `policy_server_config` (`image_resolution [180,320]`, `action_horizon 40`, `action_keys [eef_9d, gripper_position, joint_position]`). `Gr00tN1d7Pipeline` ([`pipeline_gr00t.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py)) is a thin wrapper: `load_weights` is a no-op; the whole model loads via `AutoModel.from_pretrained(..., bf16)` in [`policy.py#L64`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/policy.py#L64).
|
||||
- **Perception** [Verified]: backbone = stock HF `Qwen3VLForConditionalGeneration` configured from `nvidia/Cosmos-Reason2-2B`, forced `sdpa`, LLM **truncated to 12 layers**, `lm_head` dropped; features = pre-final-norm hidden of the last kept layer `[B, seq, 2048]` via a forward hook ([`gr00t_n1d7.py#L291`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py#L291)). Image preprocessing ([`processing_gr00t_n1d7.py#L49`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/processing_gr00t_n1d7.py#L49)): torchvision v2 `LetterBox → Resize(256) → CenterCrop(230) → Resize(256)` on uint8, then the HF `Qwen3VLProcessor` (from `Qwen/Qwen3-VL-2B-Instruct`) does patching/normalization. Cameras/frames come from checkpoint `modality_configs`; DROID test uses V=2 views × T=2 frames = 4 images/step ([`tests/helpers/client.py#L155`](../../vllm-omni/tests/helpers/client.py#L155)) [Inferred for the count].
|
||||
- **Structured inputs** [Verified]: `obs["state"]: {key: float32 (B,T,D)}` (DROID: eef_9d 9 + gripper 1 + joint 7 = 17 raw), min/max-normalized to [-1,1] from checkpoint `statistics.json`, zero-padded to `max_state_dim=132`, projected by a **per-embodiment** `CategorySpecificMLP` (`W[32,132,1024]`→`[32,1024,1536]`, index = embodiment id, DROID→24; [`embodiment_conditioned_mlp.py`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/modules/embodiment_conditioned_mlp.py)). Language: lower-cased, tokenized by the Qwen3-VL tokenizer (left pad), one chat turn with all images then text. All of it arrives as raw numpy in `extra_args["robot_obs"]` (`video|images`, `state`, `language|prompt`) — **never** `multi_modal_data` ([`pipeline_gr00t.py#L110`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py#L110)).
|
||||
- **Fusion** [Verified]: vision+language inside the VLM; `vlln` LayerNorm; state fused as one token prepended to 40 action tokens; VL features enter the DiT via cross-attention in `AlternateVLDiT` ([`modules/dit.py#L239`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/modules/dit.py#L239)): even blocks cross-attend (alternating text-only / image-only K/V masks, `attend_text_every_n_blocks=2`), odd blocks self-attend; built on `diffusers.models.attention.Attention` + `AdaLayerNorm`.
|
||||
- **Action head** [Verified]: `Gr00tN1d7ActionHead` — 16 DiT layers, 32 heads × 48 = 1536, cross-attn dim 2048; `MultiEmbodimentActionEncoder` (per-embodiment W1/W2/W3 with sinusoidal timestep), decoder `CategorySpecificMLP(1024→1024→132)`; flow matching with **4 Euler steps** (`num_inference_timesteps=4`, from model config only — request `num_inference_steps` ignored), horizon **40**, padded action dim 132, no CFG, no scheduler object ([`gr00t_n1d7.py#L123`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/modeling/gr00t_n1d7.py#L123)). RTC inpainting code exists but is unreachable at inference.
|
||||
- **Temporal state** [Verified]: none — `state_history_length=1`, `reset()` returns `{}`; every request recomputes the full VLM forward (4 images) + 4 DiT steps; frame history must be supplied by the client.
|
||||
- **Output** [Verified]: `DiffusionOutput(output={"actions": {key: float32 (B,40,D)}})` after un-normalization and relative→absolute conversion using scipy `Rotation` ([`dataio/state_action`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/dataio/state_action)); DROID shapes `eef_9d (1,40,9)`, `gripper_position (1,40,1)`, `joint_position (1,40,7)` ([`test_gr00t_openpi_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_gr00t_openpi_expansion.py)).
|
||||
- **Latency per step** [Verified]: CPU transforms + HF processor → H2D → Qwen3-VL prefill (ViT + 12 layers, SDPA, eager) → 4 × (encoder + 16-block DiT + decoder) → D2H → numpy/scipy post. Nothing compiled or graph-captured; startup warm-up returns zeros without touching the model, so the first real request pays cold start. Batch 1; one embodiment per server.
|
||||
- **Client** [Verified]: no GR00T-specific example; reuse [`examples/online_serving/pi0/openpi_client.py`](../../vllm-omni/examples/online_serving/pi0/openpi_client.py) (`openpi_client.WebsocketClientPolicy`). Serve: `vllm serve nvidia/GR00T-N1.7-3B --omni --port 8000 --served-model-name gr00t-n1d7 --deploy-config vllm_omni/deploy/Gr00tN1d7.yaml` ([recipe](../../vllm-omni/recipes/NVIDIA/GR00T-N1.7.md)); recipe reports ~6 GiB peak bf16 (repo-reported).
|
||||
|
||||
### 3.2 π0 (`pi0`)
|
||||
|
||||
- **Pipeline** [Verified]: [`pi0_pipeline_config.py#L16`](../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py#L16) single `DIFFUSION` stage, `final_output_type="action"`. Deploy [`pi0.yaml`](../../vllm-omni/vllm_omni/deploy/pi0.yaml): `dtype: float32`, `max_num_seqs 1`, `enforce_eager: true` ("flow matching needs no CUDA graph"), `model_config`: `paligemma_variant gemma_2b`, `action_expert_variant gemma_300m`, `chunk_size 50`, `max_action_dim 32`, `max_state_dim 32`, `num_inference_steps 10`, `image_resolution [224,224]`, `tokenizer_max_length 48`, `max_cameras 3`, `policy_server_config`.
|
||||
- **Model** [Verified]: `PaliGemmaWithActionExpert` = HF `PaliGemmaForConditionalGeneration` (SigLIP + Gemma-2B prefix) + `GemmaForCausalLM` Gemma-300M suffix ([`modeling_pi0.py#L372`](../../vllm-omni/vllm_omni/diffusion/models/pi0/modeling_pi0.py#L372)); hand-rolled eager attention walking Gemma layers manually (prefix-only / suffix-only helpers); flow matching Euler t=1→0, 10 steps, no CFG; prefix = `[3×256 image tokens, 48 text]` bidirectional, suffix = `[state, 50 action-time tokens]`.
|
||||
- **Inputs** [Verified]: `extra_args["robot_obs"]` — up to 3 cameras in `image_feature_keys` order (missing → −1-filled, masked), `resize_with_pad` to 224 in [-1,1] ([`processor_pi0.py#L48`](../../vllm-omni/vllm_omni/diffusion/models/pi0/processor_pi0.py#L48), numpy/PIL/torch only), `state` zero-padded to 32, `prompt` right-padded to 48 tokens; no embodiment id. Optional `extra_args["noise"]`, `["num_inference_steps"]`.
|
||||
- **State/Output/Latency** [Verified]: stateless across calls (`session_id`/`reset` ignored); prefix KV computed once per request and reused across the 10 suffix steps only. Output `{"actions": float32 (50,32)}`; un-normalization via `config.norm_stats` (identity for `pi0_base`; README notes LeRobot `policy_preprocessor.json` stats are **not** loaded). Per step: 3× SigLIP + 18-layer Gemma-2B prefix over ~816 tokens + 10 × 18-layer 300M suffix passes; float32, eager, batch 1.
|
||||
- **Client** [Verified]: [`examples/online_serving/pi0/openpi_client.py`](../../vllm-omni/examples/online_serving/pi0/openpi_client.py) loop `policy.infer(obs)`; serve `vllm serve lerobot/pi0_base --omni --port 8000 --served-model-name pi0 --deploy-config vllm_omni/deploy/pi0.yaml --enforce-eager` ([recipe](../../vllm-omni/recipes/lerobot/Pi0.md)). Parity test runs on **CPU/float32** ([`test_pi0_parity.py`](../../vllm-omni/tests/diffusion/models/pi0/test_pi0_parity.py)).
|
||||
|
||||
### 3.3 DreamZero-DROID (`dreamzero`)
|
||||
|
||||
- **Pipeline** [Verified]: [`dreamzero/pipeline.py#L11`](../../vllm-omni/vllm_omni/model_executor/models/dreamzero/pipeline.py#L11) single `DIFFUSION` stage, `final_output_type="image"` (actions ride in `multimodal_output`). Deploy [`dreamzero.yaml`](../../vllm-omni/vllm_omni/deploy/dreamzero.yaml): `engine_backend: vllm_omni.experimental.ar_diffusion.engine.ARDiffusionEngine` (mandatory, pipeline refuses others), `max_num_seqs 1`, `enforce_eager: false` (but all shipped commands add `--enforce-eager`), `cache_backend: step_cache` (skip DiT steps when velocity cosine-sim > 0.95/0.93), `policy_server_config` (`image_resolution [180,320]`, 2 external + wrist cameras, `needs_session_id`, `action_space joint_position`). [`dreamzero_tp1_cfg2.yaml`](../../vllm-omni/vllm_omni/deploy/dreamzero_tp1_cfg2.yaml) differs only by `devices "0,1"` and `parallel_config {tensor_parallel_size 1, cfg_parallel_size 2}`.
|
||||
- **Model** [Verified]: `CausalWanModel` ([`causal_wan_model.py`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/causal_wan_model.py), comment L743 "14B: 40 layers, dim 5120, heads 40, ffn 13824"; real dims from checkpoint config) — causal DiT with **action/state tokens appended after video tokens** (own RoPE tables, own `VideoActionScheduler`), `WanI2VCrossAttention` (257 CLIP tokens + 512 UMT5 tokens), fused `QKVParallelLinear` + `paged_write_attn` custom op. Text encoder UMT5-XXL (24L, d 4096, tokenizer `google/umt5-xxl`), image encoder CLIP ViT-H/14 ([`image_encoder.py#L184`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/image_encoder.py#L184)), VAE `DistributedAutoencoderKLWan` (16 ch, 8× spatial, 4 frames/latent). Denoising: `FlowUniPCMultistepScheduler`, **16 steps**, CFG 5.0 (video only; action takes positive branch), sigma shift 5.0 ([`utils.py#L6`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/utils.py#L6)). Action horizon 24, dim 8 (7 joints + gripper, [`transform/droid.py#L42`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/transform/droid.py#L42)), relative joint actions un-normalized with q01/q99 stats from `experiment_cfg/metadata.json`.
|
||||
- **Inputs** [Verified]: `extra_args["robot_obs"]` with `observation/exterior_image_{0,1}_left`, `observation/wrist_image_left` (`(H,W,3)` first call, `(4,H,W,3)` later), `observation/joint_position (7)`, `observation/gripper_position (1)`, `prompt`, `session_id`; three cameras center-cropped, resized 176×320 and **stitched to 352×640** ([`transform/droid.py#L46`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/transform/droid.py#L46)); state padded to `max_state_dim`; language wrapped in a multi-view template, tokenized to 512.
|
||||
- **Temporal state** [Verified]: per-session paged KV in the AR-Diffusion runner ([`experimental/ar_diffusion/runner.py`](../../vllm-omni/vllm_omni/experimental/ar_diffusion/runner.py)) with window 21 latent frames, two KV branches (pos/neg CFG), cross-attn caches (text 512, image 257), `session_capacity 64` (LRU), **603 MiB model-owned state per session** (Wan VAE encoder `feat_cache`) ([`pipeline_dreamzero.py#L129`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/pipeline_dreamzero.py#L129)); each request = prefill observation latents into KV → denoise `num_frame_per_block` future latents + action chunk → `current_start_frame += nfpb`; window resets keep prompt/text KV. Optional [`session_state_manager`](../../vllm-omni/docs/features/session_state_manager.md) adapter bounds the VAE history ring. Not `diffusion_kv/`.
|
||||
- **Output** [Verified]: `{"actions": float32 (24, 8), "video": normalized latents (B,16,T,H/8,W/8) bf16}`; RGB decode only via worker RPC in [`export_prediction_video.py`](../../vllm-omni/examples/offline_inference/dreamzero/export_prediction_video.py).
|
||||
- **Latency per tick** [Verified]: CPU transform → (first call) UMT5 + CLIP + VAE encode + cross-KV population; (later) incremental VAE encode → 1 DiT prefill pass per CFG branch → 16 × (1–2) DiT passes (step-cache may skip) → denorm. `torch.compile` (`reduce-overhead` encoders, `default` per DiT block) exists but is opt-in since CI/examples force eager. Batch 1, single sequence per runner.
|
||||
- **Client** [Verified]: [`openpi_client.py`](../../vllm-omni/examples/online_serving/dreamzero/openpi_client.py) (frame schedule offsets `[-23,-16,-8,0]`, cursor +24), [`droid_sim_eval_client.py`](../../vllm-omni/examples/online_serving/dreamzero/droid_sim_eval_client.py) (Isaac Lab; replans every `open_loop_horizon=8` of the 24-action chunk, gripper binarized). Serve: `vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 --served-model-name dreamzero-droid --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml --enforce-eager --disable-log-stats` ([README](../../vllm-omni/examples/online_serving/dreamzero/README.md)).
|
||||
|
||||
### 3.4 LingBot-World 2.0 (`lingbot_world`)
|
||||
|
||||
- **Pipeline** [Verified]: [`lingbot_world/pipeline.py#L23`](../../vllm-omni/vllm_omni/model_executor/models/lingbot_world/pipeline.py#L23) single `DIFFUSION` stage → `video`; deploy [`lingbot_world_v2_stepwise.yaml`](../../vllm-omni/vllm_omni/deploy/lingbot_world_v2_stepwise.yaml): `ARDiffusionEngine`, `step_execution: true`, `streaming_output: true`, `num_inference_steps 4`, `ar_diffusion_height/width 480×832`, KV `gpu_memory_fraction 0.6`.
|
||||
- **Model** [Verified]: `CausalLingBotWorldTransformer3DModel` ([`transformer.py#L675`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/transformer.py#L675)): Wan-style 40-layer / dim 5120 DiT, block-causal self-attn with **sink 9 + sliding window 18 latent frames**, `num_frames_per_block 3`, camera AdaLN injection from Plücker embeddings ([`camera.py`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/camera.py)); UMT5 text (512), `DistributedAutoencoderKLWan`, DMD 4 timesteps `(1000,750,500,250)` with 4 probe + 1 commit forwards per block ([`dmd_block.py`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/dmd_block.py)); no CFG.
|
||||
- **Controls** [Verified]: `prompt.multi_modal_data.image` (first frame, required) + `extra_args`: `action_path` (poses/intrinsics npy under a trusted root), `camera_action_script` (per-chunk WASD/IJKL lists, integrated in [`actions.py#L274`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world/actions.py#L274)), or a typed `ar_diffusion_tick`. Served via `WS /v1/realtime/video` ([`streaming_video_output_api.md`](../../vllm-omni/docs/serving/streaming_video_output_api.md)) emitting one m4s chunk per AR block; mid-session camera interaction not wired ([recipe](../../vllm-omni/recipes/Robbyant/LingBot-World-2.0.md)). This is a world model without an action head — relevant to VLA only as the KV/session template. No AMD/NPU/XPU support claimed.
|
||||
|
||||
### 3.5 InternVLA-A1 (`internvla_a1`)
|
||||
|
||||
[Verified] No `PipelineConfig`, no deploy YAML, not in `OMNI_PIPELINES`; **offline-only** via `initialize_model(OmniDiffusionConfig(model_class_name="InternVLAA1Pipeline"))` and direct `pipeline.forward` ([`examples/offline_inference/internvla_a1/end2end.py`](../../vllm-omni/examples/offline_inference/internvla_a1/end2end.py)). Model ([`model_internvla_a1.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/model_internvla_a1.py)): three Qwen3-VL experts (understanding `Qwen3VLForConditionalGeneration` 28L/2048, generation and action `Qwen3VLTextModel` 28L/1024) with joint attention (`compute_layer_complete` concatenates Q/K/V across experts), plus a vendored NVIDIA Cosmos CI8x8 continuous image tokenizer ([`cosmos_ci_torch.py`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/cosmos_ci_torch.py)) whose latents form a "middle" token stream; flow-matching action head, chunk 50, dim 32 (16 used), 10 steps, no CFG; inputs `extra_args["batch_inputs"]` with 3 cameras × T=2 frames `[B,2,3,H,W]` in [0,1], `observation.state` (pre-normalized by the example), `observation.task`. Optional `torch.compile(max-autotune)` / `regionally_compile`; bf16 eager; doc reports 38 s one-sample e2e on H200 ([`internvla_a1.md`](../../vllm-omni/docs/user_guide/examples/offline_inference/internvla_a1.md), repo-reported). `supported_models.md` lists online ✅ but no serving path exists in source [Verified]; `end2end.py` passes `prompts=` to a dataclass whose field is `prompt` [Inferred possible bug, not executed].
|
||||
|
||||
### 3.6 Cosmos3 action policy (brief)
|
||||
|
||||
[Verified] [`diffusion/models/cosmos3/action.py`](../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py): modes `policy | forward_dynamics | inverse_dynamics`, `EMBODIMENT_TO_DOMAIN_ID`; RoboLab defaults chunk 32, raw action dim 8, 540×640, 4 steps, guidance 3.0; requires external `cosmos_framework` for the action transform. Not analysed further.
|
||||
|
||||
### 3.7 VLA cross-family comparison
|
||||
|
||||
| | GR00T N1.7 | π0 | DreamZero | InternVLA-A1 |
|
||||
|---|---|---|---|---|
|
||||
| Vision/LM backbone | Qwen3-VL (Cosmos-Reason2-2B, 12/… layers), HF classes | PaliGemma (SigLIP + Gemma-2B), HF classes | CLIP ViT-H + UMT5-XXL + Wan VAE, reimplemented | Qwen3-VL ×3 experts + Cosmos CI8x8 |
|
||||
| Images / step | V×T from checkpoint (DROID 2×2), 256² | 3 × 224² | 3 cams stitched 352×640, 4-frame history | 3 × 224² × 2 frames |
|
||||
| State entry | `robot_obs["state"]` dict, per-embodiment MLP | `robot_obs["state"]` → 32-pad | `robot_obs` joint+gripper → 64-pad | `batch_inputs["observation.state"]` |
|
||||
| Action head | 16-block DiT cross-attn, flow, 4 steps, H=40 | Gemma-300M expert, flow, 10 steps, H=50 | actions as DiT tokens, 16 UniPC steps, CFG, H=24 | act expert, flow, 10 steps, H=50 |
|
||||
| Cross-step state | none | none | paged KV per session + VAE stream | none |
|
||||
| Compile/graphs | none (eager) | none (fp32 eager) | opt-in compile; eager in CI | opt-in compile |
|
||||
| Serving | OpenPI WS | OpenPI WS | OpenPI WS (session) | none (offline) |
|
||||
|
||||
### 3.8 Control-loop request/response contracts
|
||||
|
||||
Exact observation and action shapes as consumed/produced in tree (batch dimension included where the code requires it). [Verified from the cited files]
|
||||
|
||||
| Family | Observation keys (in `extra_args["robot_obs"]` / `batch_inputs`) | Shapes / dtypes | Reply | Session semantics |
|
||||
|---|---|---|---|---|
|
||||
| GR00T N1.7 | `video` or `images`: `{cam: uint8 (B,T,H,W,3)}`; `state`: `{key: float32 (B,T,D)}`; `language`: `{key: [[str]]}` or `prompt`; ([`pipeline_gr00t.py#L135`](../../vllm-omni/vllm_omni/diffusion/models/gr00t/pipeline_gr00t.py#L135)) | DROID: 2 cams × T=2 × 256² uint8; `eef_9d (1,1,9)`, `gripper_position (1,1,1)`, `joint_position (1,1,7)`; any image size accepted (letterboxed to 256²) | `{key: float32 (B,40,D)}` — `eef_9d (1,40,9)`, `gripper_position (1,40,1)`, `joint_position (1,40,7)`, un-normalized, relative→absolute applied | stateless; `reset` is a no-op; `policy_server_config.needs_session_id: true` only for client bookkeeping |
|
||||
| π0 | `observation.images.*` / `images` (≤3 cams, `image_feature_keys` order), `state`, `prompt`; ([`processor_pi0.py#L162`](../../vllm-omni/vllm_omni/diffusion/models/pi0/processor_pi0.py#L162)) | HWC uint8 224² (resized with pad), `state` float32 ≤32, prompt ≤48 tokens | `actions` float32 `(50, 32)` (batch squeezed), identity norm for `pi0_base` | stateless; `session_id`/`reset` ignored |
|
||||
| DreamZero | `observation/exterior_image_{0,1}_left`, `observation/wrist_image_left`, `observation/joint_position`, `observation/gripper_position`, `prompt`, `session_id`, optional `embodiment`; ([`pipeline_dreamzero.py#L1361`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero/pipeline_dreamzero.py#L1361)) | images `(H,W,3)` on first call then `(4,H,W,3)` history (180×320 from clients), joint `(7)`, gripper `(1)` | `actions` float32 `(24, 8)`; `video` = latents `(B,16,T,H/8,W/8)` in `images` | **session-affine**: paged KV + VAE stream per `session_id`, LRU 64; `reset` on first call / explicit / prompt change; `extra_args["close_session"]` |
|
||||
| InternVLA-A1 | `batch_inputs`: `observation.images.image{0,1,2}` `[B,2,3,H,W]` float [0,1] + `_mask`, `observation.state` (pre-normalized, 16→32 pad), `observation.task` str; `noise`, `decode_image`; ([`pipeline_internvla_a1.py#L235`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1/pipeline_internvla_a1.py#L235)) | B=1 enforced; history T=2 (offsets −15, 0) | `payload.actions` `(1, 50, 32)` (first 16 dims meaningful), optional decoded image in `metadata` | stateless; offline `pipeline.forward` only |
|
||||
| LingBot-World (for reference) | `multi_modal_data.image` + `extra_args.{action_path \| camera_action_script \| ar_diffusion_tick, flow_shift}` | first frame 480×832; per-chunk 3-frame WASD lists | video chunk (m4s) per AR block, `metadata.ar_diffusion.{session_id, chunk_index, applied_event_ids}` | session KV (sink 9 + window 9 latent frames), `session_capacity 2` |
|
||||
|
||||
Latency-relevant client behaviour [Verified]: the DreamZero sim client replans every 8 executed actions of a 24-action chunk ([`droid_sim_eval_client.py`](../../vllm-omni/examples/online_serving/dreamzero/droid_sim_eval_client.py)); the π0/GR00T clients issue one `infer` per chunk with no overlap; no client in tree pipelines the next observation while the previous chunk executes (asynchronous inference would need server-side session state that only DreamZero has) [Inferred].
|
||||
|
||||
---
|
||||
|
||||
## 4. Cross-cutting pipeline mechanics
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
C[Client: /v1/audio/speech, /v1/realtime/robot/openpi, Omni.generate] --> E[AsyncOmniEngine]
|
||||
E --> O[Orchestrator: request state, routing, abort]
|
||||
O --> S0[Stage 0 process: LLM_AR (vLLM engine + OmniARScheduler)]
|
||||
S0 -- OmniConnector put/get (SHM, msgpack, CPU tensors) --> S1[Stage 1 process: LLM_GENERATION (GPUGenerationModelRunner) or DIFFUSION worker]
|
||||
S1 --> MO[OmniRequestOutput.multimodal_output / streaming chunks]
|
||||
MO --> E
|
||||
```
|
||||
|
||||
- **Stage config format** [Verified]: `PipelineConfig(model_type, model_arch, hf_architectures, stages=(StagePipelineConfig(...),), default_deploy_config_name, endpoint_restrictions, duplex_*)` and `StagePipelineConfig(stage_id, model_stage, execution_type ∈ {LLM_AR, LLM_GENERATION, DIFFUSION}, input_sources, final_output/_type, owns_tokenizer, engine_output_type, model_arch, retains_state_across_chunks, sampling_constraints, custom_process_input_func, custom_process_next_stage_input_func, async_chunk_process_next_stage_input_func, sync_process_input_func, prompt_transform_func, prompt_expand_func, cfg_kv_collect_func, omni_kv_config, scheduler_cls, model_subdir, tokenizer_subdir, inline_diffusion, requires_full_payload_input, extras)` ([`stage_config.py#L229`](../../vllm-omni/vllm_omni/config/stage_config.py#L229)). Deploy YAML schema = `DeployConfig` (`async_chunk`, `session_mode`, `connectors`, `edges`, `stages: [StageDeployConfig]`, `platforms: {npu|rocm|xpu: {stages}}`, pipeline-wide dtype/quantization/TP/PP) with per-stage `StageDeployConfig` (devices, num_replicas, connectors, sampling defaults, engine sizing, ~60 diffusion knobs, `engine_extras` catch-all) ([`#L371`](../../vllm-omni/vllm_omni/config/stage_config.py#L371); doc [`stage_configs.md`](../../vllm-omni/docs/configuration/stage_configs.md)). Precedence: `--stage-overrides` > global CLI > platform section > `base_config` overlay > defaults. Scheduler per exec type: `OmniARScheduler`/`OmniARAsyncScheduler`, `OmniGenerationScheduler`, `None` for diffusion ([`#L202`](../../vllm-omni/vllm_omni/config/stage_config.py#L202)). Single-stage diffusion models without a registered pipeline use `_create_default_diffusion_stage_cfg` ([`async_omni_engine.py#L1013`](../../vllm-omni/vllm_omni/engine/async_omni_engine.py#L1013)).
|
||||
- **Stage output → next-stage input** [Verified]: producer side, `custom_process_next_stage_input_func` (full payload) or `async_chunk_process_next_stage_input_func` (per step) is invoked by the transfer adapter in the main thread, then `connector.put` in a background `save_loop` ([`chunk_transfer_adapter.py#L654`](../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py#L654); design [`async_chunk.md`](../../vllm-omni/docs/design/feature/async_chunk.md)); consumer side, `custom_process_input_func` / `sync_process_input_func` run in the stage client / orchestrator ([`orchestrator.py#L2548`](../../vllm-omni/vllm_omni/engine/orchestrator.py#L2548), [`stage_engine_core_client.py#L398`](../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py#L398)); async-selection logic in `_select_processor_funcs` ([`stage_config.py#L875`](../../vllm-omni/vllm_omni/config/stage_config.py#L875)). Payload schema = `OmniPayload` TypedDicts / msgspec `OmniPayloadStruct` (`hidden_states`, `embed`, `ids`, `codes.{audio,ref}`, `meta.*`, `latent`, `speaker`, `language`) validated with `forbid_unknown_fields` ([`data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py)). No `action`/`video` keys exist in `OmniPayload` — VLA/diffusion models do not use it.
|
||||
- **Connectors** [Verified]: registered names `SharedMemoryConnector`, `MooncakeStoreConnector`, `MooncakeTransferEngineConnector`, `MoriTransferEngineConnector`, `YuanrongConnector`, `YuanrongTransferEngineConnector` ([`factory.py#L129`](../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py#L129)). SHM connector writes POSIX shared memory under `/dev/shm` with a flock, serializing via `OmniSerializer` (msgpack; **tensors moved to CPU and copied as bytes**, [`serialization.py#L89`](../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py#L89); `data_entry_keys._serialize_tensor` also does `.to("cpu")`). Hence every stage hop is a D2H → IPC → H2D round trip [Inferred consequence]. Same-host only; Mooncake for cross-host.
|
||||
- **Streaming output paths** [Verified]: AR stage emits per decode step; generation stage batches chunk payloads; HTTP `/v1/audio/speech` returns wav/mp3/flac/pcm/opus via `soundfile` ([`audio_utils_mixin.py#L106`](../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py#L106), resampling with `torchaudio.functional.resample`), streams raw bytes (`stream_format="audio"`) or OpenAI `speech.audio.delta` SSE; WebSocket `/v1/audio/speech/stream` accepts incremental text and returns per-sentence PCM chunks with optional word timestamps ([`serving_speech_stream.py`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py); [`speech_api.md`](../../vllm-omni/docs/serving/speech_api.md)). Per-model request normalisation/validation lives in [`tts_adapters/`](../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters) (one adapter per TTS family, `ARTTSAdapter`/`DiffusionTTSAdapter`). Robot actions: one msgpack reply per request, no streaming.
|
||||
- **Output objects** [Verified]: `OmniRequestOutput(RequestOutput)` ([`outputs/__init__.py#L108`](../../vllm-omni/vllm_omni/outputs/__init__.py#L108)) adds `stage_id`, `replica_id`, `final_output_type` ("text"/"image"/"audio"/"latents"), `images`, `latents`, `multimodal_output` and plain-dict `metrics` to vLLM's `RequestOutput`; TTS audio arrives as `multimodal_output["audio"]`/`["model_outputs"]` + `"sr"`, robot actions as `multimodal_output["actions"]`, DreamZero video latents in `images`. Diffusion-side, `OmniRequestOutput.from_diffusion` is built by [`output_formatter.py`](../../vllm-omni/vllm_omni/diffusion/output_formatter.py) which unwraps `{"payload", "metadata"}` envelopes.
|
||||
- **Deploy precedence and platform overrides** [Verified]: `--stage-overrides '{"1": {...}}'` > explicit global CLI flags > `platforms.{npu|rocm|xpu}.stages` > `base_config` overlay > bundled YAML > parser defaults ([`stage_configs.md`](../../vllm-omni/docs/configuration/stage_configs.md)); the stage-based CLI (`--stage-id N --headless --omni-master-address/port`) runs one stage per process. TTS YAMLs already carry per-platform escape hatches (Qwen3-TTS: ROCm stage-1 eager, NPU piecewise graphs; Fish: XPU memory/len limits) — evidence that vocoder graph capture is the first thing to break off-CUDA [Inferred].
|
||||
- **Abort propagation** [Verified]: client disconnect → `serving_speech_stream` calls `engine_client.abort(request_id)` ([`#L365`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py#L365)) → `AsyncOmni._abort` ([`async_omni.py#L1181`](../../vllm-omni/vllm_omni/entrypoints/async_omni.py#L1181)) → `AsyncOmniEngine.abort_async` (RPC with ack, [`async_omni_engine.py#L1931`](../../vllm-omni/vllm_omni/engine/async_omni_engine.py#L1931)) → `Orchestrator._handle_abort` / `_abort_request_ids` → each stage pool's `abort_requests` ([`orchestrator.py#L897`](../../vllm-omni/vllm_omni/engine/orchestrator.py#L897)); partial AR tokens returned as terminal abort outputs. Model-level per-request cleanup hooks (`on_requests_finished`) release codec state slots (MOSS) / decoder caches (Qwen3-TTS).
|
||||
- **Device/dtype assumptions** [Verified]: platforms `cuda`, `rocm`, `npu` (Ascend, incl. 310P patches), `xpu`, `musa` selected by probing pynvml/amdsmi/torch.npu ([`platforms/__init__.py`](../../vllm-omni/vllm_omni/platforms/__init__.py)); **no CPU platform**. Only compiled source: `diffusion/distributed/csrc/a2a_permute.cu` JIT-built via `torch.utils.cpp_extension.load` for Ulysses all-to-all ([`a2a_permute.py#L64`](../../vllm-omni/vllm_omni/diffusion/distributed/a2a_permute.py#L64)) — not used by TTS/VLA. `setup.py` compiles nothing; it only routes requirements by detected device ([`setup.py#L43`](../../vllm-omni/setup.py#L43)). Talkers are bf16 (vLLM default); vocoders mostly fp32 (Qwen3-TTS, HiFT, NeMo RVQ-VAE) or bf16 (MOSS codec on GPU); π0 float32.
|
||||
- **Dependencies** [Verified]: [`requirements/common.txt`](../../vllm-omni/requirements/common.txt) — `transformers>=5.10.1,<5.15`, `diffusers==0.40.0`, `accelerate`, `soundfile>=0.13.1`, `openai-whisper`, `x-transformers`, `einops`, `cache-dit`, `torchsde`, `gguf`, `cosmos-guardrail`, `msgpack`, `pyzmq`; `cuda.txt` adds `onnxruntime`, `fa3-fwd`; `npu.txt` pins `torchaudio==2.10.0`; `xpu.txt` `auto-round-lib`; `musa.txt` `flash_attn_3`. vLLM itself is **not** in requirements — docs pin `vllm==0.28.0` ([`cuda.inc.md#L23`](../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md#L23)). Optional extras in [`pyproject.toml`](../../vllm-omni/pyproject.toml): `forced-aligner` (`qwen-asr`), `indextts2` (fugashi/WeTextProcessing), `fa4`, `quack`; dev extras include `step-audio2`, `voxcpm`, `s3tokenizer`, `funasr`, `mooncake-transfer-engine-cuda13`. Family-specific external packages: `fish-speech` (DAC), `tensorrt` (CosyVoice3/Step-Audio2), NeMo (Gepard/IndexTTS2), `openpi-client` (VLA clients), `cosmos_framework` (Cosmos3).
|
||||
- **Session/duplex state** [Verified]: turn-based by default; `session_mode: "duplex"` + `DuplexSessionRuntimeConfig` for MiniCPM-o/PersonaPlex/Nemotron VoiceChat ([`stage_config.py#L499`](../../vllm-omni/vllm_omni/config/stage_config.py#L499), [`fullduplex.md`](../../vllm-omni/docs/design/fullduplex.md)); AR-diffusion world models keep per-session paged KV and model-owned state under [`experimental/ar_diffusion`](../../vllm-omni/vllm_omni/experimental/ar_diffusion) / [`experimental/world_models`](../../vllm-omni/vllm_omni/experimental/world_models) ([`realtime_ar_diffusion.md`](../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md)).
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client (/v1/audio/speech stream)
|
||||
participant S0 as Stage0 Talker (LLM_AR)
|
||||
participant X as SHM connector
|
||||
participant S1 as Stage1 Code2Wav (LLM_GENERATION)
|
||||
C->>S0: prompt embeds (text + ref codes/speaker)
|
||||
S0->>S0: prefill (paged KV)
|
||||
loop each decode step
|
||||
S0->>S0: sample codebook-0, code predictor -> Q codes
|
||||
S0->>X: async_chunk func: put new frames when chunk boundary
|
||||
end
|
||||
X-->>S1: get chunk (CPU tensor), append to growing request
|
||||
S1->>S1: stateful decode (conv ctx + window KV), batched across requests
|
||||
S1-->>C: PCM chunk (first after 1 frame for Qwen3-TTS)
|
||||
S0->>X: finished=True
|
||||
S1-->>C: last chunk, release state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Portability assessment per family
|
||||
|
||||
Legend: **A** plain torch / HF (exportable to ONNX/TorchScript/ExecuTorch in principle), **B** depends on vLLM layer classes or paged-attention runtime (needs re-implementation for an edge runtime), **C** CUDA-only kernels/graphs/TRT (must be dropped or replaced). Parameter counts are labels (source in §1).
|
||||
|
||||
| Family | Sub-module | Class | Notes |
|
||||
|---|---|---|---|
|
||||
| Qwen3-TTS | 12 Hz tokenizer decoder ([`modeling_qwen3_tts_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py)) | A | torch + transformers mask utils/`DynamicCache`; SnakeBeta Triton has torch fallback; NPU patches RoPE/RMSNorm. Causal convs + 72-frame window make it a clean streaming candidate [Inferred]. |
|
||||
| | Talker (`Qwen3TTSTalkerForConditionalGeneration`) | B | vLLM `Qwen3Model`, `ParallelLMHead`, `LogitsProcessor`; ECAPA speaker encoder is plain nn. |
|
||||
| | Code predictor | A/B | math is `nn.Linear` + SDPA (design note in file); wrapped by `VocabParallelEmbedding`/`CustomOp` and `torch.compile` — bypassable [Inferred]. |
|
||||
| | Graph wrappers, forced aligner | C | `torch.cuda.CUDAGraph`; drop on edge. |
|
||||
| CosyVoice3 | CFM ([`cfm.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/cfm.py)), HiFT ([`hifigan.py`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3/code2wav_core/hifigan.py)), layers | A | plain torch (+omegaconf); HiFT has data-dependent `finalize` branches and a 300 s sine buffer that complicate static export [Inferred]. |
|
||||
| | DiT estimator ([`cosyvoice3_dit.py`](../../vllm-omni/vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py)) | A/B | standard math but uses `vllm_omni.diffusion.attention.layer.Attention`, diffusers `AdaLayerNormZero`, `x_transformers` rotary — swap attention for SDPA [Proposal]. An fp16 ONNX export already exists upstream (`yuekai/Fun-CosyVoice3-0.5B-2512-FP16-ONNX`) [Verified reference]. |
|
||||
| | Talker (`Qwen2Model`), s3tokenizer, CAM++ | B / A | talker on vLLM; CAM++ is ONNX (onnxruntime) — portable; s3tokenizer is a GPU pip package. |
|
||||
| | TRT paths | C | optional; `COSYVOICE3_TRT=0` disables. |
|
||||
| Fish S2 Pro | DAC decoder/encoder | A (external) | needs `fish-speech` pip package; plain torch inside; decoder uses `torch.amp.autocast("cuda")` literal (harmless off-CUDA) [Inferred]. |
|
||||
| | Slow AR | B (+C optional) | vLLM `Qwen3Model`; optional Triton KV backend. |
|
||||
| | Fast AR ([`fish_speech_fast_ar.py`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech/fish_speech_fast_ar.py)) | A/B | SDPA + scratch KV, but layers are vLLM `Linear/RMSNorm/RoPE (disable_tp)`; small (4 layers). |
|
||||
| MOSS-TTS | Codec ([`audio_tokenizer_v2.py`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts/audio_tokenizer_v2.py)) | A | transformers `PreTrainedModel` + SDPA, ring KV; no convs → attractive for NPU/GPU-less export [Inferred]. |
|
||||
| | Talkers, depth/local transformers | B | vLLM `Qwen3Model`; depth transformer itself is plain torch (1–4 layers). |
|
||||
| | Graph wrappers | C | CUDA-only, gated by `device.type == "cuda"`. |
|
||||
| | Nano | A (HF remote code) | not vLLM-native at all; 0.1B; `flash_attn` optional else sdpa. |
|
||||
| Gepard / VoxCPM2 / dots.tts / Ming-TTS | codecs | A / external | NeMo NanoCodec (Gepard) is an external dep; AudioVAE (Ming/dots/VoxCPM2) plain torch; CAM++ via onnxruntime. |
|
||||
| GR00T N1.7 | `policy.py`, `modeling/`, `dataio/`, `configs/` | **A** | torch + transformers (Qwen3-VL) + diffusers (`Attention`, `FeedForward`, `AdaLayerNorm`) + torchvision + scipy; no vLLM kernels; only `pipeline_gr00t.py` and two `vllm.logger` imports bind it to vLLM-Omni. Needs HF hub access for Cosmos-Reason2-2B config and Qwen3-VL-2B processor unless cached [Inferred]. |
|
||||
| π0 | `modeling_pi0.py`, `processor_pi0.py` | **A** | torch + transformers (Gemma/PaliGemma); vLLM import only for `default_weight_loader`; parity test runs on CPU/fp32. |
|
||||
| InternVLA-A1 | model + Cosmos tokenizer | A | torch/einops/transformers; `regionally_compile` optional. |
|
||||
| DreamZero | everything | B/C | vLLM TP layers, custom `paged_write_attn` (flash-attn on CUDA, aiter on ROCm, torch reference elsewhere), AR-Diffusion engine; 14B — not an edge candidate. |
|
||||
| LingBot-World | transformer | B/C | vLLM parallel layers, paged attention; 14B. |
|
||||
| Shared | `common/ops/*`, `snake_activation.py` | C with fallback | Triton kernels guarded by `HAS_TRITON` / `is_cuda`. |
|
||||
|
||||
Suggested extraction boundaries per family [Proposal] — "keep" = reuse the in-tree module as the reference implementation for export; "re-implement" = rewrite against the edge runtime's primitives; "drop" = server-only.
|
||||
|
||||
| Family | Keep (export as-is) | Re-implement on edge runtime | Drop |
|
||||
|---|---|---|---|
|
||||
| Qwen3-TTS | 12 Hz decoder (`Qwen3TTSTokenizerV2Decoder`, incl. its explicit conv/KV contexts), ECAPA speaker encoder, prompt-template logic from `prompt_embeds_builder.py` | Qwen3 talker decode loop with extra embedding tables + codebook-0 head; code predictor (5-layer, 15 sub-prefills/frame) | CUDA-graph wrappers, SHM transport, forced aligner, 25 Hz tokenizer |
|
||||
| MOSS-TTS-Realtime / Local | transformer codec with ring KV (`audio_tokenizer_v2.py`), depth/local transformer | Qwen3 backbone loop with additive multi-codebook embeddings | delay-pattern variants (8B, non-streaming), graph wrappers, reference-encoder service |
|
||||
| CosyVoice3 | HiFT vocoder, CAM++ ONNX, text tokenizer | Qwen2-0.5B talker + RAS sampler; CFM DiT with SDPA and an **incremental** (non-cumulative) chunk schedule | s3tokenizer GPU dependency (needs an ONNX speech tokenizer), TRT paths |
|
||||
| Fish S2 Pro | DAC decoder/encoder (via `fish-speech`), fast AR (4 layers) | Qwen3-36L slow AR (largest talker of the four — likely too big) | Triton KV backend, graphs |
|
||||
| GR00T N1.7 | entire `policy.py` + `modeling/` + `dataio/` (HF/torch/diffusers) | nothing mandatory; optionally cache VLM features across steps when images are unchanged | vLLM-Omni pipeline wrapper, OpenPI server (replace with in-process loop) |
|
||||
| π0 | `modeling_pi0.py` + `processor_pi0.py` | prefix KV caching across steps (not present in tree) | same as above |
|
||||
| DreamZero / LingBot-World | — | — | out of edge scope (14B, session paged KV, CFG parallel) |
|
||||
|
||||
For an edge TTS port the minimal extraction is: (1) talker weights → any small-LLM runtime with KV cache (the talker is a stock Qwen2/Qwen3 decoder plus extra embedding tables and heads); (2) the per-frame sub-AR (code predictor / fast AR / depth transformer) as a tiny stateless transformer re-prefilled per codebook (≤5 layers, 2–16 sub-steps per frame); (3) the codec decoder as a standalone streaming torch module with explicit context state (Qwen3-TTS's 2/12-frame conv contexts + windowed KV, or MOSS's ring KV, are already factored). For VLA, GR00T and π0 are directly liftable as HF/torch graphs; the serving contract to keep is the OpenPI msgpack `infer/reset` protocol with `policy_server_config` handshake.
|
||||
|
||||
---
|
||||
|
||||
## 6. Tests and examples
|
||||
|
||||
- **Layout** [Verified]: [`tests/`](../../vllm-omni/tests) — `model_executor/models/<family>/` CPU unit tests (`core_model`, `cpu` marks), `model_executor/stage_input_processors/`, `diffusion/models/<family>/`, `e2e/offline_inference/`, `e2e/online_serving/`, `e2e/accuracy/`, `entrypoints/openai_api/`, `attention/`, `platforms/`, `dfx/` (perf/stability). Markers in [`pyproject.toml`](../../vllm-omni/pyproject.toml): CI levels `core_model` (L1/L2 per PR), `advanced_model` (L3 merge), `full_model` (L4 nightly), `local_model`; module marks `tts`, `diffusion`, `omni`; hardware marks `cpu/cuda/rocm/xpu/npu/musa` + SKU (`H100`, `L4`, `MI325`, `B60`, `A2/A3`, `S5000`) + `cards_N`; a pre-commit hook enforces one level + one platform mark per test file ([`docs/contributing/README.md`](../../vllm-omni/docs/contributing/README.md)). Buildkite lanes: [`.buildkite/cuda`](../../vllm-omni/.buildkite/cuda) (ready/merge/nightly/weekly), [`amd`](../../vllm-omni/.buildkite/amd), [`npu`](../../vllm-omni/.buildkite/npu), [`intel`](../../vllm-omni/.buildkite/intel). Run per docs: `pytest tests/...` with markers; L1 CPU tests need no weights.
|
||||
- **TTS tests** [Verified]: Qwen3-TTS — [`tests/model_executor/models/qwen3_tts/`](../../vllm-omni/tests/model_executor/models/qwen3_tts) (incremental decode, CUDA-graph decoder equivalence, talker preprocess/prompt len/ref codes/silence ban, custom voice profiles), [`test_qwen3_tts_async_chunk.py`](../../vllm-omni/tests/model_executor/stage_input_processors/test_qwen3_tts_async_chunk.py), e2e [`test_qwen3_tts_base.py`](../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_base.py) / [`test_qwen3_tts_customvoice.py`](../../vllm-omni/tests/e2e/offline_inference/test_qwen3_tts_customvoice.py) and online counterparts. CosyVoice3 — [`tests/model_executor/models/cosyvoice3/`](../../vllm-omni/tests/model_executor/models/cosyvoice3) (components incl. TRT stream deps, helpers, utils, config), [`test_cosyvoice3_stage_input_processors.py`](../../vllm-omni/tests/model_executor/stage_input_processors/test_cosyvoice3_stage_input_processors.py), e2e [`test_cosyvoice3_expansion.py`](../../vllm-omni/tests/e2e/offline_inference/test_cosyvoice3_expansion.py) (L4; sync + async params) / [`online`](../../vllm-omni/tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py) (H100/MI325, `COSYVOICE3_TRT=0` on ROCm). Fish — [`tests/model_executor/models/fish_speech/`](../../vllm-omni/tests/model_executor/models/fish_speech), [`test_fish_speech_async_chunk.py`](../../vllm-omni/tests/model_executor/stage_input_processors/test_fish_speech_async_chunk.py), [`tests/attention/test_fish_kvcache_attn.py`](../../vllm-omni/tests/attention/test_fish_kvcache_attn.py). MOSS — [`tests/model_executor/models/moss_tts/`](../../vllm-omni/tests/model_executor/models/moss_tts) (codec CUDA graph, compact top-k, depth RoPE cache, fused load, per-request codes, reference encoder, logits processor); e2e [`test_moss_tts_expansion.py`](../../vllm-omni/tests/e2e/offline_inference/test_moss_tts_expansion.py), [`test_moss_tts_realtime.py`](../../vllm-omni/tests/e2e/offline_inference/test_moss_tts_realtime.py), [`test_moss_tts_nano_expansion.py`](../../vllm-omni/tests/e2e/offline_inference/test_moss_tts_nano_expansion.py) — several MOSS e2e tests are **skipped** (issues #4700, #6417). TTS API/adapters: [`test_serving_speech.py`](../../vllm-omni/tests/entrypoints/openai_api/test_serving_speech.py), [`test_tts_adapter.py`](../../vllm-omni/tests/entrypoints/openai_api/test_tts_adapter.py). Benchmarks: [`benchmarks/tts/`](../../vllm-omni/benchmarks/tts) (`vllm bench serve --omni --backend openai-audio-speech --endpoint /v1/audio/speech --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,...`; registry [`model_configs.yaml`](../../vllm-omni/benchmarks/tts/model_configs.yaml)).
|
||||
- **VLA tests** [Verified]: GR00T — [`tests/diffusion/models/gr00t/test_pipeline.py`](../../vllm-omni/tests/diffusion/models/gr00t/test_pipeline.py) (CPU, fake policy), e2e [`test_gr00t_openpi_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_gr00t_openpi_expansion.py) (L4; precision vs Isaac-GR00T reference `atol=1e-2`, `GR00T_NOISE_SEED=42`). π0 — [`test_pi0_units.py`](../../vllm-omni/tests/diffusion/models/pi0/test_pi0_units.py) (CPU), [`test_pi0_parity.py`](../../vllm-omni/tests/diffusion/models/pi0/test_pi0_parity.py) (vs LeRobot, `local_model`, atol 1e-4), e2e [`test_pi0_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_pi0_expansion.py) (H100, `DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA`). DreamZero — [`tests/diffusion/models/dreamzero/`](../../vllm-omni/tests/diffusion/models/dreamzero) (deploy configs, session state accounting/equivalence/lifecycle, fused QK norm), e2e [`test_dreamzero_expansion.py`](../../vllm-omni/tests/e2e/online_serving/test_dreamzero_expansion.py) (H100×2, expects 3 × `(24,8)`), accuracy [`test_dreamzero.py`](../../vllm-omni/tests/e2e/accuracy/test_dreamzero.py) (upstream parity, `rtol 1e-2 / atol 1e-3`). LingBot-World — [`tests/diffusion/models/lingbot_world/`](../../vllm-omni/tests/diffusion/models/lingbot_world), [`test_lingbot_world_v2_stepwise.py`](../../vllm-omni/tests/e2e/online_serving/test_lingbot_world_v2_stepwise.py). InternVLA — [`test_internvla_a1.py`](../../vllm-omni/tests/examples/offline_inference/test_internvla_a1.py) (advanced level, env-gated). OpenPI transport: [`test_openpi_serving.py`](../../vllm-omni/tests/entrypoints/openai_api/test_openpi_serving.py), [`test_openpi_connection.py`](../../vllm-omni/tests/entrypoints/openai_api/test_openpi_connection.py).
|
||||
- **Example commands (verbatim from READMEs; not run)** [Verified]:
|
||||
- Qwen3-TTS: `python examples/offline_inference/text_to_speech/qwen3_tts/end2end.py --query-type Base --audio-path /path/to/reference.wav`; `vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice --omni --port 8091`; `python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you?" --stream-audio --response-format pcm --word-timestamps` ([offline README](../../vllm-omni/examples/offline_inference/text_to_speech/README.md), [online README](../../vllm-omni/examples/online_serving/text_to_speech/README.md)).
|
||||
- CosyVoice3: `python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py --model pretrained_models/Fun-CosyVoice3-0.5B --tokenizer pretrained_models/Fun-CosyVoice3-0.5B/CosyVoice-BlankEN --ref-audio /path/to/reference.wav --prompt-text "You are a helpful assistant.<|endofprompt|>Trascript in your ref audio clip"`; `vllm serve FunAudioLLM/Fun-CosyVoice3-0.5B-2512 --omni --port 8091 --trust-remote-code`.
|
||||
- Fish: `python examples/offline_inference/text_to_speech/fish_speech/end2end.py --text "Hello, this is a streaming test." --streaming`; `vllm serve fishaudio/s2-pro --omni --port 8091` ([recipe](../../vllm-omni/recipes/fishaudio/Fish-Speech-S2-Pro.md)).
|
||||
- MOSS: `python end2end.py --model OpenMOSS-Team/MOSS-TTS --text "Hello, this is a MOSS-TTS test." --ref-audio path/to/reference.wav --output output.wav`; `vllm serve OpenMOSS-Team/MOSS-TTS-Realtime --omni --port 8091`; `vllm serve OpenMOSS-Team/MOSS-TTS-Nano --omni --port 8091` ([recipe](../../vllm-omni/recipes/OpenMOSS/MOSS-TTS.md)).
|
||||
- GR00T: `vllm serve nvidia/GR00T-N1.7-3B --omni --host 127.0.0.1 --port 8000 --served-model-name gr00t-n1d7 --deploy-config vllm_omni/deploy/Gr00tN1d7.yaml`.
|
||||
- π0: `vllm serve lerobot/pi0_base --omni --port 8000 --served-model-name pi0 --deploy-config vllm_omni/deploy/pi0.yaml --enforce-eager --disable-log-stats`; `python examples/online_serving/pi0/openpi_client.py --host 127.0.0.1 --port 8000 --prompt "pick up the red block and place it in the bin"` ([README](../../vllm-omni/examples/online_serving/pi0/README.md)).
|
||||
- DreamZero: `vllm serve GEAR-Dreams/DreamZero-DROID --omni --port 8000 --served-model-name dreamzero-droid --deploy-config vllm_omni/deploy/dreamzero_tp1_cfg2.yaml --enforce-eager --disable-log-stats`; `python examples/online_serving/dreamzero/openpi_client.py --host 127.0.0.1 --port 8000 --video-dir outputs/dreamzero/assets`.
|
||||
- InternVLA-A1: `bash run.sh --num-episodes 1` in [`examples/offline_inference/internvla_a1/`](../../vllm-omni/examples/offline_inference/internvla_a1) ([README](../../vllm-omni/examples/offline_inference/internvla_a1/README.md)).
|
||||
- LingBot-World: `vllm serve robbyant/lingbot-world-v2-14b-causal-fast-diffusers --omni --deploy-config vllm_omni/deploy/lingbot_world_v2_stepwise.yaml --port 8000` + [`streaming_video_client.py`](../../vllm-omni/examples/online_serving/streaming_video_generation/streaming_video_client.py).
|
||||
- **Apps** [Verified]: [`apps/ComfyUI-vLLM-Omni`](../../vllm-omni/apps/ComfyUI-vLLM-Omni) (ComfyUI nodes) and [`apps/lvsa-omni-showcase`](../../vllm-omni/apps/lvsa-omni-showcase) (long-video sparse attention) — neither mobile nor TTS/VLA specific. No mobile bindings exist.
|
||||
|
||||
---
|
||||
|
||||
## 7. Limitations and unknowns
|
||||
|
||||
1. **No edge/CPU/mobile execution path at all** — platforms are CUDA/ROCm/NPU/XPU/MUSA only; `cpu` requirements exist solely for docs builds. [Verified]
|
||||
2. **All talkers depend on vLLM's paged-attention runtime** (`Qwen2Model`/`Qwen3Model`, `ParallelLMHead`, `LogitsProcessor`); exporting them requires re-implementing embedding/head extensions on a different LLM runtime. [Verified]
|
||||
3. **Stage hops serialize tensors through CPU shared memory** (msgpack), so even co-located stages pay D2H/H2D per chunk. [Verified] Latency impact not measured here. [Unknown]
|
||||
4. **MOSS Delay-family streaming is inert** (`chunk_frames = 1<<30`), despite YAML `codec_streaming: true`; `MOSS_TTS_CODEC_PATH` is documented but unimplemented; Realtime online serving passes `prompt_audio_array` the talker never reads. [Verified]
|
||||
5. **CosyVoice3 streaming re-decodes the full history per chunk** (cumulative CFM + HiFT), O(n²); stage 1 is not batched; `codec_left_context_frames` is unused by its processor; two RAS implementations diverge (`common/nucleus_ras_sampling.py` vs inline). [Verified]
|
||||
6. **Qwen3-TTS `stop_token_ids [2150]` vs config default `codec_eos_token_id=4198`** — checkpoint override assumed. [Inferred] Actual talker layer counts for shipped checkpoints not in tree. [Unknown]
|
||||
7. **Fish `dac_utils.py` docstring says the DAC encoder runs on CPU; code runs it on the stage-0 GPU.** Recipe links to non-existent doc/example paths. [Verified]
|
||||
8. **VLA models have no cross-step reuse**: GR00T and π0 recompute the full VLM prefill each control step; only DreamZero/LingBot keep session KV, and those require the experimental `ARDiffusionEngine` with `max_num_seqs=1`. [Verified]
|
||||
9. **GR00T warm-up never runs the model** (dummy zeros), so first-request cold start is unmitigated; `policy_server_config.image_resolution [180,320]` is advertised but the model letterboxes to 256². `final_output_type` for GR00T resolves to `"image"` in the diffusion metadata path while the pipeline declares `"actions"`. [Verified]
|
||||
10. **InternVLA-A1 has no serving path** despite the supported-models table's online ✅; `end2end.py` may pass a wrong kwarg (`prompts=`). [Verified / Inferred]
|
||||
11. **Parameter counts are never computed in-tree**; all sizes here are HF ids, YAML headers, or comments. DreamZero's UMT5-XXL / ViT-H sizes, GLM-TTS, IndexTTS2, OmniVoice, MiniCPM-o 4.5, PersonaPlex, Local v1.5 backbone: [Unknown].
|
||||
12. **Performance numbers are repo-reported only** (Qwen3-TTS TTFP 64 ms, MOSS Realtime "TTFB ~180 ms", InternVLA 38 s/sample); nothing reproduced. [Verified labels]
|
||||
13. **DreamZero NPU/Intel checkmarks** in `supported_models.md` are not evidenced by DreamZero-specific platform code (CUDA stream sync in CFG-parallel, `torch.cuda.is_available` gates). [Inferred]
|
||||
14. **Several e2e tests are skipped** (MOSS #4700/#6417); GR00T recipe references a test file that does not exist (`test_gr00t_openpi.py`). [Verified]
|
||||
15. **Subagent-derived line numbers** were spot-checked but not all re-read individually; the mechanical check below covers path existence, not line accuracy. [Inferred]
|
||||
|
||||
---
|
||||
|
||||
## 8. Reference index
|
||||
|
||||
All links resolve relative to `analysis/engines/`; existence verified mechanically at write time.
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| [`vllm_omni/config/pipeline_registry.py`](../../vllm-omni/vllm_omni/config/pipeline_registry.py) | `OMNI_PIPELINES` model_type → `PipelineConfig` map, `register_pipeline`, `resolve_pipeline_config` |
|
||||
| [`vllm_omni/config/stage_config.py`](../../vllm-omni/vllm_omni/config/stage_config.py) | `StageExecutionType`, `StagePipelineConfig`, `PipelineConfig`, `StageDeployConfig`, `DeployConfig`, scheduler resolution, processor-func selection |
|
||||
| [`vllm_omni/model_executor/models/registry.py`](../../vllm-omni/vllm_omni/model_executor/models/registry.py) | `_OMNI_MODELS` HF-arch → module/class map |
|
||||
| [`vllm_omni/model_executor/models/output_templates.py`](../../vllm-omni/vllm_omni/model_executor/models/output_templates.py) | `OmniOutput(text_hidden_states, multimodal_outputs, ...)` |
|
||||
| [`vllm_omni/data_entry_keys.py`](../../vllm-omni/vllm_omni/data_entry_keys.py) | `OmniPayload` / `OmniPayloadStruct` inter-stage schema, tensor (de)serialization to CPU |
|
||||
| [`vllm_omni/deploy/`](../../vllm-omni/vllm_omni/deploy) | 90 bundled deploy YAMLs (`qwen3_tts.yaml`, `cosyvoice3.yaml`, `fish_qwen3_omni*.yaml`, `moss_*.yaml`, `Gr00tN1d7.yaml`, `pi0.yaml`, `dreamzero*.yaml`, `lingbot_world_v2_stepwise.yaml`, …) |
|
||||
| [`vllm_omni/model_executor/models/qwen3_tts/`](../../vllm-omni/vllm_omni/model_executor/models/qwen3_tts) | talker, code2wav, prompt builder, code predictor wrapper, 12 Hz/25 Hz tokenizers, CUDA-graph wrappers |
|
||||
| [`vllm_omni/model_executor/models/common/qwen3_code_predictor.py`](../../vllm-omni/vllm_omni/model_executor/models/common/qwen3_code_predictor.py) | shared residual-codebook predictor (`CodePredictorWrapper`) |
|
||||
| [`vllm_omni/model_executor/stage_input_processors/qwen3_tts.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/qwen3_tts.py) | `talker2code2wav_async_chunk`, `_full_payload`, `_token_only` |
|
||||
| [`vllm_omni/utils/forced_aligner.py`](../../vllm-omni/vllm_omni/utils/forced_aligner.py) | injects Qwen3-ForcedAligner pooling stage for word timestamps |
|
||||
| [`vllm_omni/model_executor/models/cosyvoice3/`](../../vllm-omni/vllm_omni/model_executor/models/cosyvoice3) | talker (`CosyVoice3Model`, RAS sampler), code2wav (CFM + HiFT), TRT helpers, tokenizer, utils |
|
||||
| [`vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py`](../../vllm-omni/vllm_omni/diffusion/models/cosyvoice3_audio/cosyvoice3_dit.py) | CFM estimator DiT used by CosyVoice3 code2wav |
|
||||
| [`vllm_omni/model_executor/stage_input_processors/cosyvoice3.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/cosyvoice3.py) | CosyVoice3 async-chunk / sync processors |
|
||||
| [`vllm_omni/model_executor/models/fish_speech/`](../../vllm-omni/vllm_omni/model_executor/models/fish_speech) | slow AR, fast AR, DAC decoder/encoder, prompt utils |
|
||||
| [`vllm_omni/model_executor/stage_input_processors/fish_speech.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/fish_speech.py) | `slow_ar_to_dac_decoder_async_chunk` |
|
||||
| [`vllm_omni/attention/fish_kvcache_backend.py`](../../vllm-omni/vllm_omni/attention/fish_kvcache_backend.py) | optional Triton decode-only KV backend for Fish slow AR |
|
||||
| [`vllm_omni/utils/speaker_cache.py`](../../vllm-omni/vllm_omni/utils/speaker_cache.py) | LRU speaker/voice cache shared by TTS families |
|
||||
| [`vllm_omni/model_executor/models/moss_tts/`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts) | Delay/Realtime/Local talkers, depth transformer, codec decoder, audio tokenizer v1/v2, graph wrappers, reference encoder |
|
||||
| [`vllm_omni/model_executor/models/moss_tts_nano/`](../../vllm-omni/vllm_omni/model_executor/models/moss_tts_nano) | single-stage Nano wrapper around HF remote code |
|
||||
| [`vllm_omni/model_executor/stage_input_processors/moss_tts.py`](../../vllm-omni/vllm_omni/model_executor/stage_input_processors/moss_tts.py) | delay/raw async processors, sync `talker2codec` |
|
||||
| [`vllm_omni/core/sched/omni_generation_scheduler.py`](../../vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py) | generation-stage scheduler (`retains_state_across_chunks` accounting) |
|
||||
| [`vllm_omni/worker/gpu_ar_model_runner.py`](../../vllm-omni/vllm_omni/worker/gpu_ar_model_runner.py) | AR runner: `talker_mtp` CUDA-graph capture, model-owned sampler hooks |
|
||||
| [`vllm_omni/model_executor/models/gr00t/pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/gr00t/pipeline.py) | `GR00T_N1D7_PIPELINE` |
|
||||
| [`vllm_omni/diffusion/models/gr00t/`](../../vllm-omni/vllm_omni/diffusion/models/gr00t) | `pipeline_gr00t.py`, `policy.py`, `modeling/gr00t_n1d7.py`, `modeling/modules/dit.py`, `embodiment_conditioned_mlp.py`, `processing_gr00t_n1d7.py`, `dataio/` |
|
||||
| [`vllm_omni/diffusion/models/pi0_pipeline_config.py`](../../vllm-omni/vllm_omni/diffusion/models/pi0_pipeline_config.py) | `PI0_PIPELINE` |
|
||||
| [`vllm_omni/diffusion/models/pi0/`](../../vllm-omni/vllm_omni/diffusion/models/pi0) | `pipeline_pi0.py`, `modeling_pi0.py`, `processor_pi0.py`, `config.py` |
|
||||
| [`vllm_omni/model_executor/models/dreamzero/pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/dreamzero/pipeline.py) | `DREAMZERO_PIPELINE` |
|
||||
| [`vllm_omni/diffusion/models/dreamzero/`](../../vllm-omni/vllm_omni/diffusion/models/dreamzero) | `pipeline_dreamzero.py`, `causal_wan_model.py`, `image_encoder.py`, `action_encoder.py`, `state_dreamzero.py`, `transform/droid.py`, `utils.py` |
|
||||
| [`vllm_omni/model_executor/models/lingbot_world/pipeline.py`](../../vllm-omni/vllm_omni/model_executor/models/lingbot_world/pipeline.py) | `LINGBOT_WORLD_PIPELINE` |
|
||||
| [`vllm_omni/diffusion/models/lingbot_world/`](../../vllm-omni/vllm_omni/diffusion/models/lingbot_world) | `pipeline.py`, `transformer.py`, `dmd_block.py`, `camera.py`, `actions.py` |
|
||||
| [`vllm_omni/diffusion/models/internvla_a1/`](../../vllm-omni/vllm_omni/diffusion/models/internvla_a1) | `pipeline_internvla_a1.py`, `model_internvla_a1.py`, `cosmos_ci_torch.py`, `config.py` |
|
||||
| [`vllm_omni/diffusion/models/cosmos3/action.py`](../../vllm-omni/vllm_omni/diffusion/models/cosmos3/action.py) | Cosmos3 action-policy modes |
|
||||
| [`vllm_omni/diffusion/request.py`](../../vllm-omni/vllm_omni/diffusion/request.py) | `OmniDiffusionRequest` |
|
||||
| [`vllm_omni/inputs/data.py`](../../vllm-omni/vllm_omni/inputs/data.py) | `OmniDiffusionSamplingParams.extra_args` |
|
||||
| [`vllm_omni/diffusion/data.py`](../../vllm-omni/vllm_omni/diffusion/data.py) | `DiffusionOutput`, `OmniDiffusionConfig`, model-class auto-detection |
|
||||
| [`vllm_omni/diffusion/output_formatter.py`](../../vllm-omni/vllm_omni/diffusion/output_formatter.py) | maps `actions/audio/trajectory` into `multimodal_output` |
|
||||
| [`vllm_omni/outputs/__init__.py`](../../vllm-omni/vllm_omni/outputs/__init__.py) | `OmniRequestOutput` (stage_id, final_output_type, images, latents, multimodal_output) |
|
||||
| [`vllm_omni/entrypoints/openpi/connection.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/connection.py) | OpenPI msgpack WebSocket connection handler |
|
||||
| [`vllm_omni/entrypoints/openpi/serving.py`](../../vllm-omni/vllm_omni/entrypoints/openpi/serving.py) | builds `OmniDiffusionRequest` with `extra_args["robot_obs"]`, extracts actions |
|
||||
| [`vllm_omni/entrypoints/openai/api_server.py`](../../vllm-omni/vllm_omni/entrypoints/openai/api_server.py) | route registration (`/v1/realtime/robot/openpi`, `/v1/realtime/video`, speech endpoints) |
|
||||
| [`vllm_omni/entrypoints/openai/serving_speech.py`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech.py) | `/v1/audio/speech` request building per family |
|
||||
| [`vllm_omni/entrypoints/openai/serving_speech_stream.py`](../../vllm-omni/vllm_omni/entrypoints/openai/serving_speech_stream.py) | WebSocket streaming-text TTS session, abort on disconnect |
|
||||
| [`vllm_omni/entrypoints/openai/audio_utils_mixin.py`](../../vllm-omni/vllm_omni/entrypoints/openai/audio_utils_mixin.py) | wav/mp3/flac/pcm/opus encoding via soundfile, torchaudio resampling |
|
||||
| [`vllm_omni/entrypoints/openai/tts_adapters/`](../../vllm-omni/vllm_omni/entrypoints/openai/tts_adapters) | per-family TTS request adapters and capability loaders |
|
||||
| [`vllm_omni/entrypoints/async_omni.py`](../../vllm-omni/vllm_omni/entrypoints/async_omni.py) | `AsyncOmni` generate/abort |
|
||||
| [`vllm_omni/engine/async_omni_engine.py`](../../vllm-omni/vllm_omni/engine/async_omni_engine.py) | engine composition root, `abort_async`, default diffusion stage cfg |
|
||||
| [`vllm_omni/engine/orchestrator.py`](../../vllm-omni/vllm_omni/engine/orchestrator.py) | cross-stage routing, `_handle_abort`, custom input func invocation |
|
||||
| [`vllm_omni/engine/stage_engine_core_client.py`](../../vllm-omni/vllm_omni/engine/stage_engine_core_client.py) | applies `custom_process_input_func` per stage |
|
||||
| [`vllm_omni/distributed/omni_connectors/factory.py`](../../vllm-omni/vllm_omni/distributed/omni_connectors/factory.py) | connector registry |
|
||||
| [`vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py`](../../vllm-omni/vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py) | async-chunk put/get lifecycle |
|
||||
| [`vllm_omni/distributed/omni_connectors/utils/serialization.py`](../../vllm-omni/vllm_omni/distributed/omni_connectors/utils/serialization.py) | `OmniSerializer` (msgpack, tensors → CPU bytes) |
|
||||
| [`vllm_omni/platforms/__init__.py`](../../vllm-omni/vllm_omni/platforms/__init__.py) | platform plugin probing (cuda/rocm/npu/xpu/musa) |
|
||||
| [`vllm_omni/diffusion/distributed/a2a_permute.py`](../../vllm-omni/vllm_omni/diffusion/distributed/a2a_permute.py) | only compiled CUDA source (JIT) in tree |
|
||||
| [`vllm_omni/experimental/ar_diffusion/`](../../vllm-omni/vllm_omni/experimental/ar_diffusion) | `ARDiffusionEngine`, runner, session, paged KV for world models |
|
||||
| [`vllm_omni/experimental/world_models/`](../../vllm-omni/vllm_omni/experimental/world_models) | session state manager and DreamZero adapter |
|
||||
| [`setup.py`](../../vllm-omni/setup.py) | platform-aware requirements routing, no extensions |
|
||||
| [`pyproject.toml`](../../vllm-omni/pyproject.toml) | optional extras, pytest markers |
|
||||
| [`requirements/common.txt`](../../vllm-omni/requirements/common.txt) | core dependency pins |
|
||||
| [`docs/design/architecture_overview.md`](../../vllm-omni/docs/design/architecture_overview.md) | stage-based architecture, Qwen3-Omni stage sizes |
|
||||
| [`docs/configuration/stage_configs.md`](../../vllm-omni/docs/configuration/stage_configs.md) | pipeline vs deploy config schema |
|
||||
| [`docs/design/feature/async_chunk.md`](../../vllm-omni/docs/design/feature/async_chunk.md) | async chunk design + repo-reported perf table |
|
||||
| [`docs/design/feature/realtime_ar_diffusion.md`](../../vllm-omni/docs/design/feature/realtime_ar_diffusion.md) | AR-diffusion session contract |
|
||||
| [`docs/design/fullduplex.md`](../../vllm-omni/docs/design/fullduplex.md) | full-duplex runtime |
|
||||
| [`docs/design/qwen3_omni_tts_performance_optimization.md`](../../vllm-omni/docs/design/qwen3_omni_tts_performance_optimization.md) | repo-reported Qwen3-TTS/Omni latency numbers |
|
||||
| [`docs/features/session_state_manager.md`](../../vllm-omni/docs/features/session_state_manager.md) | per-session state accounting (DreamZero) |
|
||||
| [`docs/serving/speech_api.md`](../../vllm-omni/docs/serving/speech_api.md) | `/v1/audio/speech`, streaming formats, voices |
|
||||
| [`docs/serving/openpi_api.md`](../../vllm-omni/docs/serving/openpi_api.md) | OpenPI robot WebSocket protocol |
|
||||
| [`docs/serving/streaming_video_output_api.md`](../../vllm-omni/docs/serving/streaming_video_output_api.md) | `/v1/realtime/video` |
|
||||
| [`docs/models/supported_models.md`](../../vllm-omni/docs/models/supported_models.md) | model/platform support table |
|
||||
| [`docs/getting_started/installation/gpu/cuda.inc.md`](../../vllm-omni/docs/getting_started/installation/gpu/cuda.inc.md) | `vllm==0.28.0` pin |
|
||||
| [`docs/contributing/README.md`](../../vllm-omni/docs/contributing/README.md) | test levels and marks |
|
||||
| [`docs/user_guide/examples/offline_inference/internvla_a1.md`](../../vllm-omni/docs/user_guide/examples/offline_inference/internvla_a1.md) | InternVLA-A1 usage + repo-reported timings |
|
||||
| [`recipes/Qwen/Qwen3-TTS.md`](../../vllm-omni/recipes/Qwen/Qwen3-TTS.md), [`recipes/fishaudio/Fish-Speech-S2-Pro.md`](../../vllm-omni/recipes/fishaudio/Fish-Speech-S2-Pro.md), [`recipes/OpenMOSS/MOSS-TTS.md`](../../vllm-omni/recipes/OpenMOSS/MOSS-TTS.md), [`recipes/OpenBMB/VoxCPM2.md`](../../vllm-omni/recipes/OpenBMB/VoxCPM2.md), [`recipes/rednote-hilab/dots.tts.md`](../../vllm-omni/recipes/rednote-hilab/dots.tts.md), [`recipes/NVIDIA/Nemotron-Labs-Audex.md`](../../vllm-omni/recipes/NVIDIA/Nemotron-Labs-Audex.md) | TTS deployment recipes (size labels, commands) |
|
||||
| [`recipes/NVIDIA/GR00T-N1.7.md`](../../vllm-omni/recipes/NVIDIA/GR00T-N1.7.md), [`recipes/lerobot/Pi0.md`](../../vllm-omni/recipes/lerobot/Pi0.md), [`recipes/Robbyant/LingBot-World-2.0.md`](../../vllm-omni/recipes/Robbyant/LingBot-World-2.0.md) | VLA / world-model recipes |
|
||||
| [`examples/offline_inference/text_to_speech/`](../../vllm-omni/examples/offline_inference/text_to_speech), [`examples/online_serving/text_to_speech/`](../../vllm-omni/examples/online_serving/text_to_speech) | per-family TTS examples and READMEs |
|
||||
| [`examples/online_serving/pi0/`](../../vllm-omni/examples/online_serving/pi0), [`examples/online_serving/dreamzero/`](../../vllm-omni/examples/online_serving/dreamzero), [`examples/offline_inference/dreamzero/`](../../vllm-omni/examples/offline_inference/dreamzero), [`examples/offline_inference/internvla_a1/`](../../vllm-omni/examples/offline_inference/internvla_a1), [`examples/online_serving/streaming_video_generation/streaming_video_client.py`](../../vllm-omni/examples/online_serving/streaming_video_generation/streaming_video_client.py) | VLA / world-model clients |
|
||||
| [`benchmarks/tts/`](../../vllm-omni/benchmarks/tts) | universal TTS benchmark CLI + model registry |
|
||||
| [`tests/`](../../vllm-omni/tests) | see §6 for per-family test files |
|
||||
| [`.buildkite/`](../../vllm-omni/.buildkite) | CI lanes (cuda/amd/npu/intel) |
|
||||
| [`apps/`](../../vllm-omni/apps) | ComfyUI nodes, LVSA showcase (no mobile) |
|
||||
Reference in new issue
Block a user