0 Fork 0

Files

T

tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork

The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

2026-09-14 14:01:12 +08:00

110 KiB

Raw Blame History

ExecuTorch (pytorch/executorch) — Engine Analysis

Field	Value
Local path	`/data/zhoutaichang/embedding_infer/executorch`
Remote	`https://github.com/pytorch/executorch.git` (origin)
Branch	`main`
HEAD	`cfc6ecd5166bec70c5fde2bad836fed26018fe52` — "Qualcomm AI Engine Direct - Enable QNN Windows ARM64 build in CI (#22438)", 2026-09-07 23:47 +0800
Declared version	`1.5.0` (version.txt)
Dirty status	clean (`git status --short` empty)
Submodules	all 23 uninitialized (`git submodule status` shows `-` for every entry), including `extension/llm/tokenizers`, `backends/xnnpack/third-party/{XNNPACK,pthreadpool,cpuinfo,FP16,FXdiv}`, `backends/mlx/third-party/mlx`, `backends/vulkan/third-party/*`, `third-party/{ao,flatbuffers,flatcc,gflags,googletest,json,pocketfft,pybind11,ios-cmake,prelude}`, `kernels/optimized/third-party/eigen`, `shim`, `backends/cadence/utils/FACTO`
Analysis date	2026-09-08 (read-only; no builds, installs or GPU use)
Repo guidance followed	CLAUDE.md (AGENTS.md is a symlink to it), `.claude/*.md` reference docs, `.wiki/index.md`

Evidence labels: [Verified] read in code/docs at this HEAD; [Inferred] reasoned from structure; [Proposal] design suggestion; [Unknown] not determinable from the tree. Repo-published numbers are labelled "repo-reported, not reproduced".

0. Summary

What it is [Verified]: ExecuTorch is an ahead-of-time (AOT) compiler + minimal C++ execution runtime for PyTorch models. Python side (exir/) lowers torch.export programs through Edge IR, partitions subgraphs to hardware "delegates" (to_backend), plans memory statically, and serializes a FlatBuffer .pte (schema/program.fbs). The C++ runtime (runtime/executor/method.cpp) walks a linear instruction list per method.
Execution runtime, not a serving stack [Verified]: the core is a single-Method, single-thread, sequential interpreter (Method::execute loops chains → instructions; comment says "Chains are executed sequentially today"). There is no request queue, batcher or server in runtime/.
Serving-adjacent layers exist above the core [Verified]: extension/llm/runner/ (TextLLMRunner / MultimodalRunner with token callbacks), an experimental extension/llm/batching/ (DecodeFirstScheduler, Runner, Session, GenerationHandle with cancel) and examples/llm_server/ (OpenAI-compatible Python control plane over JSONL worker binaries). No in-tree class implements batching::Executor outside its tests.
Backends in tree [Verified]: XNNPACK, Vulkan, WebGPU, Qualcomm QNN (HTP/GPU/LPAI), MediaTek Neuron, Samsung Exynos (ENN), Arm (Ethos-U via TOSA/Vela, VGF, Cortex-M CMSIS-NN), Cadence DSP, NXP Neutron, OpenVINO, Apple CoreML, Apple Metal (experimental), Apple "Core AI" (under construction), MLX (experimental), CUDA (AOTInductor + Triton; experimental ROCm), a "native" portable delegate, and the AOTI common library. No backends/apple/mps directory exists at this HEAD.
Device model [Verified]: backend selection happens entirely at export (partitioner). At runtime BackendDelegate::Init looks up the delegate by string id in a 16-slot static registry (kMaxRegisteredBackends = 16, runtime/backend/interface.cpp) and calls is_available(). There is no runtime device discovery/selection.
Memory [Verified]: static memory planning in Python (greedy default, optional banked planning) produces non_const_buffer_sizes; the runtime MemoryManager = method allocator + HierarchicalAllocator (planned arenas, ≤16 spans) + temp allocator. A new Device{CPU,CUDA} concept and DeviceAllocator registry let planned buffers live in CUDA memory with explicit _h2d_copy/_d2h_copy ops inserted at delegate boundaries.
Kernels [Verified]: portable (209 ops in functions.yaml), optimized (23), quantized (19), ATen wrapper (225) plus LLM custom ops (custom_sdpa, update_cache, quantized_moe_ffn, gated_delta_rule, ...). Kernel table is a static array (default 250 ops × 8 kernels), selective build generates selected_max_kernel_num.h.
Quantization [Verified]: torchao source transforms (8da4w, int8, 4w, torchao:8da*, torchao:fpa*) and PT2E quantizers per backend (XNNPACK, QNN, CoreML, Vulkan, OpenVINO, TOSA/Ethos-U/VGF). CUDA backend has int4/5/6/8 packed GEMM shims; MLX has int2/4/8 weight quant.
LLM decode loop [Verified]: TextPrefiller::prefill (chunked) → TextTokenGenerator::generate → per-token TextDecoderRunner::step → Module::execute("forward"|"decode") → Sampler → tokenizer decode → token_callback. KV cache is a mutable buffer inside the .pte (static, [B,H,max_context,D] layout), updated by update_cache/custom_sdpa.
TTS in tree [Verified]: examples/models/voxtral_tts (LM + flow head + codec decoder, 5 methods, CPU/XNNPACK and CUDA, streaming codec chunks with AudioChunkCallback) and examples/models/supertonic (MLX, 4 methods incl. vocoder, dynamic FP16). Mimi codec on QNN under examples/qualcomm/oss_scripts/moshi. Audio front end: extension/audio/mel_spectrogram.py (exportable Whisper mel), extension/asr/runner (seq2seq + transducer), extension/llm/runner/audio.h.
VLA in tree [Verified]: none. No OpenVLA/pi0/LeRobot/diffusion-policy/proprioceptive references anywhere. Vision encoders exist for VLMs (LLaVA, Llama 3.2 Vision, Gemma3, SmolVLM, InternVL3) via MultimodalRunner with vision_encoder/token_embedding/text_decoder method naming.
Edge profile [Verified]: core runtime CI thresholds ~45–52 KB stripped (Linux gcc/clang size_test), ~118 KB bare-metal Cortex-M / ~136 KB Zephyr all-in; CMake presets for android-arm64-v8a, ios, macos, linux, windows, arm-baremetal, arm-ethosu-linux, esp-baremetal, riscv64-linux, zephyr, llm-release-{cuda,rocm,metal}, mlx-release. Android AAR (Maven), iOS xcframeworks (SwiftPM), Python wheels incl. aarch64 CUDA wheels.
Jetson [Unknown]: no explicit Jetson/Orin documentation; an aarch64-Linux CUDA wheel workflow exists (.github/workflows/build-wheels-cuda-aarch64-linux.yml), so aarch64+CUDA is at least built.
Tokenizers [Verified]: C++ tokenizers (HF JSON, Tiktoken, SentencePiece, Llama2c, Tekken) live in the extension/llm/tokenizers submodule, which is not checked out here; only the .claude/tokenizers.md summary and call sites could be inspected.

1. Architecture overview

graph TD
  subgraph AOT["AOT (Python, exir/)"]
    TE["torch.export → ExportedProgram"] --> TOEDGE["to_edge / to_edge_transform_and_lower<br/>EdgeProgramManager"]
    TOEDGE --> PART["Partitioner(s) → to_backend()<br/>BackendDetails.preprocess → delegate blob"]
    PART --> PASSES["edge_to_executorch_passes<br/>ConstraintBasedSymShapeEvalPass, reinplace, PropagateDevicePass"]
    PASSES --> MP["MemoryPlanningPass (greedy / banked)"]
    MP --> EMIT["_Emitter → ExecutionPlan(FlatBuffer)"]
    EMIT --> PTE[".pte (+ .ptd via flat_tensor)"]
  end
  subgraph RT["Runtime (C++, runtime/)"]
    PTE --> DL["DataLoader (file/mmap/buffer)"]
    DL --> PROG["Program::load → Program::load_method"]
    PROG --> METH["Method::init<br/>parse_values / resolve delegates / resolve_operator"]
    METH --> EXEC["Method::execute → execute_instruction<br/>KernelCall | DelegateCall | JumpFalse | Move | Free"]
    EXEC --> KREG["Kernel registry<br/>portable / optimized / quantized / custom"]
    EXEC --> BREG["Backend registry (≤16)<br/>BackendInterface::init/execute"]
    MM["MemoryManager<br/>method alloc + HierarchicalAllocator + temp"] --> METH
    DEV["DeviceAllocatorRegistry<br/>CPU / CUDA"] --> MM
    PAL["Platform PAL (log, clock, alloc, abort)"] --> EXEC
  end
  subgraph EXT["Extensions"]
    MOD["extension/module::Module"] --> PROG
    LLM["extension/llm/runner<br/>TextLLMRunner / MultimodalRunner"] --> MOD
    BATCH["extension/llm/batching<br/>DecodeFirstScheduler / Runner"] -.-> LLM
    ASR["extension/asr/runner"] --> MOD
    THR["extension/threadpool (pthreadpool+cpuinfo)"] --> KREG
    THR --> BREG
    BIND["android JNI / apple Swift / pybindings / wasm"] --> MOD
    SRV["examples/llm_server (OpenAI HTTP)"] -.-> LLM
  end
  subgraph BE["Delegates (backends/)"]
    BREG --> XNN["XNNPACK"] & VK["Vulkan"] & QNN["Qualcomm QNN"] & CML["CoreML"] & MLX["MLX"] & CUDA["CUDA (AOTI)"] & ARM["Arm Ethos-U / VGF"] & OTH["MediaTek / Samsung / NXP / OpenVINO / Cadence / WebGPU / Metal / native"]
  end

Key modules

Module	Path	Responsibility	Key symbols
Export IR	exir/program/_program.py	`to_edge`, `to_edge_transform_and_lower`, `EdgeProgramManager.to_backend/to_executorch`, `ExecutorchProgramManager.buffer/save`	`to_edge` (L1414), `EdgeProgramManager` (L1496), `ExecutorchProgramManager` (L1823)
Backend API	exir/backend/backend_api.py, exir/backend/partitioner.py, exir/backend/backend_details.py	Partition + preprocess into delegate blobs	`to_backend` (singledispatch), `Partitioner.partition`, `BackendDetails.preprocess/preprocess_multimethod`, `PreprocessResult`
Memory planning	exir/memory_planning.py, exir/banked_memory_planning.py	Lifetime analysis, arena assignment	`greedy`, `naive`, `MemoryPlanningAlgorithmSuite`, `_partition_specs_by_device`, `TargetMemoryMap`
Config	exir/capture/_config.py	Export/backend knobs	`EdgeCompileConfig`, `ExecutorchBackendConfig`
Emitter	exir/emit/_emitter.py	FX → FlatBuffer instructions	`_Emitter`, `_TopLevelEmitter`
Schema	schema/program.fbs	`.pte` file format	`Program`, `ExecutionPlan`, `Chain`, `KernelCall`, `DelegateCall`, `NonConstBufferDevice`
Program/Method	runtime/executor/program.h, runtime/executor/method.cpp	Load, init, execute	`Program::load/load_method`, `Method::init/execute/step/set_input/get_outputs`, `BackendDelegate`
Memory	runtime/executor/memory_manager.h, runtime/core/hierarchical_allocator.h, runtime/core/memory_allocator.h	Arena allocators	`MemoryManager`, `HierarchicalAllocator::get_offset_address`, `MemoryAllocator`
Device memory	runtime/core/device_allocator.h, runtime/core/device_memory_buffer.h, runtime/core/portable_type/device.h	Non-CPU planned buffers	`DeviceAllocator`, `DeviceAllocatorRegistry`, `DeviceMemoryBuffer`, `Device{CPU,CUDA}`
Backend registry	runtime/backend/interface.h	Delegate ABI	`BackendInterface::{is_available,init,execute,set_option,get_option,destroy}`, `register_backend`
Kernel registry	runtime/kernel/operator_registry.h	Op → kernel resolution	`Kernel`, `KernelKey`, `register_kernels`, `get_op_function_from_registry`
PAL	runtime/platform/platform.h	OS abstraction	`et_pal_init/abort/current_ticks/emit_log_message/allocate/free`, `PalImpl`
Kernels	kernels/portable/functions.yaml, kernels/optimized/optimized.yaml, kernels/quantized/quantized.yaml	CPU op libraries	YAML op defs + `kernels//cpu/op_.cpp`
Selective build	tools/cmake/Codegen.cmake, codegen/gen.py	Generate registration for selected ops	`gen_selected_ops`, `EXECUTORCH_SELECT_OPS_*`
Module API	extension/module/module.h	Convenience wrapper	`Module::{load_method,execute,forward,set_input,set_output,method_meta}`, `LoadMode`
LLM runner	extension/llm/runner/	Prefill/decode loop	`IRunner`, `TextLLMRunner`, `TextPrefiller`, `TextDecoderRunner`, `TextTokenGenerator`, `IOManager`, `MultimodalRunner`, `MultimodalPrefiller`, `LLMSession`, `LLMEngine`
LLM batching	extension/llm/batching/	Experimental multi-session scheduler	`Scheduler`, `DecodeFirstScheduler`, `Executor`, `Runner`, `Session`, `GenerationHandle`
LLM cache	extension/llm/cache/	Off-graph KV cache abstractions	`CacheBase`, `SequenceCache`, `CellCache`, `RingPolicy`, `kvcache::update_and_attend`
Custom ops	extension/llm/custom_ops/custom_ops.py	LLM kernels	`sdpa_with_kv_cache`, `custom_sdpa`, `update_cache`, `quantized_moe_ffn`, `gated_delta_rule`
Threadpool	extension/threadpool/threadpool.h, runtime/kernel/thread_parallel_interface.h	CPU parallelism	`ThreadPool`, `get_threadpool`, `get_pthreadpool`, `parallel_for`
Devtools	devtools/etdump/, devtools/inspector/_inspector.py, devtools/bundled_program/	Profiling / debugging	ETDump, `Inspector`, bundled program

2. Device layer

2.1 Backends actually in tree [Verified]

Every backend registers itself with register_backend(...) into a static 16-entry table (runtime/backend/interface.cpp, kMaxRegisteredBackends = 16, with a TODO noting global statics prevent multiple executors). Registration sites:

Backend id / dir	Registration	Hardware / OS	Status note
XNNPACK — backends/xnnpack	XNNPACKBackend.cpp#L272	ARM64/ARMv7/x86 CPU on Android, iOS, macOS, Linux, Windows (xnnpack-overview.md); KleidiAI via `EXECUTORCH_XNNPACK_ENABLE_KLEIDI` (CMakeLists.txt#L34)	default CPU delegate
Vulkan — backends/vulkan	VulkanBackend.cpp#L931	Android GPUs (Adreno/Mali/PowerVR noted in wiki), desktop experimental	mature on Android
WebGPU — backends/webgpu	WebGPUBackend.cpp#L263	Dawn/Tint WGSL; browser via Emscripten, native Metal/Vulkan	under active development
Qualcomm QNN — backends/qualcomm	QnnExecuTorchBackend.cpp#L394	HTP / GPU / LPAI; 38 chipset enum entries in qc_schema.py (`QcomChipset`)	no BC guarantees stated
MediaTek Neuron — backends/mediatek	NeuronBackend.cpp#L261	Dimensity 9300/9400 NPU
Samsung Exynos — backends/samsung	enn_backend.cpp#L96	Exynos 2500 NPU/DSP (`EXECUTORCH_BUILD_ENN`)
Arm Ethos-U — backends/arm	EthosUBackend.cpp#L451	Ethos-U55/65/85 via TOSA → Vela	bare-metal
Arm VGF — same dir	VGFBackend.cpp#L803	Vulkan ML extensions (SPIR-V ML)
Arm Cortex-M — backends/cortex_m	not a delegate: an operator dialect (`cortex_m::*`) on CMSIS-NN with portable fallback	Cortex-M MCUs	Beta
Cadence — backends/cadence	no delegation; custom ops for HiFi4/FusionG3/Vision DSP (see wiki "No-delegation architecture")	Xtensa DSP
NXP Neutron — backends/nxp	NeutronBackend.cpp#L676	eIQ Neutron NPU (i.MX)
OpenVINO — backends/openvino	OpenvinoBackend.cpp#L419	Intel CPU / iGPU / dGPU / NPU
Apple CoreML — backends/apple/coreml	coreml_backend_delegate.mm#L438	iOS/macOS ANE/GPU/CPU (compute units via compile spec)
Apple Metal — backends/apple/metal	metal_backend.cpp#L696	AOTI-driven Metal	"EXPERIMENTAL"
Apple Core AI — backends/apple/coreai	Python only at this HEAD		"Under construction — not for use"
MLX — backends/mlx	MLXBackend.cpp#L570	Apple Silicon GPU, macOS ≥ 14, iOS ≥ 17 experimental (mlx-overview.md)	experimental
CUDA — backends/cuda	cuda_backend.cpp#L1429	NVIDIA Linux/Windows via AOTInductor + Triton; ROCm experimental (rocm.md)
AOTI common — backends/aoti	library, not a backend	shared shims for CUDA/Metal
native — backends/native	`NativeBackend` serializes an fx subgraph to a generic flatbuffer executed by backends/native/runtime/Program.h	portable	new
example/test — backends/example, backends/test	reference delegate + test harness

Reference table also in .claude/backends.md and docs/source/backends-overview.md.

2.2 Device discovery / selection [Verified]

AOT: the user chooses the target by passing a Partitioner to to_backend()/to_edge_transform_and_lower(). Multiple partitioners are applied in priority order; unmatched ops stay as KernelCalls for the CPU kernel library (.claude/backends.md).
Runtime: BackendDelegate::Init (method.cpp#L61) does get_backend_class(backend_id) then backend->is_available() and fails with NotFound if either check fails. There is no fallback to another backend at runtime and no enumeration of physical devices. Backend-specific selection (e.g., CoreML compute units, QNN SoC/HTP arch, CUDA graph enable) is carried in CompileSpecs embedded in the .pte and parsed in init (e.g., coreml_backend_delegate.mm#L278, kEnableCudaGraphForMethod in cuda_backend.cpp#L85).
Runtime knobs can be pushed to a backend by name with set_option(backend_name, Span<BackendOption>) (interface.h); a BackendOptionsMap can be passed at load_method time.

2.3 Memory allocation / pools [Verified]

MemoryManager(method_allocator, planned_memory, temp_allocator) (memory_manager.h); method and temp allocator must differ. planned_memory is a HierarchicalAllocator over Span<Span<uint8_t>> — one span per mem_id, hard cap kSpanArraySize = 16 for the deprecated constructor; get_offset_address(memory_id, offset, size) with overflow checks (hierarchical_allocator.h).
Sizes come from ExecutionPlan.non_const_buffer_sizes (program.fbs#L368); the app must allocate those arenas. Method::resolve_operator uses temp allocator (falls back to method allocator) for TensorMeta scratch (method.cpp#L731).
Temp allocator is reset after each instruction and at the start of execute() (method.cpp#L1697) — kernels get per-op scratch.
Planning algorithms: greedy (default, allows overlapping lifetimes), naive, MemoryPlanningAlgorithmSuite picks the minimum total; banked_memory_planning.TargetMemoryMap adds capacity-aware placement across memory banks with spill (banked_memory_planning.py). Mutable buffers (KV caches) are placeholders that are "never freed" (memory_planning.py#L492) and are planned only if alloc_mutable_buffers.
Inputs/outputs: if not memory-planned, Method::set_input aliases user memory (share_tensor_data) instead of copying; otherwise copy_tensor_data after resize_tensor (method.cpp#L1146); set_output_data_ptr lets the caller supply output buffers.

2.4 Host ↔ device transfers [Verified]

Device{type: CPU|CUDA, index} (device.h); DeviceAllocator has allocate/deallocate/copy_host_to_device/copy_device_to_host; DeviceAllocatorRegistry holds one allocator per device type; DeviceMemoryBuffer::create(size, type) RAII (device_allocator.h, device_memory_buffer.h).
ExecutionPlan.non_const_buffer_device: [NonConstBufferDevice] marks planned arenas on CUDA (program.fbs#L417); MemoryManager::has_device_memory() exposes this.
At export, ExecutorchBackendConfig.enable_non_cpu_memory_planning=True (default) partitions specs by device (_partition_specs_by_device) and PropagateDevicePass inserts _h2d_copy/_d2h_copy ops at delegate boundaries (_config.py#L126, propagate_device_pass.py, _device_copy_ops_registry.py#L27).
CUDA backend auto-registers CudaAllocator (cuda_backend.cpp#L1431, cuda_allocator.h) and uses cudaMemcpyAsync device-to-device for CUDA-graph static I/O staging (cuda_backend.cpp#L574).
Other delegates copy internally (XNNPACK: xnn_setup_runtime_v2 on external tensors; Vulkan: maybe_resize_input/staging in VulkanBackend.cpp#L529; QNN: RegisterMem/PreRegisterMem shared buffers and rpcmem in QnnManager.h#L89, rpc_mem.h).

2.5 Synchronization / streams [Verified]

Core runtime is synchronous: every KernelCall/DelegateCall returns before the next instruction. No stream abstraction in runtime/.
CUDA backend: optional CUDA-graph capture per method (CudaGraphPhase::{Warmup,Replay,Disabled}, kCudaGraphWarmupSteps, cudaGraphLaunch on a stream — cuda_backend.cpp#L434); enable with compile-spec enable_cuda_graph_for_method. Unavailable on ROCm (rocm.md).
Vulkan: ComputeGraph::submit_current_cmd[_and_wait], submit_deferred_cmds_and_wait, prepack, optional_warmup_execute, propagate_resize (ComputeGraph.h#L1103).
MLX: async_eval with keep-alive of input data (MLXBackend.cpp#L94).

2.6 Portability [Verified]

PAL functions are weak symbols (ET_INTERNAL_PLATFORM_WEAKNESS) or a PalImpl table (platform.h); defaults in runtime/platform/default. Examples of custom PALs: Arduino (examples/arduino/arduino_pal.cpp), Zephyr (zephyr/), CUDA (backends/cuda/runtime/platform/platform.cpp).
Build presets (CMakePresets.json): android-arm64-v8a, android-x86_64, macos, ios, ios-simulator, linux, windows, pybind, zephyr, arm-baremetal, arm-ethosu-linux, esp-baremetal, riscv64-linux, llm, llm-release[-cuda|-rocm|-metal], llm-debug[-cuda|-metal|-vulkan], mlx[-release|-debug], profiling.
Host OS for building: Linux x86_64, macOS x86_64/ARM64, Windows 10+ (VS2022 + Clang-CL) or WSL (using-executorch-building-from-source.md).

3. Kernel layer

3.1 Operator set & registration [Verified]

Op definitions are YAML (- op:/- func: entries): portable 209 (functions.yaml) + 2 custom (custom_ops.yaml); optimized 23 (optimized.yaml — op_add/bmm/div/elu/exp/fft_*/gelu/grid_sampler_2d/le/linear/log_softmax/mm/mul/native_layer_norm/sub/sum/where); quantized 19 (quantized.yaml); ATen wrapper 225 (kernels/aten/functions.yaml); prim ops (et_copy_index, et_view, sym ops) in kernels/prim_ops/register_prim_ops.cpp.
Codegen (codegen/gen.py, codegen/templates) emits Kernel{name, KernelKey, OpFunction} arrays registered via register_kernels(Span<const Kernel>) (operator_registry.h#L179). KernelKey encodes dtype/dim-order per tensor arg (kKernelKeyBufSize = 659); a null key is the fallback kernel.
Table capacity: MAX_KERNEL_NUM → EXECUTORCH_SELECTED_MAX_KERNEL_NUM (generated by selective build) → default 250 ops × 8 kernels (operator_registry.cpp#L30). Duplicate registration aborts (see .claude/faq.md).
Portable kernels must use ExecuTorch types (executorch::aten::Tensor, no at::Tensor) — kernels/README.md.

3.2 Dispatch [Verified]

At Method::init, each KernelCall instruction is resolved once: resolve_operator builds TensorMeta for the args and calls get_op_function_from_registry(name, meta_list); the resulting OpFunction pointer is stored in chain.kernels_[instr] (method.cpp#L731, #L1040).
At execute_instruction, KernelCall invokes chain.kernels_[idx](context, args) with a KernelRuntimeContext(event_tracer, temp_allocator); DelegateCall invokes delegates_[idx].Execute(BackendExecutionContext, args); JumpFalseCall, MoveCall, FreeCall implement control flow, aliasing and explicit frees (method.cpp#L1450).
Routing to a backend is therefore decided entirely at export: an op either became part of a delegate blob (executorch_call_delegate → DelegateCall) or stays a CPU KernelCall. There is no runtime per-op dispatcher across backends.
When the Method is loaded with an explicit kernel_registry span, resolution uses that list instead of the global table (method.h Span<const Kernel> kernel_registry_).

3.3 Quantization formats & where dequant happens [Verified]

LLM source transforms (quantize.py): int8 (weight-only), 8da4w/8da8w (int8 dynamic act + int4/int8 weight, torchao Int8DynamicActivationIntxWeightConfig), 4w, torchao:8da* and torchao:fpa* low-bit kernels for Arm, embedding quant 4,32 etc. Config in llm_config.py (QuantizationConfig.qmode, pt2e_quantize, use_qat, use_spin_quant, preq_mode values 8da4w and 8da4w_output_8da8w). SpinQuant R1/R2 rotations: apply_spin_quant_r1_r2.py.
KV-cache quant: QuantizedKVCache (int8 per-token, QuantizedCacheType), StaticQuantizedKVCache with calibration, QuantizedRingKVCache, CustomKVCacheWithAttentionSink (custom_kv_cache.py); QuantizedSDPA consumes int8 caches (sdpa.py#L104). Dequant happens inside custom_quantized_sdpa/XNNPACK kernels at runtime.
PT2E quantizers (quantizer_lib.py): XNNPACK (xnnpack_quantizer.py), QNN (get_qnn_quantizer, 8a8w/16a4w/16a8w — see wiki recipes.md), CoreML, Vulkan, OpenVINO, TOSA/Ethos-U/VGF, Cortex-M CortexMQuantizer. Q/DQ nodes are fused into the delegate blob by the backend's preprocess; ops that stay on CPU run quantized_decomposed::* kernels from kernels/quantized (see quant_fusion_pass.py in exir/passes).
GPU: CUDA int4 tile-packed (tile_packed_to_4d, _weight_int4pack_mm) plus int5/int6/int8 planar GEMM shims (backends/cuda/runtime/shims, quantize_op_dispatch); MLX INT2/4/8 weight quant via torchao (mlx-overview.md); WebGPU 4-bit weight-only, 8da4w, q8ta (webgpu README). Also extension/llm/export/{int4,mx,nvfp4,gguf}.py for GGUF loading, MX and NVFP4 formats.

3.4 Compilation / JIT / graph capture [Verified]

Everything is AOT: torch.export → to_edge (ATen→Edge dialect, decompositions, dim-order) → partition/preprocess → to_executorch (passes incl. ConstraintBasedSymShapeEvalPass, reinplace, MemoryPlanningPass, ToOutVarPass) → emit. No JIT at runtime.
Delegate compilers: XNNPACK builds an xnn_subgraph at init (xnn_create_runtime_v4 with pthreadpool, weights cache, workspace — XNNCompiler.cpp#L2296); Vulkan builds ComputeGraph from a flatbuffer, prepacks and optionally warms up (VulkanBackend.cpp#L676); QNN loads precompiled context binaries (or online-prepare) via QnnManager::Compile (QnnManager.h#L85); CoreML compiles/caches .mlmodelc at first load; CUDA dlopens the AOTInductor .so from the .ptd and calls AOTInductorModelContainerCreateWithDevice/Run (cuda_backend.cpp#L164); MLX interprets its own flatbuffer bytecode (backends/mlx/serialization).

3.5 Fallback path [Verified]

Unsupported ops simply remain KernelCalls to the portable/optimized CPU kernels (backends-overview.md: "Operators not supported by the delegate are executed using the portable CPU fallback"). If the CPU kernel is also missing (selective build), Method::init fails with OperatorMissing (0x14) (.claude/faq.md). Cortex-M's dialect explicitly relies on portable-ops fallback (cortex_m/README.md).
extension/llm/custom_ops/op_fallback.cpp provides an AOT-side fallback op (op_fallback.py).

3.6 Extension points [Verified]

Custom op: define via torch.library (see custom_ops.py), implement an out-variant C++ kernel using ExecuTorch tensors, register with EXECUTORCH_LIBRARY / YAML + codegen (gen_selected_ops, Codegen.cmake); docs kernel-library-custom-aten-kernel.md, examples/portable/custom_ops.
Custom backend: Python BackendDetails.preprocess(edge_program, compile_specs) -> PreprocessResult + a Partitioner returning PartitionResult (backend_details.py#L53, partitioner.py#L37); C++ BackendInterface subclass + static auto _ = register_backend({"Id", &instance}); template in docs/source/backends/template and the backends/example reference delegate. Multi-method delegates via preprocess_multimethod and MethodProgramsPartitionerSpec (backend_api.py#L664).
Custom memory planning: pass your own MemoryPlanningPass/algorithm list or custom mem_id pools (compiler-memory-planning.md).
Custom passes: EdgeProgramManager.transform(passes) and ExecutorchBackendConfig.passes (compiler-custom-compiler-passes.md).

4. Model runner

4.1 Loading / export / conversion pipeline [Verified]

Formats: .pte (FlatBuffer program with inline or segmented constants and delegate blobs; extended header in schema/extended_header.h; pte-file-format.md); .ptd external tensor files (extension/flat_tensor, FlatTensorDataMap; ptd-file-format.md) used for external constants (ExecutorchBackendConfig.external_constants), CUDA AOTI blobs (aoti_cuda_blob.ptd), Whisper weights, LoRA adapters.
Export API: to_edge_transform_and_lower(ep, partitioner=[...], transform_passes=..., compile_config=EdgeCompileConfig) then .to_executorch(ExecutorchBackendConfig) and .buffer/.save() (_program.py#L1280). Also executorch.export recipe API (export/).
LLM export: python -m executorch.extension.llm.export.export_llm (Hydra/YAML LlmConfig: base, model, export, quantization, backend, debug) → export_llm.py → export_llama_lib.py → LLMEdgeManager in builder.py. Model classes in ModelType (stories110m, llama2/3/3_1/3_2, llama3_2_vision, static_llama, qwen2_5*, qwen3*, qwen3_5*, phi_4_mini, smollm2, lfm2*). HF models via optimum-executorch (out of tree; export-llm-optimum.md). QNN LLMs use a separate path examples/qualcomm/oss_scripts/llama/llama.py (wiki warns against export_llama --qnn, model-specific.md).
Loading: DataLoader variants (FileDataLoader, MmapDataLoader with mlock/madvise, BufferDataLoader, FileDescriptorDataLoader — extension/data_loader); Module::LoadMode {File, Mmap, MmapUseMlock, MmapUseMlockIgnoreErrors, MmapUseMadvise} (module.h#L50); LLM runner default MmapUseMlockIgnoreErrors (llm_runner_helper.h#L106).

4.2 Execution lifecycle [Verified]

Program::load(loader, Verification) → header check, optional flatbuffer verification (program.h#L97).
Program::load_method(name, MemoryManager*, EventTracer*, NamedDataMap*) → Method::init: parse_values (materialize values_ table, tensors pointing into planned arenas / constant segments / external .ptd data), allocate and BackendDelegate::Init each delegate (calls backend->init → DelegateHandle*), resolve_operator for every KernelCall (method.cpp#L883).
Method::set_input(s) (resize + copy or alias) → Method::execute() (checks all inputs set, resets temp allocator, sequentially runs all chains/instructions, guards against >ET_MAX_INSTRUCTIONS=10 M, then reset_execution) → Method::get_outputs (shallow copies) (method.cpp#L1697).
Alternative: Method::step() executes one instruction (experimental; used by devtools/debuggers), reset_execution() after EndOfMethod.
Warmup: no core warmup phase; TextLLMRunner::warmup(prompt, max_new_tokens) runs a throwaway generation (text_llm_runner.h#L146); Vulkan optional_warmup_execute; CUDA-graph warmup steps.
Multiple Methods per program (e.g., encoder, text_decoder, prefill, decode, vision_encoder, token_embedding, TTS duration_predictor/text_encoder/vector_estimator/vocoder) are loaded independently, each with its own planned memory; a global MergedDataMap lets them share .ptd constants.

4.3 Dynamic shapes [Verified]

Export with torch.export.Dim dynamic shapes; ConstraintBasedSymShapeEvalPass (default in ExecutorchBackendConfig.sym_shape_eval_pass) converts symbolic sizes to upper bounds for memory planning (DynamicMemoryPlanningMode.UPPER_BOUND default; SYMBOLIC exists) (sym_shape_eval_pass.py#L257, dynamic_shape.py, _config.py#L56).
Runtime tensors carry TensorShapeDynamism {STATIC, DYNAMIC_BOUND, DYNAMIC_UNBOUND} (tensor_shape_dynamism.h); set_input calls resize_tensor then copies; DYNAMIC_UNBOUND is not memory-planned and must be provided by the caller.
Delegates handle dynamic shapes themselves: XNNPACK xnn_reshape_external_value + xnn_reshape_runtime + resize_outputs per call (XNNExecutor.cpp#L150); Vulkan maybe_resize_input/propagate_resize; MLX advertises dynamic shape support; QNN graphs are static (wiki known-issues). LLM examples default to static (enable_dynamic_shape=False in model_args.py#L102); the LLM runner reads enable_dynamic_shape metadata (constants.h) and otherwise pads prefill to fixed length.

4.4 State & KV cache [Verified]

In-graph mutable buffers: KVCache registers k_cache/v_cache buffers of shape [max_batch, n_heads, max_context_len, head_dim] and updates with index_copy_/slice writes (attention.py#L99); CustomKVCache uses llama::update_cache + llama::custom_sdpa (custom_kv_cache.py#L579, op_update_cache.cpp, op_sdpa.cpp). Ring buffer (CustomRingKVCache, QuantizedRingKVCache) and attention-sink variants exist; ReplaceKVCache... transforms are selected by ModelConfig flags (use_kv_cache, use_sdpa_with_kv_cache, quantize_kv_cache, use_attention_sink, use_ring_kv_cache).
The cache lives in the method's planned non-const memory (mutable buffer, never freed) and persists across execute() calls; the runner tracks pos_ and passes start_pos as an input tensor. reset() zeroes position; there is no paging, eviction or cross-request sharing in the core runner — one Method = one sequence. max_context_len vs max_seq_len (prefill chunk) are metadata methods (get_max_context_len, get_max_seq_len).
Static attention (NPU-friendly): StaticKVCache/StaticAttention/StaticAttentionIOManager in Python (static_attention.py) and C++ (static_attention_io_manager.h) keep caches outside the graph as method inputs/outputs; update styles SHIFT_POINTER and SMART_MASK (fixed I/O pointers enabling persistent AP↔NPU mapping; supports per-layer cache lengths for sliding-window layers, batch 1, lookahead n-gram cache). Used by the QNN runner (kv_manager.h, rpc_mem.h).
Off-graph cache library (new, experimental): extension/llm/cache defines CacheBase, SequenceControl::rewind, SequencePlanner::plan/commit, LayoutPolicy (FlatPolicy, RingPolicy), SequenceCache, and a paged-like CellCache (cells claimed per position with owner bitsets, kMaxSeqs = 64, free_cells()), reached from the graph through the functional custom op kvcache::update_and_attend keyed by layer_id (cache.h, cell_cache.h, sequence_cache.h, update_and_attend.py). MLX supports per-session mutable state (mlx_mutable_state.h, skip_mutable_buffer_init — MLXBackend.cpp#L298); CUDA has cuda_mutable_state.h (backends/cuda/runtime/cuda_mutable_state.h). [Inferred] these are the substrate for the multi-session LLMEngine implementations (Muse Glimmer, Qwen3.5-MoE, Gemma4-31B engines).

4.5 Multimodal stages [Verified]

MultimodalRunner (IRunner) takes std::vector<MultimodalInput> (text / tokens / Image / Audio (mel) / RawAudio); MultimodalPrefiller runs vision_encoder or audio_encoder, token_embedding, then text_decoder for prefill; decode continues with MultimodalDecoderRunner/TextTokenGenerator (multimodal_runner.h, multimodal_prefiller.cpp#L112, multimodal_input.h).
Encoders/preprocessors as separate methods or .ptes: LLaVA (export_image_encoder, CLIP preprocessing in Python; export_llava.py#L138); Llama 3.2 Vision (exportable preprocess + C++ preprocess in llama3_2_vision/preprocess); Whisper mel preprocessor.pte (extension/audio/mel_spectrogram.py); Voxtral Realtime preprocessor.pte (voxtral_realtime README).
Decoders/vocoders: Voxtral TTS codec_decoder.pte (Conv1d/ConvTranspose1d + 8 transformer layers → 24 kHz) driven by voxtral_tts_runner with a 7-step Euler flow head; Supertonic vocoder method; Mimi decoder on QNN (qnn_mimi_decoder_runner.cpp).
Diffusion: examples/models/stable_diffusion and stable_diffusion_3_5_large are model definitions/export scripts only (no runner); DiT on QNN (dit.py).

5. Scheduler

5(a) Request / token / pipeline-level scheduling

Core runtime: N/A [Verified]. One Method executes one request synchronously; execute() refuses re-entry while in_progress(); there is no queue, no batching across requests, no admission control. Batch size is baked into the exported shapes (LLM examples: batch 1).
LLM runner (extension/llm/runner) [Verified]: sequential prefill → decode per call; GenerationConfig (max_new_tokens, seq_len, temperature, echo, warming, grammar) (irunner.h); chunked prefill in TextPrefiller::prefill when prompt > max_seq_len (text_llm_runner.cpp#L143); optional two-method PTE (prefill + decode) (constants.h#L28). Cancellation: IRunner::stop() sets an atomic checked per token in TextTokenGenerator::generate (text_token_generator.h#L131); LLMSession::stop() is documented as a token-boundary cooperative stop (llm_session.h#L104). No backpressure concept (callbacks are synchronous on the calling thread).
extension/llm/batching (experimental) [Verified]: Scheduler interface (submit/has_work/get_work/cancel(sid)/clear/max_prefill_chunk_size), DecodeFirstScheduler::create(max_batch_tokens=544, max_decode_sequences=32, max_prefill_chunk_size=256) takes all pending decodes first then fills the token budget with round-robin prefill chunks (continuous batching with chunked prefill; admission rejects over-budget tasks) (decode_first_scheduler.h); Executor (open_session/close_session/set_sampling/execute(BatchInput, BatchOutput)); Runner(executor, scheduler) with open_session_async, Session::generate_async(delta, GenConfig, callback) returning GenerationHandle{cancel, done, wait, finish_reason} and FinishReason {StopToken, NewTokenLimit, Cancelled, Failed} (runner.h). Gap [Verified]: no in-tree Executor implementation outside extension/llm/batching/test; no batched .pte example wires into it yet.
examples/llm_server [Verified]: OpenAI-compatible /v1/chat/completions (streaming + non-streaming), one worker process with serialized execution, multiple isolated sessions on one weight load when the engine reports LLMServingCapacity > 1, bounded request cancellation, structured 400s for unsupported params (n>1, penalties, logprobs...) (examples/llm_server/README.md, worker_loop.h, worker_prefill_plan.h). LLMEngine implementations: muse_glimmer_engine.h (with DFlash speculative decoding), qwen35_moe_engine.h, gemma4_31b_engine.h.
Multi-stage pipelines (TTS/ASR): hand-written in each example runner (e.g., VoxtralTtsRunner::synthesize_streaming LM → flow head → codec chunks with streaming_chunk_frames_=25, streaming_initial_chunk_=5, streaming_left_context_=25 — voxtral_tts_runner.h#L121); QNN has a Python genai_pipeline with model-preparation / quantization / compilation / inference stages (genai_pipeline.py).

5(b) Graph / operator / thread-level scheduling

Op ordering [Verified]: fixed at export (topological order emitted into Chain.instructions); executed strictly sequentially by Method::execute — "Chains are executed sequentially today, but future async designs may branch and run many in parallel" (method.cpp#L1722). No inter-op parallelism, no stream assignment in the core.
Intra-op threading [Verified]: extension/threadpool wraps pthreadpool and sizes the pool from cpuinfo performant cores (get_num_performant_cores, cpuinfo_utils.cpp); ThreadPool::run(fn, range) is blocking pthreadpool_parallelize_1d; _unsafe_reset_threadpool(n) changes size (threadpool.cpp#L65); kernels use parallel_for(begin,end,grain,fn) which degrades to a serial loop without ET_USE_THREADPOOL (thread_parallel_interface.h#L53). XNNPACK shares the same get_pthreadpool() (XNNCompiler.cpp#L2300). NoThreadPoolGuard prevents nested parallelism.
Delegate-internal scheduling [Verified]: Vulkan builds command buffers per graph and submits/waits (submit_current_cmd_and_wait); CUDA optionally replays a captured CUDA graph; QNN executes a named graph on HTP with optional multi-graph context; XNNPACK runs the subgraph on the pthreadpool.
Batching [Verified]: static batch dim only; HierarchicalAllocator sizes are per-method so batch changes require re-export.

6. I/O layer

Text tokenization [Verified via docs, submodule absent]: extension/llm/tokenizers (pytorch-labs tokenizers, submodule commit 1d7ca636) provides C++ HFTokenizer (tokenizer.json), Tiktoken, SentencePiece, Llama2c, Tekken plus Python pytorch_tokenizers.get_tokenizer (.claude/tokenizers.md); llm::load_tokenizer(path, special_tokens) auto-detects (llm_runner_helper.h#L47); llama runner tries Tiktoken → SentencePiece → Llama2c (runner.cpp). WASM tokenizers bindings in extension/wasm/tokenizers.
Image preprocessing [Verified]: C++ ImageProcessor::process/process_yuv/process_into (resize + normalize, NV12/NV21, SIMD and Apple GPU variants) in extension/image/image_processor.h; Swift ExecuTorch+ImageProcessor.swift; JNI jni_layer_image.cpp; examples use stb_image (gemma3/e2e_runner.cpp) and llm::Image (image.h); custom op tile_crop (op_tile_crop.cpp).
Audio [Verified]: WhisperAudioProcessor (torch.stft + mel filterbank, exportable to whisper_preprocess.pte, 16 kHz/80 bins/30 s defaults) in extension/audio/mel_spectrogram.py; RawAudio{data, batch, n_channels, n_samples} and Audio (mel, float/bf16) types (audio.h); wav_loader.h; ASR runners Seq2SeqRunner (Whisper-style encoder + text_decoder) and TransducerRunner (Parakeet TDT) in extension/asr/runner; Silero VAD streaming runner (stream_main.cpp).
Streaming [Verified]: token-level token_callback(const std::string&) per generated token and stats_callback(const Stats&) (first_token_ms, prompt_eval_end_ms, num_generated_tokens, gpu_total_bytes — stats.h); audio-chunk AudioChunkCallback in Voxtral TTS; Voxtral Realtime processes 80 ms chunks live; Supertonic persistent mode emits JSONL results per request. HTTP SSE streaming in examples/llm_server.
Buffering [Verified]: planned arenas allocated by the app; Module allocates them internally; set_output_data_ptr for zero-copy outputs; XNNPACK weights cache for .ptd weights (XNNCompiler.cpp#L2283).
Postprocessing [Verified]: Sampler (temperature, top-k sample_topk, argmax) + LogitProcessor chain (grammar hooks) (sampler.h, logit_processor.h); tokenizer decode(prev, cur); WAV writers in TTS examples (wav_writer.h).
Application boundaries [Verified]:
- C++: runtime/ core API, extension/module::Module, extension/tensor (TensorPtr), extension/runner_util, LLM/ASR runners, executor_runner CLI (examples/portable/executor_runner/executor_runner.cpp).
- Python: executorch.runtime.Runtime/Program/Method and portable_lib._load_for_executorch (.claude/runtime-api.md, extension/pybindings/pybindings.cpp); LLM runner pybindings (extension/llm/runner/pybindings.cpp).
- Android: Kotlin Module, Tensor, EValue, ExecuTorchRuntime, BackendOptionsMap + JNI layers for runtime/llama/asr/image/training (extension/android, jni_layer_llama.cpp); shipped as AAR on Maven Central (using-executorch-android.md).
- Apple: Objective-C/Swift ExecuTorchModule/Tensor/Value/ImageProcessor/BackendOptions (extension/apple/ExecuTorch/Exported) and ExecuTorchLLM (extension/llm/apple); xcframeworks via SwiftPM (using-executorch-ios.md).
- WASM: extension/wasm/wasm_bindings.cpp.
- HTTP: examples/llm_server (Python control plane; C++ single-binary server listed as future).
- Demo apps in tree: only examples/demo-apps/react-native/rnllama (native Android/iOS demo apps have moved out of this tree; docs still reference them).

7. Execution flow

7(a) Text LLM decode (TextLLMRunner over XNNPACK-delegated `.pte`) [Verified]

sequenceDiagram
  participant App
  participant R as TextLLMRunner
  participant Tok as Tokenizer
  participant P as TextPrefiller
  participant G as TextTokenGenerator
  participant D as TextDecoderRunner
  participant M as Module/Method
  participant K as Kernels / XNNPACK delegate
  participant S as Sampler
  App->>R: create_text_llm_runner(pte, tokenizer, data_files)
  App->>R: load()
  R->>M: load_method("forward") → Method::init (parse_values, BackendDelegate::Init → XNNPACKBackend::init builds xnn runtime, resolve_operator)
  App->>R: generate(prompt, GenerationConfig, token_cb, stats_cb)
  R->>Tok: encode(prompt, bos, eos)
  R->>P: prefill(tokens, pos_) (chunked ≤ max_seq_len)
  P->>D: step(tokens_tensor, start_pos)
  D->>M: IOManager::prepare_prefill → execute("forward", {tokens, start_pos})
  M->>K: Method::execute: KernelCall/DelegateCall sequence (update_cache, custom_sdpa, xnn subgraph)
  K-->>M: logits
  M-->>D: logits tensor
  D->>S: logits_to_token(temperature)
  S-->>P: first token
  P-->>R: first token, pos_ += n
  R->>G: generate(first token, start_pos, max_new_tokens, temperature, token_cb)
  loop until EOS / max_new_tokens / stop()
    G->>D: step({cur_token}, pos)
    D->>M: execute("forward"|"decode")
    M->>K: instructions (KV cache mutable buffer updated in planned memory)
    K-->>D: logits
    D->>S: sample → cur_token
    G->>Tok: decode(prev, cur)
    G-->>App: token_cb(text)
  end
  G-->>R: n_generated
  R-->>App: stats_cb(Stats: TTFT, tok/s ...)

Source: text_llm_runner.cpp, text_prefiller.h, text_token_generator.h#L83, text_decoder_runner.cpp#L40, io_manager.h, method.cpp#L1697.

7(b) TTS: Voxtral TTS streaming synthesis (CPU or CUDA) [Verified structure; per-method details Inferred]

sequenceDiagram
  participant App
  participant TR as VoxtralTtsRunner
  participant Tok as Tekken tokenizer
  participant LM as model.pte (5 methods: LM + flow head)
  participant CD as codec_decoder.pte
  participant BE as XNNPACK / CUDA(AOTI) delegate
  App->>TR: ctor(model_path, codec_path, tokenizer, voice embedding, [aoti .ptd])
  TR->>LM: Module::load_method(...) for LM + flow-head methods
  TR->>CD: Module::load_method("forward")
  App->>TR: synthesize_streaming(text, AudioChunkCallback)
  TR->>Tok: encode(text)
  TR->>LM: prefill(text tokens + voice embedding)
  loop per audio frame (autoregressive)
    TR->>LM: decode step → hidden state
    TR->>LM: flow head: 7-step Euler ODE (x0 from xorshift64 rng) → 37 codebook tokens
    LM->>BE: DelegateCall (AOTI container run / xnn runtime)
  end
  Note over TR: after streaming_initial_chunk_ (5 frames) then every streaming_chunk_frames_ (25) with left context 25
  TR->>CD: execute(codebook tokens window) → PCM 24 kHz
  CD->>BE: DelegateCall
  TR-->>App: AudioChunkCallback(float PCM chunk)
  TR-->>App: final WAV via wav_writer

Source: voxtral_tts_runner.h, voxtral_tts_runner.cpp, README. Exact method names inside model.pte were not enumerated here [Unknown].

Multimodal VLM path (for VLA-style vision+text) is the MultimodalRunner sequence: vision_encoder(image) → token_embedding(tokens) → text_decoder(embeds, start_pos) prefill → token loop (multimodal_prefiller.cpp).

8. Tests and examples

Python tests [Verified]: pytest -n auto from repo root (CLAUDE.md); pytest.ini ignores backends/arm/** (needs examples/arm/setup.sh), some XNNPACK op tests, backends/test (WIP harness), LLaVA tests, backend-specific Muse Glimmer pipelines. Tests live next to code: exir/tests, exir/backend/test, backends/*/test, extension/llm/export/test, examples/models/llama/tests (incl. test_static_attention.py), runtime/test/test_runtime.py.
C++ tests [Verified]: gtest via CMake EXECUTORCH_BUILD_TESTS=ON then ctest --output-on-failure; Test.cmake enumerates runtime/*/test, kernels/test, extension/*/test; test/run_oss_cpp_tests.sh; size tests test/build_size_test.sh and test/size_test.cpp; end-to-end test/end2end.
Backend op-coverage harness: backends/test (suite, harness, FACTO), multi_method_delegate_test.cpp.
CI [Verified]: ~60 workflows in .github/workflows — pull.yml, trunk/periodic.yml, nightly.yml, _unittest.yml, _android.yml, apple.yml, cuda.yml, cuda-windows.yml, cuda-perf.yml, rocm.yml, metal.yml, mlx.yml, riscv64.yml, qnn-windows-msvc.yml, test-backend-{arm,coreml,cortex-m,nxp,openvino}.yml, _test_cadence.yml, _xtensa_*.yml, _llm_server.yml, _test_arduino_library.yml, wheel builds for linux/macos/windows/aarch64/cuda.
Example apps / runners [Verified]: LLM: examples/models/llama (main.cpp, runner/, export_llama.py, eval, Android perf table), examples/models/{qwen3,phi-3-mini,gemma3,gemma4,smollm2,lfm2,deepseek-r1-distill-llama-8B,...}; VLM: llava, llama3_2_vision, gemma3, smolvlm, internvl3, muse_glimmer; audio: whisper, voxtral, voxtral_realtime, parakeet, silero_vad, sortformer, granite_speech, emformer_rnnt, wav2letter; TTS: voxtral_tts, supertonic; vision: mobilenet_v2/v3, resnet, deit, dinov2, efficient_sam, yolo12/26, torchvision_vit, edsr; diffusion: stable_diffusion, stable_diffusion_3_5_large. Platform examples: examples/qualcomm (40+ oss_scripts incl. llama, whisper, t5, gemma4, moshi/mimi, DiT, EfficientSAM), examples/apple/coreml, examples/arm (Ethos-U/VGF/Cortex-M notebooks: smollm2, silero_vad, tinystories, mobilesam), examples/mediatek, examples/samsung, examples/nxp, examples/openvino, examples/vulkan, examples/xnnpack, examples/cuda (ROCm pointwise), examples/arduino, examples/espressif, examples/raspberry_pi/pico2, examples/riscv, examples/zephyr, examples/wasm, examples/llm_pte_finetuning, examples/demo-apps/react-native.
Make targets [Verified]: Makefile + per-example CMakePresets.json (make whisper-cpu|whisper-cuda|whisper-metal, make silero-vad-cpu, make sortformer-cpu, make supertonic-mlx).

9. TTS relevance

Exists [Verified]:

Voxtral-4B-TTS end-to-end: export (export_voxtral_tts.py) producing model.pte (Mistral 4B decoder + 3-layer flow-matching head, 5 methods) and codec_decoder.pte; backends portable, xnnpack, cuda, cuda-windows; quant --qlinear 4w|8w|8da4w|8da8w, --qlinear-codec, --qembedding; --streaming chunk metadata; C++ runner with synthesize_streaming + AudioChunkCallback, xorshift RNG for flow noise, WAV writer. Repo-reported (not reproduced): CUDA 4w RTF 0.31× on RTX 5080 with ~2.6 s time-to-first-audio, 3.4 GB .ptd; FP32 CUDA RTF 51× (codec on portable CPU) (README).
Supertonic 3 on MLX: single dynamic FP16 .pte with methods duration_predictor, text_encoder, vector_estimator (5 flow steps), vocoder; one-shot and persistent JSONL server mode; repo-reported RTF ≈ 0.028 on Apple Silicon (README, supertonic_runner.cpp, model/vocoder.py). macOS/Apple-Silicon only.
Neural audio codecs: Mimi (Moshi) tests (examples/models/moshi/mimi/test_mimi.py) and a QNN Mimi decoder runner (qnn_mimi_decoder_runner.cpp); Voxtral codec decoder.
Audio ops: portable/optimized FFT (op_fft_r2c/c2r in kernels/optimized/cpu, pocketfft third-party), Conv1d→Conv2d transform (backends/transforms/convert_conv1d_to_conv2d_pass.py) for XNNPACK; ConvTranspose1d handled in Voxtral codec on CPU/CUDA; mel front end exportable.
Streaming audio output: chunked codec decoding with left context (Voxtral TTS); real-time 80 ms chunk ASR (Voxtral Realtime) shows the runtime can sustain streaming loops; VAD streaming runner.
Text front end: Supertonic text_processor.cpp; no general G2P/phonemizer library.
NXP README explicitly lists TTS as a Neutron NPU use-case (backends/nxp/README.md) [Verified statement, no TTS example for NXP found].

Missing / gaps [Verified unless noted]:

No generic TTS runner abstraction (unlike extension/llm/runner or extension/asr/runner); each TTS model has a bespoke C++ runner.
No mobile (Android/iOS/QNN/CoreML) TTS example; Supertonic is MLX-only, Voxtral TTS is CPU/CUDA. [Inferred] the XNNPACK path is fp32 (--dtype fp32 default) and 4B params — heavy for phones.
No vocoder zoo (HiFi-GAN, BigVGAN, WaveRNN, etc.) or codec encoders (EnCodec/SNAC/DAC) in tree; stable_diffusion has no audio variant.
Flow-matching ODE loops are unrolled by the runner (7 Euler steps as repeated method calls), not expressed as a graph; no CFG batching helper.
No audio-specific streaming buffer/jitter management; callbacks are synchronous.
Sampling for audio tokens (multi-codebook, 37 codebooks per frame) is handled inside the example, not by Sampler.

10. VLA relevance