0 Fork 0

Files

T

tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork

The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ

2026-09-14 14:01:12 +08:00

103 KiB

Raw Blame History

MNN (alibaba/MNN) — Engine Analysis

Field	Value
Local path	`/data/zhoutaichang/embedding_infer/MNN`
Remote	`https://github.com/alibaba/MNN.git` (origin)
Branch / HEAD	`master` @ `bef71b9756a2c77549eddbe33eb97290e3b16602` (2026-09-04, "[Vulkan:Perf] Optimize INT4 cooperative matrix path")
Dirty status	clean (`git status --porcelain` empty)
Submodules	none (`.gitmodules` absent; third-party code is vendored under `3rd_party/` or fetched by CMake at build time — see §11)
Engine version	3.6.1 (MNNDefine.h)
Analysis date	2026-09-08
Reading restrictions	Per AGENTS.md, `schema/private/` and `source/internal/` were not read or cited.

Evidence labels: [Verified] = code/doc read; [Inferred] = reasoned from structure; [Proposal] = design suggestion; [Unknown] = could not determine. Links are relative to analysis/engines/.

0. Summary

What it is [Verified]: MNN is a C++11 inference engine (no training focus, -fno-rtti -fno-exceptions) with two APIs — the low-level Session API (Interpreter → createSession → runSession) and the recommended Express/Module API (Module::load → onForward(VARP)) — plus a separate transformer stack (transformers/llm/engine, transformers/diffusion/engine) built on top of Module. See AGENTS.md, Interpreter.hpp, Module.hpp.
Execution runtime vs serving stack [Verified]: the core is a single-graph, single-request runtime. There is no request scheduler, no continuous batching, no paging. Serving is thin: mls_server (POST /chat/completions, SSE) and apps/mnncli (/v1/chat/completions, /v1/messages) wrap one Llm behind a mutex (mls_server.cpp, mnncli_server.cpp).
Model format [Verified]: FlatBuffers .mnn (MNN.fbs, 186 OpType enumerators) with large weights split into a sidecar .mnn.weight addressed by Op.externalPath + per-op external:[offset,size,...]. Converter frontends: ONNX, TF, TFLite, Caffe, TorchScript, MNN-JSON (cli.cpp).
Backends in tree [Verified]: 16 dirs under source/backend: cpu (+arm82 fp16, x86 AVX2/AVX512-VNNI, RISC-V RVV, KleidiAI, SME2, bf16), metal, opencl, vulkan (image or buffer variant chosen at build), opengl, cuda, musa, tensorrt, coreml, nnapi, qnn, hiai, hexagon, rknn (plugin op), neuropilot (stub only in OSS). No LoongArch, no WebGPU.
Heterogeneous fallback [Verified]: every non-CPU Session carries a CPU "backup" backend; per-op Backend::onCreate returning nullptr triggers a retry on CPU inside _createExecutions (Pipeline.cpp#L583-L596), with WrapExecution inserting layout/device copies. Only works when the runtime's CompilerType is Compiler_Loop (full geometry decomposition); NPU-style backends (CoreML, NNAPI, QNN, HIAI, TensorRT) compile the whole subgraph and are exempt from wrapping.
Kernel dispatch [Verified]: per-backend addCreator(OpType, Creator*) maps; CPU registers 75 op types, Metal 36, OpenCL 39 (×buffer/image), Vulkan buffer 35, CUDA ~44. Geometry decomposition lowers shape-manipulation ops into OpType_Raster regions so backends only need a small compute subset (GeometryComputer.cpp, docs/contribute/backend.md).
LLM quantization [Verified]: exporter default W4 block-64 asymmetric with fp16 scales; W2/W3/W4/W8/FP16 supported (W2/W3 need ARMv8.6 i8mm); optional HQQ/AWQ/GPTQ/OmniQuant/SmoothQuant (llmexport.py). Runtime dequant is fused into the GEMM under MNN_LOW_MEMORY (weight-dequant float GEMM with memory=normal, or dynamic-int8-activation GEMM with memory=low) (ConvolutionFloatFactory.cpp). KV cache: int8 / TQ3 / TQ4 on CPU, int8 on Metal, optional mmap-to-disk.
LLM engine [Verified]: Llm (text), Omni : Embedding : Llm (vision/audio/talker), Talker : Llm (speech). Decode is a plain autoregressive loop in ArGeneration::generate (generate.cpp#L44) with optional lookahead / MTP / EAGLE / DFlash speculative strategies. Modules are cloned per (seq-len-bucket, all_logits) key to avoid resizing (mPrefillKey=100, decode key 1).
TTS is first-class [Verified]: (1) in-engine Qwen2.5-Omni thinker→talker→DiT(flow ODE)→BigVGAN and Qwen3-TTS talker→code-predictor→speech-decoder (single waveform callback after all frames, Talker::generateQwen3TTS in omni.cpp) while Qwen2.5-Omni streams chunked PCM through setWavformCallback(const float*, size_t, bool last) and a two-thread async DiT/vocoder pipeline (omni.cpp); (2) a native MNNTTSSDK (Bert-VITS2, Supertonic, Piper) under apps/frameworks/mnn_tts; (3) a full sherpa-onnx port (VITS/Kokoro/Matcha/HiFiGAN + Zipformer ASR) under apps/frameworks/sherpa-mnn.
VLA is absent [Verified]: exhaustive grep for VLA/OpenVLA/π0/SmolVLA/LeRobot/diffusion-policy/proprio/action-head yields zero real hits. Building blocks exist: 16 vision-encoder exporters (Qwen2/2.5/3/3.5-VL, Gemma3/4, InternVL, MiniCPM-V, SmolVLM/Idefics3, LFM2-VL, MobileCLIP, ...), M-RoPE, DiT/flow-matching runtime (SD3.5/Sana/Wan2.1 T2V, token2wav DiT).
Edge profile [Verified]: minimal libMNN.so links only pthread/dl (Linux) or log/m/android (Android); OpenCL/Vulkan are dlopened by default; repo-reported core size ≈ 800 KB (Android armv7a, c++_shared), 12 MB full iOS static lib; MNN_BUILD_MINI strips geometry/shape for fixed-shape models. Platforms with build scripts: Android (NDK, API ≥21 arm64 / 14 armv7), iOS/macOS (CocoaPods MNN.podspec, fat framework), Linux/Windows/OpenHarmony, WASM (emcmake), RISC-V (SpacemiT K3), CUDA sm60–120 (incl. sm87 Jetson Orin), Qualcomm QNN/Hexagon, Huawei HIAI, Rockchip RKNN, Apple ANE via CoreML.
No JIT / graph capture [Verified]: kernels are AOT (Metal source strings compiled at first use with a persisted tuning cache; Vulkan SPIR-V arrays; OpenCL online-compiled with binary+tuning cache; CUDA nvcc). No cudaGraph, no MTLIndirectCommandBuffer. GPU batching is controlled by OP_ENCODER_NUMBER_FOR_COMMIT.
Bindings [Verified]: C++ only in core (no C API); Python MNN.llm (response/generate/…, no TTS callback), Android JNI + Kotlin apps, iOS ObjC++/Swift wrapper with audio-out callback.
Concurrency model [Verified]: one Llm = one Executor + one RuntimeManager; thread-safe only if one thread drives it (test_multi_instance.cpp). Cancellation of text decode has no engine API — apps step generate(1) themselves; USER_CANCEL is set only when the wav callback returns false.
Top facts for edge TTS/VLA: TTS with RTF reporting exists today (chunk-streamed for Qwen2.5-Omni, single-callback for Qwen3-TTS, on CPU/GPU, DiT forced fp32 on GPU); the DiT ODE loop is host-driven Express code, so a diffusion action head would be implementable the same way; the missing pieces for VLA are proprio/action I/O and any low-latency control loop.

1. Architecture overview

graph TD
  subgraph Export["Export / conversion (host)"]
    HF["HF checkpoint"] --> LLMEXP["llmexport.py<br/>utils/model_mapper.py, vision.py, audio.py, talker.py, token2wav.py"]
    ONNX["ONNX / TF / TFLite / Caffe / TorchScript"] --> CONV["MNNConvert<br/>tools/converter (optimizer/merge, postconvert)"]
    LLMEXP --> CONV
    GGUF["GGUF / safetensors"] --> G2M["gguf2mnn.py / safetensors2mnn.py"] --> CONV
    CONV --> MNNF["llm.mnn + llm.mnn.weight + config.json + tokenizer.txt + embeddings_bf16.bin<br/>(+ visual.mnn, audio.mnn, talker.mnn, dit.mnn, bigvgan.mnn ...)"]
  end
  subgraph App["Application layer"]
    DEMO["llm_demo / qwen3_tts_demo / llm_bench"]
    MLS["mls_server / mnncli (HTTP, SSE)"]
    PY["pymnn MNN.llm"]
    AND["Android MnnLlmChat (JNI) / iOS MNNLLMChat / TaoAvatar"]
  end
  subgraph LLM["transformers/llm/engine"]
    LLMC["Llm (llm.cpp): load, forwardRaw, generate, response, prompt cache"]
    OMNI["Omni : Embedding : Llm (omni.cpp): vision/audio encoders, M-RoPE"]
    TALK["Talker : Llm: codec LM -> DiT -> BigVGAN / speech_decoder<br/>async token2wav workers, wav callback"]
    TOK["Tokenizer (SP / tiktoken / BERT / HF BPE / Pipeline) + jinja.hpp"]
    SAMP["Sampler (greedy/topK/topP/minP/tfs/typical/penalty/mixed)"]
    GEN["Generation strategies: AR, Lookahead, MTP, EAGLE, DFlash"]
    KVM["KVMeta (add/remove/reserve, prefix file)"]
    DISK["DiskEmbedding (bf16/int8/int4/int2/int3 rows from file)"]
  end
  subgraph Express["express/ (Module API)"]
    MOD["Module::load / clone -> NetModule -> PipelineModule -> StaticModule"]
    EXEC["Executor / RuntimeManager (shared Runtime, hints, cache, external file)"]
    EXPR["Expr / VARP lazy graph, ComputeCache"]
  end
  subgraph Core["source/core (Session API)"]
    INTERP["Interpreter -> Schedule::schedule"]
    SESS["Session (ModeGroup, resize/run)"]
    PIPE["Pipeline: encode (shape+geometry) -> allocMemory (create Execution, wrap copies) -> execute"]
    GEO["GeometryComputer (-> Raster/Loop) + SizeComputer"]
    BUF["BufferAllocator (Eager / Defer), KVCacheManager"]
    BK["Backend / Runtime registry (MNNInsertExtraRuntimeCreator)"]
  end
  subgraph Backends["source/backend/*"]
    CPU["CPU (+arm82 fp16, AVX2/512, RVV, KleidiAI, SME2, bf16)"]
    GPU["Metal | OpenCL | Vulkan | OpenGL | CUDA | MUSA | TensorRT"]
    NPU["CoreML | NNAPI | QNN | HIAI | Hexagon | RKNN(plugin) | NeuroPilot(stub)"]
  end
  MNNF --> LLMC
  DEMO --> LLMC
  MLS --> LLMC
  PY --> LLMC
  AND --> LLMC
  LLMC --> OMNI --> TALK
  LLMC --> TOK
  LLMC --> SAMP
  LLMC --> GEN
  LLMC --> KVM
  LLMC --> DISK
  LLMC --> MOD
  MOD --> EXEC
  MOD --> EXPR
  MOD --> SESS
  INTERP --> SESS --> PIPE
  PIPE --> GEO
  PIPE --> BUF
  PIPE --> BK
  BK --> CPU
  BK --> GPU
  BK --> NPU
  GPU -. "unsupported op -> CPU backup" .-> CPU

Key modules

Module	Path	Responsibility	Key symbols
Public headers	include/MNN	Session API, tensor, forward types, Express/Module API	`Interpreter`, `Tensor`, `MNNForwardType`, `BackendConfig`, `ScheduleConfig`, `Express::Module`, `Express::Executor`, `VARP`
Core scheduling	source/core/Schedule.cpp	Pick backend type, build per-pipeline op lists, mark I/O tensors	`Schedule::schedule`, `Schedule::getAppropriateType`, `Schedule::BackendCache`, `OpCacheInfo`
Session	source/core/Session.cpp	Owns pipelines, translates `SessionMode`/`HintMode` into `RuntimeHint`, resize/run	`Session::Session`, `Session::resize`, `Session::run`, `ModeGroup::setHint`, `createPipelineBackend`
Pipeline	source/core/Pipeline.cpp	shape+geometry → command buffer → execution creation (with CPU fallback) → memory plan → execute	`Pipeline::encode`, `Pipeline::allocMemory`, `_createExecutions`, `_InsertCopy`, `Pipeline::execute`, `_pushTuningTask`
Backend abstraction	source/core/Backend.hpp, Backend.cpp	Backend/Runtime interfaces, creator registry, hints	`Backend::onCreate/onAcquire/onCopyBuffer/onSync`, `Runtime`, `RuntimeHint`, `MNNInsertExtraRuntimeCreator`, `registerBackend`
Memory	source/core/BufferAllocator.hpp	Static/dynamic pools, mmap-backed allocators	`EagerBufferAllocator`, `DeferBufferAllocator`, `BufferAllocator::createMmap`
Copy/wrap	source/core/WrapExecution.cpp	Insert device/layout conversion between executions	`WrapExecution::needWrap`, `makeCopyExecution`
Geometry	source/geometry	Decompose ops into Raster/Loop primitives	`GeometryComputer`, `GeometryComputerUtils::shapeComputeAndGeometryTransform`, `registerGeometryOps`
Shape	source/shape	Output-shape inference per op	`SizeComputer`, `SizeComputerSuite`, `REGISTER_SHAPE`
KV cache core	source/core/KVMeta.hpp, KVCacheManager.hpp	Host-side KV cursor; base KV storage w/ disk mmap	`KVMeta::sync`, `KVCacheConfig`, `KVCacheManager::onAlloc/onRealloc`
CPU backend	source/backend/cpu	Kernels, thread pool, core detection, attention, KV cache	`CPUBackend::addCreator/onCreate`, `CPURuntime`, `ThreadPool`, `MNNGetCPUInfo`, `CPUAttention`, `CPUKVCacheManager`, `CoreFunctions`
ARM fp16	source/backend/arm82	fp16 variant of CPU backend (NC8HW8)	`Arm82Backend`, `Arm82Functions::init`
GPU backends	metal, opencl, vulkan, cuda	Device runtimes, kernels, tuning caches	`MetalBackend`, `OpenCLBackend`, `VulkanBackend`, `CUDABackend`
NPU backends	coreml, nnapi, qnn, hiai, hexagon, rknn	Subgraph compilation / delegation	`CoreMLBackend::buildModel`, `NNAPIBackend::buildModel`, `QnnBackend::finalizeGraph`, `NPUBackend::bulidIRModelAndLoad`, `registerRKNNPlugin`
Express	express/	Lazy VARP graph, Module implementations, RuntimeManager	`Executor::RuntimeManager::createRuntimeManager`, `StaticModule`, `PipelineModule`, `Executor::ComputeCache`
Converter	tools/converter	Frontends + optimizer passes + weight quant/coding	`MNNConverter.cpp`, `cli.cpp`, `PostConverter.cpp`, `merge/FuseAttention.cpp`, `WeightQuantAndCoding.cpp`
LLM export	transformers/llm/export	HF → MNN (text/vision/audio/talker/token2wav/draft models)	`llmexport.py`, `utils/model_mapper.py`, `utils/vision.py`, `utils/audio.py`, `utils/talker.py`, `utils/token2wav.py`, `gguf2mnn.py`
LLM engine	transformers/llm/engine	Runtime for LLM/VLM/omni/TTS/embedding/reranker	`Llm`, `Omni`, `Talker`, `Embedding`, `LlmConfig`, `Tokenizer`, `Sampler`, `Generation*`, `DiskEmbedding`
Diffusion engine	transformers/diffusion/engine	SD1.5 (PNDM), SD3.5 (flow), Sana (flow, Qwen3 text enc), Wan2.1 T2V	`Diffusion::createDiffusion`, `StableDiffusion`, `DiffusionSD35`, `SanaDiffusion`, `WanDiffusion`
CV / audio libs	tools/cv, tools/audio	OpenCV-like ops on VARP; wav I/O, fbank/whisper/conformer/usm mel	`MNN::CV::resize/imread`, `MNN::AUDIO::load/save/whisper_fbank/conformer_fbank/usm_fbank`
Apps	apps/	Android/iOS chat apps, TTS SDK, sherpa port, mnncli server, Sana app	`MnnLlmChat`, `MnnTaoAvatar`, `mnn_tts`, `sherpa-mnn`, `mnncli`
Python	pymnn/	Bindings: expr/nn/cv/audio/llm/reranker	`pymnn/src/llm.h` (`PyMNNLLM_methods`)

2. Device layer

Backend abstraction and selection

[Verified] MNNForwardType (MNNForwardType.h): CPU=0, METAL=1, CUDA=2, OPENCL=3, AUTO=4, NN=5 (NNAPI on Android / CoreML on iOS / QNN online), OPENGL=6, VULKAN=7, USER_0=8 (HIAI), USER_1=9 (TensorRT), HEXAGON=10, USER_3=11, ALL=12, CPU_EXTENSION=13, MNN_MEMORY_AHARDWAREBUFFER=14, MUSA=15; offline-convert pseudo-types MNN_CONVERT_QNN=32, MNN_CONVERT_NEUROPILOT=33, MNN_CONVERT_COREML=34. MNNGpuMode bit-flags: MNN_GPU_TUNING_{NONE,HEAVY,WIDE,NORMAL,FAST}, MNN_GPU_MEMORY_BUFFER (1<<6), MNN_GPU_MEMORY_IMAGE (1<<7), MNN_GPU_RECORD_OP (1<<8), MNN_GPU_RECORD_BATCH (1<<9) — these are OR'ed into ScheduleConfig::numThread for GPU backends (the LLM engine does numThread |= 64 | 512 for OpenCL, llm.cpp#L232).
[Verified] BackendConfig = MemoryMode {Normal, High, Low}, PowerMode {Normal, High, Low}, PrecisionMode {Normal, High, Low, Low_BF16}, and a union of void* sharedContext / size_t flags. RuntimeStatus probes: STATUS_SUPPORT_FP16, _DOT_PRODUCT, _POWER_LOW, _SIMD_GROUP_REDUCE, _FUSED_PROJ.
[Verified] class Backend (Backend.hpp): pure virtuals onCreate(inputs, outputs, op) (returns Execution* or nullptr = unsupported), onResizeBegin/End, onExecuteBegin/End, onAcquire(tensor, StorageType), onClearBuffer, onCopyBuffer(src, dst); helpers onAcquireBuffer/onReleaseBuffer store a MemObj in the tensor descriptor; StorageType {STATIC, DYNAMIC, DYNAMIC_SEPERATE, DYNAMIC_IN_EXECUTION}; GPU hooks onMapTensor/onUnmapTensor/onSync(MapType, toCpu, tensor); getMetaPtr/setMetaPtr (KV meta). class Runtime: CompilerType {Compiler_Geometry, Compiler_Origin, Compiler_Loop} (default Loop), AllocatorType {Defer, Eager}, onCreate(config, origin), onReset, onGabageCollect(level), onSetCache/onGetCache, onSetCachePath, onMeasure, async-work API (setAsyncWork/waitAsyncWork/mCancelled), onConcurrencyBegin/End.
[Verified] RuntimeHint (Backend.hpp) carries all tunables: memoryAllocatorType, winogradMemoryUsed=3, cpuDecreaseRate=50, dynamicQuantOption, attentionOption=8, kvcacheSizeLimit=-1, kvcacheDirPath, prefixcacheDirPath, midMemoryPath, weightMemoryPath, mmapFileSize=1024 MB, useCachedMmap, encorderNumForCommit=10, initThreadNumber, useArmSme2Cores, enableKleidiAI, cpuIds, divisionRatio=41. Session::ModeGroup::setHint (Session.cpp#L77) is the single translation point from Interpreter::HintMode.
[Verified] Registry: registerBackend() in Backend.cpp (under std::call_once) always registers CPU and conditionally CoreML (MNN_COREML_ENABLED), NNAPI, QNN, RKNN plugin, Hexagon, OpenCL, Metal, NeuroPilot. Vulkan, OpenGL, CUDA, MUSA, TensorRT, HIAI self-register via file-scope static initializers (e.g. VulkanRuntime.cpp, cuda/Register.cpp, GLBackend.cpp). MNNGetExtraRuntimeCreator instantiates a probe Runtime when the creator was registered with needCheck=true (OpenCL, Vulkan, HIAI) so a missing driver degrades gracefully.
[Verified] Discovery/selection: Schedule::getAppropriateType (Schedule.cpp#L111) resolves MNN_FORWARD_AUTO through the fixed priority list HIAI → CoreML/NN → TensorRT → CUDA → OpenCL → Metal → Vulkan → CPU, falls back to config.backupType if the creator is missing, and demotes OpenCL+Power_Low if STATUS_SUPPORT_POWER_LOW is false. RuntimeFactory::create is a one-liner over the registry (RuntimeFactory.cpp).

Memory allocation and pools

[Verified] BufferAllocator.hpp: EagerBufferAllocator (free-list multimap<size,Node>, 64-byte alignment via MNN_MEMORY_ALIGN_DEFAULT in MNNMemoryUtils.h, per-thread groups) and DeferBufferAllocator (symbolic plan then compute()/apply() into one arena). Factories createDefault, createMmap(dir, prefix, postfix, autoRemove, syncValid), createRecurse(parent).
[Verified] CPU: CPURuntime owns one static Eager pool and MNN_CPU_MAX_BUFFER_INDEX=2 dynamic buffers; createDynamicBufferAlloctor returns Defer vs Eager based on hint().memoryAllocatorType (HintMode::MEM_ALLOCATOR_TYPE) (CPUBackend.cpp). hint().weightMemoryPath swaps the static pool for an mmap-backed one (this is use_mmap in the LLM config → EXTERNAL_WEIGHT_DIR); midMemoryPath does the same for dynamic buffers (EXTERNAL_FEATUREMAP_DIR).
[Verified] SessionMode memory knobs: Session_Memory_Collect runs onGabageCollect(0) after every allocMemory (Session.cpp#L363); Session_Memory_Cache keeps static memory. Session_Input_User (what Module::Config::shapeMutable=true maps to) keeps inputs user-owned and stages copies through inputTensorCopyCache in Pipeline::_copyInputs.
[Verified] Express default: Executor::getGlobalExecutor() creates a 1-thread CPU runtime with memoryAllocatorType=0 (Defer) (Executor.cpp#L154).

Host↔device transfers and synchronization

[Verified] Tensor.hpp: copyFromHostTensor, copyToHostTensor, map/unmap(MapType, DimensionType), wait(MapType, finish). Tensor::map calls Backend::onSync then onMapTensor, falling back to a temp host copy when mapping is unsupported (Tensor.cpp).
[Verified] Metal (MetalBackend.mm): one MTLCommandQueue, a rolling MTLCommandBuffer, mEncoderCount reset at onExecuteBegin, encoder flushed/committed every hint().encorderNumForCommit ops (OP_ENCODER_NUMBER_FOR_COMMIT), onSync(toCpu) = flush + commit + waitUntilCompleted; onAcquire handles DYNAMIC_IN_EXECUTION from an execution pool; tuned threadgroup tables persisted through onGetCache/onSetCache (MetalTuneLevel default Wide, schema MetalCache.fbs).
[Verified] OpenCL (OpenCLBackend.cpp): setGpuMode decodes BUFFER/IMAGE (AUTO → BUFFER on Mali/Intel, IMAGE elsewhere) and tuning level; record-queue (MNN_GPU_RECORD_OP/BATCH) when the device supports it; onSync(toCpu) = commandQueue().finish(); SVM fine-grain buffers under MNN_OPENCL_SVM_ENABLE; binary+tuning cache in a FlatBuffer (CLCache schema under opencl/core/runtime). The buffer path can be compiled out with MNN_OPENCL_SIZE_CUT.
[Verified] Vulkan buffer variant (VulkanBackend.cpp): mDirect = !(gpuMode & MNN_GPU_RECORD_BATCH); all command buffers submitted in one vkQueueSubmit + single VulkanFence in _finish(); onCopyBuffer stages through a host-visible buffer; dynamic pool is an EagerBufferAllocator aligned to nonCoherentAtomSize. Image variant selected by MNN_VULKAN_IMAGE=ON (default) at build time.

Portability — what is actually in tree

Backend dir	Forward type	Registration	CMake gate	Delegation granularity / notes
cpu	CPU / CPU_EXTENSION	`registerCPURuntimeCreator`	always	per-op; `Compiler_Loop`
arm82	CPU_EXTENSION (fp16)	`registerArm82RuntimeCreator` → `Arm82Functions::init`	`MNN_ARM82` (ON)	chosen when `Precision_Low` and `fp16arith`
metal	METAL	`registerMetalRuntimeCreator`	`MNN_METAL` + APPLE	per-op
opencl	OPENCL	`registerOpenCLRuntimeCreator` (needCheck)	`MNN_OPENCL`	per-op; dlopen unless `MNN_USE_SYSTEM_LIB`
vulkan	VULKAN	static init (needCheck)	`MNN_VULKAN`, `MNN_VULKAN_IMAGE`	per-op; dlopen `libvulkan`
opengl	OPENGL	static init	`MNN_OPENGL`	per-op; links GLESv3/EGL
cuda	CUDA	static init	`MNN_CUDA` (+`MNN_CUDA_QUANT/BF16/TUNE_PARAM`)	per-op; sm 60–120
musa	MUSA=15	static init	`MNN_MUSA` (+ stub/compat/native)	Moore Threads
tensorrt	USER_1	static init	`MNN_TENSORRT`	whole subgraph at `onResizeEnd`
coreml	NN	`registerCoreMLRuntimeCreator`	`MNN_COREML`	whole subgraph: `buildModel()` at `onResizeEnd`, `invokeModel()` at `onExecuteEnd`
nnapi	NN	`registerNNAPIRuntimeCreator`	`MNN_NNAPI` (API ≥ 29)	whole graph via `ANeuralNetworksCompilation`; `Compiler_Geometry`
qnn	NN (online) / CONVERT_QNN (offline)	`registerQNNRuntimeCreator`	`MNN_QNN`, `MNN_QNN_ONLINE_FINALIZE`, `MNN_QNN_CONVERT_MODE`	`Compiler_Origin`; graph finalized once; context binary cached via `onGetCache/onSetCache`; SDK via prepare_qnn_deps.sh
hiai	USER_0	static init (needCheck)	`MNN_NPU` (not an `option()`)	whole graph `bulidIRModelAndLoad`; `Compiler_Origin`
hexagon	HEXAGON=10	`registerHexagon` (probes DSP lib)	`MNN_HEXAGON`	per-op over decomposed stream (`Compiler_Loop`), forced 1 thread
rknn	(CPU + Plugin op)	`registerRKNNPlugin`	`MNN_RKNN` + `MNN_WITH_PLUGIN`	one plugin op wraps a whole `.rknn` file
neuropilot	CONVERT_NEUROPILOT=33	—	forced OFF (CMakeLists.txt)	not shipped in OSS

[Verified] NNAPI and CoreML both claim MNN_FORWARD_NN; MNNInsertExtraRuntimeCreator rejects duplicates, so only one can be linked per build [Inferred from registry assert].
[Verified] CPU ISA paths: source/backend/cpu/arm/arm64 (NEON, *_ARMV82_* sdot, *_ARMV86_* i8mm/smmla, low_memory/ W2/W3/W4 GEMM, sme2_asm/), arm32, x86_x64/{sse,avx,avxfma,avx512} (AVX512-VNNI split), riscv/rvv (+ SpacemiT IME2), kleidiai/ (fetched v1.16.0 by cmake/KleidiAI.cmake), bf16/. Runtime feature detection MNNGetCPUInfo() (CPURuntime.cpp): fp16arith, dot, i8mm, sve2, sme2, rvv/zvfh, groups (big/little clusters by cpufreq), smeCoreNumber, perfCoreNumber.

Multi-backend fallback (unsupported op on GPU)

[Verified] Session::createPipelineBackend (Session.cpp#L26) creates cache.first (target) and, unless the target is CPU, a separate CPU cache.second with flags=4 (MNN_CPU_USE_DEFAULT_BACKEND).
[Verified] _createExecutions (Pipeline.cpp#L538): try OpCommonUtils::createExecutionWithExternal(mBackend, …); if nullptr, // Try Backup on mBackupBackend; if still null → NOT_SUPPORT. _InsertCopy then walks every command's inputs/outputs and, when WrapExecution::needWrap(t, curBackend) is true, appends an OpType_Copy command built by WrapExecution::makeCopyExecution (const tensors cached via copyConstCache). needWrap returns false for MNN_FORWARD_NN and types ≥ MNN_CONVERT_QNN, and for CPU↔CPU_EXTENSION only when bytes or pack (on NC4HW4 data) differ (WrapExecution.cpp).
[Verified] Coarser fallback under Session_Backend_Auto: Pipeline::allocMemory asks Runtime::onMeasure per op; if any reports initCostLong it prints "Turn back to cpu", starts _pushTuningTask asynchronously and swaps the whole pipeline backend to CPU (Pipeline.cpp#L1079).
[Verified] Only Compiler_Loop runtimes get op-level CPU fallback (docs/contribute/backend.md); NPU backends compile the whole graph and refuse (return null) unsupported ops.

[Verified] Executor::RuntimeManager::createRuntimeManager(config) (Executor.cpp#L334) calls _getOrCreateRuntime(type, …) on the current Executor — an existing runtime of the same forward type is reused (onReset with new thread count), and mRuntime.second (CPU backup) is always the executor's CPU runtime. StaticModule::clone reuses the RuntimeManager's runtime and shared const tensors, so N cloned Modules share one thread pool/allocator/tuning cache while owning separate Backend/Session objects (StaticModule.cpp). Executor::newExecutor + ExecutorScope gives a fully independent runtime (used by the Talker's async worker threads).

CPU threading and core binding

[Verified] MNN_USE_THREAD_POOL=ON (default) → own pool (ThreadPool.cpp): pools keyed by affinity mask, MNN_THREAD_POOL_MAX_TASKS=2 concurrent task slots, sleeping-worker bitmask. CPURuntime::_resetThreadPool may return fewer threads than requested. CPURuntime::_validateCpuIds forbids mixing little and big cores; Power_Low → little cluster, Power_High → highest clusters, Power_Normal → no pinning; MNNSetSchedAffinity via raw sched_setaffinity syscall. Work split by computeDivideSizes weights little cores with CPU_LITTLECORE_DECREASE_RATE (default 50%).

3. Kernel layer

Operator set and registration

[Verified] MNN.fbs: enum OpType with 186 entries; transformer block Attention=299, FmhaV2=300, Fmhca=301, SeqLen2Spatial=302, SplitGeLU=303, GroupNorm=304, LinearAttention=305, RoPE=306, FusedLinear=307, GatedRMSNorm=308; quant ops ConvInt8=513, Int8ToFloat=514, DepthwiseConvInt8=515, FloatToInt8=517, EltwiseInt8=518, DynamicQuant=155; control While=600, If=601, LayerNorm=603, GridSample=604; escape hatches Plugin=256, Extra=512; Stft=156 is the only audio-ish op. table Op { inputIndexes, main:OpParameter, name, outputIndexes, type, defaultDimentionFormat=NHWC, externalPath }; table Net { bizCode, extraTensorDescribe, extraInfo, oplists, outputName, preferForwardType, sourceType, tensorName, tensorNumber, usage (INFERENCE/TRAIN/INFERENCE_STATIC), subgraphs, mnn_uuid }. AttentionParam { kv_cache, kv_shared_layer, layer_index, kv_shared_layer_index, mhq_quant, output_c4, attnScale }, RoPEParam, FusedLinearParam, LinearAttentionParam (gate folding), LoopParam/RegionCommand (the Loop IR). Other schema files: CaffeOp.fbs (IDSTQuan, Convolution2D), TensorflowOp.fbs, Tensor.fbs (MNN_DATA_FORMAT {NCHW, NHWC, NC4HW4, NHWC4}, Blob.external), UserDefine.fbs, TFQuantizeOp.fbs, Type.fbs, ExtraInfo.fbs, TrainInfo.fbs.
[Verified] Registration per backend: CPU REGISTER_CPU_OP_CREATOR(name, OpType) → CPUBackend::addCreator into a std::map<OpType, Creator*>; generated CPUOPRegister.cpp registers 75 op types (incl. Attention, LinearAttention, RoPE, Stft, Plugin, ConvInt8, DynamicQuant, While). Metal MetalOPRegister.mm 36 (transformer ops under MNN_SUPPORT_TRANSFORMER_FUSE). OpenCL keys by pair<OpType, GpuMemObject> (buffer vs image) — OpenCLOPRegister.cpp 66 entries / 39 types. Vulkan buffer 35 / image 26 (self-registering statics). CUDA ~44 (static CUDACreatorRegister). NNAPI 23, CoreML 22, QNN 41.

Dispatch mechanism

[Verified] Execution (Execution.hpp): onResize, onExecute (pure), onClone (capability probe when dst==nullptr), plus a string-keyed insertExtraCreator/searchExtraCreator. CPUBackend::onCreate (CPUBackend.cpp) rewrites Convolution → ConvInt8 when I/O quantAttr.type == DT_INT8 and the conv has quantized weights, then looks up gCreator.
[Verified] Precision → CPU variant (CPURuntime::onCreate): Precision_Low + fp16arith → Arm82Backend (pack 8, bytes 2); Precision_Low_BF16 → BF16 core functions; flags==4 → plain CPU; else AVX2Backend on x86. All variants share the CoreFunctions table (CommonOptFunction.h: pack, bytes, supportFp16arith/SDot/I8mm/SME2, kernel pointers) — so NC4HW4 is really NC{pack}HW{pack}: 4 (fp32 NEON), 8 (fp16 / AVX2), 16 (AVX512).
[Verified] The flow per op at resize: SizeComputer::computeOutputSize → GeometryComputer::onCompute (or identity DefaultGeometryComputer) → Backend::onCreate → Execution::onResize; at run: Execution::onExecute in Pipeline::execute (Pipeline.cpp#L1167).

Geometry decomposition

[Verified] GeometryComputer.hpp: GeometryComputer::Context holds one pre-built OpType_Raster op used for every emitted region command; registerGeometryComputer(computer, types, CompilerType); two tables (Compiler_Geometry, Compiler_Loop), Compiler_Origin bypasses geometry. 51 files (GeometryConv2D, GeometryBatchMatMul, GeometryReshape, GeometryPermute, GeometryBinary, GeometryReduce, GeometryLayernorm, GeometryFusedProj, GeometryGatedRMSNorm, …), 84 registrations in GeometryOPRegister.cpp. GEOMETRY_COMPUTE_MASK bits: FUSEREGION, FUSEREGION_MULTI, USELOOP, OPENCACHE.
[Verified] Contract (docs/contribute/op.md, docs/contribute/backend.md): schema → shape → (optional) geometry → per-backend execution; if geometry is written, no per-backend kernel is required. [Inferred] Minimal backend residue: Raster, Convolution(+Depthwise), MatMul, BinaryOp, UnaryOp, Pooling, Reduction, Softmax, ReLU/PReLU, Scale, While/Loop — the intersection of all backend registries.
[Verified] Usage_INFERENCE_STATIC models (converted with --saveStaticModel) skip shape+geometry entirely (needComputeGeometry=false in Schedule::schedule; Pipeline::encode copies commands verbatim) — this is what MNN_SKIPBUILD_GEOMETRY/MNN_BUILD_MINI rely on.

Shape inference / dynamic shapes

[Verified] SizeComputer.hpp: onComputeSize, onComputeFlops, REGISTER_SHAPE(_INPUTS) (ops whose shape depends on input content, e.g. Stft, mark mNeedContentInputIndex); 82 files, 122 registrations (ShapeRegister.cpp); transformer shapes gated by MNN_SUPPORT_TRANSFORMER_FUSE.
[Verified] Runtime shape change: Interpreter::resizeTensor → Session::setNeedResize; Session::resize() (Session.cpp#L306) re-runs Pipeline::encode (shape+geometry) then allocMemory (execution creation cached in executionCache, memory re-planned). Session_Resize_Check/Fix (Module::traceOrOptimize) records a resize trace and freezes it. For Express Modules, shapeMutable=true → Session_Input_User.

Layout

[Verified] Tensor::DimensionType {TENSORFLOW (NHWC), CAFFE (NCHW), CAFFE_C4 (NC4HW4)}; NativeInsideDescribe::dimensionFormat defaults to MNN_DATA_FORMAT_NC4HW4 (TensorUtils.hpp); Region is the raster primitive. Arm82Backend reinterprets NC4HW4 as NC8HW8 (Arm82Backend.hpp). Layout conversion between ops/backends is automatic via WrapExecution (§2).

Quantization formats and where dequant happens

[Verified] Classic PTQ int8: tools/quantization (quantized.out, calibration.cpp: keys quant_bits, feature_quantize_method (KL/ADMM/EMA), weight_quantize_method, feature_clamp_value, weight_clamp_value, winogradOpt, skip_quant_op_names). Weights stored as IDSTQuan { buffer, alpha, type (1 int8 / 2 sparse / 3 fp16 / 4 weightInt8 / 8 external low-bit), quantScale, scaleIn/Out, aMaxOrBits, aMin, readType, alphaFp16, scaleStorage } (CaffeOp.fbs) and decoded by ConvolutionCommon::load (ConvolutionCommon.cpp) into Int8Common { weight, alpha, canUseInt4/Int3/Int2, asymmetric }.
[Verified] LLM weight quant (export): --quant_bit {2,3,4,8,16} (default 4), --quant_block (default 64; 0 = per-channel), --lm_quant_bit/--lm_quant_block, --visual_quant_bit/--visual_quant_block, --embed_bit {16,8,4}, --sym, --scale_bit {16,32}, --awq, --hqq, --gptq_path, --omni, --smooth, --calib_data, --act_bit {8,16}, --quant_config (per-op JSON) — llmexport.py; packing in utils/mnn_converter.py (quant, write_header, build_weight) and utils/mnn_utils.py (repack_low_bits). MNNConvert equivalents: --weightQuantBits, --weightQuantBlock, --weightQuantAsymmetric, --hqq, --weightQuantScaleBit (cli.cpp).
[Verified] Runtime dequant on CPU (MNN_LOW_MEMORY, forced ON by MNN_BUILD_LLM): ConvolutionFloatFactory::_createUnit (ConvolutionFloatFactory.cpp) — with Memory_Low → DenseConvInt8TiledExecutor(isDynamicQuant=true) (activations quantized to int8 per forward: BatchAsyDynamicQuant/BatchSymDynamicQuant, DYNAMIC_QUANT_OPTIONS 1 = per-tensor, 2 = block-quant input), else DenseConvolutionTiledExecutor where block-wise scale/bias pointers are threaded into the packed GEMM (selectLowMemoryMatmulFunc) — no materialized float weights. ARM64 kernels: arm64/low_memory (MNNGemmInt8AddBiasScale_ARMV82_w{2,3,4}_Unit.S, _ARMV86_w{2,3,4}_, MNNDynamicQuantFP32_Pack{4,8}.S). GPU: Metal MetalConvolution1x1.mm accepts mDequantBits ∈ {2,3,4,8} (Q4 16-byte GEMV); OpenCL ConvBufLowMemoryExecution/ConvLowMemoryExecution + gemm_int.cl/gemv_conv1x1_buf.cl; Vulkan buffer gemv_dequant_int4.comp, dynamic_w8a8_coop_gemm_fused.comp, VulkanConv1x1CoopA8; CUDA weight_only_quant/ConvFpAIntBExecution.cu + CUTLASS int8.
[Verified] fp16: MNN_ARM82 (ON; Precision_Low), bf16 MNN_SUPPORT_BF16 (OFF). KleidiAI: MNN_KLEIDIAI (ON) but used only if RuntimeHint::enableKleidiAI (CPU_ENABLE_KLEIDIAI, default from MNN_KLEIDIAI_DEFAULT_ON=OFF) and the conv is 1×1 stride-1. SME2: MNN_SME2 (ON), hints CPU_SME2_INSTRUCTIONS, CPU_SME2_NEON_DIVISION_RATIO (default 41).
[Verified] Attention/KV quant: RuntimeHint::attentionOption (default 8): %8 → 0 none / 1 K int8 / 2 K+V int8 / 3 K TQ3 / 4 K+V TQ3 / 5 K TQ4 / 6 K+V TQ4; /8 → flash attention on/off (CPUAttention.cpp, KVQuantMode in CPUKVCacheManager.hpp). LLM config key attention_mode (legacy quant_qkv). Metal supports only int8 KV and flash attention for head_dim ∈ {64,128,256} (docs/transformers/llm.md); other GPU backends ignore attention_mode [Verified, repo doc].
[Verified] Fused transformer ops: MNN_SUPPORT_TRANSFORMER_FUSE; converter merge passes in optimizer/merge (FuseAttention, FuseFmhaV2, FuseFmhca, FuseGeLu, FuseSplitGeLu, FuseGroupNorm, FuseLayerNorm{,V2,V3,RMS,RMSGamm,WithoutGammBeta}, MergeDynamicQuantV1/V2, ConvertMatMulToConv2D) and postconvert/FuseTransformerC4.cpp; RoPE, FusedLinear (QKV / gate-up / LN+proj), LinearAttention gate folding, GatedRMSNorm are emitted by the exporter (--disable_fuse_qkv_proj, --disable_fuse_gate_up_proj, --disable_fuse_ln_proj, --disable_transformer_c4). Cross-layer KV sharing: AttentionParam.kv_shared_layer_index → Execution::onClone from a registry in _createExecutions.

Compilation / JIT / graph capture

[Verified] No machine-code JIT. codegen/ (codegen/CMakeLists.txt: MNN_CODEGEN_OPENCL/METAL/CUDA, all OFF) is source-level op fusion emitted at Session::resize when Session_Codegen_Enable. Metal shaders are embedded C strings (AllShader.cpp) compiled by the OS with pipeline-state + tuning cache; Vulkan SPIR-V pre-generated (buffer/compiler); OpenCL online-built with CLCache; CUDA AOT .cu. No cudaGraph/MTLIndirectCommandBuffer anywhere. Interpreter::setCacheFile / RuntimeManager::setCache persist tuning across runs (LLM engine writes tmp_path/mnn_cachefile.bin for non-CPU backends).

Fallback when op unsupported

[Verified] Per-op CPU backup (§2); if CPU also lacks the op → NOT_SUPPORT at resize. Unresolved OpType_Extra is a converter-time failure (only Metal registers an Extra creator; CPU has none).

Extension points: custom op / backend

[Verified] Plugin ops: MNN_WITH_PLUGIN; include/MNN/plugin (PluginKernel.hpp: CPUComputeKernel::init/compute/resize, REGISTER_PLUGIN_COMPUTE_KERNEL(name, kernel); PluginShapeInference.hpp: REGISTER_PLUGIN_OP(name, inferShapeKernel)), string-keyed; OpType_Plugin { type:string, attr } executed by CPUPlugin.cpp. RKNN and offline-QNN reuse this path. No registerCustomOp API exists.
[Verified] Native op: schema + REGISTER_SHAPE + optional geometry + REGISTER_{CPU,METAL,OPENCL}_OP_CREATOR / VulkanBackend::addCreator (docs/contribute/op.md); converter --customOpLibs/--allowCustomOp.
[Verified] New backend: subclass Backend + Runtime, register via MNNInsertExtraRuntimeCreator(type, creator, needCheck) (docs/contribute/backend.md).

4. Model runner

Model loading / export / conversion

[Verified] .mnn = FlatBuffer Net; Interpreter::createFromFile/createFromBuffer, createSession(ScheduleConfig{type, numThread|gpuMode, backendConfig, backupType, saveTensors, path}), setSessionMode, setSessionHint, setCacheFile, setExternalFile, resizeTensor/resizeSession, runSession(WithCallBack), updateSessionToModel (Interpreter.hpp). SessionMode values 0–17 (Session_Debug/Release, Input_Inside/User, Output_Inside/User, Resize_Direct/Defer, Backend_Fix/Auto, Memory_Collect/Cache, Codegen_Disable/Enable, Resize_Check/Fix, Module_Forward_Separate/Combine); HintMode 0–17 (see §2); ExternalPathType {KVCACHE_DIR, FEATUREMAP_DIR, WEIGHT_DIR, NPU_FILE_DIR, PREFIXCACHE_DIR}.
[Verified] Module API: Module::load(inputs, outputs, file|buffer, rtMgr, Config{dynamic=false, shapeMutable=true, rearrange=false, backend, base}) (Module.hpp); implementation NetModule → PipelineModule (sub-graphs, If/While/MoE modules) → StaticModule (wraps a Session) (Module.cpp, PipelineModule.hpp, StaticModule.cpp). Module::load auto-sets the external file to <file>.weight if none set. Module::clone(module, shareParams) shares weights, new Session. Lazy Express eval: Executor::LazyMode {LAZY_FULL, LAZY_CONTENT, LAZY_COMPUTE_ONCE}; Executor::ComputeCache in express/Utils.hpp.
[Verified] External weights: Op.externalPath + Convolution2D.external=[offset, weight_bytes, bias_bytes], Blob.external=[offset, bytes]; writer writeFb.cpp (--saveExternalData, auto for large models, threshold externalTreshold=64 KB in config.hpp); loaded lazily by FileLoader at execution creation (createExecutionWithExternal).
[Verified] Converter: MNNConverter.cpp → Cli::convertModel; frontends onnx/, tensorflow/, tflite/, caffe/, torch/ (TorchScript, MNN_BUILD_TORCH); flags --fp16, --weightQuantBits, --weightQuantBlock, --hqq, --optimizeLevel 0/1/2, --optimizePrefer, --keepInputFormat, --saveStaticModel, --saveExternalData, --transformerFuse, --transformerFuseC4, --transformerFuseQkvProj/GateUpProj/LnProj, --alignDenormalizedValue, --detectSparseSpeedUp, --customOpLibs, --allowCustomOp, --testdir/--thredhold, --JsonFile, --rknn. Pass pipeline in PostConverter.cpp (RemoveUnusefulOp → … → MergeBNToConvolution → … → AddTensorFormatConverter → ReIndexTensor), 41 merge passes, 33 onnxextra/ rewrites.
[Verified] GGUF import (gguf2mnn.py): F16, Q4_0, Q4_1, Q4_K, Q5_0, Q5_1, Q6_K, Q8_0 → MNN low-bit blobs (others assert); safetensors2mnn.py for pre-quantized/plain safetensors.
[Verified] LLM export pipeline (llmexport.py: export_talker → export_vision → export_audio → export_eagle → export_dflash → export_language → export_mtp → export_tokenizer → export_config) produces llm.mnn, llm.mnn.weight, llm_config.json, config.json (runtime defaults: backend_type=cpu, thread_num=4, precision=low, memory=low, sampler_type=mixed, temperature=0.8, top_k=40, top_p=0.9, min_p=0.05, …), tokenizer.txt (or .mtok binary), embeddings_bf16.bin / embeddings_int{8,4}.bin, optional visual.mnn, audio.mnn, talker.mnn, predit.mnn, dit.mnn, bigvgan.mnn, spk_dict.mnn, code_predictor.mnn, speech_decoder.mnn, speaker_encoder.mnn, mtp.mnn, EAGLE/DFlash draft files, ple_embeddings_bf16.bin (Gemma4). Supported families in model_mapper.py: llama, qwen/qwen2/qwen3/qwen3_moe/qwen3_5(_moe/_text), qwen2_vl/qwen2_5_vl/qwen3_vl(_moe), qwen2_audio, qwen3_asr, qwen2_5_omni, qwen3_tts, internlm, mobilellm, baichuan, deepseek-vl, llama4_text, mimo, poi_qwen2_mtp, chatglm/chatglm2, phi/phi-msft, internvl_chat, gemma2/gemma3(_text)/gemma4, openelm, idefics3/smolvlm, llava_qwen2 (FastVLM), hunyuan_v1_dense/hunyuan_vl, gpt_oss, minicpm/minicpmv, funaudiochat, glm_ocr, lfm2/lfm2_moe/lfm2_vl/lfm2_audio. NPU export helpers: export/npu (generate_llm_qnn.py, generate_llm_mlda.py) driving compilefornpu.cpp.

Execution lifecycle (init → warmup → step)

[Verified] Llm::createLLM(config.json) → Omni if is_visual || is_audio || has_talker else Llm (llm.cpp#L88). Llm::load() (llm.cpp#L319): check files → clear module pool before re-creating runtime (Backend lifetime) → initRuntime() (ScheduleConfig from backend_type/thread_num/precision/power/memory, RuntimeManager, hints from setRuntimeHint) → Tokenizer::createTokenizer → merge context.json into jinja context → DiskEmbedding (+ PLE) → chat template → Sampler → Module::load({input_ids, attention_mask, position_ids, logits_index[, deepstack_embeds, ple_embeddings]}, {logits[, talker_embeds][, hidden_states]}) with shapeMutable = !(opencl|vulkan|npu), rearrange=true, external file = llm_weight → setSpeculativeConfig (spec only if the model declares a dimensioned logits_index input) → GenerationStrategyFactory::create → clone decode module (key <1,false>) and spec module (<draft+1,true>) → preallocate mask/position VARPs → mGenerationStrategy->load → status RUNNING.
[Verified] No implicit warmup. Optional Llm::tuning(OP_ENCODER_NUMBER, candidates) benchmarks commit granularity on Metal/CPU (skipped on OpenCL) then resets KV; demos call it with {1,5,10,20,30,50,100}.
[Verified] Step: Llm::forwardRaw(hidden, mask, pos, extra) (llm.cpp#L569) picks module by (seqLenKey = inDecode ? seq_len : mPrefillKey(100), allLogits), chooses logitsAllIdx/logitsLastIdx, onForward, validates outputs, optional Tensor::wait when async=false, then mMeta->sync(). forwardVec sets mMeta->add = seq_len and splits into size_limit blocks for chunked prefill.
[Verified] Runtime hints actually set by the LLM engine (llm.cpp#L181): INIT_THREAD_NUMBER=4, MEM_ALLOCATOR_TYPE=0, ATTENTION_OPTION, USE_CACHED_MMAP, DYNAMIC_QUANT_OPTIONS, KVCACHE_INFO (ptr), CPU_SME2_NEON_DIVISION_RATIO, MMAP_FILE_SIZE, OP_ENCODER_NUMBER_FOR_COMMIT (0 during prefill / 512 during decode on OpenCL); external paths KVCACHE_DIR (kvcache_mmap), PREFIXCACHE_DIR, WEIGHT_DIR (use_mmap), NPU_FILE_DIR. Never set: KVCACHE_SIZE_LIMIT, CPU_CORE_IDS, CPU_ENABLE_KLEIDIAI.

Dynamic shapes

[Verified] Strategy is bucketed module clones, not per-step resize: prefill of any length maps to key 100 (the Session inside resizes on shape change since Session_Input_User), decode always key 1 (no resize), spec-verify key draft+1. Mask/position VARPs are reused when dims match (needNewVar). Chunked prefill (size_limit: [block, 1]) keeps shapes to a handful of values, needed on OpenCL/Vulkan/NPU where shapeMutable=false.

State & KV cache

[Verified] KVMeta (KVMeta.hpp): block=4096, previous, remove, reserve[] (pairs begin/len), n_reserve, add, file_name, file_flag {NoChange, PendingWrite, PendingRead}, seqlen_in_disk, layer_index, layer_nums, attn_scale, spec_block; sync() applies previous = previous - remove + add + Σreserve_len after each forward. Llm::setKVCacheInfo, eraseHistory(begin,end) (partial rollback via reserve segments), reset() (remove = previous), reuse_kv (multi-turn KV retention), prompt_cache (text-prefix diff → only the token delta is prefilled, llm.cpp#L1167), setPrefixCacheFile (KV prefix to disk; requires kvcache_mmap=true).
[Verified] Storage: KVCacheManager.hpp (KVCacheConfig { mKVCacheDir, mPrefixCacheDir, mExpandChunk=64, mBlockNum=1, mKvAlignNum }, key layout {numhead, [maxlen/hP, headdim, hP]} (or int8 [maxlen/hP8, headdim/lP8, hP8, lP8]), value {numhead, [headdim/hP, maxlen, hP]}, mmap FDs) and CPUKVCacheManager.cpp (onAlloc/onRealloc grow by mExpandChunk on demand, expandKVCacheInMem/InDisk, moveKVCacheFromMemToDisk when kvcacheSizeLimit exceeded, quant via MNNQuantAttentionKey/Value). Layout is contiguous per head — no paging, no eviction policy; the KV is a growable array trimmed only by explicit remove/reserve. Per-Llm instance (each cloned Module has its own attention executions → own KV). Metal/OpenCL/Vulkan/CUDA have their own attention executions (MetalAttention.mm, AttentionBufExecution.cpp, VulkanAttention.cpp, AttentionExecution.cu).
[Verified] Embedding lookup is host-side from a file (DiskEmbedding, diskembedding.cpp: bf16/int8/int4/int2/int3 rows, buffered FileLoader reads, tie-embeddings read from inside llm.mnn.weight), then fed as input_embeds.

Sampling and speculative decoding

[Verified] sampler.cpp: greedy, temperature, topK, topP, minP, tfs, typical, penalty (repetition/presence/frequency + n-gram), mixed pipeline (penalty forced first), logit_bias, banned_tokens, GPU top-k prefilter.
[Verified] speculative_decoding: ArGeneration (plain AR), LookaheadGeneration (n-gram lookahead, draft_predict_length, ngram_match_maxlen, lookup_file), MtpGeneration (mtp.mnn, needs hidden_states), EagleGeneration (tree drafting: eagle_model/eagle_fc/eagle_d2t, eagle_depth=3, eagle_topk), DFlashGeneration (block diffusion-style drafts: dflash_model/fc/kvmat, dflash_block_size=8). Verification = greedy longest prefix (Generation::draftVerify), KV rollback via KVMeta.reserve/spec_block.

Multimodal stages

[Verified] Omni (omni.cpp): separate mProcessorRuntimeManager from the mllm sub-config (can put encoders on a different backend/thread count); visual.mnn (mVisionModule) with per-family preprocess defaultVisionProcess (CV resize+normalize), qwen2VisionProcess (smart-resize to pixel budget, patch 14 / 16 for Qwen3-VL, window mask), gemma4VisionProcess, smolvlmVisionProcess, minicpmVisionProcess, hunyuanVisionProcess, qwenVideoProcess (fps sampling, temporal patch 2, video_max_frames=768); audio.mnn (mAudioModule) fed by whisper_fbank / conformer_fbank / usm_fbank (16 kHz; Qwen2-Audio clipped to 3000 frames), audio window mask for 2-input encoders; M-RoPE position streams (MropeInfo); prompt syntax <img>…</img>, <audio>…</audio>, <video>…</video> or MultimodalPrompt{images, audios, videos}; embeddings spliced into pad-token runs in Omni::embedding.
[Verified] Talker (§9) for speech output; Embedding/reranker (sentence_embeddings output pooled inside the graph, embedding.cpp, reranker.hpp).
[Verified] Diffusion (diffusion.hpp): DiffusionModelType {STABLE_DIFFUSION_1_5, STABLE_DIFFUSION_TAIYI_CHINESE, SANA_DIFFUSION, WAN2_1_T2V}; StableDiffusion = text_encoder → unet (PNDM/PLMS, scheduler.hpp) → vae_decoder with module release by memoryMode and Session_Resize_Check/Fix to freeze the denoise-loop resize plan; DiffusionSD35 (3 text encoders, DiT transformer, flow matching step_flow_match); SanaDiffusion (Qwen3-0.6B text encoder, DiT, flow matching, img2img; sana_diffusion.hpp); WanDiffusion (T2V, runVideo, frame dump; wan_diffusion.hpp, wan_diffusion_demo.cpp). Export: diffusion/export. Build: MNN_BUILD_DIFFUSION (forces MNN_LOW_MEMORY, MNN_SUPPORT_TRANSFORMER_FUSE, MNN_BUILD_OPENCV).

5. Scheduler

(a) Request / token / pipeline-level scheduling

Request scheduling: N/A [Verified]. One Llm object serves one conversation at a time; there is no queue, no batching across requests, no continuous batching, no prefill/decode disaggregation. mls_server and mnncli serialize HTTP requests with a std::mutex around llm->response (mls_server.cpp). Batch dim is fixed to 1 in the exported graph (input_ids [1, seq]; llmexport.py patches the batch dim).
Concurrency [Verified]: N independent Llm instances in N threads work (each has its own Executor/RuntimeManager; test_multi_instance.cpp); ExecutorScope makes the executor thread-local. Runtimes created from the global executor share the CPU runtime and thread pool (_getOrCreateRuntime). LlmContext::mutex only protects token vectors for external readers.
Prefill vs decode [Verified]: prefill = one forwardVec over all prompt embeddings (optionally chunked by size_limit block size — Llm::generate(ids) loops generate(chunk_embeds, 0)), decode = ArGeneration::generate loop of sample → is_stop → tokenizer_decode → stream to os → forwardVec({token}) (generate.cpp#L44). Stage-specific tuning: Llm::switchMode(Prefill|Decode) toggles OpenCL commit batching.
Multi-stage pipelines [Verified]: (1) Omni::responseInterleaved interleaves one thinker decode step with one talker step in a single loop (while (!thinker_done || !talker_done), omni.cpp#L2286); (2) Talker async token2wav: main thread runs the codec LM and enqueues WavChunks (60 codec tokens + 24/12 context) → ditWorkerLoop thread (own Executor::newExecutor, cloned mPreDit/mDit) → mMelQueue → vocoderWorkerLoop thread (cloned BigVGAN) → wav callback (omni.cpp#L2944). Enabled only when the talker's backend is CPU and a processor runtime distinct from the talker runtime exists; GPU backends run token2wav synchronously. Talker::finalize waits up to 60 s for the last chunk.
Cancellation / abort [Verified]: LlmStatus::USER_CANCEL exists and stops every generation loop, but the engine sets it in exactly one place — when the wav callback returns false (omni.cpp#L3403). Text decode has no cancel API; response() blocks until stop/max tokens/timeout_ms. Apps work around this by calling response(..., max_new_tokens=0) (prefill only) and stepping llm->generate(1) per token under their own flag (llm_session.cpp#L300). timeout_ms config aborts with TIMEOUT after prefill or during decode. Async resize tuning can be cancelled through Runtime::mCancelled.
Backpressure [Verified]: none for text (stream is an std::ostream*, writes are synchronous). For audio, the wav callback's return value is the only backpressure/abort signal; the async queues are unbounded std::queues.

(b) Graph / operator / thread-level scheduling

[Verified] Schedule::schedule (Schedule.cpp#L291): builds ScheduleInfo = const-tensor init (with external weight loader) + one PipelineInfo per ScheduleConfig (multi-path sessions supported via createMultiPathSession), ops kept in model order (generateScheduleGraph filters by path/saveTensors), marks INPUT/OUTPUT usage, computes needInputContentForShape, and for static models disables shape/geometry.
[Verified] Pipeline::encode → shapeComputeAndGeometryTransform (shape + geometry + const folding, with OpResizeCache for shape-change replay) → allocMemory (_createExecutions → _InsertCopy → _SetTensorBackend → per-execution onResize between onResizeBegin/End, memory planned by the Defer/Eager allocator with _recycleDynamicMemory) → execute (linear walk of executeBuffer.command calling onExecute, between onExecuteBegin/End on both backends). Op order is topological as emitted by the converter; there is no reordering or multi-stream assignment.
[Verified] Thread pools: CPU ThreadPool per affinity mask, numThread per ScheduleConfig, big/little weighting, SME2 core selection (useArmSme2Cores, divisionRatio). GPU "streams": one Metal command queue with commit every N encoders; one OpenCL command queue with optional record queue; one Vulkan submit per onExecuteEnd; no multi-stream overlap. Async: Runtime::setAsyncWork for background tuning; INIT_THREAD_NUMBER for parallel module load.

6. I/O layer

Image preprocessing [Verified]: CV::ImageProcess (ImageProcess.hpp: 15 formats incl. NV21/NV12/I420, Filter {NEAREST, BILINEAR, BICUBIC}, affine Matrix, mean/normal, convert(src → Tensor|raw), draw), kernels in ImageProcessFunction.cpp. OpenCV-like MNN::CV on VARP (tools/cv/include/cv/cv.hpp: resize/warpAffine/cvtColor/GaussianBlur/threshold/findContours/imread/imwrite/solvePnP…), MNN_BUILD_OPENCV, MNN_IMGCODECS (stb-based).
Audio preprocessing [Verified]: MNN::AUDIO (audio.hpp): load/save (wav), hamming_window/hann_window, melscale_fbanks, spectrogram, mel_spectrogram, fbank, whisper_fbank, conformer_fbank (NeMo-style), usm_fbank (Gemma4), resamplers (linear, soxr-HQ); MNN_BUILD_AUDIO. Python exposes all but the two fork additions (pymnn/src/audio.h).
Tokenization [Verified]: in-house tokenizer.cpp: tokenizer.txt with magic 430 and type SENTENCEPIECE | TIKTOIKEN | BERT | HUGGINGFACE | PIPELINE (.mtok binary: Normalizer/PreTokenizer/TokenizerModel/TokenDecoder, added tokens, byte-level BPE); special/stop tokens in the file; is_stop(); chat templates rendered by a vendored single-header Jinja2 engine jinja.hpp (jinja.cpp, not minja) with tools support; template/eos/bos read from the tokenizer file or config.json["jinja"]; stripThinkBlocks for prompt-cache stability (prompt_cache_utils.hpp).
Streaming [Verified]: text tokens are written to std::ostream* per token (apps subclass std::streambuf, e.g. llm_stream_buffer.hpp, with UTF-8 reassembly); no token callback API. Audio: setWavformCallback(bool(const float* pcm, size_t n, bool last)), chunked at ~120 mel frames / 60 codec tokens (Qwen2.5-Omni) or once at the end (Qwen3-TTS, single speech_decoder call). Perf counters in LlmContext (prefill_us, decode_us, sample_us, ttfa_us, audio_us).
Buffering [Verified]: disk-streamed embeddings; mmap weights (use_mmap → EXTERNAL_WEIGHT_DIR, mmap_size, use_cached_mmap); KV to disk (kvcache_mmap); OpenCL/Metal tuning cache file.
Postprocessing [Verified]: detokenize per token; wav writing is the caller's job (MNN::AUDIO::save(path, VARP, 24000) in qwen3_tts_demo.cpp); diffusion writes PNG via MNN::CV::imwrite.
Application boundaries [Verified]: C++ (Interpreter, Express, Llm; no C API beyond MNNSharedContext.h); Python MNN.llm (pymnn/src/llm.h: create/load/forward/generate/response/get_context/set_config/erase_history/txt_embedding/create_lora…, GIL released in response; no TTS); HTTP mls (POST /chat/completions, SSE) and mnncli (/v1/models, /v1/chat/completions, Anthropic /v1/messages); CLIs llm_demo, llm_bench, embedding_demo, reranker_demo, qwen3_tts_demo, multi_lora_demo, rollback_demo, tokenizer_demo, apply_template, llm_logits_diff (demo/); Android JNI + Kotlin (MnnLlmChat, MnnTaoAvatar), iOS ObjC++ wrapper (LLMInferenceEngineWrapper.h with setAudioWaveformCallback), engine iOS demo (engine/ios); OpenHarmony build only (no app); no WASM/JS binding (build recipe only).

7. Execution flow

(a) Text LLM: one `response()` call

sequenceDiagram
  participant App
  participant Llm as Llm (llm.cpp)
  participant Tok as Tokenizer+jinja
  participant Emb as DiskEmbedding
  participant Mod as Module pool (StaticModule/Session)
  participant Pipe as Pipeline/Backend
  participant Attn as CPUAttention+KVCacheManager
  participant Gen as ArGeneration
  participant Samp as Sampler
  App->>Llm: response(ChatMessages, os, end_with, max_new)
  Llm->>Tok: apply_chat_template(messages) -> prompt
  Llm->>Llm: prompt_cache? diff vs mCachedPromptText -> delta tokens
  Llm->>Tok: tokenizer_encode(prompt|delta)
  Llm->>Llm: generate_init(os): reset ctx; if !reuse_kv: mMeta.remove=previous
  Llm->>Emb: embedding(ids) -> VARP [seq,1,hidden]
  Llm->>Llm: forwardVec: mMeta.add=seq; gen_attention_mask; gen_position_ids
  Llm->>Mod: select key (100, all_logits) -> onForward({embeds,mask,pos,logits_index})
  Mod->>Pipe: Session resize (shape+geometry, exec cache) if dims changed; execute
  Pipe->>Attn: OpType_Attention onExecute: onAlloc(KVMeta) grows KV by expandChunk; quant K/V; flash-attn blocks
  Pipe-->>Mod: logits (last token)
  Mod-->>Llm: outputs; mMeta.sync() (previous += add - remove)
  Llm->>Gen: generate(max_new) [OpenCL: OP_ENCODER_NUMBER_FOR_COMMIT=512]
  loop until stop token / max_new / USER_CANCEL / TIMEOUT
    Gen->>Samp: sample(logits) -> token
    Gen->>Llm: is_stop(token)? (sets NORMAL_FINISHED)
    Gen->>Tok: tokenizer_decode(token)
    Gen->>App: os << piece (flush)
    Gen->>Llm: forwardVec({token}) -> module key (1,false), mMeta.add=1
    Llm->>Mod: onForward (no resize; KV append)
    Mod-->>Gen: next logits
  end
  Gen-->>Llm: status MAX_TOKENS_FINISHED | NORMAL_FINISHED
  Llm->>Llm: updateCachedPromptText (prompt_cache)
  Llm-->>App: return (output_tokens in LlmContext)

(b) TTS: Qwen2.5-Omni thinker → talker → DiT → BigVGAN (streaming)

sequenceDiagram
  participant App
  participant Omni as Omni (omni.cpp)
  participant Thinker as Llm modules (llm.mnn)
  participant Talker as Talker (talker.mnn)
  participant Q as WavChunk queues
  participant DiT as ditWorkerLoop (predit.mnn + dit.mnn, own Executor)
  participant Voc as vocoderWorkerLoop (bigvgan.mnn clone)
  App->>Omni: setWavformCallback(cb)
  App->>Omni: response(prompt, os)
  Omni->>Talker: generate_init()
  Omni->>Thinker: prefill (+vision/audio embeddings via visual.mnn/audio.mnn)
  Thinker-->>Omni: logits + talker_embeds -> Talker.addTalkerEmbeds
  loop thinker decode (text streamed to os)
    Omni->>Thinker: forwardVec(token)
    Thinker-->>Omni: talker_embeds[i]
  end
  App->>Omni: generateWavform()  (or interleaved: talker steps inside the text loop)
  Omni->>Talker: generate(): stepPrefill(concat(talker_embeds[0], text_bos+codec_pad, talker_embeds[1]+codec_bos))
  loop codec tokens (max talker_max_new_tokens) until 8292/8294
    Talker->>Talker: stepForward: embedding(prev)+talker_embeds[i]|text_pad -> forward -> sample (penalty,topK,topP,temp)
    Talker->>Q: trySubmitChunkAsync: when >= 24+60+12 tokens push WavChunk(codec_tokens, noise slice)
  end
  Talker->>Q: finalize(): push last chunk, wait mWavLastDone (<=60s)
  Q->>DiT: pop WavChunk
  DiT->>DiT: mPreDit({cond,spk,code}) -> code_embeds,rope,mask; dit_steps-1 ODE steps (Euler or RK4) of mDit({x,code_embeds,rope,mask,t}) -> mel[80, 2*codec]
  DiT->>Q: push chunk with mel (left/right padding sliced)
  Q->>Voc: pop mel chunk; concat into mMelBuffer
  Voc->>Voc: mBigvgan(mel) -> waveform; trim 8*240 samples context each side
  Voc->>App: cb(pcm_ptr, n_samples, is_last)  (return false => USER_CANCEL)
  Omni-->>App: prints talker prefill/decode, token2wav time, tts rtf

[Verified] Qwen3-TTS variant (Omni::generateTTS → Talker::generateQwen3TTS, omni.cpp#L3222): prompt embedder (codec prefix + text + tts tokens, optional ECAPA speaker embedding from --ref_audio) → per-frame loop: talker → first code (2048 + EOS) → code_predictor autoregressively over 16 groups → codec_embedder next input → after all frames one speech_decoder({codes[1,16,frames]}) → waveform → single callback with last=true. Not chunk-streamed.

8. Tests and examples

[Verified] Unit tests: test/ (op/ 103 files, expr/ 21, speed/ 17, grad/, core/, plugin/, model/, cv/, kleidiai/, sharedmem/); runner ./run_test.out [test_name] [backend 0=CPU,3=OpenCL] [precision] [thread/gpuMode] [flag] [memory] (test/main.cpp); MNN_TEST_SKIP env; attention/KV tests AttentionTest.cpp, GEMM/GEMV microbenchmarks GemmSpeed.cpp, GemvBWTest.cpp.
[Verified] Driver test.sh (static | local | android <serial> [filter]) with stages in test_stages.json (unit → lowmem → smokeA → smokeB → bench → llm, filters cpu|opencl-image|opencl-buffer|vulkan|lowmem|llm, documented driver-bug skip rationales for Mali/OpenCL-image); docs/testing.md; skill skills/test-ci/SKILL.md incl. real-device iPhone LLM bench (ios_llm_bench.sh).
[Verified] Model/tool tests: tools/cpp (MNNV2Basic, ModuleBasic, backendTest, testModel, timeProfile, GetMNNInfo, MNN2QNNModel, compilefornpu); converter round-trips tools/script (testMNNFromOnnx.py/Tf/Tflite/Torch, testPTQ.py, testTransformerC4Switches.py); pymnn pymnn/test.
[Verified] LLM: engine/test (test_tokenizer, test_omni_video_prompt, test_multi_instance), evals transformers/llm/eval (perplexity, C-Eval), llm_bench.cpp (-p 512 pp, -n 128 tg, -pg), bench_ttfa.py (time to first audio), nightly llm_nightly.sh.
[Verified] CI (.github/workflows): linux.yml builds 3 configs and runs run_test.out (incl. LLM/vision/audio/OpenCL/Vulkan-enabled build); macos.yml, windows.yml, android.yml (build-only), ios.yml, pymnn wheels, code-format.yml. No on-device GPU/NPU CI in GitHub Actions.
[Verified] Examples: demo/exec (pictureRecognition, segment, multiPose, transformerDemo, expressDemo…), benchmark/ (benchmark.out, Android script), apps: MnnLlmChat (LLM/VLM/audio-in/audio-out/diffusion, model market HF+ModelScope+Modelers), MnnTaoAvatar (LLM+ASR+TTS+A2BS+NNR avatar), iOS MNNLLMChat, apps/sana, apps/mnncli, TTS SDK demo mnn_tts/demo/android.

9. TTS relevance

What exists [Verified]

Engine-native speech output (Talker : Llm, omni.hpp):
- Qwen2.5-Omni: talker.mnn (codec-token LM conditioned on thinker hidden states), spk_dict.mnn (speakers Chelsie, Ethan), predit.mnn + dit.mnn (flow-matching DiT, dit_steps=5, dit_solver 1 = Euler / 4 = RK4, forced Precision_High on GPU), bigvgan.mnn vocoder; 50 codec tokens ≈ 1 s (repo doc); chunked streaming (60-token DiT chunks, 24/12 context, vocoder 8/8 context at 240× upsample), async two-thread token2wav on CPU, interleaved thinker/talker mode (interleaved=true), RTF reporting.
- Qwen3-TTS (talker_type=qwen3_tts): talker + code_predictor (16 groups × 2048) + codec_embedder + speech_decoder (upsample 1920) + speaker_encoder (ECAPA-TDNN; reference audio required — voice cloning only); languages auto/chinese/english/german/italian/portuguese/spanish/japanese/korean/french/russian; API Llm::generateTTS(text, language, max_new_tokens, ref_audio); demo qwen3_tts_demo.cpp.
- Export: utils/talker.py (Qwen2_5OmniTalker, Qwen3TTSTalker), utils/token2wav.py (DiTBlock, DiTAttention, DitRotary, BigVGAN anti-alias surgery, ECAPA_TDNN, Qwen2_5OmniToken2Wav, Qwen3TTSToken2Wav).
- Config keys (llmconfig.hpp#L201): has_talker, talker_model/weight/embedding_file, talker_type, predit_model, dit_model, bigvgan_model, spk_dict, talker_speaker, talker_max_new_tokens, dit_steps, dit_solver, interleaved, code_predictor_*, speech_decoder_*, speaker_encoder_*, tts_{bos,eos,pad}_token_id.
- App integration: Android LlmSession_setWavformCallbackNative writes output.wav @24 kHz and plays chunks (llm_session.cpp, AudioChunksPlayer.kt); iOS setAudioWaveformCallback; docs llm.md (AudioQueue playback example).
Native TTS SDK (apps/frameworks/mnn_tts): MNNTTSSDK::Process(text) → (sample_rate, Audio) with backends Bert-VITS2 (Chinese/English G2P, BERT front-ends, TTSGenerator), Supertonic (duration predictor / text encoder / vector estimator / vocoder, multi-voice styles), Piper (espeak-ng phonemizer; currently commented out); Android service + JNI, iOS, desktop. Shipped models bert-vits2-MNN (1.39 GB) and supertonic-tts-mnn (464 MB) in model_market.json.
sherpa-mnn (apps/frameworks/sherpa-mnn): sherpa-onnx ported onto MNN: offline TTS VITS / Kokoro / Matcha + HiFiGAN, MeloTTS/Piper lexicons, streaming Zipformer ASR, VAD, speaker ID, C/Kotlin/Python APIs. Used by both Android apps for voice chat (ASR in, TTS out).
Audio input (ASR side): audio.mnn encoders for Qwen2-Audio, Qwen3-ASR (qwen3_asr), Qwen2.5-Omni audio, FunAudioChat, LFM2-Audio, Gemma4 audio (utils/audio.py); MNN::AUDIO front-ends (whisper/conformer/usm); Whisper itself is not a shipped pipeline (only its fbank).
Ops: OpType_Stft (shape/CPU kernel, ShapeStft.cpp); everything else (mel, iSTFT, vocoder convs/transposed convs/snake activations) is expressed as ordinary graph ops after export.

What's missing [Verified unless noted]

No CosyVoice / Fish-Speech / F5-TTS / SparkTTS / ChatTTS / Kokoro-native (only via sherpa) / SNAC / EnCodec / DAC / WavTokenizer; no Qwen3-Omni.
Qwen3-TTS path is not chunk-streamed (single speech_decoder call) and has no preset speakers (ref audio mandatory).
No sample-rate conversion on output; fixed 24 kHz assumption in demos.
No engine-level audio backpressure/queue bounds; no USER_CANCEL for text other than via wav callback.
Python binding lacks setWavformCallback/generateTTS.
[Proposal] For vLLM-Omni-style stage separation, the natural seams are Talker::stepForward (codec LM) and the WavChunk queues (DiT / vocoder) — they are already separate Modules with their own RuntimeManager/Executor, so they could be placed on CPU vs GPU or different cores.

10. VLA relevance