vllm-omni/benchmarks/mammoth_moda2/README.md

MammothModa2 startup benchmark

Measures Omni() initialization and first/subsequent image requests for MammothModa2-Preview. This is a measurement baseline for #7075; it changes no runtime code.

Run

From the repository root, with the model downloaded and vLLM-Omni installed:

set -o pipefail
python benchmarks/mammoth_moda2/bench_startup.py \
    --model /path/MammothModa2-Preview \
    --deploy-config vllm_omni/deploy/mammoth_moda2.yaml \
    --height 512 --width 512 --seed 42 \
    --extra-body '{"text_guidance_scale": 4.0, "cfg_range": [0.0, 1.0], "num_inference_steps": 8}' \
    --repeat 1 --label baseline --output-json baseline.json --save-image baseline.png \
    2>&1 | tee baseline.log
python benchmarks/mammoth_moda2/parse_startup_log.py baseline.log --require-complete --markdown

Use OMP_NUM_THREADS=16 to set CPU threads and --parallel-stage-init to initialize stages concurrently. For first/subsequent requests, use --height 1024 --width 1024 --repeat 3 and 50 inference steps. Memory polling is opt-in (--sample-memory); collect it in separate runs to avoid perturbing the timing comparison. --save-image saves only the first request's image. Compare that image across runs; a fixed seed alone does not guarantee equality across execution modes or software versions.

Measurement boundaries

Metric	Boundary / limitation
Imports	Before torch import → benchmark imports complete; excludes interpreter startup
Time to ready	Omni() call → return; excludes client imports and requests
Time to first image	Benchmark clock start → first generate() return, including request setup
First / subsequent request	Each generate() call → return; same prompt and settings
Stage spawn / device init	Log timestamps: launch → world_size= → Starting to load model; 1 s resolution
Weight load	Engine-reported Loading weights took; read/cast/copy are merged
Model setup estimate	Reported model-load time minus weight-load time; not independently instrumented
Profile + KV + warmup	One merged interval after loading; compilation/capture may be nested within it
Compile / graph capture	Explicit engine log durations when present; N/A for eager execution
Memory	Optional 1 s samples: sum of process-tree RSS and maximum device memory used across GPUs reported by nvidia-smi; includes other GPU users and can miss transient peaks

Stage timelines overlap with parallel initialization; do not sum them to calculate time to ready. Missing log fields stay unmeasured. The parser marks logs without BENCH_JSON as incomplete; --require-complete makes this an error.

Reference results

A800-SXM4-80GB; Ubuntu 22.04; 16-CPU cgroup quota on a 128-core host; vLLM 0.28.0+cu129, torch 2.13.0+cu129, vLLM-Omni 34aa1d2a. Checkpoint: 34.5 GiB, 8 shards, mostly fp32, on local NVMe. Committed eager deploy config, GPU memory fractions 0.5 / 0.3.

The 2026-09-10 rerun used the corrected benchmark: one warm-up process, then three rounds of four configurations with order rotated between rounds. The checkpoint page cache was warmed before the matrix; no cache eviction was requested during it. Each fresh process sent one 512×512, 8-step request (guidance 4.0, seed 42). Memory polling was disabled. All 12 measured runs are retained in the results. Values below are Omni() seconds, mean ± sample standard deviation, n=3; client imports took approximately another 10 s.

CPU threads	Serial init	Parallel init
Default (64)	60.1 ± 0.2	41.1 ± 1.0
16	56.5 ± 0.9	36.7 ± 0.2

On this configuration, parallel initialization reduced time to ready by about 32%; combining it with 16 threads reduced it by about 39%. This supports trying these settings on similar deployments, not changing defaults for all models or hardware. Fixed 512×512 samples matched across all 12 measured runs; this is a smoke check, not a broad quality evaluation.

For context, in the earlier serial baseline AR / DiT spent approximately 11 / 11 s in spawn and imports, 6 / 6 s in device setup, 6 / 4 s loading weights, and 5 / 2 s in profile + KV + warmup. About 6 s followed the last stage becoming ready. These coarse log intervals locate work; they are not independent operator timings.

Separate observations from the earlier experiment (not rerun in this matrix):

• Requested page-cache eviction: 80.8 s; FlashInfer JIT cache removed: 129.1 s (means, n=3 each). These are separate cache conditions, not guaranteed cold-machine startup or isolated disk/JIT costs.
• Three 1024×1024, 50-step eager requests in one process took 95.2 / 94.9 / 94.6 s. No large first-request penalty was observed in that run.
• With AR enforce_eager: false and serial init, time to ready was 62.5 ± 0.1 s (warm caches, n=3); logged graph capture was 4 s / 0.20 GiB. The model reported torch.compile unsupported. KV capacity fell from about 15.7 to 2.0 GiB, and one parallel-init trial failed. The reported non-KV profiling difference is not a measurement of graph allocation. Graph-mode samples differed from eager. This needs separate memory and quality analysis before recommending the configuration.

Loading diagnostic and storage scenarios

OMP_NUM_THREADS=16 python benchmarks/mammoth_moda2/raw_load_bench.py \
    --model /path/MammothModa2-Preview --shards 6,7,8 --mode cpu_cast
# Compare --mode gpu_cast in a fresh process with the same cache conditions.

MODEL_LOCAL=/local/MammothModa2-Preview OUT_DIR=./startup-bench \
    bash benchmarks/mammoth_moda2/bench_storage_scenarios.sh
# Optional MODEL_NFS=/nfs/MammothModa2-Preview; SCENARIOS="local-cold local-warm".

The diagnostic clones each mmap-backed tensor into owned CPU memory before casting. materialize_s includes page faults, CPU allocation and copying; cast_s measures dtype conversion; h2d_s includes GPU allocation and copy. CUDA context/operation warmup is outside the timer. Total also includes file opening and bookkeeping. This extra CPU copy differs from the engine loader, so these numbers do not decompose its loading time or measure disk bandwidth. The revised diagnostic was run three times each for CPU cast with 64 / 16 threads and GPU cast with 16 threads; per-run values are in the results file. Both CPU materialization and conversion were faster with 16 threads here; this does not isolate how much of the engine-level improvement comes from either.

Storage scenarios explicitly read shards before warm runs and request posix_fadvise(DONTNEED) before cold runs. Eviction is not guaranteed, and server-side NFS caches are uncontrolled. The script stops on failure and summarizes only logs from the current invocation.