anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
45a3e77351b9ef2b36612159bb0d5a3df812466b
vllm-omni/recipes/MammothModa2/MammothModa2.md
T

10 KiB
Raw Blame History

MammothModa2

MammothModa2-Preview and MammothModa2-Dev unified understanding and generation

Summary

  • Vendor: ByteDance Research
  • Models: bytedance-research/MammothModa2-Preview, bytedance-research/MammothModa2-Dev
  • Tasks: Preview and Dev text-to-image (AR → DiT); Dev text/image understanding
  • Mode: Offline inference
  • Maintainer: Community

When to use this recipe

Use this recipe to run MammothModa2-Preview through shared task-oriented offline examples. Text-to-image uses the shared image example (text_to_image.py) instead of a model-specific script. The generic example formats the AR prompt, drives the AR → DiT stage pipeline, and forwards MammothModa2-specific generation parameters through the pipeline-declared extra_body contract.

MammothModa2's DiT stage consumes its inputs through the multi-stage kwargs interface (not OmniDiffusionRequest), so its generation knobs (text_guidance_scale, cfg_range, num_inference_steps) are passed via --extra-body rather than the standard --num-inference-steps / --cfg-scale flags. Image size uses the standard --height / --width flags.

References

Hardware Support

The default deploy config runs both the AR and DiT stages on a single GPU (devices: "0"). The committed gpu_memory_utilization split (stage-0 AR 0.5, stage-1 DiT 0.3) is sized for an ~80 GB GPU. The model also fits on a 48 GB GPU after rebalancing the split so the AR weights (~23 GB) leave room for the KV cache — see the note under 1x L40S 48GB.

GPU

Optional FP8 AR KV cache

For CUDA deployments, mammoth_moda2_fp8_kv.yaml is an opt-in preset that stores the Stage 0 AR KV cache as FP8 E4M3. Stage 1 remains on kv_cache_dtype=auto; its DiT execution is unaffected. This setting quantizes only the autoregressive KV cache. It is neither FP8 weight/activation quantization nor vLLM-Omni diffusion KV-cache quantization.

Use the preset in place of the default deploy config:

python examples/offline_inference/text_to_image/text_to_image.py \
  --model ./MammothModa2-Preview \
  --deploy-config vllm_omni/deploy/mammoth_moda2_fp8_kv.yaml \
  --prompt "A stylish woman riding a motorcycle in NYC, movie poster style" \
  --height 1024 \
  --width 1024 \
  --seed 42 \
  --extra-body '{"text_guidance_scale": 4.0, "cfg_range": [0.0, 1.0], "num_inference_steps": 50}' \
  --output mammoth_t2i.png

The preset was validated on one NVIDIA H800 80GB with CUDA and FlashAttention 3. The native and FP8 runs used the same model, code revision, and downstream configuration.

Metric Native BF16 (kv_cache_dtype=auto) FP8 E4M3 (fp8_e4m3) Change
KV cache memory 15.59 GiB 15.55 GiB -0.04 GiB
GPU KV cache size 145,904 tokens 291,232 tokens +99.6%
Maximum concurrency at 8,192 tokens 17.81x 35.55x +99.6%
Steady-state AR median 71.191 s 79.816 s +12.1%
Steady-state end-to-end median 83.840 s 92.473 s +10.3%
Steady-state DiT median 12.574 s 12.561 s effectively unchanged

The FP8 run used kv_cache_dtype=fp8_e4m3 only for Stage 0; Stage 1 used kv_cache_dtype=auto. The nearly unchanged reserved cache memory holds almost twice as many tokens because FP8 reduces the bytes per cached token. This is a capacity/concurrency tradeoff: the measured AR and end-to-end latencies were higher than the native-BF16 baseline.

1024x1024 fixed-seed smoke test completed successfully with no obvious visual failure. FP8 is lossy, so numerical or image-quality equivalence with BF16 is not implied.

1x L40S 48GB

48 GB config adjustment: the committed vllm_omni/deploy/mammoth_moda2.yaml uses gpu_memory_utilization 0.5 / 0.3 (sized for ~80 GB). To fit on a 48 GB L40S, set the stage-0 (AR) value to 0.8 and the stage-1 (DiT) value to 0.16 before running. (On an ~80 GB GPU, leave the defaults unchanged.)

Environment

  • OS: Linux
  • Python: Match the repository requirements for your checkout
  • Driver / runtime: NVIDIA CUDA environment with one L40S 48 GB (verified) or an ~80 GB GPU for the default config
  • vLLM version: Match the repository requirements for your checkout
  • vLLM-Omni version or commit: Use the commit you are deploying from

Offline Commands

Download the model:

hf download bytedance-research/MammothModa2-Preview --local-dir ./MammothModa2-Preview

Run text-to-image with the shared offline example from the repository root. The deploy config sets trust_remote_code, so no extra flag is needed. Forward the MammothModa2 generation parameters as a JSON object through --extra-body:

python examples/offline_inference/text_to_image/text_to_image.py \
  --model ./MammothModa2-Preview \
  --deploy-config vllm_omni/deploy/mammoth_moda2.yaml \
  --prompt "A stylish woman riding a motorcycle in NYC, movie poster style" \
  --height 1024 \
  --width 1024 \
  --extra-body '{"text_guidance_scale": 4.0, "cfg_range": [0.0, 1.0], "num_inference_steps": 50}' \
  --output mammoth_t2i.png

The --extra-body JSON forwards MammothModa2-specific parameters into OmniDiffusionSamplingParams.extra_args. Keys are filtered against the model's declared extra_body_params (see vllm_omni/model_extras/mammothmodal2_preview.py), so unknown keys for MammothModa2 are silently dropped:

  • text_guidance_scale — classifier-free guidance scale for the DiT stage (default 9.0; CFG is active only when > 1.0).
  • cfg_range — relative step range [start, end] over which CFG is applied (default [0.0, 1.0]).
  • num_inference_steps — number of DiT denoising steps (default 50).

--height and --width must be multiples of 16.

Run text-to-text through the shared understanding example. It recognizes the MammothModa2 checkpoint and automatically selects mammoth_moda2_ar.yaml:

python examples/offline_inference/x_to_text/x_to_text.py \
  --model ./MammothModa2-Preview \
  --prompt "Explain multimodal generation in three sentences."

Add an image for image-to-text or image summarization. The shared example uses MammothModa2's chat and vision-token template:

python examples/offline_inference/x_to_text/x_to_text.py \
  --model ./MammothModa2-Preview \
  --image /path/to/input.jpg \
  --prompt "Please summarize the content of this image."

Verification

The example writes the generated image to the --output path. Confirm the file exists and is a valid image:

ls -lh mammoth_t2i.png
python -c "from PIL import Image; print(Image.open('mammoth_t2i.png').size)"

1x AMD MI300X, MammothModa2 Preview

Environment

  • OS: Linux 6.8.0-134-generic, x86_64
  • Container: official ROCm image built from docker/Dockerfile.rocm
  • Python: 3.12.13
  • PyTorch: 2.11.0+gitd0c8b1f
  • Driver / runtime: AMD 6.19.14.31400000 / ROCm 7.2.53211
  • GPU: one AMD Instinct MI300X, gfx942:sramecc+:xnack-, 191.69 GiB visible HBM
  • vLLM version: 0.27.0+rocm723
  • vLLM Omni version or commit: 73e1368c7bb940efe1a025859c9d6c8eeeb2e3f0
  • Installed vLLM Omni package metadata: 0.27.0rc2.dev44+g55abdade9.rocm

Offline Commands

The checked run used the committed stage split, with gpu_memory_utilization set to 0.5 for AR and 0.3 for DiT:

python3 examples/offline_inference/text_to_image/text_to_image.py \
    --model bytedance-research/MammothModa2-Preview \
    --deploy-config vllm_omni/deploy/mammoth_moda2.yaml \
    --prompt "A stylish woman riding a motorcycle in NYC, movie poster style" \
    --height 1024 \
    --width 1024 \
    --seed 42 \
    --extra-body '{"text_guidance_scale": 4.0, "cfg_range": [0.0, 1.0], "num_inference_steps": 50}' \
    --enable-diffusion-pipeline-profiler \
    --log-stats \
    --output mammoth_t2i.png

Verification

The first request took 85.224 seconds. The AR stage generated 4,161 visual tokens in 72.996 seconds, and the DiT stage took 12.163 seconds. AR weight loading used 21.4 GiB and took 8.250 seconds. DiT weight loading used 5.49 GiB and took 1.824 seconds. The largest one second whole device memory sample was 106.57 GiB, including the AR KV cache reserved by the 0.5 memory setting.

The output was a valid 1024 by 1024 RGB PNG.

MammothModa2-Dev unified inference

MammothModa2-Dev uses a Qwen3-VL AR backbone, while MammothModa2-Preview uses Qwen2.5-VL. vLLM-Omni selects the matching implementation from the nested llm_config.model_type; no checkpoint edits or trust_remote_code flag are required.

Text-to-text and image-to-text use the AR-only deploy. Text-to-image loads the Qwen3 generation experts (gen_mlp), extra visual vocabulary and image head, then sends the generated visual tokens and hidden states to the DiT stage.

Download the checkpoint:

hf download bytedance-research/MammothModa2-Dev --local-dir ./MammothModa2-Dev

Run text-to-text through the shared understanding example. It recognizes the Dev checkpoint as MammothModa2 and automatically selects mammoth_moda2_ar.yaml:

python examples/offline_inference/x_to_text/x_to_text.py \
  --model ./MammothModa2-Dev \
  --prompt "Explain multimodal generation in three sentences."

Add an image for image-to-text or image summarization:

python examples/offline_inference/x_to_text/x_to_text.py \
  --model ./MammothModa2-Dev \
  --image ./image.png \
  --prompt "Please summarize the content of this image."

The Dev checkpoint is approximately 47.55 GiB on disk. In the verified AR-only run, loaded model weights used approximately 16.97 GiB of GPU memory before KV and encoder caches. Allow additional GPU memory for those caches and the input image.

Reference in new issue View Git Blame Copy Permalink