vllm-omni/recipes/MammothModa2/MammothModa2-Preview.md

MammothModa2-Preview

MammothModa2-Preview text-to-image generation through the shared offline image example

Summary

• Vendor: ByteDance Research
• Model: bytedance-research/MammothModa2-Preview
• Task: Text-to-image generation (AR → DiT two-stage pipeline), plus image-to-text understanding (AR stage)
• Mode: Offline inference
• Maintainer: Community

When to use this recipe

Use this recipe to run MammothModa2-Preview text-to-image through the shared offline image example (text_to_image.py) instead of a model-specific script. The generic example formats the AR prompt, drives the AR → DiT stage pipeline, and forwards MammothModa2-specific generation parameters through the pipeline-declared extra_body contract.

MammothModa2's DiT stage consumes its inputs through the multi-stage kwargs interface (not OmniDiffusionRequest), so its generation knobs (text_guidance_scale, cfg_range, num_inference_steps) are passed via --extra-body rather than the standard --num-inference-steps / --cfg-scale flags. Image size uses the standard --height / --width flags.

References

• Upstream model: bytedance-research/MammothModa2-Preview
• Related offline example: examples/offline_inference/text_to_image/text_to_image.py
• Declared parameters: vllm_omni/model_extras/mammothmodal2_preview.py
• Deploy config: vllm_omni/deploy/mammoth_moda2.yaml

Hardware Support

The default deploy config runs both the AR and DiT stages on a single GPU (devices: "0"). The committed gpu_memory_utilization split (stage-0 AR 0.5, stage-1 DiT 0.3) is sized for an ~80 GB GPU. The model also fits on a 48 GB GPU after rebalancing the split so the AR weights (~23 GB) leave room for the KV cache — see the note under 1x L40S 48GB.

GPU

1x L40S 48GB

48 GB config adjustment: the committed vllm_omni/deploy/mammoth_moda2.yaml uses gpu_memory_utilization 0.5 / 0.3 (sized for ~80 GB). To fit on a 48 GB L40S, set the stage-0 (AR) value to 0.8 and the stage-1 (DiT) value to 0.16 before running. (On an ~80 GB GPU, leave the defaults unchanged.)

Environment

• OS: Linux
• Python: Match the repository requirements for your checkout
• Driver / runtime: NVIDIA CUDA environment with one L40S 48 GB (verified) or an ~80 GB GPU for the default config
• vLLM version: Match the repository requirements for your checkout
• vLLM-Omni version or commit: Use the commit you are deploying from

Offline Commands

Download the model:

hf download bytedance-research/MammothModa2-Preview --local-dir ./MammothModa2-Preview

Run text-to-image with the shared offline example from the repository root. The deploy config sets trust_remote_code, so no extra flag is needed. Forward the MammothModa2 generation parameters as a JSON object through --extra-body:

python examples/offline_inference/text_to_image/text_to_image.py \
  --model ./MammothModa2-Preview \
  --stage-configs-path vllm_omni/deploy/mammoth_moda2.yaml \
  --prompt "A stylish woman riding a motorcycle in NYC, movie poster style" \
  --height 1024 \
  --width 1024 \
  --extra-body '{"text_guidance_scale": 4.0, "cfg_range": [0.0, 1.0], "num_inference_steps": 50}' \
  --output mammoth_t2i.png

The --extra-body JSON forwards MammothModa2-specific parameters into OmniDiffusionSamplingParams.extra_args. Keys are filtered against the model's declared extra_body_params (see vllm_omni/model_extras/mammothmodal2_preview.py), so unknown keys for MammothModa2 are silently dropped:

• text_guidance_scale — classifier-free guidance scale for the DiT stage (default 9.0; CFG is active only when > 1.0).
• cfg_range — relative step range [start, end] over which CFG is applied (default [0.0, 1.0]).
• num_inference_steps — number of DiT denoising steps (default 50).

--height and --width must be multiples of 16.

Verification

The example writes the generated image to the --output path. Confirm the file exists and is a valid image:

ls -lh mammoth_t2i.png
python -c "from PIL import Image; print(Image.open('mammoth_t2i.png').size)"

Image-to-Text (I2T, simplest)

For the simplest offline image-understanding run, use a short inline Python snippet from the repository root:

from PIL import Image
from vllm import SamplingParams
from vllm.multimodal.image import convert_image_mode
from vllm_omni import Omni

def main():
    prompt = (
        "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
        "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>"
        "Summarize this image.<|im_end|>\n"
        "<|im_start|>assistant\n"
    )

    omni = Omni(
        model="./MammothModa2-Preview",
        deploy_config="vllm_omni/deploy/mammoth_moda2_ar.yaml",
    )
    try:
        outputs = list(
            omni.generate(
                [{
                    "prompt": prompt,
                    "multi_modal_data": {"image": convert_image_mode(Image.open("./image.png"), "RGB")},
                    "additional_information": {"omni_task": ["chat"]},
                }],
                [SamplingParams(temperature=0.2, top_p=0.9, top_k=-1, max_tokens=512, seed=42)],
            )
        )
    finally:
        omni.close()

    ro = getattr(outputs[-1], "request_output", outputs[-1])
    print(ro.outputs[0].text.strip())

if __name__ == "__main__":
    main()