[Model] Support MammothModa2-Dev (#5411)
Signed-off-by: princepride <wangzhipeng628@gmail.com>
This commit is contained in:
13 files changed
+525
-64
No files matched your search
+61
-37
@@ -1,13 +1,12 @@
|
||||
# MammothModa2-Preview
|
||||
# MammothModa2
|
||||
|
||||
> MammothModa2-Preview generation and understanding through shared offline examples
|
||||
> MammothModa2-Preview and MammothModa2-Dev unified understanding and generation
|
||||
|
||||
## Summary
|
||||
|
||||
- Vendor: ByteDance Research
|
||||
- Model: `bytedance-research/MammothModa2-Preview`
|
||||
- Task: Text-to-image generation (AR → DiT two-stage pipeline), plus
|
||||
image-to-text understanding (AR stage)
|
||||
- Models: `bytedance-research/MammothModa2-Preview`, `bytedance-research/MammothModa2-Dev`
|
||||
- Tasks: Preview and Dev text-to-image (AR → DiT); Dev text/image understanding
|
||||
- Mode: Offline inference
|
||||
- Maintainer: Community
|
||||
|
||||
@@ -29,7 +28,9 @@ flags. Image size uses the standard `--height` / `--width` flags.
|
||||
|
||||
- Upstream model:
|
||||
[`bytedance-research/MammothModa2-Preview`](https://huggingface.co/bytedance-research/MammothModa2-Preview)
|
||||
- Related text-to-image example:
|
||||
- Dev model:
|
||||
[`bytedance-research/MammothModa2-Dev`](https://huggingface.co/bytedance-research/MammothModa2-Dev)
|
||||
- Related offline example:
|
||||
[`examples/offline_inference/text_to_image/text_to_image.py`](../../examples/offline_inference/text_to_image/text_to_image.py)
|
||||
- Related T2T/I2T example:
|
||||
[`examples/offline_inference/x_to_text/x_to_text.py`](../../examples/offline_inference/x_to_text/x_to_text.py)
|
||||
@@ -131,10 +132,24 @@ ls -lh mammoth_t2i.png
|
||||
python -c "from PIL import Image; print(Image.open('mammoth_t2i.png').size)"
|
||||
```
|
||||
|
||||
### Image-to-Text (I2T, simplest)
|
||||
## MammothModa2-Dev unified inference
|
||||
|
||||
For the simplest offline image-understanding run, use a short inline Python
|
||||
snippet from the repository root:
|
||||
MammothModa2-Dev uses a Qwen3-VL AR backbone, while MammothModa2-Preview uses
|
||||
Qwen2.5-VL. vLLM-Omni selects the matching implementation from the nested
|
||||
`llm_config.model_type`; no checkpoint edits or `trust_remote_code` flag are
|
||||
required.
|
||||
|
||||
Text-to-text and image-to-text use the AR-only deploy. Text-to-image loads the
|
||||
Qwen3 generation experts (`gen_mlp`), extra visual vocabulary and image head,
|
||||
then sends the generated visual tokens and hidden states to the DiT stage.
|
||||
|
||||
Download the checkpoint:
|
||||
|
||||
```bash
|
||||
hf download bytedance-research/MammothModa2-Dev --local-dir ./MammothModa2-Dev
|
||||
```
|
||||
|
||||
Run image-to-text inference from the repository root:
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
@@ -142,35 +157,44 @@ from vllm import SamplingParams
|
||||
from vllm.multimodal.image import convert_image_mode
|
||||
from vllm_omni import Omni
|
||||
|
||||
def main():
|
||||
prompt = (
|
||||
"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
|
||||
"<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>"
|
||||
"Summarize this image.<|im_end|>\n"
|
||||
"<|im_start|>assistant\n"
|
||||
)
|
||||
prompt = (
|
||||
"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
|
||||
"<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>"
|
||||
"Summarize this image.<|im_end|>\n"
|
||||
"<|im_start|>assistant\n"
|
||||
)
|
||||
|
||||
omni = Omni(
|
||||
model="./MammothModa2-Preview",
|
||||
deploy_config="vllm_omni/deploy/mammoth_moda2_ar.yaml",
|
||||
)
|
||||
try:
|
||||
outputs = list(
|
||||
omni.generate(
|
||||
[{
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {"image": convert_image_mode(Image.open("./image.png"), "RGB")},
|
||||
"additional_information": {"omni_task": ["chat"]},
|
||||
}],
|
||||
[SamplingParams(temperature=0.2, top_p=0.9, top_k=-1, max_tokens=512, seed=42)],
|
||||
)
|
||||
omni = Omni(
|
||||
model="./MammothModa2-Dev",
|
||||
deploy_config="vllm_omni/deploy/mammoth_moda2_ar.yaml",
|
||||
)
|
||||
try:
|
||||
outputs = list(
|
||||
omni.generate(
|
||||
[{
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {
|
||||
"image": convert_image_mode(Image.open("./image.png"), "RGB"),
|
||||
},
|
||||
"additional_information": {"omni_task": ["chat"]},
|
||||
}],
|
||||
[SamplingParams(
|
||||
temperature=0.2,
|
||||
top_p=0.9,
|
||||
top_k=-1,
|
||||
max_tokens=512,
|
||||
seed=42,
|
||||
)],
|
||||
)
|
||||
finally:
|
||||
omni.close()
|
||||
)
|
||||
finally:
|
||||
omni.close()
|
||||
|
||||
ro = getattr(outputs[-1], "request_output", outputs[-1])
|
||||
print(ro.outputs[0].text.strip())
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
request_output = getattr(outputs[-1], "request_output", outputs[-1])
|
||||
print(request_output.outputs[0].text.strip())
|
||||
```
|
||||
|
||||
The Dev checkpoint is approximately 47.55 GiB on disk. In the verified AR-only
|
||||
run, loaded model weights used approximately 16.97 GiB of GPU memory before KV
|
||||
and encoder caches. Allow additional GPU memory for those caches and the input
|
||||
image.
|
||||
Reference in new issue
Block a user