[Refactor]: Mammothmoda2-dev use x_to_text.py in examples (#5454)
Signed-off-by: princepride <wangzhipeng628@gmail.com>
This commit is contained in:
2 files changed
+26
-40
No files matched your search
@@ -11,6 +11,7 @@ Generate text from text or image inputs with vLLM-Omni's shared offline entrypoi
|
||||
| `ByteDance-Seed/BAGEL-7B-MoT` | Yes | Yes | `vllm_omni/deploy/bagel.yaml` |
|
||||
| `tencent/HunyuanImage-3.0-Instruct` | Yes | Yes | `vllm_omni/deploy/hunyuan_image3_ar.yaml` |
|
||||
| `bytedance-research/MammothModa2-Preview` | Yes | Yes | `vllm_omni/deploy/mammoth_moda2_ar.yaml` |
|
||||
| `bytedance-research/MammothModa2-Dev` | Yes | Yes | `vllm_omni/deploy/mammoth_moda2_ar.yaml` |
|
||||
|
||||
The script recognizes these model families from `config.json`, applies the
|
||||
model-specific prompt format, and selects an AR-only deploy for HunyuanImage-3
|
||||
@@ -42,6 +43,14 @@ python examples/offline_inference/x_to_text/x_to_text.py \
|
||||
--prompt "Explain multimodal inference in three concise sentences."
|
||||
```
|
||||
|
||||
The same command supports the Dev checkpoint:
|
||||
|
||||
```bash
|
||||
python examples/offline_inference/x_to_text/x_to_text.py \
|
||||
--model bytedance-research/MammothModa2-Dev \
|
||||
--prompt "Explain multimodal inference in three concise sentences."
|
||||
```
|
||||
|
||||
## Image-To-Text
|
||||
|
||||
Pass one image with `--image`. The default question can be replaced with any
|
||||
@@ -70,6 +79,9 @@ python examples/offline_inference/x_to_text/x_to_text.py \
|
||||
--prompt "Describe this image in detail."
|
||||
```
|
||||
|
||||
Use `bytedance-research/MammothModa2-Dev` in the same command to run I2T with
|
||||
the Dev checkpoint.
|
||||
|
||||
## Key Arguments
|
||||
|
||||
| Argument | Default | Description |
|
||||
|
||||
@@ -149,49 +149,23 @@ Download the checkpoint:
|
||||
hf download bytedance-research/MammothModa2-Dev --local-dir ./MammothModa2-Dev
|
||||
```
|
||||
|
||||
Run image-to-text inference from the repository root:
|
||||
Run text-to-text through the shared understanding example. It recognizes the
|
||||
Dev checkpoint as MammothModa2 and automatically selects
|
||||
`mammoth_moda2_ar.yaml`:
|
||||
|
||||
```python
|
||||
from PIL import Image
|
||||
from vllm import SamplingParams
|
||||
from vllm.multimodal.image import convert_image_mode
|
||||
from vllm_omni import Omni
|
||||
```bash
|
||||
python examples/offline_inference/x_to_text/x_to_text.py \
|
||||
--model ./MammothModa2-Dev \
|
||||
--prompt "Explain multimodal generation in three sentences."
|
||||
```
|
||||
|
||||
prompt = (
|
||||
"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
|
||||
"<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>"
|
||||
"Summarize this image.<|im_end|>\n"
|
||||
"<|im_start|>assistant\n"
|
||||
)
|
||||
Add an image for image-to-text or image summarization:
|
||||
|
||||
omni = Omni(
|
||||
model="./MammothModa2-Dev",
|
||||
deploy_config="vllm_omni/deploy/mammoth_moda2_ar.yaml",
|
||||
)
|
||||
try:
|
||||
outputs = list(
|
||||
omni.generate(
|
||||
[{
|
||||
"prompt": prompt,
|
||||
"multi_modal_data": {
|
||||
"image": convert_image_mode(Image.open("./image.png"), "RGB"),
|
||||
},
|
||||
"additional_information": {"omni_task": ["chat"]},
|
||||
}],
|
||||
[SamplingParams(
|
||||
temperature=0.2,
|
||||
top_p=0.9,
|
||||
top_k=-1,
|
||||
max_tokens=512,
|
||||
seed=42,
|
||||
)],
|
||||
)
|
||||
)
|
||||
finally:
|
||||
omni.close()
|
||||
|
||||
request_output = getattr(outputs[-1], "request_output", outputs[-1])
|
||||
print(request_output.outputs[0].text.strip())
|
||||
```bash
|
||||
python examples/offline_inference/x_to_text/x_to_text.py \
|
||||
--model ./MammothModa2-Dev \
|
||||
--image ./image.png \
|
||||
--prompt "Please summarize the content of this image."
|
||||
```
|
||||
|
||||
The Dev checkpoint is approximately 47.55 GiB on disk. In the verified AR-only
|
||||
|
||||
Reference in new issue
Block a user