92 lines
2.4 KiB
Markdown
92 lines
2.4 KiB
Markdown
# MiniCPM-o 4.5: Offline inference
|
|
|
|
Two-stage pipeline: **thinker** (multimodal understanding) → **talker + Token2Wav**
|
|
(24 kHz speech). Deploy config auto-loads from
|
|
`vllm_omni/deploy/minicpmo_4_5.yaml` (2-GPU default).
|
|
|
|
## Setup
|
|
|
|
- Install talker deps: `pip install stepaudio2-minicpmo`
|
|
- `--trust-remote-code` is always passed by `end2end.py` (required for MiniCPMO)
|
|
- See [pipeline and deploy configuration docs](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) for memory tuning
|
|
|
|
## Run examples
|
|
|
|
### Single prompt
|
|
|
|
```bash
|
|
cd examples/offline_inference/minicpmo
|
|
bash run_single_prompt.sh
|
|
```
|
|
|
|
### Multiple prompts
|
|
|
|
```bash
|
|
bash run_multiple_prompts.sh
|
|
```
|
|
|
|
### Thinker tensor parallel (3-GPU)
|
|
|
|
```bash
|
|
bash run_single_prompt_tp.sh
|
|
```
|
|
|
|
### Modality control
|
|
|
|
Text-only (skip talker / no `<|tts_bos|>`):
|
|
|
|
```bash
|
|
python end2end.py --query-type use_audio --modalities text
|
|
```
|
|
|
|
Text + speech (default — appends `<|tts_bos|>`):
|
|
|
|
```bash
|
|
python end2end.py --query-type text --modalities text,audio
|
|
```
|
|
|
|
### Local media files
|
|
|
|
```bash
|
|
python end2end.py --query-type use_video --video-path /path/to/video.mp4
|
|
python end2end.py --query-type use_image --image-path /path/to/image.jpg
|
|
python end2end.py --query-type use_audio --audio-path /path/to/audio.wav
|
|
python end2end.py --query-type use_mixed_modalities \
|
|
--video-path /path/to/video.mp4 \
|
|
--image-path /path/to/image.jpg \
|
|
--audio-path /path/to/audio.wav
|
|
```
|
|
|
|
Supported `--query-type` values:
|
|
|
|
| Query type | Inputs |
|
|
|---|---|
|
|
| `text` | Text only |
|
|
| `use_image` | Image + text |
|
|
| `use_audio` | Audio + text |
|
|
| `use_video` | Video + text |
|
|
| `use_multi_audios` | Two audio clips |
|
|
| `use_mixed_modalities` | Audio + image + video |
|
|
|
|
### Custom deploy config
|
|
|
|
```bash
|
|
python end2end.py --query-type text \
|
|
--deploy-config /path/to/vllm_omni/deploy/minicpmo_4_5_8x4090.yaml
|
|
```
|
|
|
|
## Notes
|
|
|
|
- Speech requires `<|tts_bos|>` on the assistant prefix (offline equivalent of
|
|
online `chat_template_kwargs.use_tts_template=true`). Without it, the talker
|
|
gets an empty TTS span.
|
|
- Output WAV is **24 kHz mono**.
|
|
- Placeholders in the prompt are MiniCPM-style:
|
|
`(<image>./</image>)`, `(<audio>./</audio>)`, `(<video>./</video>)`.
|
|
- Default layout needs **2 GPUs**. Async chunking is off in the bundled YAMLs.
|
|
|
|
## Online serving
|
|
|
|
See [`examples/online_serving/minicpmo/`](../../online_serving/minicpmo/) and
|
|
the recipe [`recipes/OpenBMB/MiniCPM-o-4_5.md`](../../../recipes/OpenBMB/MiniCPM-o-4_5.md).
|