amy-why-3459andCursor
64bcdc2186
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com> Co-authored-by: Cursor <cursoragent@cursor.com>
160 lines
4.4 KiB
Markdown
160 lines
4.4 KiB
Markdown
# MiniCPM-o 4.5: Online serving
|
|
|
|
OpenAI-compatible `/v1/chat/completions` serving for **MiniCPM-o 4.5**, plus a
|
|
Gradio UI and curl / Python clients.
|
|
|
|
Inputs: text / image / audio / video. Outputs: text and optional **24 kHz** speech.
|
|
|
|
## Installation
|
|
|
|
Please refer to [README.md](../../../README.md). Install the talker extra:
|
|
|
|
```bash
|
|
pip install 'vllm-omni[minicpmo]'
|
|
```
|
|
|
|
## Launch the server
|
|
|
|
The deploy config auto-loads via `--omni`; the default
|
|
`vllm_omni/deploy/minicpmo_4_5.yaml` targets a single-GPU layout (thinker
|
|
and talker + t2w co-located on GPU 0). For other hardware layouts pick
|
|
one of the deploy variants below.
|
|
|
|
| deploy config | GPUs | Notes |
|
|
|---|---|---|
|
|
| `minicpmo_4_5.yaml` (default) | 1 | Thinker and talker+t2w co-located on GPU0. |
|
|
| `minicpmo_4_5_2gpu.yaml` | 2 | Thinker on GPU0, talker+t2w on GPU1. |
|
|
| `minicpmo_4_5_3gpu.yaml` | 3 | Thinker 2-way TP on GPU0/1, talker+t2w share GPU2. |
|
|
| `minicpmo_4_5_8x4090.yaml` | 8 | Full 8x4090 layout. |
|
|
|
|
Default (single-GPU):
|
|
|
|
```bash
|
|
vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
|
--trust-remote-code \
|
|
--host 0.0.0.0 --port 8099
|
|
```
|
|
|
|
Other layouts:
|
|
|
|
```bash
|
|
vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
|
--deploy-config vllm_omni/deploy/minicpmo_4_5_3gpu.yaml \
|
|
--trust-remote-code \
|
|
--host 0.0.0.0 --port 8099
|
|
```
|
|
|
|
### Stage-based CLI (optional)
|
|
|
|
Stage 0 (thinker + API) and stage 1 (talker) can run in separate processes:
|
|
|
|
```bash
|
|
# Stage 0
|
|
CUDA_VISIBLE_DEVICES=0 vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
|
--trust-remote-code --port 8099 --stage-id 0 \
|
|
--omni-master-address 127.0.0.1 --omni-master-port 26000
|
|
|
|
# Stage 1 (headless)
|
|
CUDA_VISIBLE_DEVICES=1 vllm serve openbmb/MiniCPM-o-4_5 --omni \
|
|
--trust-remote-code --stage-id 1 --headless \
|
|
--omni-master-address 127.0.0.1 --omni-master-port 26000
|
|
```
|
|
|
|
### Per-stage overrides
|
|
|
|
```bash
|
|
vllm serve openbmb/MiniCPM-o-4_5 --omni --trust-remote-code --port 8099 \
|
|
--stage-overrides '{"0": {"gpu_memory_utilization": 0.65}}'
|
|
```
|
|
|
|
## Send multimodal requests
|
|
|
|
```bash
|
|
cd examples/online_serving/minicpmo
|
|
```
|
|
|
|
### curl
|
|
|
|
```bash
|
|
bash run_curl_multimodal_generation.sh text
|
|
bash run_curl_multimodal_generation.sh use_image
|
|
bash run_curl_multimodal_generation.sh use_audio '["text"]' # text-only
|
|
```
|
|
|
|
Text + speech smoke test (TTS needs top-level `chat_template_kwargs`):
|
|
|
|
```bash
|
|
curl http://localhost:8099/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "openbmb/MiniCPM-o-4_5",
|
|
"messages": [{"role": "user", "content": "Say hello, then introduce vLLM in one sentence."}],
|
|
"modalities": ["text", "audio"],
|
|
"chat_template_kwargs": {"use_tts_template": true}
|
|
}'
|
|
```
|
|
|
|
### Python client
|
|
|
|
```bash
|
|
python openai_chat_completion_client_for_multimodal_generation.py \
|
|
--query-type use_image \
|
|
--port 8099 \
|
|
--host localhost
|
|
|
|
# Text-only (faster; no <|tts_bos|>)
|
|
python openai_chat_completion_client_for_multimodal_generation.py \
|
|
--query-type text \
|
|
--modalities text \
|
|
--prompt "Briefly introduce yourself."
|
|
```
|
|
|
|
Shared helpers also work if you pass MiniCPM defaults yourself:
|
|
|
|
```bash
|
|
python ../openai_chat_completion_client_for_multimodal_generation.py \
|
|
--model openbmb/MiniCPM-o-4_5 \
|
|
--query-type text \
|
|
--port 8099
|
|
```
|
|
|
|
(Note: the shared client does **not** set `use_tts_template`; prefer the
|
|
MiniCPM-specific client above for speech.)
|
|
|
|
### Gradio demo
|
|
|
|
```bash
|
|
bash run_gradio_demo.sh
|
|
|
|
# Or:
|
|
python gradio_demo.py \
|
|
--minicpmo45-api-base http://localhost:8099/v1 \
|
|
--minicpmo45-model openbmb/MiniCPM-o-4_5 \
|
|
--port 7862
|
|
```
|
|
|
|
Open `http://<host>:7862`. Uncheck **"Generate speech output (TTS)"** for
|
|
text-only responses.
|
|
|
|
## Modality control
|
|
|
|
| Modalities | Output |
|
|
|---|---|
|
|
| `["text"]` | Text only (no TTS bos) |
|
|
| `["text", "audio"]` / unset | Text + 24 kHz speech |
|
|
|
|
Speech requires `chat_template_kwargs.use_tts_template=true` so the chat
|
|
template appends `<|tts_bos|>`. For **curl**, put that field at the request
|
|
root; nested `extra_body` is ignored. The OpenAI Python SDK may use
|
|
`extra_body` because it merges those fields into the root.
|
|
|
|
## Notes
|
|
|
|
- Stage 1 is capped at `max_num_seqs: 1` in the deploy YAML (talker shares
|
|
request-0 audio metadata).
|
|
- Output audio is base64 WAV in `message.audio.data` (24 kHz mono).
|
|
- Offline counterpart:
|
|
[`examples/offline_inference/minicpmo/`](../../offline_inference/minicpmo/)
|
|
- Recipe:
|
|
[`recipes/OpenBMB/MiniCPM-o-4_5.md`](../../../recipes/OpenBMB/MiniCPM-o-4_5.md)
|