ZhanshenzhuiandSy03
48ae412a2f
Signed-off-by: Zhanshenzhui <99891002+zyforsure@users.noreply.github.com> Co-authored-by: Sy03 <1370724210@qq.com>
115 lines
4.2 KiB
Markdown
115 lines
4.2 KiB
Markdown
# Mage-VL
|
|
|
|
> Codec-native image/video understanding plus proactive full-duplex streaming.
|
|
|
|
## Summary
|
|
|
|
- Vendor: Microsoft
|
|
- Model: [`microsoft/Mage-VL`](https://huggingface.co/microsoft/Mage-VL)
|
|
- Task: Image understanding, video understanding, codec-native video understanding,
|
|
and event-gated streaming commentary
|
|
- Mode: Full-duplex WebSocket server and session adapter in
|
|
`vllm_omni/experimental/fullduplex/mage_vl`
|
|
- Maintainer: Community
|
|
|
|
## Status
|
|
|
|
This experimental integration provides the Mage-VL adapter contract: bounded causal
|
|
visual windows, codec-window input, rolling gate evaluation, explicit query handling,
|
|
per-session state isolation, and stale-output cancellation through the shared
|
|
full-duplex runtime.
|
|
|
|
The production transport exposes `WebSocket /v1/mage-vl/duplex`, with bearer-token
|
|
authentication, bounded concurrent sessions and message sizes, idle timeouts,
|
|
disconnect cancellation, per-session state cleanup, and serialized access to the
|
|
shared Transformers checkpoint. Native vLLM model execution remains follow-up work.
|
|
|
|
The default `frames` backend uses decoded frames and has no codec preprocessing
|
|
requirement. The optional `codec` backend requires `ffmpeg` and `ffprobe`, plus the
|
|
`cv-preinfer` executable provided by `codec-video-prep>=0.2.5`, on `PATH`. Server
|
|
startup fails with an actionable error when `cv-preinfer` is unavailable; use
|
|
`--video-backend frames` when codec preprocessing is not installed.
|
|
|
|
## Serving
|
|
|
|
```bash
|
|
python -m vllm_omni.experimental.fullduplex.mage_vl.serving.server \
|
|
--model microsoft/Mage-VL --host 0.0.0.0 --port 8090 \
|
|
--auth-token "$MAGE_VL_API_KEY" --max-sessions 32 \
|
|
--video-backend frames --num-frames 8 --target-fps 1 \
|
|
--gate-threshold 0.5 --attn-impl sdpa
|
|
|
|
python examples/online_serving/mage_vl/duplex_client.py \
|
|
--url ws://127.0.0.1:8090/v1/mage-vl/duplex \
|
|
--auth-token "$MAGE_VL_API_KEY" \
|
|
--video /path/to/segment.mp4 \
|
|
--prompt "Describe this video segment in detail."
|
|
```
|
|
|
|
Each input window is a JSON `input.append` event whose `data.data.video_base64`
|
|
contains an MP4 segment. A text `input.append` before the video requests an explicit
|
|
answer; without text, the StreamMind gate decides whether to respond proactively.
|
|
The server emits `session.created`, `response.created`, one or more
|
|
`response.delta`, and `response.done` events. Clients can send `response.cancel` for
|
|
barge-in and `close` for graceful shutdown.
|
|
|
|
The adapter can still be wired to another backend through `--adapter module:factory`,
|
|
including:
|
|
|
|
- a Transformers `trust_remote_code=True` backend for offline/local testing;
|
|
- the model author's SGLang-compatible serving path;
|
|
- a future native vLLM `mage_vl` backend.
|
|
|
|
## Adapter Shape
|
|
|
|
```python
|
|
from vllm_omni.experimental.fullduplex.core.session import DuplexSession, DuplexSessionConfig
|
|
from vllm_omni.experimental.fullduplex.mage_vl import MageVLDuplexAdapter, MageVLDuplexRuntime
|
|
|
|
|
|
adapter = MageVLDuplexAdapter(gate=run_mage_gate, generate=run_mage_decoder)
|
|
session = DuplexSession(
|
|
"stream-1",
|
|
DuplexSessionConfig(
|
|
input_modalities=("codec_window", "video", "image", "text"),
|
|
output_modalities=("text",),
|
|
proactive=True,
|
|
),
|
|
)
|
|
runtime = MageVLDuplexRuntime(session, adapter)
|
|
await runtime.run(input_events, emit_event)
|
|
```
|
|
|
|
Input events use the shared full-duplex protocol. For decoded video/image, pass the frame
|
|
payload directly. For codec-native streaming, pass `modality="codec_window"` and include
|
|
fields such as `segment_id`, `pts_ms`, `duration_ms`, `codec`, and `metadata`.
|
|
|
|
```python
|
|
{
|
|
"type": "input.append",
|
|
"modality": "codec_window",
|
|
"data": {
|
|
"kind": "h264",
|
|
"segment_id": "seg-42",
|
|
"pts_ms": 42000,
|
|
"duration_ms": 1000,
|
|
"codec": {
|
|
"motion_vectors": ...,
|
|
"residual_energy": ...,
|
|
},
|
|
"metadata": {"gop": "P"},
|
|
},
|
|
}
|
|
```
|
|
|
|
The gate callable receives the current rolling window and returns either a
|
|
`MageVLGateDecision`, a mapping with `should_respond` / `text` / `event_id`, a bool, or
|
|
a text string. Repeated `event_id` values are de-duplicated at the session adapter.
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
pytest tests/e2e/features/fullduplex/test_mage_vl_adapter.py \
|
|
tests/examples/offline_inference/test_mage_vl.py
|
|
```
|