vllm-omni/recipes/Microsoft/Mage-VL.md

Mage-VL

Codec-native image/video understanding plus proactive full-duplex streaming.

Summary

• Vendor: Microsoft
• Model: microsoft/Mage-VL
• Task: Image understanding, video understanding, codec-native video understanding, and event-gated streaming commentary
• Mode: Full-duplex WebSocket server and session adapter in vllm_omni/experimental/fullduplex/mage_vl
• Maintainer: Community

Status

This experimental integration provides the Mage-VL adapter contract: bounded causal visual windows, codec-window input, rolling gate evaluation, explicit query handling, per-session state isolation, and stale-output cancellation through the shared full-duplex runtime.

The production transport exposes WebSocket /v1/mage-vl/duplex, with bearer-token authentication, bounded concurrent sessions and message sizes, idle timeouts, disconnect cancellation, per-session state cleanup, and serialized access to the shared Transformers checkpoint. Native vLLM model execution remains follow-up work.

The default frames backend uses decoded frames and has no codec preprocessing requirement. The optional codec backend requires ffmpeg and ffprobe, plus the cv-preinfer executable provided by codec-video-prep>=0.2.5, on PATH. Server startup fails with an actionable error when cv-preinfer is unavailable; use --video-backend frames when codec preprocessing is not installed.

Serving

python -m vllm_omni.experimental.fullduplex.mage_vl.serving.server \
  --model microsoft/Mage-VL --host 0.0.0.0 --port 8090 \
  --auth-token "$MAGE_VL_API_KEY" --max-sessions 32 \
  --video-backend frames --num-frames 8 --target-fps 1 \
  --gate-threshold 0.5 --attn-impl sdpa

python examples/online_serving/mage_vl/duplex_client.py \
  --url ws://127.0.0.1:8090/v1/mage-vl/duplex \
  --auth-token "$MAGE_VL_API_KEY" \
  --video /path/to/segment.mp4 \
  --prompt "Describe this video segment in detail."

Each input window is a JSON input.append event whose data.data.video_base64 contains an MP4 segment. A text input.append before the video requests an explicit answer; without text, the StreamMind gate decides whether to respond proactively. The server emits session.created, response.created, one or more response.delta, and response.done events. Clients can send response.cancel for barge-in and close for graceful shutdown.

The adapter can still be wired to another backend through --adapter module:factory, including:

• a Transformers trust_remote_code=True backend for offline/local testing;
• the model author's SGLang-compatible serving path;
• a future native vLLM mage_vl backend.

Adapter Shape

from vllm_omni.experimental.fullduplex.core.session import DuplexSession, DuplexSessionConfig
from vllm_omni.experimental.fullduplex.mage_vl import MageVLDuplexAdapter, MageVLDuplexRuntime


adapter = MageVLDuplexAdapter(gate=run_mage_gate, generate=run_mage_decoder)
session = DuplexSession(
    "stream-1",
    DuplexSessionConfig(
        input_modalities=("codec_window", "video", "image", "text"),
        output_modalities=("text",),
        proactive=True,
    ),
)
runtime = MageVLDuplexRuntime(session, adapter)
await runtime.run(input_events, emit_event)

Input events use the shared full-duplex protocol. For decoded video/image, pass the frame payload directly. For codec-native streaming, pass modality="codec_window" and include fields such as segment_id, pts_ms, duration_ms, codec, and metadata.

{
    "type": "input.append",
    "modality": "codec_window",
    "data": {
        "kind": "h264",
        "segment_id": "seg-42",
        "pts_ms": 42000,
        "duration_ms": 1000,
        "codec": {
            "motion_vectors": ...,
            "residual_energy": ...,
        },
        "metadata": {"gop": "P"},
    },
}

The gate callable receives the current rolling window and returns either a MageVLGateDecision, a mapping with should_respond / text / event_id, a bool, or a text string. Repeated event_id values are de-duplicated at the session adapter.

Verification

pytest tests/e2e/features/fullduplex/test_mage_vl_adapter.py \
  tests/examples/offline_inference/test_mage_vl.py