vllm-omni/recipes/JD/JoyAI-VL-Interaction.md

JoyAI-VL-Interaction

Real-time streaming video-language interaction (proactive speak / silence / delegate)

Summary

• Vendor: JD (Joy Future Academy)
• Model: jdopensource/JoyAI-VL-Interaction-Preview (8B, Qwen3-VL architecture with weights retrained for streaming interaction)
• Task: Per-tick proactive interaction over a live video stream — the model decides on its own each second to speak, stay silent, or delegate a hard question
• Mode: Online serving — an OpenAI-compatible interaction orchestrator in front of a plain vllm serve backend
• Maintainer: Community

When to use this recipe

Use this to stand up the streaming-interaction serving layer (vllm_omni/experimental/fullduplex/). The model is served unchanged by vllm serve; this layer adds session state, 3-tier summary memory, the per-tick decision, and pluggable ASR / TTS / delegation. For the framework internals and how to add another full-duplex model, see vllm_omni/experimental/fullduplex/README.md.

Environment

• OS: Linux
• Python: 3.10+
• Hardware: 1x GPU. The default T_s=100 frame window + --max-model-len 131072 wants ≈48GB+; for ~24GB, lower --chunk-frames, --max-model-len, and the image limit together, and/or load the weights in fp8 (see "Smaller GPU: fp8 weights" below — ~9.9 GiB vs 16.8 GiB)
• vLLM / vLLM-Omni: versions from your current checkout

Start server

From repository root:

# 1. Serve the model (plain `vllm serve`, NOT --omni; it uses the Qwen3-VL architecture).
#    The image limit must cover the short-term frame window (chunk_frames, default 100 = T_s);
#    prefix caching keeps the accumulating window cheap. Lower both for smaller GPUs.
vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
  --served-model-name JoyAI-VL-Interaction-Preview --port 8061 \
  --max-model-len 131072 --enable-prefix-caching \
  --limit-mm-per-prompt '{"image":256,"video":1}'

# 2. Interaction orchestrator (OpenAI-compatible, :8070)
python -m vllm_omni.experimental.fullduplex.joyvl.serving.server --port 8070 \
  --main-backend-url http://127.0.0.1:8061/v1 --main-model JoyAI-VL-Interaction-Preview

Smaller GPU: fp8 weights

Add --quantization fp8 to the step-1 vllm serve to load the weights in fp8 online from the bf16 checkpoint (no separate quantized checkpoint needed; uses vLLM's generic fp8 path, since the model is a standard Qwen3-VL VLM):

vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
  --served-model-name JoyAI-VL-Interaction-Preview --port 8061 \
  --quantization fp8 --max-model-len 131072 --enable-prefix-caching \
  --limit-mm-per-prompt '{"image":256,"video":1}'

Measured here: model weights 9.9 GiB vs 16.8 GiB bf16 (−41%), which lets the 8B fit a much smaller card. This is a memory option, not a speed one: at one request per tick (the streaming regime) fp8 was not faster (≈19% slower in our test) because single-stream decode is memory-latency-bound and pays fp8's dynamic-scale overhead; its throughput gain needs batching. Decisions stay coherent, but fp8 can shift the speak/silence boundary, so for timing-critical alerting validate output against bf16 first.

Optional one-shot launch (model + orchestrator + JD webui + ASR/TTS/background, all env-configurable). The JD webui frontend is external — set WEBUI_DIR to point at it:

bash examples/online_serving/joyvl_interaction/scripts/start_all.sh

Using the model

The orchestrator is OpenAI-compatible: send one user turn per video frame (~1 fps) to /v1/chat/completions with an x-session-id header, and attach an optional standing instruction as a text part. Each reply carries an interaction block — action is silence / response / delegate and text is what to say:

curl -s http://127.0.0.1:8070/v1/chat/completions \
  -H 'x-session-id: s1' -H 'content-type: application/json' -d '{
    "messages": [{"role": "user", "content": [
      {"type": "text", "text": "Alert me if a fire breaks out"},
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
    ]}]}' | jq .interaction

Send the standing instruction once (it persists for the session); subsequent turns send just the frame. Ready-made headless client: cli/run_cli_demo.py. Reset a session with POST /reset {"session_id": "..."}.

What to ask it — give a standing task and let it act on its own each second:

• Proactive alerting — "tell me when someone enters", "alert me if a fire breaks out"
• Streaming Q&A — ask about what's on screen; it answers once it has the evidence
• Time & memory — "how many people have walked past?", "what did you see earlier?" (the 3-tier summary memory lets it recall beyond the live frame window)
• Read & translate — "translate the on-screen text as it appears"
• Delegate hard questions — handed to a background brain (see below)
• Live commentary — --persona talkative for continuous, danmaku-style narration

Personas (--persona): default (speak on meaningful events or to answer), silent (answer only when asked, never delegate), talkative (proactively narrate).

Tuning: --chunk-frames (short-term window T_s), --response-dedup-threshold (lower drops more near-duplicate narration), and --no-memory to disable the 3-tier summaries. The model stays silent until the first instruction arrives (force_silence_before_query, on by default).

Delegation (background brain)

When the model judges a question too hard, it emits </delegation> <question> and the orchestrator hands it to a background brain — any OpenAI-compatible endpoint you self-host. Enable it by pointing the orchestrator at one:

python -m vllm_omni.experimental.fullduplex.joyvl.serving.server --port 8070 \
  --main-backend-url http://127.0.0.1:8061/v1 --main-model JoyAI-VL-Interaction-Preview \
  --delegation-backend-url <brain-endpoint>/v1 \
  --delegation-model <brain-model> --delegation-kind chat

--delegation-kind picks the bridge:

• chat — a stronger text/VL model answers (/chat/completions)
• image — a text-to-image model generates a picture (/images/generations, e.g. Qwen-Image)
• edit — an image-edit model restyles the current frame (e.g. Qwen-Image-Edit)
• router — dispatch each request to chat / image / edit by inspecting it (set --delegation-image-url / --delegation-edit-url for the latter two)
• stub — canned answers for tests/demos only (no backend needed)

chat/image/edit/router each need a backend URL — without one, delegation stays off (the model's delegate note is still spoken, but nothing is folded back). The brain is bring-your-own: a larger vLLM you serve, or any OpenAI-compatible API (e.g. --delegation-backend-url https://api.anthropic.com/v1/ --delegation-model claude-... --delegation-api-key …). The reference deployment instead drives the codex CLI as the brain via a separate background-agent service; that agent runs with its own credentials and bypasses its sandbox, so it is not bundled here — self-host a plain OpenAI-compatible endpoint instead.

Host the WebUI demo

For the full browser experience — live webcam / RTSP input, voice (ASR/TTS), and the per-tick decision stream — run JD's official WebUI in front of the orchestrator. Clone the model repo and start its WebUI pointed at the orchestrator (:8070):

git clone https://github.com/jd-opensource/JoyAI-VL-Interaction.git
cd JoyAI-VL-Interaction/services/webui
uv venv && uv pip install -e .
bash scripts/start_server.sh --api-base http://127.0.0.1:8070/v1

Open the printed HTTPS URL, allow the camera (or enter an RTSP URL), and give a standing instruction. examples/online_serving/joyvl_interaction/scripts/start_all.sh can launch the model + orchestrator + WebUI together — point WEBUI_DIR at your clone.

Verification

# headless: stream a clip and print the per-tick decision timeline
# (the CLI reads video frames via OpenCV: `uv pip install opencv-python`)
python examples/online_serving/joyvl_interaction/cli/run_cli_demo.py \
  path/to/video.mp4 --query "Alert me if a fire breaks out"

pytest tests/e2e/features/fullduplex   # framework + JoyVL unit tests

Testing with an RTSP stream (optional)

RTSP is a webui-side input — the browser pulls the stream and feeds frames to the orchestrator over the normal API; no serving-layer code is involved. To simulate an RTSP camera from a local video file (no physical IP camera needed), use the helper scripts in examples/online_serving/joyvl_interaction/rtsp/, which wrap MediaMTX + ffmpeg:

cd examples/online_serving/joyvl_interaction/rtsp

# 1. Local RTSP server (MediaMTX, listens on :8554)
bash ./mediamtx.sh

# 2. Push a local video file as an RTSP stream (another terminal)
bash ./rtsp.sh ./videos/example.mp4 rtsp://127.0.0.1:8554/fire1

# 3. In the WebUI RTSP box, enter:  rtsp://127.0.0.1:8554/fire1
#    (replace 127.0.0.1 with the MediaMTX host IP if the webui runs on another machine)

See the directory's README.md for streaming a whole video folder (rtsp_all.sh) and the audio-track caveat.

Notes

• --omni is not used: the model keeps the Qwen3-VL architecture (only the weights are retrained), so stock vllm serve runs the forward pass; this recipe only adds the interaction/serving layer.
• On a host without nvcc / ninja, vllm serve of the 8B can crash engine-core in the FlashInfer sampler JIT (FileNotFoundError: 'ninja') during profile_run. Set VLLM_USE_FLASHINFER_SAMPLER=0 (or install ninja) to work around it.
• force_silence_before_query is on by default — the model stays silent until an instruction arrives; give a standing task (e.g. "translate the on-screen text") to arm proactive output.
• Frame resolution is the main latency knob. The system/memory prefix is prefix-cached, so per-tick new compute is dominated by the new frame's vision tokens. Measured on the 8B: ~256×192 frames hit Qwen3-VL's min-pixel floor (~72 vision tokens, ~17 ms/tick) vs ~302 tokens / ~38 ms at 640×480 — about 2× cheaper. Downsample frames to ~256×192 for the tightest latency / highest concurrency; one GPU then sustains ~150–180 concurrent 1 fps streams with p95 < 200 ms. Per-tick latency is already far inside the 1 fps budget, so serving is rarely the bottleneck — resolution is the lever if you need more headroom.
• Speech is external and pluggable: point ASR_URL / TTS_URL at the bridges in examples/online_serving/joyvl_interaction/bridges/ or any compatible service.
• The decision prompts, sampling, and 3-tier summary memory (T_s=100, mid→long every 5 chunks, key_frames=0 = summarize all chunk frames) are aligned to the JoyVL reference adapter so per-tick behavior matches the released model; the framework only supplies the serving structure.