[Fix] JoyVL serving: align with reference engine (bounded long-term memory, timestamps, max_pixels) + recipe updates (#4623)

Signed-off-by: lishunyang12 <125541396+lishunyang12@users.noreply.github.com>
This commit is contained in:
SYLAR authored and GitHub committed 2026-06-24 07:04:17 +08:00
1 parent 0958e62ecc
commit d35cdd2dab
9 files changed
+116 -28

No files matched your search

+34 -5
View File
@@ -5,7 +5,7 @@
## Summary
- Vendor: JD (Joy Future Academy)
- Model: [`jdopensource/JoyAI-VL-Interaction-Preview`](https://huggingface.co/jdopensource/JoyAI-VL-Interaction-Preview) (8B, vanilla Qwen3-VL weights)
- Model: [`jdopensource/JoyAI-VL-Interaction-Preview`](https://huggingface.co/jdopensource/JoyAI-VL-Interaction-Preview) (8B, Qwen3-VL architecture with weights retrained for streaming interaction)
- Task: Per-tick proactive interaction over a live video stream — the model decides
on its own each second to speak, stay silent, or delegate a hard question
- Mode: Online serving — an OpenAI-compatible interaction orchestrator in front of
@@ -25,7 +25,8 @@ framework internals and how to add another full-duplex model, see
- OS: Linux
- Python: 3.10+
- Hardware: 1x GPU. The default `T_s=100` frame window + `--max-model-len 131072` wants
≈48GB+; for ~24GB, lower `--chunk-frames`, `--max-model-len`, and the image limit together
≈48GB+; for ~24GB, lower `--chunk-frames`, `--max-model-len`, and the image limit together,
and/or load the weights in fp8 (see "Smaller GPU: fp8 weights" below — ~9.9 GiB vs 16.8 GiB)
- vLLM / vLLM-Omni: versions from your current checkout
## Start server
@@ -33,7 +34,7 @@ framework internals and how to add another full-duplex model, see
From repository root:
```bash
# 1. Serve the model (plain `vllm serve`, NOT --omni; it is vanilla Qwen3-VL).
# 1. Serve the model (plain `vllm serve`, NOT --omni; it uses the Qwen3-VL architecture).
# The image limit must cover the short-term frame window (chunk_frames, default 100 = T_s);
# prefix caching keeps the accumulating window cheap. Lower both for smaller GPUs.
vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
@@ -46,6 +47,26 @@ python -m vllm_omni.experimental.fullduplex.joyvl.serving.server --port 8070 \
--main-backend-url http://127.0.0.1:8061/v1 --main-model JoyAI-VL-Interaction-Preview
```
### Smaller GPU: fp8 weights
Add `--quantization fp8` to the step-1 `vllm serve` to load the weights in fp8 online
from the bf16 checkpoint (no separate quantized checkpoint needed; uses vLLM's generic
fp8 path, since the model is a standard Qwen3-VL VLM):
```bash
vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
--served-model-name JoyAI-VL-Interaction-Preview --port 8061 \
--quantization fp8 --max-model-len 131072 --enable-prefix-caching \
--limit-mm-per-prompt '{"image":256,"video":1}'
```
Measured here: model weights **9.9 GiB** vs **16.8 GiB** bf16 (**−41%**), which lets the 8B
fit a much smaller card. This is a **memory** option, not a speed one: at one request per
tick (the streaming regime) fp8 was **not** faster (≈19% slower in our test) because
single-stream decode is memory-latency-bound and pays fp8's dynamic-scale overhead; its
throughput gain needs batching. Decisions stay coherent, but fp8 can shift the
speak/silence boundary, so for timing-critical alerting validate output against bf16 first.
Optional one-shot launch (model + orchestrator + JD webui + ASR/TTS/background,
all env-configurable). The JD webui frontend is external — set `WEBUI_DIR` to point at it:
@@ -176,14 +197,22 @@ the audio-track caveat.
## Notes
- `--omni` is **not** used: the model is standard Qwen3-VL, so stock `vllm serve`
runs the forward pass; this recipe only adds the interaction/serving layer.
- `--omni` is **not** used: the model keeps the Qwen3-VL architecture (only the
weights are retrained), so stock `vllm serve` runs the forward pass; this recipe
only adds the interaction/serving layer.
- On a host without `nvcc` / `ninja`, `vllm serve` of the 8B can crash engine-core in the
FlashInfer sampler JIT (`FileNotFoundError: 'ninja'`) during `profile_run`. Set
`VLLM_USE_FLASHINFER_SAMPLER=0` (or install `ninja`) to work around it.
- `force_silence_before_query` is on by default — the model stays silent until an
instruction arrives; give a standing task (e.g. "translate the on-screen text")
to arm proactive output.
- **Frame resolution is the main latency knob.** The system/memory prefix is prefix-cached,
so per-tick *new* compute is dominated by the new frame's vision tokens. Measured on the
8B: ~256×192 frames hit Qwen3-VL's min-pixel floor (~72 vision tokens, ~17 ms/tick) vs
~302 tokens / ~38 ms at 640×480 — about 2× cheaper. Downsample frames to ~256×192 for the
tightest latency / highest concurrency; one GPU then sustains ~150–180 concurrent 1 fps
streams with p95 < 200 ms. Per-tick latency is already far inside the 1 fps budget, so
serving is rarely the bottleneck — resolution is the lever if you need more headroom.
- Speech is external and pluggable: point `ASR_URL` / `TTS_URL` at the bridges in
`examples/online_serving/joyvl_interaction/bridges/` or any compatible service.
- The decision prompts, sampling, and 3-tier summary memory (`T_s=100`, mid→long every