[Fix] JoyVL serving: align with reference engine (bounded long-term memory, timestamps, max_pixels) + recipe updates (#4623)
Signed-off-by: lishunyang12 <125541396+lishunyang12@users.noreply.github.com>
This commit is contained in:
1 parent
0958e62ecc
commit
d35cdd2dab
9 files changed
+116
-28
No files matched your search
@@ -5,7 +5,7 @@
|
||||
## Summary
|
||||
|
||||
- Vendor: JD (Joy Future Academy)
|
||||
- Model: [`jdopensource/JoyAI-VL-Interaction-Preview`](https://huggingface.co/jdopensource/JoyAI-VL-Interaction-Preview) (8B, vanilla Qwen3-VL weights)
|
||||
- Model: [`jdopensource/JoyAI-VL-Interaction-Preview`](https://huggingface.co/jdopensource/JoyAI-VL-Interaction-Preview) (8B, Qwen3-VL architecture with weights retrained for streaming interaction)
|
||||
- Task: Per-tick proactive interaction over a live video stream — the model decides
|
||||
on its own each second to speak, stay silent, or delegate a hard question
|
||||
- Mode: Online serving — an OpenAI-compatible interaction orchestrator in front of
|
||||
@@ -25,7 +25,8 @@ framework internals and how to add another full-duplex model, see
|
||||
- OS: Linux
|
||||
- Python: 3.10+
|
||||
- Hardware: 1x GPU. The default `T_s=100` frame window + `--max-model-len 131072` wants
|
||||
≈48GB+; for ~24GB, lower `--chunk-frames`, `--max-model-len`, and the image limit together
|
||||
≈48GB+; for ~24GB, lower `--chunk-frames`, `--max-model-len`, and the image limit together,
|
||||
and/or load the weights in fp8 (see "Smaller GPU: fp8 weights" below — ~9.9 GiB vs 16.8 GiB)
|
||||
- vLLM / vLLM-Omni: versions from your current checkout
|
||||
|
||||
## Start server
|
||||
@@ -33,7 +34,7 @@ framework internals and how to add another full-duplex model, see
|
||||
From repository root:
|
||||
|
||||
```bash
|
||||
# 1. Serve the model (plain `vllm serve`, NOT --omni; it is vanilla Qwen3-VL).
|
||||
# 1. Serve the model (plain `vllm serve`, NOT --omni; it uses the Qwen3-VL architecture).
|
||||
# The image limit must cover the short-term frame window (chunk_frames, default 100 = T_s);
|
||||
# prefix caching keeps the accumulating window cheap. Lower both for smaller GPUs.
|
||||
vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
|
||||
@@ -46,6 +47,26 @@ python -m vllm_omni.experimental.fullduplex.joyvl.serving.server --port 8070 \
|
||||
--main-backend-url http://127.0.0.1:8061/v1 --main-model JoyAI-VL-Interaction-Preview
|
||||
```
|
||||
|
||||
### Smaller GPU: fp8 weights
|
||||
|
||||
Add `--quantization fp8` to the step-1 `vllm serve` to load the weights in fp8 online
|
||||
from the bf16 checkpoint (no separate quantized checkpoint needed; uses vLLM's generic
|
||||
fp8 path, since the model is a standard Qwen3-VL VLM):
|
||||
|
||||
```bash
|
||||
vllm serve jdopensource/JoyAI-VL-Interaction-Preview \
|
||||
--served-model-name JoyAI-VL-Interaction-Preview --port 8061 \
|
||||
--quantization fp8 --max-model-len 131072 --enable-prefix-caching \
|
||||
--limit-mm-per-prompt '{"image":256,"video":1}'
|
||||
```
|
||||
|
||||
Measured here: model weights **9.9 GiB** vs **16.8 GiB** bf16 (**−41%**), which lets the 8B
|
||||
fit a much smaller card. This is a **memory** option, not a speed one: at one request per
|
||||
tick (the streaming regime) fp8 was **not** faster (≈19% slower in our test) because
|
||||
single-stream decode is memory-latency-bound and pays fp8's dynamic-scale overhead; its
|
||||
throughput gain needs batching. Decisions stay coherent, but fp8 can shift the
|
||||
speak/silence boundary, so for timing-critical alerting validate output against bf16 first.
|
||||
|
||||
Optional one-shot launch (model + orchestrator + JD webui + ASR/TTS/background,
|
||||
all env-configurable). The JD webui frontend is external — set `WEBUI_DIR` to point at it:
|
||||
|
||||
@@ -176,14 +197,22 @@ the audio-track caveat.
|
||||
|
||||
## Notes
|
||||
|
||||
- `--omni` is **not** used: the model is standard Qwen3-VL, so stock `vllm serve`
|
||||
runs the forward pass; this recipe only adds the interaction/serving layer.
|
||||
- `--omni` is **not** used: the model keeps the Qwen3-VL architecture (only the
|
||||
weights are retrained), so stock `vllm serve` runs the forward pass; this recipe
|
||||
only adds the interaction/serving layer.
|
||||
- On a host without `nvcc` / `ninja`, `vllm serve` of the 8B can crash engine-core in the
|
||||
FlashInfer sampler JIT (`FileNotFoundError: 'ninja'`) during `profile_run`. Set
|
||||
`VLLM_USE_FLASHINFER_SAMPLER=0` (or install `ninja`) to work around it.
|
||||
- `force_silence_before_query` is on by default — the model stays silent until an
|
||||
instruction arrives; give a standing task (e.g. "translate the on-screen text")
|
||||
to arm proactive output.
|
||||
- **Frame resolution is the main latency knob.** The system/memory prefix is prefix-cached,
|
||||
so per-tick *new* compute is dominated by the new frame's vision tokens. Measured on the
|
||||
8B: ~256×192 frames hit Qwen3-VL's min-pixel floor (~72 vision tokens, ~17 ms/tick) vs
|
||||
~302 tokens / ~38 ms at 640×480 — about 2× cheaper. Downsample frames to ~256×192 for the
|
||||
tightest latency / highest concurrency; one GPU then sustains ~150–180 concurrent 1 fps
|
||||
streams with p95 < 200 ms. Per-tick latency is already far inside the 1 fps budget, so
|
||||
serving is rarely the bottleneck — resolution is the lever if you need more headroom.
|
||||
- Speech is external and pluggable: point `ASR_URL` / `TTS_URL` at the bridges in
|
||||
`examples/online_serving/joyvl_interaction/bridges/` or any compatible service.
|
||||
- The decision prompts, sampling, and 3-tier summary memory (`T_s=100`, mid→long every
|
||||
|
||||
Reference in new issue
Block a user