anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity
1,809 Commits 235 Branches 31 Tags
lance-model-support
Commit Graph
4 Commits
Author SHA1 Message Date
lishunyang12 a326cd0d57 [Lance] Restore deploy YAML + patch post-main-merge API drift 
After merging main into ``lance-model-support`` (commit c15f51a2),
Bagel's internal API was aggressively trimmed: ``forward_cache_update_text``,
``forward_cache_update_vit``, ``generate_image``, ``generate_text``, and
``Qwen2MoTForCausalLM.forward`` no longer accept ``packed_text_indexes`` /
``packed_key_value_indexes`` / ``key_values_lens`` / ``packed_query_indexes``
— they're now derived from ``past_key_values``.  Lance's pipeline was
still passing those kwargs and crashed on every task.

Changes:

- ``vllm_omni/deploy/lance.yaml`` (restored): Lance's HF ``config.json``
  is descriptive metadata with no ``model_type`` field, so the loader
  can't auto-detect the pipeline.  This YAML is the minimal hook that
  tells the loader "use the Lance pipeline" — one stage, no multi-stage
  orchestration.
- ``examples/offline_inference/lance/gradio_demo.py``: pass
  ``deploy_config`` again so ``Omni`` picks up the YAML.
- ``vllm_omni/diffusion/models/bagel/bagel_transformer.py``: in the
  batched CFG setup, concat ``branches_pid`` along ``dim=-1`` (the
  sequence axis) instead of the default ``dim=0`` so Lance's 3-D mRoPE
  position ids ``(3, S)`` aren't collapsed into ``(6, S)`` —
  ``BagelRotaryEmbedding`` enforces exactly 3 axes for mRoPE.
- ``vllm_omni/diffusion/models/bagel/pipeline_bagel.py``: mirror the
  generated PIL image into ``custom_output["image"]`` so callers reading
  through the orchestrator IPC boundary (which strips the bare
  ``output`` field) can still recover it.  ``video_frames`` already used
  this pattern.
- ``vllm_omni/diffusion/models/lance/lance_transformer.py``: drop
  ``packed_query_indexes`` / ``key_values_lens`` /
  ``packed_key_value_indexes`` from the ``language_model.forward`` call
  inside ``forward_cache_update_vae`` — derived now.
- ``vllm_omni/diffusion/models/lance/pipeline_lance.py``:
    * drop the same 3 fields from the inline ``inp`` dicts that build
      ``forward_cache_update_text`` arguments in ``_forward_image_edit``,
      ``_forward_i2v``, ``_forward_video_edit``, and
      ``_x2t_raw_text_prefill``.
    * drop the 6 ``cfg_text_packed_query_indexes`` /
      ``cfg_text_key_values_lens`` / ``cfg_text_packed_key_value_indexes``
      (+ ``cfg_img_*`` triple) kwargs from the four ``generate_image``
      call sites.
    * filter ``key_values_lens`` / ``packed_indexes`` /
      ``packed_key_value_indexes`` out of the ``gen_input_lat`` /
      ``gen_input_vit`` dicts before unpacking into ``generate_image``
      and ``forward_cache_update_vit`` (the ``prepare_*_latent`` /
      ``prepare_vit_*`` helpers still return them for the
      backwards-compatible parent-class path).
    * drop ``key_values_lens`` / ``packed_key_value_indexes`` from
      ``_x2t_prepare_assistant_start``'s returned dict so
      ``Bagel.generate_text`` doesn't reject them.

Verification (local online inference, single GPU pair, CFG=4, 10 steps):

| Task          | Status                                          |
|---------------|-------------------------------------------------|
| t2i           | OK  (~2 s)                                      |
| image_edit    | OK  (~2.5 s)                                    |
| i2v           | OK  (~8 s @ 25 frames)                          |
| t2v           | OK  (~7 s @ 25 frames)                          |
| video_edit    | OK  (~60 s)                                     |
| x2t_image     | RUNS but generates degenerate token repetition  |
| x2t_video     | RUNS but generates degenerate token repetition  |

x2t degeneration is a deeper semantic mismatch with main's new
``past_key_values``-derived cache layout: Lance's x2t prefill stitches
the chat template across multiple ``forward_cache_update_*`` calls
(system → ViT → question → ``assistant\\n``), but the new API expects a
single contiguous KV layout.  Fixing requires reworking the segmented
prefill to match main's new cache contract — left as a follow-up since
the 5 generative tasks now work.

Signed-off-by: lishunyang12 <lishunyang12@163.com>
 2026-05-31 13:28:11 +00:00
lishunyang12 7c934b6188 [Lance] Add Gradio demo and align inference with upstream 
- Add ``examples/offline_inference/lance/gradio_demo.py`` — a unified 7-task UI
  (t2i / t2v / i2v / image_edit / video_edit / x2t_image / x2t_video) that
  loads ``Lance_3B`` and ``Lance_3B_Video`` on two GPUs so image and video
  tasks run concurrently.  Surfaces official Lance examples per task; aspect
  ratio is auto-snapped to one of upstream's 6 buckets from the input
  image/video.
- Fix x2t_image / x2t_video output format: route x2t_image through a new
  ``_forward_x2t_image`` handler (was falling through to the img2img path
  and emitting garbage), build the chat prefill in three segments
  (prefix → ViT → suffix) so the ``<|im_end|>`` markers line up with
  upstream's template, and start generation from a newline token instead of
  ``<|im_start|>`` so the assistant turn doesn't close itself with an
  immediate EOS.  Add a ``system_prompt`` override to ``render_lance_prompt``
  so per-example QA instructions surface ("Look at the image carefully and
  answer the question.") instead of the caption-style default.
- ``_forward_image_edit``: also stash the PIL image into ``custom_output``
  so callers reading ``outputs[0].output`` *or* ``custom_output["image"]``
  both work — the orchestrator IPC strips the bare ``output`` field across
  process boundaries.
- ``pipeline_registry.py``, ``model_executor/models/lance/*``, registry
  cleanup: remove the dead Lance-specific autoregressive shim
  (``lance.py``) and its stage-input processor; the diffusion pipeline owns
  the full flow now.
- Aspect-ratio buckets aligned with upstream's ``BucketResize`` (6 ratios:
  21:9 / 16:9 / 4:3 / 1:1 / 3:4 / 9:16, sized for ``resolution_vae`` ∈
  {480, 640, 768}).  3:2 / 2:3 dropped — upstream snaps those to 4:3 / 3:4.
- Gradio defaults match upstream's official scripts (cfg_text_scale=4.0,
  validation_num_timesteps=30, validation_timestep_shift=3.5, seed=42); only
  the demo *duration* is intentionally shorter than upstream's 121-frame
  default to keep the UI snappy (user can extend to 10s via slider for
  byte-identical reproduction).

Signed-off-by: lishunyang12 <lishunyang12@163.com>

Signed-off-by: WeiQing Chen <david6666666@users.noreply.github.com>
 2026-05-31 11:40:40 +00:00
lishunyang12 7b885f9d13 [Lance] Precision matching, recipe, and README cleanup 
- Precision matching across the Lance transformer and pipeline so vLLM-Omni
  outputs match the upstream reference end-to-end (text2img / img2img /
  text2video / video2video / image and video understanding).
- Add recipes/ByteDance/Lance.md covering all single-stage modalities for
  offline + online; register it in recipes/README.md.
- Offline README: document --height/--width and update the default
  resolution to 1024x1024 (matches the upstream inference_lance.sh).
- Gitignore lance_compare/ (local precision-matching dump scripts; not part
  of the repo).
- Remove LANCE_PR.md; the PR description lives on GitHub.

Signed-off-by: lishunyang12 <lishunyang12@163.com>
 2026-05-21 20:07:40 +00:00
lishunyang 69bcc6416d [New Model] Lance (ByteDance) — unified AR+diffusion multimodal (#3697) 
Adds Lance (bytedance-research/Lance), a 3B unified AR+diffusion
multimodal model on a Qwen2.5-VL backbone. Lance is BAGEL-lineage:
it reuses vLLM-Omni's BAGEL Mixture-of-Transformers core verbatim and
specializes only the ViT (Qwen2.5-VL vision), the VAE (Wan2.2) and the
checkpoint layout.

All six single-stage Lance modalities verified end-to-end on 1x B300:
t2i, t2v, image_edit, video_edit, x2t_image (image understanding) and
x2t_video (video understanding). 3-D mRoPE is wired through
BagelRotaryEmbedding for the x2t / video / edit paths. Includes the
upstream gap-analysis quality fixes (task-specific system prompts, full
Qwen chat template, upstream inference defaults and aspect-ratio
buckets).

Ships an offline example (examples/offline_inference/lance/), an
OpenAI-compatible online-serving example
(examples/online_serving/lance/) and an e2e test
(tests/e2e/online_serving/test_lance.py).

See LANCE_PR.md for the architecture rationale, per-modality status and
the two remaining follow-ups (two-stage AR+DiT, video_edit polish).

Closes #3697.

Signed-off-by: lishunyang <lishunyang03@gmail.com>
 2026-05-18 22:09:14 +00:00