Commit Graph
39 Commits
Author SHA1 Message Date
tzhouamandClaude Opus 5 3834de3105 Rename the project to edge-infer, and complete the packaging audit
Renamed from vllm-embedding: nothing here is about embeddings, and the name had
come from the parent working directory. edge-infer says what it does. The
GitHub repository is renamed to match; nothing had been pushed yet.

The audit found the packaging was incomplete in two ways.

Scripts: 56 of them produced results that are published in this repo but were
not included, so the numbers could be read and not checked. Added under
benchmarks/edge_harness/experiments/ (one-off drivers: the AI Hub on-device
runs, the 4-bit GEMM microbenchmark, fidelity and parity checks, checkpoint
build and round-trip, activation capture, and the load generator used to prove
the contention guard actually refuses) and benchmarks/edge_harness/runs/ (the
24 shell drivers behind published grids). Both directories carry a README
saying plainly that they are a record of what was run rather than a supported
interface -- most hardcode absolute paths from the machine they ran on. The
supported entry points remain the tools one level up.

One broken relative link fixed (comparison.md is under docs/engines/, not
docs/analysis/). All other links and every script reference in the docs now
resolve inside the repo.

Secret scan: the token in ~/.qai_hub/client.ini is absent from the worktree and
from all 61 commits. Worth noting that an earlier version of my own check
reported a false positive, because `grep -rl ... | head` takes its exit status
from head, which always succeeds -- the same class of mistake as the measurement
errors this project keeps recording. The AI Hub scripts authenticate by reading
that file at runtime and contain no credential.

README reframed for the new name and the wider scope: the repository is the
engine changes, the harness, the raw results, the baselines and the analyses,
not just a patch set.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:21:07 +08:00
tzhouamandClaude Opus 5 9264ff81ab Include the baselines, comparisons and study documents, not just the fork
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.

Added:

  baselines/                 llama.cpp runs (thread sweeps, NUMA sweeps, the
                             interleaved A/B passes behind the headline decode
                             figure) and the quality references: each build
                             against its own bf16 over 12 greedy prompts.
  benchmarks/.../results/    102 artifacts backing the published figures, each
                             with a provenance block naming the model,
                             quantization, thread set and build SHAs.
  docs/engines/              the engine analyses -- llama.cpp, ExecuTorch, MNN,
                             ncnn, vLLM, vLLM-Omni -- and the cross-engine and
                             device-simulation comparisons.
  docs/analysis/             the full study documents the summaries distil.
  docs/edge/comparisons.md   where this fork sits against llama.cpp and why:
                             decode is llama.cpp's, prefill is ours by ~2x, and
                             since neither engine exceeds 39% of the measured
                             222 GB/s ceiling the decode gap is the execution
                             model rather than the kernels.
  docs/edge/environment.md   pins and machine, and the correction below.

On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.

.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.

No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
2026-09-14 14:01:12 +08:00
wangyu 2e096788a6 [CI]Remove SoulX-Singer support (#6362)
Signed-off-by: wangyu <410167048@qq.com>
2026-08-23 07:16:32 +08:00
MaciejBalaNV b3270dec8d Cosmos3 transfer fix (#5614)
Signed-off-by: Maciej Bala <mbala@nvidia.com>
2026-08-20 11:40:51 -07:00
wangyu 273ffce130 [CI/Build] Strengthen pre-commit with markdownlint, SPDX, and policy hooks (#6273)
Signed-off-by: wangyu <410167048@qq.com>
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: wangyu <53896905+yenuo26@users.noreply.github.com>
2026-08-20 21:07:04 +08:00
SYLAR ce6ab0ca46 [Model] Add LTX-2.5 support (#6070)
Signed-off-by: lishunyang12 <lishunyang12@163.com>
2026-08-15 18:27:23 +08:00
2b08419d8b [CI] Per-model, per-entry-mode code coverage collection (#5593)
Signed-off-by: Shenglei Fu <sfu@confluent.io>
Signed-off-by: Shenglei Fu <117230642+ShengleiFu@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Alicia <115451386+congw729@users.noreply.github.com>
Co-authored-by: wangyu <53896905+yenuo26@users.noreply.github.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-08-14 15:36:56 +08:00
7d5376f215 [CI/Perf] Optimize Docker Caching (#5133)
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Signed-off-by: Doug Smith <dougbtv@gmail.com>
Co-authored-by: dougbtv <dosmith@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-22 19:14:55 -06:00
Shi Boao Bowieandbjf-frz a432b8a325 [Refactor][OutputProcessor 1/8]: clean up multimodal payload accumulation and move payload logic into MultimodalPayload & directory refactor (#4980)
Signed-off-by: Boao Shi <aoibosh@connect.hku.hk>
Signed-off-by: bjf-frz <frz123db@gmail.com>
Co-authored-by: bjf-frz <frz123db@gmail.com>
2026-07-15 15:50:19 +08:00
Zhou Taichang 86bdcaf3d5 [BDE] Phase 1: engine-level KV cache management for DreamZero (RFC #4366) (#4534) 2026-07-05 07:17:58 +08:00
1951dca786 [Model] add soulx-singer support (#3862)
Signed-off-by: Mr.Dong <1042542469@qq.com>
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
Co-authored-by: Mr.Dong <1042542469@qq.com>
Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>
Co-authored-by: Yueqian Lin <linyueqian@outlook.com>
2026-06-24 05:36:45 +00:00
b76291d6a1 [New Model] Add Lance (ByteDance) (#3710)
Signed-off-by: lishunyang <lishunyang03@gmail.com>
Signed-off-by: lishunyang12 <lishunyang12@163.com>
Signed-off-by: WeiQing Chen <david6666666@users.noreply.github.com>
Co-authored-by: lishunyang <lishunyang03@gmail.com>
Co-authored-by: Haco <923390377@qq.com>
2026-06-01 16:01:25 +08:00
amy-why-3459 2e8e3057bc [skip ci][Tests] Splitting Qwen3-omni's performance test cases (#3501)
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
2026-05-13 17:54:21 +08:00
Yueqian Lin b1ff695029 [Bugfix][CI] Vendor Qwen3-TTS and CosyVoice3 reference audio fixtures (#3267)
Signed-off-by: Yueqian Lin <linyueqian@outlook.com>
2026-04-30 08:44:05 +08:00
Hongsheng Liu 25b451a552 Codex revert pr reviewer (#2959)
Signed-off-by: hsliu <liuhongsheng4@huawei.com>
2026-04-21 21:43:15 +08:00
bjf-frz 71d81d475a [Enhancement] add pytorch profiler ops and memory record (#2472)
Signed-off-by: bjf-frz <frz123db@gmail.com>
2026-04-20 20:58:14 +08:00
a3903810fe [Config Refactor][2/N] Pipeline + Deploy Config Schema (#2383)
Signed-off-by: lishunyang <lishunyang12@163.com>
Signed-off-by: reidliu41 <reid201711@gmail.com>
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: xiaohajiayou <75477391+xiaohajiayou@users.noreply.github.com>
Co-authored-by: Alex Brooks <albrooks@redhat.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
2026-04-19 14:36:00 +12:00
Alex BrooksandDidan Deng 536f59b560 [Bugfix] Fix cache dit for Longcat & LTX2 (#2860)
Signed-off-by: Alex Brooks <albrooks@redhat.com>
Co-authored-by: Didan Deng <33117903+wtomin@users.noreply.github.com>
2026-04-17 20:18:00 +08:00
Hongsheng Liu f87674aa44 [skip ci]add skills (#2710)
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
2026-04-14 20:26:27 +08:00
Zhengyuan Su (苏政渊) dd9ca6feef [Bugfix] Include uv.lock in .gitignore (#2493)
Signed-off-by: Zhengyuan Su (苏政渊) <su.zhengyuan@u.nus.edu>
2026-04-05 07:43:00 +00:00
TJian 7ac8d613f6 [Version] Auto generate version using setuptool_scm (#1224)
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
2026-02-27 14:34:27 +08:00
75770c9309 [Rebase] Rebase to vllm v0.16.0 (#1357)
ComfyUI Integration -- Publish to Comfy registry / Publish Custom Node to registry (push) Failing after 44s
Signed-off-by: Taichang Zhou <tzhouam@connect.ust.hk>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
Co-authored-by: 汪志鹏 <wangzhipeng628@gmail.com>
Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: ZJY0516 <zhu.jiangyun@foxmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
2026-02-13 18:05:07 +08:00
Zeyu Huang | 黃澤宇 2538ae6332 [ComfyUI]: ComfyUI integration (#1113)
Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>
2026-02-09 09:54:41 +08:00
8939eb835d [Rebase] Rebase to vllm 0.12.0 (#335)
Signed-off-by: Rein Yang <ruiruyang2@gmail.com>
Signed-off-by: tzhouam <tzhouam@connect.ust.hk>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Rein Yang <ruiruyang2@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
2025-12-19 16:42:02 +08:00
65ca13102d [Feature] teacache integration (#179)
Signed-off-by: Prajwal A <prajwalanagani@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Signed-off-by: Samit <285365963@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Samit <285365963@qq.com>
2025-12-12 04:29:56 +00:00
NATURE f995211da0 [Feature] Omni Connector + ray supported (#215)
Signed-off-by: wzliu <wzliu@connect.hku.hk>
2025-12-10 20:15:03 +08:00
Jiangyun ZhuandKevin H. Luu 630cbe1ee6 [CI] add diffusion ci (#174)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2025-12-04 15:23:22 +00:00
Gao Han 88c00fb271 [Bugfix] Remove mandatory flash-attn dependency and optimzie docs (#113)
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
2025-11-30 18:20:44 +08:00
Roger WangandGao Han a5800cd2c2 [Doc] Misc documentation polishing (#98)
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Gao Han <gaohan19@huawei.com>
Co-authored-by: Gao Han <gaohan19@huawei.com>
2025-11-29 20:42:39 +08:00
Roger Wang 99c46e1adc [CI] Buildkite setup (#83)
Signed-off-by: Roger Wang <hey@rogerw.io>
2025-11-26 16:20:00 -08:00
Gaohan123 d4a364f052 fix huggingface download problem for spk_dict.pt
Signed-off-by: Gaohan123 <gaohan19@huawei.com>
2025-11-01 16:45:34 +08:00
Gaohan123 eafc7c44c3 fix error for pre-commit 2025-10-24 16:21:50 +08:00
hsliu dd40b97336 add RFC template 2025-10-18 08:09:19 +08:00
hsliu 81f04d3cf9 update ar_dit_test 2025-09-28 21:58:48 +08:00
hsliu ff9c7d4536 update gitignore 2025-09-26 17:54:52 +08:00
hsliu f403cec650 mv omni_llm.py to entrypoints 2025-09-26 17:39:00 +08:00
hsliu afe5c2692c feat: enhance serving functionality and add comprehensive examples
- Improve API server with better error handling and response formatting
- Enhance CLI with additional options for DiT stages and configuration
- Add comprehensive examples in examples/basic/ including:
  - API client with health checks and text generation
  - Docker setup and usage examples
  - Simple usage patterns for different scenarios
- Add utility scripts for model downloading and Docker setup
- Update documentation with implementation details and testing guidelines
- Fix configuration validation issues in OmniLLM
- Improve stage configuration handling for AR and DiT stages
- Add proper error handling and fallback mechanisms

Tested with Qwen3-0.6B model:
- Server starts successfully on port 8000
- Health and info endpoints working correctly
- Text generation with various parameters functioning
- API client examples working as expected
- CLI help and configuration options working properly
2025-09-26 15:47:55 +08:00
hsliu 154d9ca4ff Initial vLLM-omni repository structure
- Add complete directory structure for multimodal and non-autoregressive processing
- Include AsyncLLM, OmniEngine, and DiffusionEngine modules
- Add processors, schedulers, executors, and output processors
- Add API interfaces for Higress, Gradio, and ComfyUI
- Add configuration management and CLI interface
- Add comprehensive test structure and documentation
- Add pyproject.toml with dependencies and project configuration
- Add examples and documentation structure
2025-09-11 14:32:51 +08:00
Hongsheng Liu 089fa0ee95 Initial commit 2025-09-11 08:34:44 +08:00