tzhouamandClaude Opus 5
9264ff81ab
The packaging commit shipped the code and a summary of the numbers. It left out
everything the numbers are measured *against*, which is most of what makes them
checkable.
Added:
baselines/ llama.cpp runs (thread sweeps, NUMA sweeps, the
interleaved A/B passes behind the headline decode
figure) and the quality references: each build
against its own bf16 over 12 greedy prompts.
benchmarks/.../results/ 102 artifacts backing the published figures, each
with a provenance block naming the model,
quantization, thread set and build SHAs.
docs/engines/ the engine analyses -- llama.cpp, ExecuTorch, MNN,
ncnn, vLLM, vLLM-Omni -- and the cross-engine and
device-simulation comparisons.
docs/analysis/ the full study documents the summaries distil.
docs/edge/comparisons.md where this fork sits against llama.cpp and why:
decode is llama.cpp's, prefill is ours by ~2x, and
since neither engine exceeds 39% of the measured
222 GB/s ceiling the decode gap is the execution
model rather than the kernels.
docs/edge/environment.md pins and machine, and the correction below.
On "the modified vllm": there isn't one, and the doc says so plainly because it
is the obvious assumption. The vLLM checkout is clean upstream (0 commits
ahead, 0 uncommitted) and the installed wheel is stock 0.28.0+cpu. This fork
changes vLLM's CPU *behaviour* through vLLM's own extension points at import
time -- three CPU patches in vllm_omni/patch.py, the quantization-config
registry, the IR op registry, platform hooks -- specifically so the changes
survive a wheel upgrade rather than being lost on it. One of those patches is
why "KV sizing is not a memory lever" looked true for months: CpuPlatform
overwrote kv_cache_memory_bytes unconditionally, so the setting had no effect.
There is no patch series to apply; pin the version and let patch.py do the rest.
.gitignore gains explicit exceptions: upstream's blanket `*.json` and `results/`
rules were silently eating the measurement artifacts, which in this fork are the
substance rather than build output. Binary captures stay ignored.
No vLLM or llama.cpp source is redistributed -- only output produced with them.
NOTICE records both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1ax5wYLA6XbqEL1i8L9gZ
298 lines
5.4 KiB
Plaintext
298 lines
5.4 KiB
Plaintext
# Byte-compiled / optimized / DLL files
|
|
__pycache__/
|
|
*.py[cod]
|
|
*$py.class
|
|
|
|
# C extensions
|
|
*.so
|
|
|
|
# Distribution / packaging
|
|
.Python
|
|
build/
|
|
develop-eggs/
|
|
dist/
|
|
downloads/
|
|
eggs/
|
|
.eggs/
|
|
lib/
|
|
lib64/
|
|
parts/
|
|
sdist/
|
|
var/
|
|
wheels/
|
|
vllm/
|
|
share/python-wheels/
|
|
*.egg-info/
|
|
.installed.cfg
|
|
*.egg
|
|
MANIFEST
|
|
|
|
# PyInstaller
|
|
# Usually these files are written by a python script from a template
|
|
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
*.manifest
|
|
*.spec
|
|
|
|
# Installer logs
|
|
pip-log.txt
|
|
pip-delete-this-directory.txt
|
|
|
|
# Unit test / coverage reports
|
|
htmlcov/
|
|
.tox/
|
|
.nox/
|
|
.coverage
|
|
.coverage.*
|
|
.cache
|
|
nosetests.xml
|
|
coverage.xml
|
|
coverage-*.xml
|
|
coverage-*.xml.gz
|
|
*.cover
|
|
*.py,cover
|
|
.hypothesis/
|
|
.pytest_cache/
|
|
cover/
|
|
|
|
# Translations
|
|
*.mo
|
|
*.pot
|
|
|
|
# Django stuff:
|
|
*.log
|
|
local_settings.py
|
|
db.sqlite3
|
|
db.sqlite3-journal
|
|
|
|
# Flask stuff:
|
|
instance/
|
|
.webassets-cache
|
|
|
|
# Scrapy stuff:
|
|
.scrapy
|
|
|
|
# Sphinx documentation
|
|
docs/_build/
|
|
|
|
# PyBuilder
|
|
.pybuilder/
|
|
target/
|
|
|
|
# Jupyter Notebook
|
|
.ipynb_checkpoints
|
|
|
|
# IPython
|
|
profile_default/
|
|
ipython_config.py
|
|
|
|
# uv
|
|
uv.lock
|
|
|
|
# pyenv
|
|
# For a library or package, you might want to ignore these files since the code is
|
|
# intended to run in multiple environments; otherwise, check them in:
|
|
# .python-version
|
|
|
|
# pipenv
|
|
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
|
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
|
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
|
# install all needed dependencies.
|
|
#Pipfile.lock
|
|
|
|
# poetry
|
|
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
|
|
# This is especially recommended for binary packages to ensure reproducibility, and is more
|
|
# commonly ignored for libraries.
|
|
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
|
|
#poetry.lock
|
|
|
|
# pdm
|
|
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
|
|
#pdm.lock
|
|
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
|
|
# in version control.
|
|
# https://pdm.fming.dev/#use-with-ide
|
|
.pdm.toml
|
|
|
|
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
|
|
__pypackages__/
|
|
|
|
# Celery stuff
|
|
celerybeat-schedule
|
|
celerybeat.pid
|
|
|
|
# SageMath parsed files
|
|
*.sage.py
|
|
|
|
# Environments
|
|
.env
|
|
.venv
|
|
env/
|
|
venv/
|
|
ENV/
|
|
env.bak/
|
|
venv.bak/
|
|
|
|
# Spyder project settings
|
|
.spyderproject
|
|
.spyproject
|
|
|
|
# Rope project settings
|
|
.ropeproject
|
|
|
|
# mkdocs documentation
|
|
/site
|
|
|
|
# mypy
|
|
.mypy_cache/
|
|
.dmypy.json
|
|
dmypy.json
|
|
|
|
# Pyre type checker
|
|
.pyre/
|
|
|
|
# pytype static type analyzer
|
|
.pytype/
|
|
|
|
# Cython debug symbols
|
|
cython_debug/
|
|
|
|
# Claude
|
|
CLAUDE.md
|
|
/.claude/*
|
|
!.claude/skills/
|
|
!.claude/skills/readme.md
|
|
!.claude/skills/add-diffusion-model/
|
|
!.claude/skills/add-diffusion-model/SKILL.md
|
|
!.claude/skills/add-diffusion-model/references/
|
|
!.claude/skills/add-diffusion-model/references/*.md
|
|
!.claude/skills/add-tts-model/
|
|
!.claude/skills/add-tts-model/SKILL.md
|
|
!.claude/skills/review-pr/
|
|
!.claude/skills/review-pr/SKILL.md
|
|
!.claude/skills/review-pr/references/
|
|
!.claude/skills/review-pr/references/*.md
|
|
|
|
# Codex
|
|
AGENTS.md
|
|
.codex
|
|
.codex/
|
|
|
|
# cursor
|
|
.cursor/
|
|
|
|
# scripts
|
|
/scripts/
|
|
|
|
# PyCharm
|
|
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
|
|
# be added to the global gitignore or merged into this project gitignore. For a PyCharm
|
|
# project, it is recommended to include the following files:
|
|
# .idea/
|
|
# *.iml
|
|
# *.ipr
|
|
# *.iws
|
|
# .idea/
|
|
|
|
# vLLM-Omni specific
|
|
# Model files and checkpoints
|
|
checkpoints/
|
|
*.bin
|
|
*.safetensors
|
|
*.pt
|
|
*.pth
|
|
|
|
# Cache directories
|
|
cache/
|
|
!vllm_omni/diffusion/cache/
|
|
!tests/diffusion/cache/
|
|
.cache/
|
|
diffusion_cache/
|
|
kv_cache/
|
|
!vllm_omni/experimental/ar_diffusion/kv_cache/
|
|
|
|
# Logs
|
|
logs/
|
|
*.log
|
|
|
|
# Temporary files
|
|
tmp/
|
|
temp/
|
|
.tmp/
|
|
|
|
# IDE files
|
|
.vscode/
|
|
.idea/
|
|
*.swp
|
|
*.swo
|
|
*~
|
|
|
|
# OS files
|
|
.DS_Store
|
|
.DS_Store?
|
|
._*
|
|
.Spotlight-V100
|
|
.Trashes
|
|
ehthumbs.db
|
|
Thumbs.db
|
|
|
|
# Data files
|
|
data/
|
|
datasets/
|
|
*.csv
|
|
*.json
|
|
!recipes/cosmos3/negative_prompt.json
|
|
!tests/dfx/perf/tests/*.json
|
|
!apps/ComfyUI-vLLM-Omni/example_workflows/*.json
|
|
*.jsonl
|
|
*.parquet
|
|
|
|
# Output files
|
|
outputs/
|
|
!vllm_omni/outputs/
|
|
results/
|
|
generated/
|
|
output_*/
|
|
/tests/e2e/accuracy/artifacts/
|
|
|
|
# Configuration overrides
|
|
configs/local.yaml
|
|
configs/production.yaml
|
|
configs/development.yaml
|
|
|
|
# Docker
|
|
Dockerfile.dev
|
|
discussion
|
|
tmp_test
|
|
|
|
# Auto-generated version file (created by setuptools_scm during build)
|
|
vllm_omni/_version.py
|
|
# output files
|
|
*.wav
|
|
# Vendored test reference audio (small, intentionally checked in)
|
|
!tests/assets/**/*.wav
|
|
.worktrees/
|
|
# CI overlay yamls materialized from tests/utils.py:_CI_OVERLAYS at test time
|
|
tests/.ci_generated/
|
|
|
|
# Local Lance precision-matching dump scripts (not part of the repo)
|
|
lance_compare/
|
|
|
|
# Linting binaries downloaded by local pre-commit hooks
|
|
shellcheck*/
|
|
|
|
# --- vllm-embedding ---------------------------------------------------------
|
|
# Measurement artifacts are the substance of this fork, not build output: every
|
|
# published figure is traceable to a file below. They are tracked despite the
|
|
# blanket `*.json` and `results/` rules above. Binary captures (.npz, .onnx,
|
|
# .wav, .npy) stay ignored.
|
|
!benchmarks/edge_harness/results/
|
|
!benchmarks/edge_harness/results/**
|
|
!baselines/
|
|
!baselines/**
|
|
benchmarks/edge_harness/results/**/*.npz
|
|
benchmarks/edge_harness/results/**/*.onnx
|
|
benchmarks/edge_harness/results/**/*.npy
|
|
benchmarks/edge_harness/results/**/*.wav
|