anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Merge vLLM-Omni v0.29.0rc1 into edge-infer/omni-0.29.0

Browse Source 
The pair for vLLM v0.29.0: rc1's docs pin vllm==0.29.0 and a static scan of
rc1's vllm.* imports against the v0.29.0 tree finds nothing missing, while
our 0.28-based tree needs seven modules v0.29.0 no longer has
(vllm.inputs.preprocess, entrypoints.launcher, openai.cli_args,
openai.engine.protocol, serve.utils.server_utils, layers.batch_invariant,
models.hunyuan_v1).

Ten files conflicted. Upstream's structure is the base everywhere; the
fork's features are re-attached to it:
- deploy_profile / apply_deploy_profile run before rc1's resolve_omni_config
  pops deploy_config (async_omni_engine, cli/serve); the deploy-profile
  selector and the resolve_model_config_path chain it needs stay in
  entrypoints/utils.py; stage_overrides takes rc1's dict type.
- config_factory keeps the lazy diffusion output-type import; the diffusion
  registry import in rc1's new config/resolver.py is made lazy for the same
  WP6 reason (TTS path must not load diffusion).
- requirements: our core/media split kept; rc1's transformers>=5.13 floor
  and kernels==0.16.1 applied in core.txt / media.txt.
- TTS: rc1's split_granularity and bounded seed plus our output_mode=
  codec_tokens, _SessionInbox cancel handling and codec DELTA accumulation;
  our private _conditioning_cache_salt is superseded by rc1's public
  tts_adapters/base.py::conditioning_cache_salt.
- stage_runtime: rc1's single ExitStack G2 path (lock released before READY)
  replaces our two-path block, with the four worker_phase names the init
  profiler consumes still emitted; _cpu_placed / host admission kept.

Not yet run against vLLM 0.29.0: the tests listed in the W6 record.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
tzhouamandClaude Fable 5.1 committed 2026-09-18 09:57:20 +08:00
2 parents 4e3e8a1ebc aff7d64948
commit 33de5d8629
513 files changed
+35146 -10927
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.buildkite/amd/scripts/bootstrap-amd-omni.sh
+78 -88
View File
@@ -14,6 +14,8 @@ set -euo pipefail
# shellcheck disable=SC1091
source .buildkite/common/scripts/resolve_skip_ci.sh
AMD_SUITE_SELECTOR=".buildkite/amd/scripts/select_test_suites.py"
if [[ -z "${RUN_ALL:-}" ]]; then
RUN_ALL=0
fi
@@ -30,31 +32,20 @@ if [[ -z "${AMD_MIRROR_HW:-}" ]]; then
AMD_MIRROR_HW="amdproduction"
fi
fail_fast() {
DISABLE_LABEL="ci-no-fail-fast"
# If BUILDKITE_PULL_REQUEST != "false", then we check the PR labels using curl and jq
if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
PR_LABELS=$(curl -s "https://api.github.com/repos/vllm-project/vllm-omni/pulls/$BUILDKITE_PULL_REQUEST" | jq -r '.labels[].name')
if [[ $PR_LABELS == *"$DISABLE_LABEL"* ]]; then
echo false
else
echo true
fi
else
echo false # not a PR or BUILDKITE_PULL_REQUEST not set
PR_LABELS=""
# Explicit debug suites do not depend on PR labels.
if [[ "${BUILDKITE_PULL_REQUEST:-false}" != "false" && -z "${DEBUG_TEST_YAML:-}" ]]; then
if ! PR_LABELS=$(curl -fsSL \
"https://api.github.com/repos/vllm-project/vllm-omni/pulls/$BUILDKITE_PULL_REQUEST" \
| jq -r '.labels[].name'); then
echo "ERROR: Could not read PR labels; refusing to select an incorrect AMD test tier." >&2
exit 1
fi
}
fi
check_run_all_label() {
RUN_ALL_LABEL="ready-run-all-tests"
# If BUILDKITE_PULL_REQUEST != "false", then we check the PR labels using curl and jq
if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
PR_LABELS=$(curl -s "https://api.github.com/repos/vllm-project/vllm-omni/pulls/$BUILDKITE_PULL_REQUEST" | jq -r '.labels[].name')
if [[ $PR_LABELS == *"$RUN_ALL_LABEL"* ]]; then
echo true
else
echo false
fi
fail_fast() {
if [[ "${BUILDKITE_PULL_REQUEST:-false}" != "false" ]]; then
echo true
else
echo false # not a PR or BUILDKITE_PULL_REQUEST not set
fi
@@ -64,6 +55,57 @@ if [[ -z "${COV_ENABLED:-}" ]]; then
COV_ENABLED=0
fi
resolve_test_specs() {
local selected_specs
if ! selected_specs=$(python3 "$AMD_SUITE_SELECTOR" \
--branch "$BUILDKITE_BRANCH" \
--labels "$PR_LABELS" \
--debug-test-yaml "${DEBUG_TEST_YAML:-}"); then
exit 1
fi
TEST_SPECS=()
while IFS= read -r suite_spec; do
[[ -z "$suite_spec" ]] && continue
TEST_SPECS+=("$suite_spec")
done <<< "$selected_specs"
}
filter_test_specs_by_skip_ci() {
local suite_spec level decision
local skip_all=0
local -a runnable_specs=()
for suite_spec in "${TEST_SPECS[@]}"; do
case "$suite_spec" in
READY_TESTS:*) level="l2" ;;
MERGE_TESTS:*) level="l3" ;;
*)
echo "ERROR: unknown AMD test suite spec '$suite_spec'" >&2
exit 1
;;
esac
if decision=$(python3 "$SKIP_CI_PY" gate amd "$level"); then
echo "Skipping AMD $level suite due to $decision decision."
[[ "$decision" == "skip-all" ]] && skip_all=1
else
runnable_specs+=("$suite_spec")
fi
done
TEST_SPECS=("${runnable_specs[@]}")
if [[ ${#TEST_SPECS[@]} -eq 0 ]]; then
if [[ $skip_all -eq 1 ]]; then
buildkite-agent annotate \
":memo: CI skipped — docs or pytest skip-mark changes only" \
--style "info" 2>/dev/null || true
fi
echo "No AMD L2/L3 suites remain after skip-ci filtering."
exit 0
fi
}
upload_pipeline() {
echo "Uploading pipeline..."
# Install minijinja
@@ -84,7 +126,7 @@ upload_pipeline() {
# (WIP) Use pipeline generator instead of jinja template
if [ -e ".buildkite/amd/pipeline_generator/pipeline_generator.py" ]; then
python -m pip install click pydantic
python .buildkite/amd/pipeline_generator/pipeline_generator.py --run_all=$RUN_ALL --list_file_diff="$LIST_FILE_DIFF" --nightly="$NIGHTLY" --mirror_hw="$AMD_MIRROR_HW"
python .buildkite/amd/pipeline_generator/pipeline_generator.py --run_all="$RUN_ALL" --list_file_diff="$LIST_FILE_DIFF" --nightly="$NIGHTLY" --mirror_hw="$AMD_MIRROR_HW"
buildkite-agent pipeline upload .buildkite/amd/pipeline.yaml
exit 0
fi
@@ -97,49 +139,12 @@ upload_pipeline() {
cd .buildkite/amd
# Select test definition file: merge suite for main, ready suite for PRs.
# For debugging, DEBUG_TEST_YAML accepts a comma-separated list containing
# "merge" and/or "ready" (case-insensitive). Multiple suites are combined
# before rendering so they share one amd-build step.
if [[ -n "${DEBUG_TEST_YAML:-}" ]]; then
declare -a DEBUG_TEST_SPECS=()
declare -A SEEN_DEBUG_TESTS=()
IFS=',' read -ra REQUESTED_DEBUG_TESTS <<< "${DEBUG_TEST_YAML,,}"
for requested_test in "${REQUESTED_DEBUG_TESTS[@]}"; do
# Permit readable values such as "ready, merge," and ignore the
# empty item produced by a trailing comma.
requested_test="${requested_test//[[:space:]]/}"
[[ -z "$requested_test" ]] && continue
if [[ -n "${SEEN_DEBUG_TESTS[$requested_test]:-}" ]]; then
echo "ERROR: duplicate DEBUG_TEST_YAML suite '$requested_test'" >&2
exit 1
fi
SEEN_DEBUG_TESTS[$requested_test]=1
case "$requested_test" in
ready)
DEBUG_TEST_SPECS+=("READY_TESTS:test-amd-ready.yml")
;;
merge)
DEBUG_TEST_SPECS+=("MERGE_TESTS:test-amd-merge.yml")
;;
*)
echo "ERROR: DEBUG_TEST_YAML entries must be 'merge' or 'ready', got '$requested_test'" >&2
exit 1
;;
esac
done
if [[ ${#DEBUG_TEST_SPECS[@]} -eq 0 ]]; then
echo "ERROR: DEBUG_TEST_YAML did not contain a test suite" >&2
exit 1
elif [[ ${#DEBUG_TEST_SPECS[@]} -eq 1 ]]; then
TEST_YAML="${DEBUG_TEST_SPECS[0]#*:}"
else
TEST_YAML=$(mktemp "${TMPDIR:-/tmp}/amd-debug-tests.XXXXXX.yml")
python - "$TEST_YAML" "${DEBUG_TEST_SPECS[@]}" <<'PY'
# Multiple label-selected or debug-selected suites share one image build.
if [[ ${#TEST_SPECS[@]} -eq 1 ]]; then
TEST_YAML="${TEST_SPECS[0]#*:}"
else
TEST_YAML=$(mktemp "${TMPDIR:-/tmp}/amd-selected-tests.XXXXXX.yml")
python - "$TEST_YAML" "${TEST_SPECS[@]}" <<'PY'
import sys
import yaml
@@ -172,13 +177,8 @@ for suite_spec in suite_specs:
with open(output_path, "w", encoding="utf-8") as output_file:
yaml.safe_dump(combined, output_file, sort_keys=False)
PY
fi
echo "DEBUG_TEST_YAML override: using ${DEBUG_TEST_SPECS[*]}"
elif [[ $BUILDKITE_BRANCH == "main" ]]; then
TEST_YAML="test-amd-merge.yml"
else
TEST_YAML="test-amd-ready.yml"
fi
echo "AMD test suites: ${TEST_SPECS[*]}"
(
set -x
@@ -198,7 +198,7 @@ PY
> pipeline.yaml
)
cat pipeline.yaml
if [[ "$TEST_YAML" == "${TMPDIR:-/tmp}/amd-debug-tests."*.yml ]]; then
if [[ "$TEST_YAML" == "${TMPDIR:-/tmp}/amd-selected-tests."*.yml ]]; then
rm -f -- "$TEST_YAML"
fi
buildkite-agent artifact upload pipeline.yaml
@@ -219,12 +219,10 @@ if [[ $BUILDKITE_BRANCH == "main" ]]; then
file_diff=$(get_diff_main)
fi
# Early exit: unified skip-ci (docs / skip marks) and CI-yaml-only level targeting.
if [[ $BUILDKITE_BRANCH == "main" ]]; then
gate_bootstrap_ci amd l3
else
gate_bootstrap_ci amd l2
fi
# Resolve PR tier labels before skip-ci so ready and merge suites can be
# filtered independently for CI-YAML-only changes.
resolve_test_specs
filter_test_specs_by_skip_ci
patterns=(
"docker/Dockerfile"
@@ -274,14 +272,6 @@ for file in $file_diff; do
fi
done
# Check for ready-run-all-tests label
LABEL_RUN_ALL=$(check_run_all_label)
if [[ $LABEL_RUN_ALL == true ]]; then
RUN_ALL=1
NIGHTLY=1
echo "Found 'ready-run-all-tests' label. Running all tests including optional tests."
fi
# Decide whether to use precompiled wheels
# Relies on existing patterns array as a basis.
if [[ -n "${VLLM_USE_PRECOMPILED:-}" ]]; then
.buildkite/amd/scripts/select_test_suites.py
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env python3
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Select AMD L2/L3 suites after Buildkite has created an AMD build.
This repository-side selector does not decide which GitHub label events start
the external ``vllm-omni-amd-ci`` pipeline. Its Buildkite PR build condition
must admit the relevant label event before this code can inspect the labels.
"""
from __future__ import annotations
import argparse
import sys
from collections.abc import Iterable
SUITE_SPECS = {
"ready": "READY_TESTS:test-amd-ready.yml",
"merge": "MERGE_TESTS:test-amd-merge.yml",
}
def _parse_debug_suites(value: str) -> tuple[str, ...]:
suites = tuple(item.strip().lower() for item in value.split(",") if item.strip())
if not suites:
raise ValueError("DEBUG_TEST_YAML did not contain a test suite")
seen: set[str] = set()
for suite in suites:
if suite not in SUITE_SPECS:
raise ValueError(f"DEBUG_TEST_YAML entries must be 'merge' or 'ready', got {suite!r}")
if suite in seen:
raise ValueError(f"duplicate DEBUG_TEST_YAML suite {suite!r}")
seen.add(suite)
return suites
def select_amd_test_suites(
*,
branch: str,
labels: Iterable[str],
debug_test_yaml: str = "",
) -> tuple[str, ...]:
"""Return ordered suite names while preserving legacy unlabeled PRs."""
if debug_test_yaml:
return _parse_debug_suites(debug_test_yaml)
if branch == "main":
return ("merge",)
label_set = {label.strip() for label in labels if label.strip()}
selected = tuple(suite for label, suite in (("ready", "ready"), ("merge-test", "merge")) if label in label_set)
# AMD historically uploaded L2 for every PR build. Keep that behavior for
# triggers such as amd-test while adding tier selection for ready/merge-test.
return selected or ("ready",)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--branch", required=True)
parser.add_argument("--labels", default="")
parser.add_argument("--debug-test-yaml", default="")
args = parser.parse_args()
labels = tuple(args.labels.splitlines())
try:
suites = select_amd_test_suites(
branch=args.branch,
labels=labels,
debug_test_yaml=args.debug_test_yaml,
)
except ValueError as exc:
parser.error(str(exc))
label_set = {label.strip() for label in labels if label.strip()}
if not args.debug_test_yaml and args.branch != "main" and not ({"ready", "merge-test"} & label_set):
print(
"No AMD L2/L3 tier label found; preserving the legacy ready-suite fallback.",
file=sys.stderr,
)
if "nightly-test" in label_set:
print(
"AMD nightly-test has no suite yet and does not affect L2/L3 selection.",
file=sys.stderr,
)
for suite in suites:
print(SUITE_SPECS[suite])
return 0
if __name__ == "__main__":
raise SystemExit(main())
.buildkite/amd/test-amd-ready.yml
+12 -1
View File
@@ -107,6 +107,11 @@ steps:
- export VLLM_LOGGING_LEVEL=DEBUG
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- timeout 20m pytest -s -v tests/diffusion/distributed/test_sequence_parallel.py -m core_model
- >-
timeout 20m pytest -s -v
tests/diffusion/models/ltx2/test_ltx2_transformer_ulysses.py
-m 'core_model and rocm and cards_2'
--run-level "core_model"
- label: "Diffusion GPU Worker Test"
agent_pool: mi300_2
@@ -122,7 +127,13 @@ steps:
mirror_hardwares: [amdproduction]
grade: Blocking
commands:
- timeout 15m pytest -sv tests/diffusion/models/ -m 'core_model and cuda' --run-level "core_model"
# Preserve legacy one-GPU tests without cards_1 while excluding every
# registered multi-GPU marker from this one-GPU job.
- >-
timeout 15m pytest -sv tests/diffusion/models/
-m 'core_model and cuda and not
(cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)'
--run-level "core_model"
- label: "Diffusion · Wan22 Test"
agent_pool: mi300_1
.buildkite/common/ci_mirror_hardwares.yml
+66 -12
View File
@@ -17,6 +17,8 @@
# a2b3_npu_1 — ascend-a2b3, resource_class npu-1 (A2/B3 CI image)
# a2b3_npu_4 — ascend-a2b3, resource_class npu-4
# a3_npu_2 — ascend-a3, resource_class npu-2 (A3 CI image tag …:$BUILDKITE_COMMIT-a3)
# a5_npu_2 — ascend-a5, resource_class npu-2
# 310p_npu_1 — ascend-310p, resource_class npu-1
#
# B200 (b200-k8s): matches vLLM device ``b200-k8s`` from ci-infra
# (AgentQueue.B200_K8S + b200_plugin_template). Queue routes the K8s pool;
@@ -26,6 +28,9 @@
# packed onto g6.12xlarge L4 nodes (``vllm.ci/gpu-pool=l4x4``). GPU / CPU /
# memory / shm requests scale with card count (1 GPU = 10 CPU / 40Gi / 24Gi shm).
# Retry covers lost agents / killed pods, not pytest or OOM (exit 1 / 137).
# The Kubernetes HF secret deliberately uses VLLM_CI_HF_TOKEN. Scheduled
# build-level HF_TOKEN values can override pod env entries with the same name;
# upload_pipeline.py restores the secret-backed value inside the job shell.
x-npu-k8s-plugins: &npu_k8s_plugins
- kubernetes:
@@ -84,7 +89,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /fsx/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -135,7 +140,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /fsx/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -186,7 +191,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /fsx/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -237,7 +242,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /fsx/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -271,7 +276,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /root/.cache/huggingface
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -306,7 +311,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /root/.cache/huggingface
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -341,7 +346,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /root/.cache/huggingface
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -376,7 +381,7 @@ mirror_hardwares:
env:
- name: HF_HOME
value: /root/.cache/huggingface
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -426,7 +431,7 @@ mirror_hardwares:
value: "0"
- name: HF_HOME
value: /raid/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -477,7 +482,7 @@ mirror_hardwares:
value: "0"
- name: HF_HOME
value: /raid/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -528,7 +533,7 @@ mirror_hardwares:
value: "0"
- name: HF_HOME
value: /raid/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -579,7 +584,7 @@ mirror_hardwares:
value: "0"
- name: HF_HOME
value: /raid/hf_cache
- name: HF_TOKEN
- name: VLLM_CI_HF_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
@@ -646,3 +651,52 @@ mirror_hardwares:
resource_class: "npu-16"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a3"
plugins: *npu_k8s_plugins
a5_npu_2:
agents:
queue: "ascend-a5"
resource_class: "npu-2"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
plugins: *npu_k8s_plugins
a5_npu_4:
agents:
queue: "ascend-a5"
resource_class: "npu-4"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
plugins: *npu_k8s_plugins
a5_npu_8:
agents:
queue: "ascend-a5"
resource_class: "npu-8"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
plugins: *npu_k8s_plugins
a5_npu_16:
agents:
queue: "ascend-a5"
resource_class: "npu-16"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
plugins: *npu_k8s_plugins
310p_npu_1:
agents:
queue: "ascend-310p"
resource_class: "npu-1"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-310p"
plugins: *npu_k8s_plugins
310p_npu_2:
agents:
queue: "ascend-310p"
resource_class: "npu-2"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-310p"
plugins: *npu_k8s_plugins
310p_npu_4:
agents:
queue: "ascend-310p"
resource_class: "npu-4"
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-310p"
plugins: *npu_k8s_plugins
.buildkite/common/scripts/upload_pipeline.py
+22 -1
View File
@@ -70,7 +70,15 @@ from tests.helpers.mark import ( # noqa: E402
LOG = "upload_pipeline"
BOOTSTRAP_STEPS_FILENAME = "bootstrap-upload-steps.yml"
BOOTSTRAP_IMAGE_BUILD_KEYS = frozenset({"image-build", "image-build-a2", "image-build-a3"})
BOOTSTRAP_IMAGE_BUILD_KEYS = frozenset(
{
"image-build",
"image-build-a2",
"image-build-a3",
"image-build-a5",
"image-build-310p",
}
)
BOOTSTRAP_UPLOAD_IF_KEYS = {
"upload-ready-pipeline": "ready",
"upload-merge-pipeline": "merge",
@@ -79,6 +87,8 @@ BOOTSTRAP_UPLOAD_IF_KEYS = {
}
E2E_GROUP_MARKER = "E2E Test"
CI_MIRROR_HARDWARES_PATH = ROOT / ".buildkite/common/ci_mirror_hardwares.yml"
CUDA_HF_TOKEN_ENV = "VLLM_CI_HF_TOKEN"
CUDA_HF_TOKEN_EXPORT = f'if [ -n "$${{{CUDA_HF_TOKEN_ENV}:-}}" ]; then export HF_TOKEN="$${{{CUDA_HF_TOKEN_ENV}}}"; fi'
# Bootstrap Buildkite ``if`` expressions.
# ``*_MAIN_IF``: main + env schedule. ``*_LABEL_IF``: PR label (and/or composed with MAIN).
@@ -501,6 +511,17 @@ def _expand_mirror_hardwares(step: dict[str, Any]) -> dict[str, Any] | None:
expanded = copy.deepcopy(preset)
merged = {key: value for key, value in step.items() if key != "mirror_hardwares"} | expanded
# A scheduled build's HF_TOKEN may override a pod env entry with the same
# name. The preset injects the Kubernetes secret under a collision-proof
# alias; restore the conventional name after Buildkite has merged env.
if any(preset_name == chip or preset_name.startswith(f"{chip}_") for chip in _cuda_mirror_chips()):
commands = merged.get("commands")
if isinstance(commands, list):
merged["commands"] = [CUDA_HF_TOKEN_EXPORT, *commands]
elif commands is None:
merged["commands"] = [CUDA_HF_TOKEN_EXPORT]
else:
merged["commands"] = [CUDA_HF_TOKEN_EXPORT, commands]
# Preset retry (K8S_RETRY on l4_*) must not clobber a step that opted out.
if "retry" in step:
merged["retry"] = step["retry"]
.buildkite/cuda/test-merge.yml
+1
View File
@@ -155,6 +155,7 @@ steps:
source_file_dependencies:
- tests/e2e/offline_inference/test_minicpmo_4_5.py
- tests/e2e/online_serving/test_minicpmo_4_5.py
- vllm_omni/transformers_utils/chat_templates/minicpmo45_native.jinja
- vllm_omni/model_executor/models/minicpmo_4_5/
- vllm_omni/model_executor/models/step_audio2/
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
.buildkite/cuda/test-ready.yml
+2 -1
View File
@@ -93,7 +93,7 @@ steps:
commands:
- |
timeout 15m bash -c "
pytest -sv tests/engine/ tests/model_executor/ -m 'core_model and cuda' --run-level 'core_model'
pytest -sv tests/engine/ tests/model_executor/ tests/worker/test_batched_omni_output.py -m 'core_model and cuda' --run-level 'core_model'
"
mirror_hardwares: l4_1
@@ -188,6 +188,7 @@ steps:
- label: "Omni · MiniCPM-o 4.5 Online Test"
source_file_dependencies:
- tests/e2e/online_serving/test_minicpmo_4_5.py
- vllm_omni/transformers_utils/chat_templates/minicpmo45_native.jinja
- vllm_omni/model_executor/models/minicpmo_4_5/
- vllm_omni/model_executor/models/step_audio2/
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
.buildkite/npu/bootstrap-upload-steps.yml
+30 -104
View File
@@ -3,10 +3,10 @@
steps:
- label: ":buildkit: Build and Push NPU Test Image - A2"
key: image-build-a2
agents:
agents: &npu-image-build-agents
queue: "ascend-a2b3"
resource_class: "npu-1"
plugins:
plugins: &npu-image-build-plugins
- kubernetes:
metadata:
annotations:
@@ -46,123 +46,47 @@ steps:
IMAGE_NAME: "vllm-omni-ci-npu"
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
command: |
set -ex
echo "--- Building and pushing NPU Test Image"
echo "Image: $${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
echo "buildkitd address: $${BUILDKITD_ADDR}"
if ! command -v buildctl &> /dev/null; then
echo "Installing buildctl..."
mkdir -p /tmp/buildkit
BUILDKIT_VERSION="v0.29.0"
wget -q "https://github.com/moby/buildkit/releases/download/$${BUILDKIT_VERSION}/buildkit-$${BUILDKIT_VERSION}.linux-arm64.tar.gz" -O /tmp/buildkit.tar.gz
tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
cp /tmp/buildkit/bin/buildctl /usr/local/bin/
fi
export DOCKER_CONFIG=/home/user/.docker
buildctl \
--addr="$${BUILDKITD_ADDR}" \
--tlscacert=/home/user/.docker/ca.pem \
--tlscert=/home/user/.docker/cert.pem \
--tlskey=/home/user/.docker/key.pem \
build \
--frontend dockerfile.v0 \
--local context=. \
--local dockerfile=./docker \
--opt filename=Dockerfile.npu.ci \
--secret id=dockerconfig,src=/home/user/.docker/config.json \
--output type=image,name=$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG},push=true \
--progress=plain
echo "--- Image pushed successfully"
echo "$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}"
- label: ":buildkit: Build and Push NPU Test Image - A3"
key: image-build-a3
agents:
queue: "ascend-a2b3"
resource_class: "npu-1"
plugins:
- kubernetes:
metadata:
annotations:
vault.hashicorp.com/agent-init-first: "true"
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/agent-inject-perms-ca.pem: "0400"
vault.hashicorp.com/agent-inject-perms-cert.pem: "0400"
vault.hashicorp.com/agent-inject-perms-config.json: "0400"
vault.hashicorp.com/agent-inject-perms-key.pem: "0400"
vault.hashicorp.com/agent-inject-secret-ca.pem: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-secret-cert.pem: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-secret-config.json: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-secret-key.pem: internal/data/ascend/buildkitd
vault.hashicorp.com/agent-inject-template-ca.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.RootCA }}\n{{- end }}"
vault.hashicorp.com/agent-inject-template-cert.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaCert }}\n{{- end }}"
vault.hashicorp.com/agent-inject-template-config.json: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.dockerConfig }}\n{{- end }}"
vault.hashicorp.com/agent-inject-template-key.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaKey }}\n{{- end }}"
vault.hashicorp.com/agent-pre-populate-only: "true"
vault.hashicorp.com/agent-run-as-group: "1000"
vault.hashicorp.com/agent-run-as-user: "1000"
vault.hashicorp.com/agent-service-account-token-volume-name: token-vol
vault.hashicorp.com/role: ascend-gha-runners
vault.hashicorp.com/secret-volume-path: /home/user/.docker/
vault.hashicorp.com/tls-skip-verify: "true"
podSpecPatch:
volumes:
- name: token-vol
projected:
defaultMode: 420
sources:
- serviceAccountToken:
audience: api
expirationSeconds: 600
path: token
agents: *npu-image-build-agents
plugins: *npu-image-build-plugins
env:
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}-a3"
IMAGE_NAME: "vllm-omni-ci-npu"
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
command: |
set -ex
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}" -a3
echo "--- Building and pushing NPU Test Image"
echo "Image: $${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
echo "buildkitd address: $${BUILDKITD_ADDR}"
- label: ":buildkit: Build and Push NPU Test Image - A5"
key: image-build-a5
agents: *npu-image-build-agents
plugins: *npu-image-build-plugins
env:
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}-a5"
IMAGE_NAME: "vllm-omni-ci-npu"
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}" -a5
if ! command -v buildctl &> /dev/null; then
echo "Installing buildctl..."
mkdir -p /tmp/buildkit
BUILDKIT_VERSION="v0.29.0"
wget -q "https://github.com/moby/buildkit/releases/download/$${BUILDKIT_VERSION}/buildkit-$${BUILDKIT_VERSION}.linux-arm64.tar.gz" -O /tmp/buildkit.tar.gz
tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
cp /tmp/buildkit/bin/buildctl /usr/local/bin/
fi
export DOCKER_CONFIG=/home/user/.docker
buildctl \
--addr="$${BUILDKITD_ADDR}" \
--tlscacert=/home/user/.docker/ca.pem \
--tlscert=/home/user/.docker/cert.pem \
--tlskey=/home/user/.docker/key.pem \
build \
--frontend dockerfile.v0 \
--local context=. \
--local dockerfile=./docker \
--opt filename=Dockerfile.npu.ci.a3 \
--secret id=dockerconfig,src=/home/user/.docker/config.json \
--output type=image,name=$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG},push=true \
--progress=plain
echo "--- Image pushed successfully"
echo "$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
- label: ":buildkit: Build and Push NPU Test Image - 310P"
key: image-build-310p
agents: *npu-image-build-agents
plugins: *npu-image-build-plugins
env:
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}-310p"
IMAGE_NAME: "vllm-omni-ci-npu"
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}" -310p
- label: "Upload Ready Pipeline"
depends_on:
- image-build-a2
- image-build-a3
- image-build-a5
- image-build-310p
key: upload-ready-pipeline
agents:
queue: "cpu_queue_premerge"
@@ -174,6 +98,8 @@ steps:
depends_on:
- image-build-a2
- image-build-a3
- image-build-a5
- image-build-310p
key: upload-nightly-pipeline
commands:
- python3 .buildkite/common/scripts/upload_pipeline.py --upload .buildkite/npu/test-npu-nightly.yml
.buildkite/npu/scripts/build-npu-image.sh
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
set -ex
dockerfile=${1:?Dockerfile is required}
image_tag=${2:?Image tag is required}
device_suffix=${3:-}
image_name=${IMAGE_NAME:-vllm-omni-ci-npu}
image_registry=${IMAGE_REGISTRY:-swr.cn-southwest-2.myhuaweicloud.com/modelfoundry}
buildkitd_addr=${BUILDKITD_ADDR:-tcp://buildkitd-service.buildkitd:1234}
echo "--- Building and pushing NPU Test Image"
echo "Image: ${image_registry}/${image_name}:${image_tag}"
echo "Dockerfile: ${dockerfile}"
if ! command -v buildctl &> /dev/null; then
mkdir -p /tmp/buildkit
buildkit_version=v0.29.0
wget -q \
"https://github.com/moby/buildkit/releases/download/${buildkit_version}/buildkit-${buildkit_version}.linux-arm64.tar.gz" \
-O /tmp/buildkit.tar.gz
tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
cp /tmp/buildkit/bin/buildctl /usr/local/bin/
fi
build_args=()
if [[ -n "${device_suffix}" ]]; then
build_args+=(--opt "build-arg:VLLM_ASCEND_DEVICE_SUFFIX=${device_suffix}")
fi
export DOCKER_CONFIG=/home/user/.docker
buildctl \
--addr="${buildkitd_addr}" \
--tlscacert=/home/user/.docker/ca.pem \
--tlscert=/home/user/.docker/cert.pem \
--tlskey=/home/user/.docker/key.pem \
build \
--frontend dockerfile.v0 \
--local context=. \
--local dockerfile=./docker \
--opt "filename=${dockerfile}" \
"${build_args[@]}" \
--secret id=dockerconfig,src=/home/user/.docker/config.json \
--output "type=image,name=${image_registry}/${image_name}:${image_tag},push=true" \
--progress=plain
echo "--- Image pushed successfully"
echo "${image_registry}/${image_name}:${image_tag}"
.buildkite/npu/test-npu-ready.yml
+53 -21
View File
@@ -1,27 +1,59 @@
steps:
- label: "Platforms Test"
key: ready-platforms-test
- group: ":card_index_dividers: Simple Test"
depends_on: upload-ready-pipeline
mirror_hardwares: a3_npu_2
commands:
- >-
pytest -s -v
tests/platforms/npu/test_minicpmo_code2wav_npugraph.py
-m 'core_model and npu and A3 and omni'
--run-level 'core_model'
steps:
- label: "Platforms Test"
key: ready-platforms-test
mirror_hardwares: a3_npu_2
commands:
- >-
pytest -s -v
tests/platforms/npu/test_minicpmo_code2wav_npugraph.py
-m 'core_model and npu and A3 and omni'
--run-level 'core_model'
- label: "Diffusion NPU layer tests"
key: ready-diffusion-npu-layer-tests
depends_on: upload-ready-pipeline
mirror_hardwares: a3_npu_2
commands:
- >-
pytest -s -v
tests/diffusion/layers/test_activation.py
tests/diffusion/layers/test_fused_qk_norm_rope_npu.py
tests/diffusion/models/minimax_h3/test_minimax_h3_qwen3vl_swiglu.py
-m 'core_model and npu and A3'
--run-level 'core_model'
- label: "Diffusion NPU layer tests - A2"
key: ready-diffusion-npu-layer-tests-a2
mirror_hardwares: a2b3_npu_1
commands:
- >-
pytest -s -v
tests/diffusion/layers/test_activation.py
-m 'core_model and npu and A2'
--run-level 'core_model'
- label: "Diffusion NPU layer tests - A3"
key: ready-diffusion-npu-layer-tests
mirror_hardwares: a3_npu_2
commands:
- >-
pytest -s -v
tests/diffusion/layers/test_activation.py
tests/diffusion/layers/test_fused_qk_norm_rope_npu.py
tests/diffusion/models/minimax_h3/test_minimax_h3_qwen3vl_swiglu.py
tests/diffusion/models/minimax_h3/test_minimax_h3_modulation.py
-m 'core_model and npu and A3'
--run-level 'core_model'
- label: "Diffusion NPU layer tests - A5"
key: ready-diffusion-npu-layer-tests-a5
mirror_hardwares: a5_npu_2
commands:
- >-
pytest -s -v
tests/diffusion/layers/test_activation.py
-m 'core_model and npu and A5'
--run-level 'core_model'
- label: "Diffusion NPU layer tests - 310P"
key: ready-diffusion-npu-layer-tests-310p
mirror_hardwares: 310p_npu_1
commands:
- >-
pytest -s -v
tests/diffusion/layers/test_activation.py
-m 'core_model and npu and 310P'
--run-level 'core_model'
- group: ":card_index_dividers: Omni Model Test"
key: ready-omni-group
.claude/skills/review-pr/SKILL.md
+6 -5
View File
@@ -123,10 +123,11 @@ it changes again, report the churn and wait for a stable target.
For a trusted PR head, materialize the pinned head in an isolated detached
worktree. A worktree freezes identity but is not a security sandbox. Treat fork
heads as untrusted unless the user and environment policy explicitly establish
otherwise: execute them only in a disposable, secret-free sandbox with restricted
filesystem, network, and resources; without one, use static SHA-addressed reads
and CI evidence only. For a local review, freeze the committed, index, worktree,
heads as untrusted until the user and environment policy explicitly establish
trust: execute their code only after that trust is recorded for this host,
with the trusted SHA noted in the review state and secrets kept out of the
execution scope; without recorded trust, use static SHA-addressed reads and CI
evidence only. For a local review, freeze the committed, index, worktree,
and NUL-safe in-scope untracked contents. Follow
[review-execution.md](references/process/review-execution.md) for trust gates,
state fingerprints, and byte-for-byte staleness checks.
@@ -196,7 +197,7 @@ style preferences as simplification findings.
Before each validation group, verify the frozen SHA plus the tracked, index,
untracked, and ignored-file fingerprint, or recreate a pristine snapshot. On a
trusted head or inside the required sandbox, run an import/version preflight,
head the user explicitly trusted for this host, run an import/version preflight,
then the narrowest relevant tests and low-cost static checks. Bind every result
to the head SHA, snapshot fingerprint, and environment fingerprint. Never run
imports, tests, builds, hooks, or repo-configurable tooling from an untrusted
.claude/skills/review-pr/references/checks/perf-verification.md
+107
View File
@@ -22,6 +22,19 @@ Keep exact commands and report variability rather than a cherry-picked run.
Use the repository's benchmark when it covers the claim; otherwise use the
smallest reproducible workload that reaches the changed production path.
## Isolation per claim
Isolate one claim per comparison. When a PR stacks several changes (a fused or
restructured load path, a distilled step schedule, a new cache), request one
column per claim at equal steps/workload instead of one blended base-vs-head
delta, and name the unchanged stage or metric that pins equivalence.
When the changed path is runtime-switchable (feature flag or environment
variable), run every column on the same head and toggle the switch rather than
diffing against a base commit that cannot separate the stacked changes:
base, head with the switch off, head with the switch on, then head with the
switch on plus each further claim.
## Evidence table
| Dimension | Typical evidence |
@@ -34,6 +47,100 @@ smallest reproducible workload that reaches the changed production path.
Every optimization needs correctness or quality evidence. A faster result on a
different workload, precision, seed, or topology is not a valid comparison.
## E2e stage-attribution table
For an end-to-end latency claim, request the stage-attribution table the
repository's perf PRs use:
- Rows carry the benchmark's own field names: total e2e, per-stage execution,
postprocess, response encoding, and a request-to-output roll-up. Add a stage
handoff/idle row when the pipeline spans stages — connector transfer and
scheduling gaps are invisible in per-stage latency deltas.
- Sanity rows pin comparability: inference steps, output frames/pixels/samples,
and generated tokens held identical across columns.
- Report P50 and P100 from the stated warmup and measured counts. One
population per number: never mix cold-start and warm samples into one value.
- Name the baseline and candidate commit SHAs, and close with an isolation
statement naming the metric that stayed unchanged ("X remained effectively
unchanged, isolating the gain to Y").
When the serving path caches, captures, or regenerates at runtime (CUDA-graph
bucket growth, cache rebuilds, recompilation), medians hide the resulting tail.
Also request the event count and the maximum observed stall.
Component-level tables pair with, not replace, the e2e table: per-op reference
vs candidate timings (with the numerical-equality result per op) plus one
complete-pipeline check under the production shapes.
## Proportionality
When a PR adds a model and explicitly labels its numbers as smoke observations
rather than claims, do not demand the attribution machinery. Ask only for
evidence protecting the model's stated operating contract — for a
realtime/duplex model that is sustained cadence over a realistic session
(enough ticks to leave warmup), with any mid-session capture/regeneration
event and its stall reported, since a cadence that collapses mid-session is a
functional failure, not a perf claim.
## Realtime/duplex world models
Request the operating-contract evidence as serving metrics, not claims:
- Sustained per-tick latency over one realistic session: median, max, and the
argmax tick.
- Stall events with their causes.
- Peak memory at first versus last tick — session-owned state must not grow
per tick.
- Control latency from a submitted action to the tick that applies it.
- Realtime-versus-offline parity under the same controls, using the
repository's video-similarity helpers. Parity plotted over tick index is the
drift-over-horizon curve and the honest measure of session lifetime;
distribution-level suites (FVD, VBench) belong to model evaluation, not the
serving integration.
Watch the contract boundaries the model documents: if a control input is
tick-API-only, the parity pair must use a horizon both paths can run.
## Speech-generation (TTS) models
The streaming contract outranks the average — a synthesis that pauses
mid-stream is broken for realtime use regardless of its averages. Request:
- The benchmark's own speech metrics over one long utterance (enough audio to
leave warmup, tens of seconds): `audio_ttfp` (time to first packet),
`audio_rtf` (real-time factor; above 1.0 cannot keep up), `e2el`, and
`ttft`/`tpot` where tokens precede audio.
- Mid-utterance stalls as the benchmark's continuity fields, not prose:
`audio_underrun` percentiles and the `audio_continuity_ok` rate
(`vllm_omni/benchmarks/metrics/metrics.py`), computed by
`vllm_omni/benchmarks/audio_continuity.py` — the companion to `audio_ttfp`
and `audio_rtf`. A synthesis that bursts chunks fast enough on average but
underruns mid-stream fails the streaming contract even when the averages
pass.
- Quality as a paired comparison against the reference implementation under
the same text, speaker prompt, and seed; report sample rate and frame
count, and assert output completeness — truncated or dropped trailing audio
is a known failure class, not a quality nuance.
- For realtime or duplex TTS: interruption (barge-in) latency and the
per-session memory trend across turns.
## Omni models
Apply the e2e stage-attribution table directly to the voice-to-voice path of
omni (speech-to-speech and multimodal AR) models:
- TTFT for text output, `audio_ttfp` for audio output, and per-stage
ITL/TPOT cadence across the perception-understanding-generation chain.
- Cross-modal parity — the omni analogue of realtime-versus-offline parity:
the same semantic request through text and audio inputs, compared on output
content or timing.
- For duplex omni sessions: turn-taking latency, interruption handling, the
concurrent session count actually validated, and a per-turn memory trend —
session-owned KV state must not grow per turn. When either turn produces
audio, continuity is judged with the same `audio_underrun` and
`audio_continuity_ok` fields as TTS, since turn-taking latency alone does
not expose mid-utterance underruns.
## Reviewer verification ladder
1. Run base/head A/B on suitable hardware when affordable.
.claude/skills/review-pr/references/process/general-checks.md
+13
View File
@@ -121,6 +121,19 @@ Search bounded callers and sibling implementations before reporting dead code,
duplication, or a missed consumer. Keep new abstractions only when they have a
distinct live caller, invariant, or compatibility need.
Verify absence and equality claims with whole-file evidence. A line window, a
grep for one symbol, or a commit message proves presence, never absence: to
assert that a file matches another version or lacks a change, diff or hash the
full files (for example `git diff <sha1>:<path> <sha2>:<path>` or the contents
API's size and SHA fields) before anchoring the finding. A verification
shortcut that would itself be wrong to run is not evidence.
For a value written or finalized in more than one place, establish which site
executes first on each entry path — and whether the earlier site makes the
later ones unreachable — before naming where a fix belongs. Enumerating all
call sites correctly still misses the bug when a path the enumeration ranked
last actually finalizes first and clears the state the proposed fix checks.
## Finding bar
Anchor each finding to a changed `path:line`; name the trigger or call path,
.claude/skills/review-pr/references/process/review-execution.md
+7 -7
View File
@@ -39,13 +39,13 @@ isolated detached worktree. Run every source read, `rg` search, import, and test
from that snapshot, not the caller's checkout. A detached worktree provides
snapshot isolation, not a security boundary.
Treat a fork head as untrusted unless the user and environment policy explicitly
establish trust. Never run its imports, tests, builds, hooks, package setup, or
repo-configurable linters/plugins on the reviewer host. Execute them only in a
disposable sandbox or VM with no credentials or inherited secrets, no host agent
or service sockets, minimal read-only host mounts, disabled or explicitly
allowlisted network, resource/time limits, and destruction after the run. If
that boundary is unavailable, limit the review to the remote diff, SHA-addressed
Treat a fork head as untrusted until the user and environment policy
explicitly establish trust. Without recorded trust, never run its imports,
tests, builds, hooks, package setup, or repo-configurable linters/plugins on
the reviewer host. Once the user explicitly trusts this pinned commit for
this host, record the trusted SHA with the review state and execute with
credentials, agent sockets, and other secrets out of scope. Without recorded
trust, limit the review to the remote diff, SHA-addressed
reads such as `git show <head_sha>:<path>`, and existing CI evidence; report all
executable validation as a gap.
CONTRIBUTING.md
+2
View File
@@ -2,6 +2,8 @@
You may find information about contributing to vLLM-Omni on [Contributing](https://vllm-omni.readthedocs.io/en/latest/contributing/).
AI-assisted contributions must follow the [AI Usage Policy](docs/contributing/README.md#ai-usage-policy), including disclosure, human review, and validation requirements.
Local `pre-commit` gates (SPDX, forbidden imports, `torch.cuda`, shellcheck,
mypy, test marks, TTS adapter ratchet, markdownlint, Buildkite schema, and
how to extend allowlists) are documented in the
benchmarks/diffusion/README.md
+30 -15
View File
@@ -17,18 +17,18 @@ The main entrypoint is:
vllm serve Qwen/Qwen-Image --omni --port 8099
```
2. Run a minimal benchmark:
1. Run a minimal benchmark:
```bash
python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
--base-url http://localhost:8099 \
--model Qwen/Qwen-Image \
--task t2i \
--dataset vbench \
--num-prompts 5
--base-url http://localhost:8099 \
--model Qwen/Qwen-Image \
--task t2i \
--dataset vbench \
--num-prompts 5
```
**Notes**
### Notes
- By default, image tasks talk to `http://<host>:<port>/v1/chat/completions`; video tasks talk to `/v1/videos`.
- If you run the server on another host or port, pass `--base-url` accordingly.
@@ -52,14 +52,14 @@ Example (`t2v`):
```bash
python3 benchmarks/diffusion/diffusion_benchmark_serving.py \
--base-url http://localhost:8099 \
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--task t2v \
--dataset vbench \
--num-prompts 50 \
--width 640 --height 480 \
--num-frames 81 --fps 16 \
--num-inference-steps 40
--base-url http://localhost:8099 \
--model Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--task t2v \
--dataset vbench \
--num-prompts 50 \
--width 640 --height 480 \
--num-frames 81 --fps 16 \
--num-inference-steps 40
```
Note: `vbench` can also be used for other tasks such as `t2i` / `i2v` (and `i2i`). For `t2i`, the loader reuses VBench t2v text prompts; for `i2v` / `i2i`, it loads the VBench i2v dataset (with image paths).
@@ -140,6 +140,21 @@ Traffic / concurrency flags:
- `--request-rate`: Target request rate (requests/second). If set to `inf`, the script sends all requests immediately.
- `--max-concurrency`: Max number of in-flight requests (default: `1`). This can hard-cap the achieved QPS: if it is too small, requests will queue behind the semaphore, and both achieved throughput and observed SLO attainment can be skewed.
### Video job timeout and failures
`--video-job-timeout` controls how long each `/v1/videos` job may be polled,
including time queued on the server (default: `900` seconds). For long video
jobs at high concurrency, use a larger budget, for example
`--video-job-timeout 1800`. This also applies to warmup jobs. The polling budget
starts after job creation; it excludes time waiting for the client concurrency
semaphore. Individual HTTP calls still use the aiohttp session timeout.
An expired job is counted as failed and deleted, which cancels queued or running
server generation. The benchmark continues processing the remaining requests.
The progress bar counts both successful and failed requests. The final report
shows the failure count and the first 10 errors; `--output-file` saves all errors
with their request IDs in `request_errors`.
### Batched warmup note
For batched serving runs, warm the same in-flight shape you plan to measure.
benchmarks/diffusion/backends.py
+64 -57
View File
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import asyncio
import base64
import json
@@ -43,6 +46,7 @@ class RequestFuncInput:
video_paths: list[str] | None = None
request_id: str = field(default_factory=lambda: str(uuid.uuid4()))
default_bot_task: str | None = DEFAULT_EDITS_BOT_TASK
video_job_timeout: float = 900.0
@dataclass
@@ -331,64 +335,64 @@ async def async_request_v1_videos(
output = RequestFuncOutput()
output.start_time = time.perf_counter()
files = dict(input.extra_body)
if input.prompt:
files.setdefault("prompt", input.prompt)
if input.width and input.height:
files.setdefault("height", input.height)
files.setdefault("width", input.width)
if input.num_frames:
files.setdefault("num_frames", input.num_frames)
if input.num_inference_steps:
files.setdefault("num_inference_steps", input.num_inference_steps)
if input.seed is not None:
files.setdefault("seed", input.seed)
if input.fps:
files.setdefault("fps", input.fps)
form = aiohttp.FormData()
for k, v in files.items():
form.add_field(k, str(v))
image_file = None
if input.image_paths and input.video_paths:
output.error = "Only one of image_paths or video_paths can be provided"
output.success = False
return output
job_id = None
try:
files = dict(input.extra_body)
if input.prompt:
files.setdefault("prompt", input.prompt)
if input.width and input.height:
files.setdefault("height", input.height)
files.setdefault("width", input.width)
if input.num_frames:
files.setdefault("num_frames", input.num_frames)
if input.num_inference_steps:
files.setdefault("num_inference_steps", input.num_inference_steps)
if input.seed is not None:
files.setdefault("seed", input.seed)
if input.fps:
files.setdefault("fps", input.fps)
if input.image_paths and len(input.image_paths) > 0:
image_path = input.image_paths[0]
image_file = open(image_path, "rb")
form.add_field(
"input_reference",
image_file,
filename=os.path.basename(image_path),
content_type=_guess_mime_type(image_path),
)
elif input.video_paths and len(input.video_paths) > 0:
video_path = input.video_paths[0]
if _is_minimax_h3_mixed_reference(input.model):
with open(video_path, "rb") as reference_file:
video_b64 = base64.b64encode(reference_file.read()).decode("ascii")
form.add_field(
"video_reference",
json.dumps([{"video_url": f"data:video/mp4;base64,{video_b64}"}]),
)
else:
image_file = open(video_path, "rb")
form = aiohttp.FormData()
for k, v in files.items():
form.add_field(k, str(v))
if input.image_paths and input.video_paths:
output.error = "Only one of image_paths or video_paths can be provided"
output.success = False
return output
if input.image_paths and len(input.image_paths) > 0:
image_path = input.image_paths[0]
image_file = open(image_path, "rb")
form.add_field(
"input_reference",
image_file,
filename=os.path.basename(video_path),
content_type=_guess_mime_type(video_path),
filename=os.path.basename(image_path),
content_type=_guess_mime_type(image_path),
)
elif input.video_paths and len(input.video_paths) > 0:
video_path = input.video_paths[0]
if _is_minimax_h3_mixed_reference(input.model):
with open(video_path, "rb") as reference_file:
video_b64 = base64.b64encode(reference_file.read()).decode("ascii")
form.add_field(
"video_reference",
json.dumps([{"video_url": f"data:video/mp4;base64,{video_b64}"}]),
)
else:
image_file = open(video_path, "rb")
form.add_field(
"input_reference",
image_file,
filename=os.path.basename(video_path),
content_type=_guess_mime_type(video_path),
)
job_id = None
job_status = None
poll_json = {}
resp_json = {}
job_status = None
poll_json = {}
resp_json = {}
try:
# invoke a post request (POST /v1/videos)
async with session.post(input.api_url, data=form) as response:
if response.status == 200:
@@ -406,7 +410,7 @@ async def async_request_v1_videos(
# invoke a poll request (GET /v1/videos/{video_id})
poll_interval = 2.0 # Unit(s)
timeout_seconds = 600.0
timeout_seconds = input.video_job_timeout
deadline = time.perf_counter() + timeout_seconds
job_url = f"{input.api_url}/{job_id}"
@@ -422,8 +426,11 @@ async def async_request_v1_videos(
poll_json = await poll_response.json()
job_status = poll_json.get("status")
if time.perf_counter() >= deadline:
output.error = f"Timed out waiting for video job {job_id} to complete."
if job_status not in {"completed", "failed"} and time.perf_counter() >= deadline:
output.error = (
f"Timed out after {timeout_seconds:g}s waiting for video job {job_id} to complete. "
"Increase --video-job-timeout for long-running or queued video jobs."
)
output.success = False
return output
@@ -452,7 +459,7 @@ async def async_request_v1_videos(
elif "peak_memory_mb" in resp_json:
output.peak_memory_mb = resp_json["peak_memory_mb"]
except Exception as e:
output.error = str(e)
output.error = f"{type(e).__name__}: {e}"
output.success = False
finally:
if image_file is not None:
@@ -465,13 +472,13 @@ async def async_request_v1_videos(
except Exception as e:
print(f"Failed to clean up video job {job_id}: {e}")
output.latency = time.perf_counter() - output.start_time
output.latency = time.perf_counter() - output.start_time
if pbar is not None:
pbar.update(1)
if output.success and input.slo_ms is not None:
output.slo_achieved = (output.latency * 1000.0) <= float(input.slo_ms)
if pbar:
pbar.update(1)
return output
benchmarks/diffusion/diffusion_benchmark_serving.py
+26 -1
View File
@@ -1,6 +1,6 @@
# adapted from fastvideo
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""
Benchmark online serving for diffusion models (Image/Video Generation).
@@ -87,6 +87,7 @@ import base64
import glob
import json
import logging
import math
import os
import random
import tempfile
@@ -1116,6 +1117,11 @@ def calculate_metrics(
"duration": total_duration,
"completed_requests": num_success,
"failed_requests": len(error_outputs),
"request_errors": [
{"request_id": req.request_id, "error": out.error}
for req, out in zip(requests_list, outputs)
if not out.success
],
"throughput_qps": num_success / total_duration if total_duration > 0 else 0,
"latency_mean": np.mean(latencies) if latencies else 0,
"latency_median": np.median(latencies) if latencies else 0,
@@ -1261,6 +1267,9 @@ def _default_endpoint_for_task(task: str) -> str:
async def benchmark(args):
if not math.isfinite(args.video_job_timeout) or args.video_job_timeout <= 0:
raise ValueError("--video-job-timeout must be a positive finite number of seconds.")
# Construct base_url if not provided
if args.base_url is None:
args.base_url = f"http://{args.host}:{args.port}"
@@ -1312,6 +1321,8 @@ async def benchmark(args):
print("Loading requests...")
requests_list = dataset.get_requests()
print(f"Prepared {len(requests_list)} requests from {args.dataset} dataset.")
for req in requests_list:
req.video_job_timeout = args.video_job_timeout
if args.return_stage_metrics and args.endpoint in _STAGE_METRICS_ENDPOINTS:
for req in requests_list:
@@ -1376,6 +1387,8 @@ async def benchmark(args):
metrics["model"] = args.model
metrics["dataset"] = args.dataset
metrics["task"] = args.task
if args.endpoint == "/v1/videos":
metrics["video_job_timeout"] = args.video_job_timeout
if args.endpoint == "/v1/images/edits":
metrics["bot_task"] = args.bot_task
@@ -1398,6 +1411,11 @@ async def benchmark(args):
)
)
print("{:<40} {}/{:<15}".format("Successful requests:", metrics["completed_requests"], len(requests_list)))
print("{:<40} {:<15}".format("Failed requests:", metrics["failed_requests"]))
for failure in metrics["request_errors"][:10]:
print(f"Request {failure['request_id']} failed: {failure['error']}")
if metrics["failed_requests"] > 10:
print("Showing first 10 failures; use --output-file to save all request errors.")
# Section 3: Performance Metrics
print(f"{'-' * 50}")
@@ -1481,6 +1499,13 @@ if __name__ == "__main__":
help="Path to local dataset file (optional).",
)
parser.add_argument("--num-prompts", type=int, default=10, help="Number of prompts to benchmark.")
parser.add_argument(
"--video-job-timeout",
type=float,
default=900.0,
help="Maximum polling time in seconds per video job, including server queue time (default: 900). "
"Increase this for long-running video generation or high concurrency.",
)
parser.add_argument(
"--max-concurrency",
type=int,
benchmarks/tts/README.md
+19
View File
@@ -85,6 +85,25 @@ vllm bench serve --omni \
--save-result --result-dir ./results
```
#### Set the sample rate for models that are not 24 kHz
Streamed PCM carries no header, so the harness derives `audio_duration` from the
byte count and a **default 24000 Hz** assumption
(`vllm_omni/metrics/definitions.py:DEFAULT_AUDIO_SAMPLE_RATE`). For a model that
emits another rate, export the real one or every duration-derived metric is
wrong by `real_sr / 24000`:
```bash
# Audio8 TTS, Fish Speech S2 Pro, Ming-omni-tts, ... all decode at 44.1 kHz
export VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=44100
```
Without it, a 44.1 kHz model reports `audio_duration` 1.84x too long, so
`audio_rtf` looks 1.84x *better* than reality and `audio_underrun` /
`Streaming continuity OK rate` are measured against inflated chunk budgets.
A quick sanity check: compare `Mean AUDIO_DURATION (s)` against the actual
length of a saved WAV for the same prompt.
#### Streaming continuity (`audio_underrun`)
`audio_underrun` (seconds) is the per-request worst-case buffer deficit
benchmarks/tts/model_configs.yaml
+11
View File
@@ -17,6 +17,17 @@ models:
extra_params:
lang: en
text_normalization: true
# Audio8 TTS Preview: DualAR, 44.1 kHz. trust_remote_code stays false.
# Bench at 44.1 kHz: export VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=44100, otherwise
# the streamed-PCM duration defaults to 24 kHz and RTF looks 1.84x too good.
Audio8/Audio8-TTS-Preview-0.6b:
supported_tasks: [default_voice, voice_clone]
backend: openai-audio-speech
endpoint: /v1/audio/speech
trust_remote_code: false
task_extra_body:
default_voice: {}
voice_clone: {}
# -CustomVoice checkpoints lack speaker_encoder weights, so voice_clone is
# NOT supported (an attempt raises ValueError from _extract_speaker_embedding
docker/Dockerfile.ci
+6 -23
View File
@@ -1,27 +1,10 @@
# Target vllm version: v0.28.0 (tag v0.28.0 @ 2cf0a6915ce5).
# Keep VLLM_BASE_TAG aligned with the target vLLM release on every rebase.
#
# Release-boundary form: there is no precompiled-wheel pin here, because the
# base image already ships exactly the target vLLM. Everything the runtime
# needs — torch, flashinfer, nvJitLink, numpy — comes from that image as one
# self-consistent set.
#
# This block reappears only when the rebase targets a vLLM commit that is NOT a
# released tag. During the 0.27 rebase, targeting v0.27.0rc1 on top of the
# v0.26.0 base image required reinstalling vLLM from wheels.vllm.ai and then
# hand-repairing the fallout: pinning torch ahead of flashinfer (installing it
# after bound flashinfer's TVM FFI runtime to the old torch and aborted engine
# core init), registering libnvJitLink.so.13 with ldconfig, and re-pinning numpy
# under numba's ceiling. All of it was compensation for a base image and a vLLM
# that disagreed. At a real release boundary they agree, so none of it is
# needed. The 0.28 rebase repeated the cycle: the v0.28.0rc / final-tag wheels
# rode on the v0.27.0 base image with the full reinstall block until the
# official v0.28.0 image shipped, at which point this file dropped back to the
# form you are reading. If you pin an untagged commit again, restore the
# reinstall block — agent/tasks/20_update_wheel_url.sh fails the run rather
# than letting the Dockerfile silently ship the wrong vLLM.
# Target vLLM version: v0.29.0 (tag @ 98dff2a81d74).
# Use the matching release image and its coherent CUDA/PyTorch/FlashInfer
# runtime. Do not reinstall a commit wheel or repair dependencies on top of
# an older base image at a release boundary. For a future unreleased target,
# restore an explicit wheel pin and validate its complete runtime stack.
ARG VLLM_BASE_IMAGE=vllm/vllm-openai
ARG VLLM_BASE_TAG=v0.28.0
ARG VLLM_BASE_TAG=v0.29.0
# We use registry-based layer caching for the dependencies,
# where the tag is determined based on the hash of the contents
docker/Dockerfile.npu
+4 -16
View File
@@ -1,20 +1,7 @@
ARG VLLM_ASCEND_IMAGE=quay.io/atlas-ci/vllm-ascend
ARG VLLM_ASCEND_TAG=v0.28.0
FROM ${VLLM_ASCEND_IMAGE}:${VLLM_ASCEND_TAG}
# WORKDIR /vllm-workspace/vllm
# RUN git fetch origin --tags && git checkout v0.18.0
# WORKDIR /vllm-workspace/vllm-ascend
# RUN git fetch origin releases/v0.18.0 && git checkout d781902ce9dbda8ab1e11bb0f2f0c1bc508fee7a
# # Install vllm-ascend
# # Append `libascend_hal.so` path (devlib) to LD_LIBRARY_PATH
# RUN export PIP_EXTRA_INDEX_URL=https://mirrors.huaweicloud.com/ascend/repos/pypi && \
# source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
# source /usr/local/Ascend/nnal/atb/set_env.sh && \
# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/`uname -i`-linux/devlib && \
# python3 -m pip install -v -e /vllm-workspace/vllm-ascend/ --extra-index https://download.pytorch.org/whl/cpu/ && \
# python3 -m pip cache purge
ARG VLLM_ASCEND_VERSION=v0.28.0
ARG VLLM_ASCEND_DEVICE_SUFFIX=
FROM ${VLLM_ASCEND_IMAGE}:${VLLM_ASCEND_VERSION}${VLLM_ASCEND_DEVICE_SUFFIX}
ARG APP_DIR=/vllm-workspace/vllm-omni
WORKDIR ${APP_DIR}
@@ -24,6 +11,7 @@ RUN export PIP_EXTRA_INDEX_URL=https://mirrors.huaweicloud.com/ascend/repos/pypi
source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
source /usr/local/Ascend/nnal/atb/set_env.sh && \
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/`uname -i`-linux/devlib && \
python3 -m pip install -v "mindiesd==3.1.0" && \
python3 -m pip install -v -e /vllm-workspace/vllm-omni/ --no-build-isolation
RUN python3 -m pip install -v --no-deps \
docker/Dockerfile.npu.a3
-36
View File
@@ -1,36 +0,0 @@
ARG VLLM_ASCEND_IMAGE=quay.io/atlas-ci/vllm-ascend
ARG VLLM_ASCEND_TAG=v0.28.0-a3
FROM ${VLLM_ASCEND_IMAGE}:${VLLM_ASCEND_TAG}
# WORKDIR /vllm-workspace/vllm
# RUN git fetch origin --tags && git checkout v0.18.0
# WORKDIR /vllm-workspace/vllm-ascend
# RUN git fetch origin releases/v0.18.0 && git checkout d781902ce9dbda8ab1e11bb0f2f0c1bc508fee7a
# # Install vllm-ascend
# # Append `libascend_hal.so` path (devlib) to LD_LIBRARY_PATH
# RUN export PIP_EXTRA_INDEX_URL=https://mirrors.huaweicloud.com/ascend/repos/pypi && \
# source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
# source /usr/local/Ascend/nnal/atb/set_env.sh && \
# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/`uname -i`-linux/devlib && \
# python3 -m pip install -v -e /vllm-workspace/vllm-ascend/ --extra-index https://download.pytorch.org/whl/cpu/ && \
# python3 -m pip cache purge
ARG APP_DIR=/vllm-workspace/vllm-omni
WORKDIR ${APP_DIR}
COPY . .
RUN export PIP_EXTRA_INDEX_URL=https://mirrors.huaweicloud.com/ascend/repos/pypi && \
source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
source /usr/local/Ascend/nnal/atb/set_env.sh && \
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/`uname -i`-linux/devlib && \
python3 -m pip install -v -e /vllm-workspace/vllm-omni/ --no-build-isolation
RUN python3 -m pip install -v --no-deps \
"stepaudio2-minicpmo>=0.1.1" \
"step-audio2>=1.0.0"
ENV VLLM_TARGET_DEVICE=npu
ENV VLLM_WORKER_MULTIPROC_METHOD=spawn
ENTRYPOINT []
docker/Dockerfile.npu.ci
+12 -8
View File
@@ -1,10 +1,10 @@
ARG VLLM_ASCEND_IMAGE=quay.io/atlas-ci/vllm-ascend
ARG VLLM_ASCEND_TAG=v0.28.0
FROM ${VLLM_ASCEND_IMAGE}:${VLLM_ASCEND_TAG}
ARG VLLM_ASCEND_VERSION=v0.28.0
ARG VLLM_ASCEND_DEVICE_SUFFIX=
FROM ${VLLM_ASCEND_IMAGE}:${VLLM_ASCEND_VERSION}${VLLM_ASCEND_DEVICE_SUFFIX}
ARG APP_DIR=/vllm-workspace/vllm-omni
WORKDIR ${APP_DIR}
COPY . .
RUN sed -i "s@^\(deb.*\)https\?://[a-z0-9.-]*\.ubuntu\.com@\1http://cache-service.nginx-pypi-cache.svc.cluster.local:8081@g" /etc/apt/sources.list
@@ -22,11 +22,9 @@ RUN apt-get update && \
rm -rf /var/cache/apt/* && \
rm -rf /var/lib/apt/lists/*
RUN source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
source /usr/local/Ascend/nnal/atb/set_env.sh && \
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/`uname -i`-linux/devlib && \
python3 -m pip install -U pip && \
RUN python3 -m pip install -U pip && \
python3 -m pip install -v \
"mindiesd==3.1.0" \
"pytest>=7.0.0" \
"pytest-asyncio>=0.21.0" \
"pytest-cov>=4.0.0" \
@@ -48,7 +46,13 @@ RUN source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
"datasets>=3.0.0" \
--index-url http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
--default-timeout=60 && \
--default-timeout=60
COPY . .
RUN source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
source /usr/local/Ascend/nnal/atb/set_env.sh && \
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/`uname -i`-linux/devlib && \
python3 -m pip install -v -e /vllm-workspace/vllm-omni/ --no-build-isolation \
--index-url http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
docker/Dockerfile.npu.ci.a3
-72
View File
@@ -1,72 +0,0 @@
ARG VLLM_ASCEND_IMAGE=quay.io/atlas-ci/vllm-ascend
ARG VLLM_ASCEND_TAG=v0.28.0-a3
FROM ${VLLM_ASCEND_IMAGE}:${VLLM_ASCEND_TAG}
ARG APP_DIR=/vllm-workspace/vllm-omni
WORKDIR ${APP_DIR}
COPY . .
RUN sed -i "s@^\(deb.*\)https\?://[a-z0-9.-]*\.ubuntu\.com@\1http://cache-service.nginx-pypi-cache.svc.cluster.local:8081@g" /etc/apt/sources.list
RUN apt-get update && \
apt-get install -y --no-install-recommends \
ffmpeg \
libgl1 \
libglib2.0-0 \
libgomp1 \
libsm6 \
libxext6 \
libxrender1 \
libxcb1 \
espeak-ng && \
rm -rf /var/cache/apt/* && \
rm -rf /var/lib/apt/lists/*
RUN source /usr/local/Ascend/ascend-toolkit/set_env.sh && \
source /usr/local/Ascend/nnal/atb/set_env.sh && \
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/Ascend/ascend-toolkit/latest/`uname -i`-linux/devlib && \
python3 -m pip install -U pip && \
python3 -m pip install -v \
"pytest>=7.0.0" \
"pytest-asyncio>=0.21.0" \
"pytest-cov>=4.0.0" \
"pytest-mock>=3.10.0" \
"openpyxl>=3.0.0" \
"psutil>=7.2.0" \
"soundfile>=0.13.1" \
"pyttsx3>=2.99" \
"opencc==1.4.1" \
"opencv-python>=4.12.0.88" \
"imageio[ffmpeg]>=0.6.0" \
"jiwer>=3.0.0" \
"zhon>=2.0.0" \
"zhconv>=1.4.2" \
"funasr>=1.0.0" \
"hyperpyyaml>=1.2.0" \
"onnx>=1.16.0" \
"librosa>=0.10.0" \
"torchmetrics>=1.9.0" \
"datasets>=3.0.0" \
--index-url http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
--default-timeout=60 && \
python3 -m pip install -v -e /vllm-workspace/vllm-omni/ --no-build-isolation \
--index-url http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
--default-timeout=60
# MiniCPM-o 4.5 Stage-2 vocoder assets: s3tokenizer plus flashcosyvoice, which
# is shipped by step-audio2. --no-deps keeps pip from replacing the Ascend torch
# build and from shadowing onnxruntime-cann with the vanilla onnxruntime wheel;
# the deps these two actually need are installed above.
RUN python3 -m pip install -v --no-deps \
"s3tokenizer>=0.3.0" \
"step-audio2>=1.0.0" \
--index-url http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
--default-timeout=60
ENV VLLM_TARGET_DEVICE=npu
ENV VLLM_WORKER_MULTIPROC_METHOD=spawn
ENTRYPOINT []
docs/cli/bench/serve.md
+70
View File
@@ -436,6 +436,76 @@ This should be seen as an edge case, and if this behavior can be avoided by sett
</details>
### Image / video generation
For `/v1/images/generations`, `/v1/images/edits`, and `/v1/videos`, set `--endpoint` to that path and omit `--backend`. The endpoint path is registered as the bench request adapter, so a separate named backend is not required.
Example (`/v1/images/generations`):
```bash
vllm bench serve --omni \
--endpoint /v1/images/generations \
--dataset-name random \
--model ~/models/Qwen/Qwen-Image \
--tokenizer ~/models/Qwen/Qwen-Image/tokenizer \
--max-concurrency 1 \
--num-warmups 2 \
--num-prompts 10 \
--random-input-len 64 \
--random-output-len 1 \
--ignore-eos \
--percentile-metrics e2el \
--extra-body '{
"num_inference_steps": 20,
"seed": 42,
"true_cfg_scale": 4.0
}'
```
If successful, following output is like:
```text
============ Serving Benchmark Result ============
Successful requests: 3
Failed requests: 0
Maximum request concurrency: 1
Benchmark duration (s): 8.42
Request throughput (req/s): 0.36
Peak concurrent requests: 2.00
-------------------Peak Memory--------------------
Mean PEAK_MEMORY_MB (MB): 58832.00
Median PEAK_MEMORY_MB (MB): 58832.00
P99 PEAK_MEMORY_MB (MB): 58832.00
----------------End-to-end Latency----------------
Mean E2EL (ms): 2798.16
Median E2EL (ms): 2799.83
P99 E2EL (ms): 2800.03
================== Image Result ==================
Total images generated: 3
Image throughput (img/s): 0.36
Average pixels per image: 1048576.00
Mean denoise step latency (ms): 136.59
---------------- Image Generation ----------------
Mean IMAGE_GENERATION (ms): 2731.84
Median IMAGE_GENERATION (ms): 2732.02
P99 IMAGE_GENERATION (ms): 2734.65
==================================================
```
Use the same pattern with `--endpoint /v1/images/edits` or `--endpoint /v1/videos` (and model-specific `--extra-body` as needed). Pure image/video runs omit the Text Result section when there is no generated text.
`/v1/images/edits` defaults to **non-streaming JSON** (`stream=false`) so single-stage edit models are not rejected by the server. For multi-stage pipelines that need SSE (AR TTFT / image chunks), pass `"stream": true` in `--extra-body`.
`/v1/videos` is an async job API: the client creates a job, then polls until `completed`/`failed`. Measured **e2el** therefore includes client poll sleep and any overshoot after the job actually finishes. Tune polling via `--extra-body`:
- `poll_interval_s` (default `2.0`): sleep between status polls
- `poll_timeout_s` (default `21600`, i.e. 6 hours): give up waiting for the job
**VIDEO_RTF** prefers server-reported generation time when available:
- `video_rtf = video_generation_time_ms / 1000 / video_duration`, where `video_generation_time_ms` comes from response `stage_durations` (or `inference_time_s` as fallback)
- If generation time is missing, falls back to `e2el / video_duration` (this fallback *does* include poll overhead)
### Multi-Stage Benchmark
<details class="admonition abstract" markdown="1">
docs/configuration/README.md
+1 -1
View File
@@ -2,7 +2,7 @@
This section lists the most common options for running vLLM-Omni.
For options within a vLLM Engine, please refer to the [vLLM 0.28 configuration guide](https://docs.vllm.ai/en/v0.28.0/configuration/index.html).
For options within a vLLM Engine, please refer to the [vLLM 0.29 configuration guide](https://docs.vllm.ai/en/v0.29.0/configuration/index.html).
Each model defines fixed topology in a registered `PipelineConfig` and runtime
overrides in a deploy YAML.
docs/configuration/environment_variables.md
+1 -1
View File
@@ -159,7 +159,7 @@ their keys only.
## Inherited vLLM variables
vLLM-Omni also reads variables through its aligned vLLM dependency. Refer to
the [vLLM 0.28 environment-variable reference](https://docs.vllm.ai/en/v0.28.0/configuration/env_vars.html)
the [vLLM 0.29 environment-variable reference](https://docs.vllm.ai/en/v0.29.0/configuration/env_vars.html)
for their definitions. This includes vLLM launch, cache, logging, plugin, ROCm,
XPU, ModelScope, and FlashInfer workspace settings.
docs/contributing/README.md
+22
View File
@@ -262,6 +262,28 @@ or serving, explain the required L2/L3 environment and provide the exact command
Repository skills guide the agent, but they do not replace contributor judgment, the accepted issue or RFC, required test evidence, or maintainer review. Review all generated changes before submitting them.
## AI Usage Policy
AI tools are welcome in vLLM-Omni contributions. Contributors, including maintainers, remain responsible for everything they submit. These requirements apply to pull requests, issues, discussions, and review comments:
- **Disclose AI assistance.** Name the tools used and describe their role in the submission, including assistance with code, tests, documentation, or writing.
- **Understand your changes.** Review every generated change and be able to explain its behavior, design choices, and interaction with vLLM-Omni without relying on AI to answer for you.
- **Validate before submitting.** Run relevant checks and report actual results. For model, kernel, or performance changes, include the hardware, model, configuration, and commands needed to reproduce accuracy or benchmark evidence. Clearly identify checks you could not run; generated claims are not test results.
- **Keep communication useful.** Verify technical claims, search for existing reports, and edit AI-assisted text for accuracy and brevity. Submit only content you have personally reviewed, and take responsibility for responding to review feedback.
- **Label generated media.** Model-generated images, video, and audio are valid examples or test artifacts for this multimodal project. Identify them as generated output and provide the model, inputs, and reproduction settings. Do not present them as ground-truth references or independent evidence of correctness.
For example, a PR description could include:
```text
AI assistance: Used Codex to draft the regression test and PR description.
I reviewed the changes and verified the test reproduces the reported failure.
Validation: <commands, results, hardware, and any checks not run>.
```
Maintainer review time is limited. Contributions that are unverified, misleading, or beyond the author's understanding may be closed. Repeated violations may lead to contribution restrictions. New contributors are welcome to ask for help and submit small, well-understood changes.
This policy is adapted from [Ghostty's AI Usage Policy](https://github.com/ghostty-org/ghostty/blob/main/AI_POLICY.md) for vLLM-Omni's contribution and multimodal validation workflows. The [coding-agent workflow](#using-repository-skills-with-coding-agents) provides practical guidance for applying it.
## Issues
If you encounter a bug or have a feature request, please search existing issues first to see if it has already been reported. If not, please file a new issue, providing as much relevant information as possible.
docs/contributing/ci/ci_settings.md
+10 -3
View File
@@ -129,7 +129,7 @@ Canonical layout (prefer these paths for new changes):
=== "NPU"
**Bootstrap:** [`npu/pipeline-npu.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/npu/pipeline-npu.yml) + [`npu/bootstrap-upload-steps.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/npu/bootstrap-upload-steps.yml)—same split as CUDA; builds A2/B3 and A3 CI images, then uploads child test pipelines.
**Bootstrap:** [`npu/pipeline-npu.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/npu/pipeline-npu.yml) + [`npu/bootstrap-upload-steps.yml`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/npu/bootstrap-upload-steps.yml)—same split as CUDA; builds A2/B3, A3, A5, and 310P CI images, then uploads child test pipelines.
**Test YAML:** `npu/test-npu-ready.yml` (L2), `test-npu-nightly.yml` (L4).
@@ -137,7 +137,12 @@ Canonical layout (prefer these paths for new changes):
**Upload:** same as CUDA—`upload_pipeline.py --upload`.
**Hardware in YAML:** `mirror_hardwares` preset (string), expanded to `agents`, top-level `image`, and `plugins`. Presets: `a2b3_npu_1`, `a2b3_npu_4`, `a3_npu_2` in `common/ci_mirror_hardwares.yml`.
**Hardware in YAML:** `mirror_hardwares` preset (string), expanded to `agents`, top-level `image`, and `plugins`. NPU presets in `common/ci_mirror_hardwares.yml` are:
- A2: `a2b3_npu_1`, `a2b3_npu_4`, `a2b3_npu_8`
- A3: `a3_npu_2`, `a3_npu_4`, `a3_npu_8`, `a3_npu_16`
- A5: `a5_npu_2`, `a5_npu_4`, `a5_npu_8`, `a5_npu_16`
- 310P: `310p_npu_1`, `310p_npu_2`, `310p_npu_4`
**Conventions**
@@ -161,7 +166,9 @@ Canonical layout (prefer these paths for new changes):
**Bootstrap:** [`amd/scripts/bootstrap-amd-omni.sh`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/scripts/bootstrap-amd-omni.sh)—skip-ci, diff filtering, Jinja render, then `buildkite-agent pipeline upload`.
**Test YAML (data):** `amd/test-amd-ready.yml` (L2 / PR), `test-amd-merge.yml` (L3 / main).
**Test YAML (data):** `amd/test-amd-ready.yml` (L2 / `ready`), `test-amd-merge.yml` (L3 / `merge-test` and `main`). On a PR carrying both labels, AMD combines both suites behind one image build. PR builds without either tier label retain the legacy L2 fallback for compatibility with the `amd-test` trigger. `DEBUG_TEST_YAML` remains an explicit override. AMD has no L4 file yet, so `nightly-test` does not select an additional suite.
**Trigger boundary:** AMD label handling has two layers. First, the external `vllm-omni-amd-ci` Buildkite pipeline condition decides whether a GitHub label event creates a build. Only after that build starts does this repository's bootstrap inspect all current PR labels and select L2, L3, or both. Repository-side selection therefore cannot make a `merge-test` event start AMD CI by itself. The external condition must admit both `ready` and `merge-test`, while preserving the legacy `amd-test` trigger and non-PR `main` / scheduled builds. For PR builds without a `DEBUG_TEST_YAML` override, an unreadable label set fails the bootstrap instead of silently selecting the wrong tier. Debug overrides skip label lookup and are validated directly; invalid values, including whitespace-only values, fail the bootstrap. Keep `nightly-test` out of the AMD condition until an AMD L4 suite exists; otherwise it would start a build without providing nightly coverage.
**Rendering:** [`test-template-amd-omni.j2`](https://github.com/vllm-project/vllm-omni/blob/main/.buildkite/amd/test-template-amd-omni.j2) wraps data steps with `amd-build` image build and `amd_<agent_pool>` queues. Do **not** hand-edit generated `pipeline.yaml`.
docs/contributing/ci/test_writing_guide.md
+3 -1
View File
@@ -48,6 +48,8 @@ Defined in `pyproject.toml`:
| `MI325` | Tests that require MI325 GPU (AMD/ROCm) * |
| `A2` | Tests that require A2 NPU * |
| `A3` | Tests that require A3 NPU * |
| `A5` | Tests that require A5 NPU * |
| `310P` | Tests that require 310P NPU * |
| `cards_{n}` | Tests that require *n* accelerator cards (`cards_1` … `cards_8`); auto-added from `num_cards` * |
| `slow` | Slow tests (may skip in quick CI) |
| `benchmark` | Benchmark tests (decorator on runner test functions; perf JSON uses `full_model` + type marker instead) |
@@ -76,7 +78,7 @@ def test_video_to_audio()
This decorator is intended to make hardware-aware, cross-platform test authoring easier and more robust for CI/CD environments. The `hardware_test` decorator in `vllm-omni/tests/helpers/mark.py` performs the following actions:
1. **Applies platform and resource markers**
Adds the appropriate pytest markers for each specified hardware platform (e.g., `cuda`, `rocm`, `xpu`, `npu`) and resource type (e.g., `L4`, `H100`, `H200`, `B200`, `MI325`, `B60`, `A2`, `A3`).
Adds the appropriate pytest markers for each specified hardware platform (e.g., `cuda`, `rocm`, `xpu`, `npu`) and resource type (e.g., `L4`, `H100`, `H200`, `B200`, `MI325`, `B60`, `A2`, `A3`, `A5`, `310P`).
```python
@pytest.mark.cuda
@pytest.mark.L4
docs/design/architecture_overview.md
+16 -7
View File
@@ -323,18 +323,24 @@ inputs, fields, and ownership rules are described below.
```mermaid
flowchart TB
layer1["Layer 1 · Authoring inputs<br/>PipelineConfig + DeployConfig"]
layer2["Layer 2 · Resolve once<br/>StageConfigFactory.create_from_model()<br/>VllmOmniConfig.from_pipeline_config()"]
layer3["Layer 3 · Transport-safe control plane<br/>VllmOmniConfig"]
layer2["Layer 2 · Production resolution boundary<br/>resolve_omni_config()<br/>(StageConfigFactory + VllmOmniConfig internally)"]
layer3["Layer 3 · Startup hand-off<br/>OmniConfigResolution<br/>PipelineConfig + temporary OmegaConf stage bridge"]
layer4["Layer 4 · Runtime launch planning<br/>StageRuntime"]
layer5["Layer 5 · Engine materialization<br/>VllmConfig / OmniDiffusionConfig"]
layer1 --> layer2 --> layer3 --> layer4 --> layer5
```
The single resolution boundary is implemented by
`StageConfigFactory.create_from_model()` and
`VllmOmniConfig.from_pipeline_config()` in
The production resolution boundary is `resolve_omni_config()` in
[`vllm_omni/config`](https://github.com/vllm-project/vllm-omni/tree/main/vllm_omni/config).
It delegates typed construction to `StageConfigFactory.create_from_model()` and
`VllmOmniConfig.from_pipeline_config()`, then returns an
`OmniConfigResolution` consumed by both `AsyncOmniEngine` and headless startup.
Until `StageRuntime` consumes typed stage configs directly, this envelope carries
the effective `PipelineConfig` alongside OmegaConf-compatible `stage_configs` as
a temporary runtime bridge; both views describe the same resolved topology,
including injected stages.
The legacy `stage_args` YAML path has been removed. Model topology now resolves
through `PipelineConfig`, with runtime overrides supplied by `DeployConfig`.
@@ -354,9 +360,12 @@ The important ownership rules are:
deploy-time defaults; it does not redefine the model graph.
3. CLI and Python overrides are applied at the resolution boundary, with
per-stage overrides taking precedence over global values where supported.
4. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan`
4. `OmniConfigResolution` is the sole startup hand-off. Its `pipeline_config`
and temporary `stage_configs` compatibility view must describe the same
topology.
5. `StageRuntime` owns launch planning and replica lifecycle. `ReplicaInitPlan`
is runtime-private state, not a user configuration object.
5. `VllmConfig` and the enriched `OmniDiffusionConfig` are materialized in the
6. `VllmConfig` and the enriched `OmniDiffusionConfig` are materialized in the
process that owns the corresponding engine.
## Main features
docs/design/feature/offloader/README.md
+24 -9
View File
@@ -53,6 +53,11 @@ must use distinct hook names and remove only hooks they own.
## Discovery and topology
`resolve_offload_plan(pipeline, config)` is the single topology entry point.
Every backend calls it once when `enable()` starts and then consumes the
resolved components, block stacks, resident heads, and staged-residency flags
it returns. Backends do not re-read model declarations.
Pipeline component discovery prefers `SupportsComponentDiscovery` declarations:
- `_dit_modules`;
@@ -61,16 +66,26 @@ Pipeline component discovery prefers `SupportsComponentDiscovery` declarations:
- `_resident_modules`.
Dotted paths are supported. Legacy pipelines may use the fallback scan of
well-known attribute names, but new integrations should declare components
explicitly.
well-known attribute names; it now warns once per pipeline class, and new
integrations must declare components explicitly.
Both layerwise backends first consume pipeline `OffloadPlan` metadata. A
plan's `block_attrs` maps each DiT path to its ordered block containers, while
`encoder_block_attrs` declares streamable encoder stacks. DiTs absent from the
plan fall back to `_layerwise_offload_blocks_attrs` (including the deprecated
singular-name compatibility path). Discovery metadata describes structure
only; the backend remains responsible for transfer, synchronization, and
storage ownership.
Block topology comes from the pipeline `OffloadPlan` first: `block_attrs` maps
each DiT path to its ordered block containers, `encoder_block_attrs` declares
streamable encoder stacks, `on_demand_component_paths` marks pipeline-managed
residency, `resident_dit_paths` marks the DiTs that may hold resident layers,
and `encoder_dlo_weight_replication` marks the encoders whose loader-produced
weights are safe for AllGather. DiTs absent from the plan fall back to
`_layerwise_offload_blocks_attrs` (including the deprecated singular-name
compatibility path).
The resolver is pure: it moves no tensor, installs no hook, writes no module
attribute, and reads no process group — multi-rank facts come from
`OffloadConfig`. It owns topology validation, so a configuration the model
cannot serve fails before the first component is placed. Explicit component
selection turns every declaration problem into an error, while the
compatibility topology (no component selector) keeps the historical
warn-and-skip behavior. Metadata still describes structure only; the backend
remains responsible for transfer, synchronization, and storage ownership.
## Cross-strategy invariants
docs/design/fullduplex.md
+4 -4
View File
@@ -66,7 +66,8 @@ The checkpoint keeps these verified contracts:
- scheduler data-plane append over a resumable request;
- stale epoch/turn/response fencing;
- `/v1/duplex` and OpenAI Realtime projection;
- optional OpenAI `server_vad` turn detection backed by per-session Silero VAD;
- optional OpenAI `server_vad` turn detection backed by a process-shared Silero
model and per-session VAD state;
- existing JoyVL behavior.
The checkpoint does not claim:
@@ -652,7 +653,7 @@ EOS decisions advance the model conversation.
Explicit Realtime clients may still use `input_audio_buffer.commit` to create a
conversation item. The translator validates that wire input before producing a
commit carrying `realtime_item_id`. Native auto-response may already have
commit carrying `item_id`. Native auto-response may already have
streamed those PCM samples into the runtime, so the runner accepts a validated
commit even when no runtime-side chunk remains. A truly empty explicit buffer
continues to return `input_audio_buffer_empty`.
@@ -677,8 +678,7 @@ multi-turn runtime evidence.
During auto-response overlap, `preserve_realtime_input` distinguishes "do not
append this silent chunk to the native buffer" from "clear the open Realtime
item". Silent overlap no longer discards earlier user PCM. A separate explicit
`server_vad` policy is documented in the Realtime contract above.
item". Silent overlap no longer discards earlier user PCM.
The first chunk of one overlapping input item also reserves its target model
turn. A later Realtime commit uses that reserved identity even if response EOS
docs/getting_started/installation/README.md
+1 -1
View File
@@ -1,7 +1,7 @@
# Installation
!!! important
vLLM-Omni is released against the matching upstream vLLM major/minor version. vLLM-Omni 0.28.x requires vLLM 0.28.x; the stable v0.28.0 instructions pin vLLM 0.28.0.
vLLM-Omni is released against the matching upstream vLLM major/minor version. The 0.29 development line requires vLLM 0.29.x; the source-install instructions pin vLLM 0.29.0. Published vLLM-Omni 0.28.0 wheels and images still require vLLM 0.28.x.
vLLM-Omni supports the following hardware platforms:
docs/getting_started/installation/gpu.md
+2
View File
@@ -34,6 +34,8 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr
### Pre-built wheels
The source-install instructions target vLLM 0.29.0. Until vLLM-Omni 0.29.0 artifacts are published, the CUDA/ROCm pre-built examples below remain on the matching 0.28.0 release pair.
Note: Pre-built wheels are currently available for vLLM-Omni 0.11.0rc1, 0.12.0rc1, 0.14.0rc1, 0.14.0, 0.16.0, 0.18.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.26.0, and 0.28.0. If you need a newer unreleased revision, please [build from source](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/installation/gpu/#build-wheel-from-source).
=== "NVIDIA CUDA"
docs/getting_started/installation/gpu/cuda.inc.md
+12 -10
View File
@@ -5,7 +5,7 @@
# --8<-- [end:requirements]
# --8<-- [start:set-up-using-python]
vLLM-Omni depends on the matching major/minor release of vLLM. The vLLM-Omni 0.28.x release line uses vLLM 0.28.x.
vLLM-Omni depends on the matching major/minor release of vLLM. The 0.29 development line uses vLLM 0.29.x. Published 0.28.0 wheels and images use vLLM 0.28.x.
!!! note
PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
@@ -18,6 +18,8 @@ Therefore, it is recommended to install vLLM and vLLM-Omni with a **fresh new**
#### Installation of vLLM
These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.29 development line, use the source-install instructions below.
vLLM-Omni is built based on vLLM. Please install it with command below.
```bash
uv pip install vllm==0.28.0 --torch-backend=auto
@@ -26,12 +28,12 @@ uv pip install vllm==0.28.0 --torch-backend=auto
#### Installation of vLLM-Omni
```bash
uv pip install vllm-omni
uv pip install vllm-omni==0.28.0
```
To run Gradio demos, also install the optional extras:
```bash
uv pip install 'vllm-omni[demo]'
uv pip install 'vllm-omni[demo]==0.28.0'
```
# --8<-- [end:pre-built-wheels]
@@ -39,13 +41,13 @@ uv pip install 'vllm-omni[demo]'
# --8<-- [start:build-wheel-from-source]
#### Installation of vLLM
If you do not need to modify source code of vLLM, you can directly install the stable 0.28.0 release version of the library
If you do not need to modify source code of vLLM, you can directly install the stable 0.29.0 release version of the library
```bash
uv pip install vllm==0.28.0 --torch-backend=auto
uv pip install vllm==0.29.0 --torch-backend=auto
```
The 0.28.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
The 0.29.0 release of vLLM ships CUDA 13.0-compatible binaries by default. If you need a different CUDA variant or want to reuse an existing PyTorch installation, build vLLM from source instead.
#### Installation of vLLM-Omni
Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -66,12 +68,12 @@ If you want to check, modify or debug with source code of vLLM, install the libr
```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.28.0
git checkout v0.29.0
```
Set up environment variables to get pre-built wheels. If there are internet problems, just download the whl file manually. And set `VLLM_PRECOMPILED_WHEEL_LOCATION` as your local absolute path of whl file.
```bash
#For CUDA 13.0 (the default for v0.28.0; the wheel filename has no `+cu130` suffix)
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0-cp38-abi3-manylinux_2_28_x86_64.whl
#For CUDA 13.0 (the default for v0.29.0; the wheel filename has no `+cu130` suffix)
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://github.com/vllm-project/vllm/releases/download/v0.29.0/vllm-0.29.0-cp38-abi3-manylinux_2_28_x86_64.whl
```
Install vllm with command below (If you have no existing PyTorch).
```bash
@@ -131,7 +133,7 @@ If you want to specify the base vLLM version:
```bash
DOCKER_BUILDKIT=1 docker build \
-f docker/Dockerfile.cuda \
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.28.0 \
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.29.0 \
-t vllm-omni-cuda .
```
docs/getting_started/installation/gpu/rocm.inc.md
+7 -5
View File
@@ -11,6 +11,8 @@ vLLM-Omni current recommends the steps in under setup through Docker Images.
#### Installation of vLLM
These pre-built wheel instructions install the published vLLM-Omni 0.28.0 release. For the 0.29 development line, use the source-install instructions below.
vLLM-Omni is built based on vLLM. Please install it with command below.
```bash
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
@@ -22,7 +24,7 @@ uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/roc
# we need to add --no-build-isolation as the torch
# is not obtained from pypi, we have to install using the
# torch installed in our environment
uv pip install vllm-omni
uv pip install vllm-omni==0.28.0
# Optional if want to run Qwen3 TTS
uv pip uninstall onnxruntime # should be removed before we can install onnxruntime-rocm
@@ -34,13 +36,13 @@ uv pip install onnxruntime-rocm
# --8<-- [start:build-wheel-from-source]
#### Installation of vLLM
If you do not need to modify source code of vLLM, you can directly install the stable 0.28.0 release version of the library
If you do not need to modify source code of vLLM, you can directly install the stable 0.29.0 release version of the library
```bash
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
uv pip install vllm==0.29.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.29.0/rocm723
```
The pre-built 0.28.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
The pre-built 0.29.0 vLLM wheel targets ROCm 7.2.3. If you need a different ROCm stack or want to reuse an existing PyTorch installation, build vLLM from source instead.
#### Installation of vLLM-Omni
Since vllm-omni is rapidly evolving, it's recommended to install it from source
@@ -58,7 +60,7 @@ If you want to check, modify or debug with source code of vLLM, install the libr
```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.28.0
git checkout v0.29.0
python3 -m pip install -r requirements/rocm.txt
python3 setup.py develop
```
docs/getting_started/installation/npu/npu.inc.md
+156 -42
View File
@@ -7,13 +7,34 @@ For detailed hardware and software requirements, please refer to the [vLLM-Ascen
The recommended way to use vLLM-Omni on NPU is through the vLLM-Ascend pre-built Docker images:
=== "A2"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0
```
=== "A3"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-a3
```
=== "A5"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-a5
```
=== "310P (Experimental)"
```bash
export IMAGE=quay.io/atlas-ci/vllm-ascend:v0.28.0-310p
```
```bash
# vLLM-Ascend image
# Atlas A2: quay.io/atlas-ci/vllm-ascend:v0.28.0
# Atlas A3: quay.io/atlas-ci/vllm-ascend:v0.28.0-a3
docker run --rm \
--name vllm-omni-npu \
--shm-size=1g \
--shm-size=64g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
@@ -27,8 +48,8 @@ docker run --rm \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-p 8000:8000 \
-it quay.io/atlas-ci/vllm-ascend:v0.28.0-a3 bash
-p 8091:8091 \
-it "$IMAGE" bash
# Inside the container, install vLLM-Omni from source
cd /vllm-workspace
@@ -72,53 +93,146 @@ Supported images as following.
| image name | Hardware | OS |
|-|-|-|
| image-tag | Atlas A2 | Ubuntu |
| image-tag-a3 | Atlas A3 | Ubuntu |
| image-tag | A2 | Ubuntu |
| image-tag-a3 | A3 | Ubuntu |
| image-tag-a5 | A5 | Ubuntu |
| image-tag-310p | 310P (experimental) | Ubuntu |
Here's an example deployment command that has been verified on 4 x NPUs:
```bash
# Atlas A2: quay.io/ascend/vllm-omni:v0.28.0
# Atlas A3: quay.io/ascend/vllm-omni:v0.28.0-a3
docker run --rm \
--name vllm-omni-a3 \
--shm-size=4g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v ~/.cache:/root/.cache \
-p 8091:8091 \
-it quay.io/ascend/vllm-omni:v0.28.0-a3 bash
```
=== "A2"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0
docker run --rm \
--name vllm-omni-a2 \
--shm-size=64g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v ~/.cache:/root/.cache \
-p 8091:8091 \
-it "$IMAGE" bash
```
=== "A3"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-a3
docker run --rm \
--name vllm-omni-a3 \
--shm-size=64g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v ~/.cache:/root/.cache \
-p 8091:8091 \
-it "$IMAGE" bash
```
=== "A5"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-a5
docker run --rm \
--name vllm-omni-a5 \
--shm-size=64g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v ~/.cache:/root/.cache \
-p 8091:8091 \
-it "$IMAGE" bash
```
=== "310P (Experimental)"
```bash
export IMAGE=quay.io/ascend/vllm-omni:v0.28.0-310p
docker run --rm \
--name vllm-omni-310p \
--shm-size=64g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v ~/.cache:/root/.cache \
-p 8091:8091 \
-it "$IMAGE" bash
```
!!! tip
You can use this docker image to serve models the same way you would with in vLLM! To do so, make sure you overwrite the default entrypoint (`vllm serve --omni`) which works only for models supported in the vLLM-Omni project.
Or build IMAGE from **source code**:
Or build the image from **source code**:
```bash
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
# A2
# docker build -t vllm-omni-dev-image:latest -f ./docker/Dockerfile.npu .
# A3
docker build -t vllm-omni-dev-image:latest -f ./docker/Dockerfile.npu.a3 .
```
All hardware variants use `docker/Dockerfile.npu`. Choose the device suffix for your hardware:
=== "A2"
```bash
docker build -t vllm-omni-dev-image:latest -f docker/Dockerfile.npu .
```
=== "A3"
```bash
docker build -t vllm-omni-dev-image:latest -f docker/Dockerfile.npu \
--build-arg VLLM_ASCEND_DEVICE_SUFFIX=-a3 .
```
=== "A5"
```bash
docker build -t vllm-omni-dev-image:latest -f docker/Dockerfile.npu \
--build-arg VLLM_ASCEND_DEVICE_SUFFIX=-a5 .
```
=== "310P (Experimental)"
```bash
docker build -t vllm-omni-dev-image:latest -f docker/Dockerfile.npu \
--build-arg VLLM_ASCEND_DEVICE_SUFFIX=-310p .
```
# --8<-- [end:pre-built-images]
docs/getting_started/quickstart.md
+3 -3
View File
@@ -19,10 +19,10 @@ uv venv --python 3.12 --seed
source .venv/bin/activate
# On CUDA
uv pip install vllm==0.28.0 --torch-backend=auto
uv pip install vllm==0.29.0 --torch-backend=auto
# On ROCm
uv pip install vllm==0.28.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.28.0/rocm723
uv pip install vllm==0.29.0+rocm723 --extra-index-url https://wheels.vllm.ai/rocm/0.29.0/rocm723
git clone https://github.com/vllm-project/vllm-omni.git
cd vllm-omni
@@ -34,7 +34,7 @@ For additional installation methods — please see the [installation guide](inst
!!! note
It is important to install the same major & minor version of vLLM and vLLM Omni, otherwise things may not work as expected. If the versions are misaligned, you will see a warning when you import vLLM Omni.
If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.28.0` and vLLM Omni `0.28.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.
If you are seeing strange behavior with the `vllm` command not handling the `--omni` flag correctly, you most likely have a version mismatch with vLLM < `0.29.0` and vLLM Omni `0.29.0`, as vLLM Omni no longer hijacks the vLLM entrypoint. Updating vLLM should resolve this issue.
## Offline Inference
docs/mkdocs/hooks/generate_argparse.py
+1
View File
@@ -124,6 +124,7 @@ def extract_omni_serve_subparser_init():
"make_arg_parser": lambda parser: parser, # no-op for doc
"_ensure_vllm_platform": lambda: None, # no-op for doc
"_nonneg_finite_float": float,
"_parse_stage_overrides": __import__("json").loads,
"VLLM_SUBCMD_PARSER_EPILOG": "",
"logger": logger,
"DummySubparsers": DummySubparsers,
docs/models/supported_models.md
+2
View File
@@ -64,6 +64,7 @@ th {
| `LongCatImageEditPipeline` | LongCat-Image-Edit | `meituan-longcat/LongCat-Image-Edit` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `LongCatVideoAvatarPipeline` | LongCat-Video-Avatar-1.5 A2V/AI2V (native single-speaker and multi-speaker AI2V/AVC) | `meituan-longcat/LongCat-Video-Avatar-1.5` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/meituan-longcat/LongCat-Video-Avatar-1.5.md) |
| `BooguImagePipeline` | Boogu-Image | `Boogu/Boogu-Image-0.1-Base`, `Boogu/Boogu-Image-0.1-Edit`, `Boogu/Boogu-Image-0.1-Edit-Turbo` | ✅︎ | | | ✅︎ | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) |
| `BooguImageTurboPipeline` | Boogu-Image Turbo distilled T2I | `Boogu/Boogu-Image-0.1-Turbo` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/Boogu/Boogu-Image.md) |
| `StableDiffusionXLPipeline` | Stable-Diffusion-XL | `stabilityai/stable-diffusion-xl-base-1.0` | ✅︎ | ✅︎ | ✅︎ | ✅︎ | — |
| `StableDiffusion3Pipeline` | Stable-Diffusion-3 | `stabilityai/stable-diffusion-3.5-medium` | ✅︎ | ✅︎ | | ✅︎ | — |
| `CosyVoice3Model` | CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | ✅︎ | ✅︎ | | ✅︎ | — |
@@ -96,6 +97,7 @@ th {
| `MiMoV2ASRForCausalLM` | MiMo-V2.5-ASR | `XiaomiMiMo/MiMo-V2.5-ASR` | ✅︎ | ✅︎ | | | — |
| `Flux2Pipeline` | FLUX.2-dev | `black-forest-labs/FLUX.2-dev` | ✅︎ | ✅︎ | | | — |
| `FishSpeechSlowARForConditionalGeneration` | Fish Speech S2 Pro | `fishaudio/s2-pro` | ✅︎ | ✅︎ | | ✅︎ | — |
| `Audio8TTSSlowARForConditionalGeneration` | Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✅︎ | | | | — |
| `SenseNovaU1Pipeline` | SenseNova-U1 (DiT-only), SenseNova-U1.5 | `SenseNova/SenseNova-U1-8B-MoT`, `sensenova/SenseNova-U1.5-8B-MoT` | ✅︎ | | | | [Repository](https://github.com/vllm-project/vllm-omni/blob/main/recipes/SenseNova/SenseNova-U1.5.md) |
| `LancePipeline` | Lance | `bytedance-research/Lance` | ✅︎ | | | | — |
| `HunyuanVideo15Pipeline` | HunyuanVideo-1.5-T2V | `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v`, `hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-720p_t2v` | ✅︎ | ✅︎ | | | — |
docs/serving/openpi_api.md
+7 -1
View File
@@ -64,7 +64,7 @@ connect
<- msgpack(policy_server_config)
infer
-> msgpack({"endpoint": "infer", "session_id": "...", ...observation})
-> msgpack({"endpoint": "infer", "session_id": "...", "seed": <optional int>, ...observation})
<- msgpack(ndarray | dict[str, ndarray])
reset
@@ -92,6 +92,12 @@ frames are not observation messages.
- The policy pipeline owns observation transforms and persistent model state,
normally keyed by `session_id`; the API layer forwards the raw observation
dictionary.
- An optional integer `seed` in an inference message becomes the engine
request's `sampling_params.seed`. A policy whose sampling consumes the
request generator (GR00T-N1.7) returns the same action chunk for the same
observation and seed. If omitted, the engine assigns a random per-request
seed, so unseeded inferences are independent of each other. The API layer
consumes `seed`; it is not forwarded inside the observation.
## Limits and Errors
docs/serving/realtime_duplex_api.md
+24 -17
View File
@@ -1,17 +1,17 @@
# Realtime Duplex API
vLLM-Omni serves full-duplex speech models over a WebSocket endpoint,
`/v1/realtime?duplex=1`, that speaks the OpenAI Realtime vocabulary plus a
`/v1/realtime`, that speaks the OpenAI Realtime vocabulary plus a
set of duplex extensions: the model decides when to listen and when to speak,
the user can talk over the assistant, playback progress is reported back so
history stays honest, and a dropped connection can resume the same session.
This page covers how to run a duplex deployment, how to drive it from Python
with `vllm_omni.clients.duplex.DuplexClient`, and the complete wire contract.
The endpoint is served for models that ship a duplex plugin — currently
MiniCPM-o 4.5, PersonaPlex, and Nemotron VoiceChat — and only for deploy
configurations that declare `session_mode: duplex`. The runtime architecture
is described in [Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
The handler is enabled by either a model-native `session_mode: duplex`
deployment or an explicit `duplex_session` configuration for turn-based
Server VAD. The runtime architecture is described in
[Full-Duplex Runtime (MiniCPM-o 4.5)](../design/fullduplex.md).
## Quick Start
@@ -27,7 +27,7 @@ vllm-omni serve openbmb/MiniCPM-o-4_5 \
`vllm_omni/deploy/minicpmo_4_5.yaml` declares `session_mode: duplex` and
`duplex_session.max_sessions: 4`, which mounts the WebSocket routes
`ws://<host>:8099/v1/realtime?duplex=1` (this page) and `ws://<host>:8099/v1/duplex`
`ws://<host>:8099/v1/realtime` (this page) and `ws://<host>:8099/v1/duplex`
(the native dialect for raw-protocol clients) next to the usual HTTP API.
PersonaPlex and Nemotron VoiceChat use `vllm_omni/deploy/personaplex.yaml` and
`vllm_omni/deploy/nemotron_labs_voicechat_duplex.yaml` in the same way.
@@ -240,7 +240,7 @@ on the same helpers.
## API Reference
This section is the complete wire contract of `/v1/realtime?duplex=1`: how
This section is the complete wire contract of `/v1/realtime`: how
the dialect relates to the OpenAI Realtime protocol, which surfaces each
model's capability flags gate, a catalogue of every client-to-server and
server-to-client event, and a JSON example for each. `DuplexClient` speaks
@@ -253,10 +253,11 @@ its protocol tests.
### Endpoint and transport
The endpoint is `ws(s)://<host>/v1/realtime?duplex=1`; the `duplex` query
parameter (`1`, `true`, or `on`) selects the duplex session handler, and the
route is mounted only for deployments whose deploy configuration declares
`session_mode: duplex`. Optional query parameters are `model`, `session_id`,
The endpoint is `ws(s)://<host>/v1/realtime`. Bare connections select the
session handler when the deployment declares either `session_mode: duplex`
or `duplex_session`. `?duplex=1` remains a compatibility alias, while
`?duplex=0` selects the legacy Realtime handler. Optional query parameters
are `model`, `session_id`,
`autostart` (`0` means resume-only), `resume`, and `native_duplex`
(`minicpmo45_native_duplex` is accepted as a deprecated alias and folded
into the canonical name). Every message is one JSON object per WebSocket
@@ -342,9 +343,14 @@ Semantic divergences hidden behind shared names:
before any commit; a commit may end in `response.listen` and no
response, and the model may open a response with no commit at all.
OpenAI: commit ⇒ item, `response.create` ⇒ exactly one response.
- `turn_detection.interrupt_response=false` is rejected;
`create_response` is ignored; `semantic_vad` is unsupported; the VAD
runs per session (Silero) and implies `overlap_policy=barge_in_on_speech`.
- With `turn_detection={"type":"server_vad"}`, omitted `interrupt_response`
defaults to `false` on turn-based models such as Qwen3-Omni, enabling
endpointing and automatic responses without interruption. Explicit `true`
is rejected there. Model-native duplex keeps the `true` default and rejects
`false`; its VAD implies `overlap_policy=barge_in_on_speech`.
This capability-dependent default differs from OpenAI's `true` default;
the effective configuration is returned in `session.created` / `session.updated`.
`semantic_vad` is unsupported.
- `rate_limits.updated` is always an empty list (compatibility only).
- Cancellation never reuses a `response_id`; truncation is driven by
`playback.ack` as well as `conversation.item.truncate`.
@@ -1201,9 +1207,10 @@ Realtime projector renames and fans out these events before they reach a
barge-in, or audio truncation; Nemotron VoiceChat does not support barge-in
or audio truncation; camera frames are consumed only by MiniCPM-o 4.5; tool
calls are produced only by Nemotron VoiceChat.
- `turn_detection` supports only `server_vad` with `interrupt_response=true`;
`semantic_vad`, `interrupt_response=false`, and `create_response` are not
supported. Server VAD requires a model-native session.
- `turn_detection` supports `server_vad` or `null`, not `semantic_vad`.
Turn-based models support endpointing without interruption, including
`create_response=false` for explicit response creation. Model-native duplex
requires `interrupt_response=true` and `create_response=true`.
- `input_audio_transcription` and `input_audio_noise_reduction` are accepted
and echoed but no separate transcription or noise-reduction stage runs;
transcripts come from the model.
docs/serving/speech_api.md
+31 -12
View File
@@ -279,10 +279,15 @@ curl -X POST http://localhost:8091/v1/audio/voices \
## Streaming Text Input (WebSocket)
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally and generates audio per sentence as boundaries are detected.
The `/v1/audio/speech/stream` WebSocket endpoint accepts text incrementally.
By default (`split_granularity=none`) it buffers until `input.done` and
synthesizes that flush as **one** TTS request, which keeps long-form timbre
stable. Set `split_granularity` to `sentence` or `clause` to emit a request
at each detected boundary (lower time-to-first-audio for STT/LLM pipelines).
> Note: text input is always streamed incrementally. Audio output remains sentence-scoped:
> use `stream_audio=false` for one binary frame per sentence, or `stream_audio=true` for one or more PCM chunks per sentence.
> Note: `stream_audio` only changes how **audio bytes** are framed (one WAV/PCM
> payload vs chunked PCM). Text segmentation is controlled separately by
> `split_granularity`.
### WebSocket Protocol
@@ -314,14 +319,16 @@ upstream LLM) pays the WebSocket handshake once instead of once per utterance.
- The session config is sticky. Send `input.text` again straight after
`session.done` to reuse it, or send another `session.config` first to change
voice, format, or reference audio. A `session.config` sent while text is
still buffered is rejected so no pending input is silently dropped.
- An utterance is the flush unit, not a linguistic one: it is whatever text was
buffered when `input.done` arrived, of any length, synthesized as one request.
`utterance_index` counts those flushes across the connection, so it tells you
which `input.done` a frame belongs to. `sentence_index` counts within one
flush and so pairs with `total_sentences`, which means every utterance reports
`sentence_index: 0` of `total_sentences: 1` (or `0` for an empty buffer).
voice, format, or reference audio. A `session.config` sent in the middle of
an utterance is rejected so no pending input is silently dropped and so a
split utterance cannot end up half in one voice and half in another.
- An utterance is the flush unit, not a linguistic one: it is one `input.done`
cycle. `utterance_index` counts those flushes across the connection, so it
tells you which `input.done` a frame belongs to. `sentence_index` counts the
TTS requests inside one flush and so pairs with `total_sentences`: with the
default `split_granularity=none` that is always `sentence_index: 0` of
`total_sentences: 1` (or `0` for an empty buffer), while `sentence` or
`clause` counts the linguistic units actually synthesized.
- End the connection with `session.close`, or by closing the socket. An idle
connection is still closed after the server's idle timeout, which now also
applies to the gap between utterances.
@@ -332,7 +339,15 @@ All REST API parameters are supported, plus:
| Parameter | Type | Default | Description |
| ----------- | ------ | --------- | ------------- |
| `stream_audio` | bool | false | Stream one or more PCM chunks for the buffered input over WebSocket |
| `stream_audio` | bool | false | Stream one or more PCM chunks for each TTS request over WebSocket |
| `split_granularity` | string | `"none"` | `"none"`: one request per `input.done`. `"sentence"`: split on `.!?` plus CJK `。！？…`, Indic danda `।॥`, and Arabic `؟`. `"clause"`: also split on `,;，；،؛`. |
| `seed` | integer | null | Forwarded to the speech engine for this session |
ASCII punctuation only ends a unit when whitespace or `input.done` follows it,
and decimals (`3.14`), thousands separators (`1,000`), abbreviations (`Dr.`,
`e.g.`) and initials (`J. R.`) are not treated as boundaries. A punctuation run
and any closing quote or bracket stay with the unit they close, so `Wait...`
and `He said "Hello."` are one request each.
```bash
DELETE /v1/audio/voices/{name}
@@ -752,6 +767,10 @@ Fish Speech uses `ref_audio` and `ref_text` for voice cloning (no `task_type` ne
| ------- | ------------- |
| `k2-fsa/OmniVoice` | Pure-diffusion TTS. Supports voice cloning via `ref_audio` (with optional `ref_text`); no built-in voice presets. |
OmniVoice uses packed variable-length attention for batched generator execution. The attention operator accepts FP16 and BF16 inputs, so its query,
key, and value tensors are evaluated in BF16 even when the stage is configured with `dtype: float32`; the attention output is converted back to the model's
hidden-state dtype before the output projection. Consequently, a float32 stage configuration does not imply FP32 attention arithmetic.
### VoxCPM2
| Model | Description |
docs/user_guide/diffusion/attention_backends/huggingface_hub.md
+1 -1
View File
@@ -12,7 +12,7 @@ implementations to reduce numerical drift and sampling divergence.
## Installation
```bash
pip install kernels==0.15.2
pip install kernels==0.16.1
```
If `kernels` is unavailable, an explicit `FLASH_ATTN_HUB` or
docs/user_guide/examples/offline_inference/text_to_speech.md
+1 -5
View File
@@ -126,11 +126,7 @@ python examples/offline_inference/text_to_speech/cosyvoice3/end2end.py \
## Fish Speech S2 Pro
4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz.
### Prerequisites
```bash
pip install fish-speech
```
No extra packages are required; the DAC codec is vendored in vLLM-Omni.
### Quick start
```bash
docs/user_guide/examples/online_serving/text_to_speech.md
+2 -7
View File
@@ -155,13 +155,8 @@ The client supports `--api-base`, `--model`, `--text`, `--ref-audio`, `--ref-tex
## Fish Speech S2 Pro
4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec.
### Prerequisites
```bash
pip install fish-speech
```
4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec, which is vendored in
vLLM-Omni (no extra packages required).
### Launch
docs/user_guide/examples/online_serving/text_to_video.md
+9
View File
@@ -199,12 +199,21 @@ curl -X POST http://localhost:8091/v1/videos \
| `boundary_ratio` | float | None | Boundary split ratio for low/high DiT (Wan2.2) |
| `flow_shift` | float | None | Scheduler flow shift (Wan2.2) |
| `seed` | int | None | Random seed (reproducible) |
| `num_outputs_per_prompt` | int | 1 | Number of videos to generate (1-10, MiniMax H3); only the first is returned, see below |
| `lora` | object | None | LoRA configuration |
| `enable_frame_interpolation` | bool | false | Enable RIFE frame interpolation before MP4 encoding |
| `frame_interpolation_exp` | int | 1 | Interpolation exponent; 1=2x temporal resolution, 2=4x |
| `frame_interpolation_scale` | float | 1.0 | RIFE inference scale; use 0.5 for high-resolution inputs |
| `frame_interpolation_model_path` | str | None | Local directory or Hugging Face repo ID with `flownet.pkl`; defaults to `elfgum/RIFE-4.22.lite` |
!!! note "Only the first output is returned"
`num_outputs_per_prompt` is forwarded to the pipeline, so the model does
generate that many videos and you pay the generation cost for all of them.
`/v1/videos` and `/v1/videos/sync` then return only the **first** one,
because an async video job stores a single file per video id. The server
logs a warning when it discards the extra outputs. Keep this at `1`
unless you have a specific reason to generate videos you will not receive.
## Frame Interpolation
Frame interpolation is an optional post-processing step for `/v1/videos` and
examples/offline_inference/qwen2_5_omni/end2end.py
+27 -6
View File
@@ -5,10 +5,12 @@ This example shows how to use vLLM-Omni for running offline inference
with the correct prompt format on Qwen2.5-Omni
"""
import argparse
import functools
import json
import os
import time
from typing import NamedTuple
from typing import Any, NamedTuple
import numpy as np
import soundfile as sf
@@ -26,6 +28,20 @@ from vllm_omni.utils.tracking_parser import TrackingArgumentParser
SEED = 42
def parse_json_object(value: str, flag_name: str = "argument") -> dict[str, Any]:
"""Parse a CLI value as a JSON object, attributing errors to ``flag_name``."""
try:
config = json.loads(value)
except json.JSONDecodeError as e:
raise argparse.ArgumentTypeError(f"{flag_name} must be valid JSON: {e}") from e
if not isinstance(config, dict):
raise argparse.ArgumentTypeError(f"{flag_name} must be a JSON object")
return config
parse_profiler_config = functools.partial(parse_json_object, flag_name="--profiler-config")
class QueryResult(NamedTuple):
inputs: dict
limit_mm_per_prompt: dict[str, int]
@@ -378,11 +394,10 @@ def main(args):
for i, prompt in enumerate(prompts):
prompt["modalities"] = output_modalities
profiler_enabled = bool(os.getenv("VLLM_TORCH_PROFILER_DIR"))
if profiler_enabled and hasattr(omni, "start_profile"):
profiler_enabled = args.profiler_config is not None
if profiler_enabled:
print("[Profiler] Starting profiling...")
omni.start_profile(stages=[0])
elif profiler_enabled:
print("[Warn] VLLM_TORCH_PROFILER_DIR is set, but current engine does not support profiler controls.")
omni_generator = omni.generate(prompts, sampling_params_list, py_generator=args.py_generator)
# Determine output directory: prefer --output-dir; fallback to --output-wav
@@ -418,7 +433,7 @@ def main(args):
print(f"Request ID: {request_id}, Saved audio to {output_wav}")
processed_count += 1
if profiler_enabled and hasattr(omni, "stop_profile") and processed_count >= total_requests:
if profiler_enabled and processed_count >= total_requests:
print(f"[Info] Processed {processed_count}/{total_requests}. Stopping profiler inside active loop...")
# Stop the profiler while workers are still alive
omni.stop_profile()
@@ -476,6 +491,12 @@ def parse_args():
default=300,
help="Timeout for initializing stages in seconds (default: 300)",
)
parser.add_argument(
"--profiler-config",
type=parse_profiler_config,
default=None,
help='JSON profiler config for torch/cuda profiling, e.g. \'{"profiler":"torch","torch_profiler_dir":"./perf"}\'.',
)
parser.add_argument(
"--shm-threshold-bytes",
type=int,
examples/offline_inference/text_to_speech/README.md
+53 -5
View File
@@ -14,6 +14,7 @@ list of supported architectures across all modalities, see
| Model | HuggingFace repo | Stages | Voice cloning | Streaming | Special modes | Sample rate |
|---|---|---|---|---|---|---|
| Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | dual-AR | ✓ | ✓ | 11 languages | 44.1 kHz |
| CosyVoice3 | `FunAudioLLM/Fun-CosyVoice3-0.5B-2512` | 2 (talker + code2wav) | ✓ | ✓ | — | 24 kHz |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | dual-AR | ✓ | ✓ | — | 44.1 kHz |
| Gepard-1.0 | `nineninesix/gepard-1.0` | single (native AR + NanoCodec) | — (zero-shot; cloning WIP) | — (serving WIP) | zero-shot | 22.05 kHz |
@@ -44,6 +45,57 @@ python examples/offline_inference/text_to_speech/<model>/end2end.py \
---
## Audio8 TTS Preview
0.6B DualAR text-to-speech model (Audio8) with its own 44.1 kHz neural audio
codec: a slow AR transformer predicts one semantic token per codec frame, and a
4-layer fast AR predicts that frame's remaining 9 codebooks.
### Prerequisites
No extra packages: the codec architecture ships in tree and its weights
(`codec.pth`) come with the checkpoint.
### Quick start
```bash
python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
--text "Welcome to Audio8 TTS, a compact multilingual text to speech model."
```
### Voice cloning
```bash
python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
--text "Welcome to Audio8 TTS." \
--ref-audio /path/to/reference.wav \
--ref-text "The exact transcript of the reference recording."
```
The reference transcript must match what is actually spoken in the clip;
a mismatch degrades both stability and speaker similarity.
### Streaming
```bash
python examples/offline_inference/text_to_speech/audio8_tts/end2end.py \
--text "Welcome to Audio8 TTS." --streaming
```
### Notes
- Output: 44.1 kHz mono WAV; the codec emits ~21.5 frames/s (2048 samples per frame).
- Context is 2048 packed text+audio positions, so `max_model_len` is 2048.
- Greedy decoding (`temperature: 0`) is degenerate for this checkpoint — it
ends the utterance almost immediately. Keep the deploy defaults
(`temperature 0.7`, `top_k 50`, `top_p 0.9`), which also match
`generation_config.json`.
- Repetition-Aware Sampling is applied to the semantic token: a repeat inside
the last 10 frames is re-drawn from a flatter distribution instead of being
masked out.
- `--num-prompts 4` smoke-tests concurrency (stage 0 runs `max_num_seqs: 4`).
### Measured behaviour (1x H20, shared GPU)
`--streaming` on a 58-character English sentence: TTFA ~0.74 s (4-frame first
chunk), 7 chunks, 1.55 s wall for 3.16 s of audio (RTF ~0.49). Treat these as
smoke-test magnitudes, not a benchmark.
---
## CosyVoice3
2-stage TTS pipeline (`talker` + `code2wav`) at 24 kHz.
@@ -130,11 +182,7 @@ Text → [Stage 0: AR] → Speech Tokens → [Stage 1: DiT + HiFT] → Audio (24
## Fish Speech S2 Pro
4B dual-AR text-to-speech model from FishAudio with the DAC codec at 44.1 kHz.
### Prerequisites
```bash
pip install fish-speech
```
No extra packages are required; the DAC codec is vendored in vLLM-Omni.
### Quick start
```bash
examples/offline_inference/text_to_speech/audio8_tts/end2end.py
+237
View File
@@ -0,0 +1,237 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Offline inference demo for Audio8 TTS Preview 0.6B via vLLM Omni.
Usage:
# Text-only synthesis
python end2end.py --text "Welcome to Audio8 TTS."
# Zero-shot voice cloning (the transcript must match the reference audio)
python end2end.py --text "Welcome to Audio8 TTS." \
--ref-audio reference.wav --ref-text "The exact transcript of the reference recording."
# Streaming (per-chunk TTFA / inter-chunk timings)
python end2end.py --text "Welcome to Audio8 TTS." --streaming
"""
import asyncio
import os
import time
import numpy as np
import soundfile as sf
import torch
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"
from vllm_omni import AsyncOmni, Omni
from vllm_omni.model_executor.models.audio8_tts.codec_utils import estimate_reference_code_frames
from vllm_omni.model_executor.models.audio8_tts.prompt_utils import (
build_text_only_prompt_ids,
estimate_voice_clone_prompt_len,
normalize_text,
)
from vllm_omni.utils.tracking_parser import TrackingArgumentParser
DEFAULT_MODEL = "Audio8/Audio8-TTS-Preview-0.6b"
def build_prompt(
text: str,
ref_audio_path: str | None = None,
ref_text: str | None = None,
model_name: str = DEFAULT_MODEL,
) -> dict:
"""Build an engine prompt, using the same protocol as online serving."""
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)
if ref_audio_path is None and ref_text is None:
prompt_ids, normalized_text = build_text_only_prompt_ids(tokenizer, text)
return {
"prompt_token_ids": prompt_ids,
"additional_information": {"text": [normalized_text]},
}
if not ref_audio_path or not ref_text:
raise ValueError("Audio8 TTS voice cloning requires both --ref-audio and --ref-text")
normalized_text = normalize_text(text)
normalized_ref_text = normalize_text(ref_text, add_default_speaker=True)
ref_audio_wav, ref_audio_sr = sf.read(ref_audio_path, dtype="float32", always_2d=False)
if ref_audio_wav.ndim > 1:
ref_audio_wav = ref_audio_wav.mean(axis=-1)
ref_frames = estimate_reference_code_frames(int(ref_audio_wav.shape[0]), int(ref_audio_sr))
placeholder_len = estimate_voice_clone_prompt_len(tokenizer, normalized_text, normalized_ref_text, ref_frames)
# The clone prompt embeds the reference audio's own codec codes, so the
# model builds it in preprocess(); we only reserve the exact length here.
return {
"prompt_token_ids": [1] * placeholder_len,
"additional_information": {
"text": normalized_text,
"ref_text": normalized_ref_text,
"ref_audio_wav": torch.from_numpy(np.asarray(ref_audio_wav, dtype=np.float32)),
"ref_audio_sr": int(ref_audio_sr),
"audio8_structured_voice_clone": True,
},
}
def extract_audio(multimodal_output: dict) -> tuple[torch.Tensor, int]:
"""Pull the waveform and sample rate out of a multimodal output.
The output processor concatenates the per-step delta tensors under
``model_outputs``; ``audio`` is the older alias. Never use ``a or b`` on
these values -- truthiness on a tensor raises.
"""
audio = multimodal_output.get("model_outputs")
if audio is None:
audio = multimodal_output.get("audio")
if audio is None:
raise ValueError(f"No audio in multimodal_output: {sorted(multimodal_output)}")
if isinstance(audio, list):
audio = torch.cat([torch.as_tensor(chunk).reshape(-1) for chunk in audio], dim=0)
audio = torch.as_tensor(audio).reshape(-1)
sr_raw = multimodal_output.get("sr")
if isinstance(sr_raw, list):
sr_raw = sr_raw[-1] if sr_raw else None
if sr_raw is None:
raise ValueError("Missing sample rate in multimodal_output")
sample_rate = int(sr_raw.item()) if hasattr(sr_raw, "item") else int(sr_raw)
return audio, sample_rate
def _save_wav(output_dir: str, request_id: str, audio: torch.Tensor, sample_rate: int) -> None:
out_wav = os.path.join(output_dir, f"output_{request_id}.wav")
sf.write(out_wav, audio.float().cpu().numpy(), samplerate=sample_rate, format="WAV")
duration = audio.numel() / sample_rate
# print(), not logging: vLLM reconfigures logging on engine start and
# disables loggers created before that, which would swallow these lines.
print(f"Request {request_id}: saved {out_wav} (sr={sample_rate}, {duration:.2f}s)")
def _omni_kwargs(args, model_name: str) -> dict:
kwargs = {
"model": model_name,
"log_stats": args.log_stats,
"stage_init_timeout": args.stage_init_timeout,
}
if args.deploy_config:
kwargs["deploy_config"] = args.deploy_config
return kwargs
def main(args) -> None:
os.makedirs(args.output_dir, exist_ok=True)
model_name = args.model or DEFAULT_MODEL
prompts = [
build_prompt(text, args.ref_audio, args.ref_text, model_name) for text in ([args.text] * args.num_prompts)
]
omni = Omni(**_omni_kwargs(args, model_name))
t_start = time.perf_counter()
total_audio_seconds = 0.0
for stage_outputs in omni.generate(prompts):
request_output = stage_outputs
if request_output is None or not request_output.outputs:
continue
audio, sample_rate = extract_audio(request_output.outputs[0].multimodal_output)
assert audio.numel() > 0, "Audio8 TTS produced an empty waveform"
total_audio_seconds += audio.numel() / sample_rate
_save_wav(args.output_dir, request_output.request_id, audio, sample_rate)
elapsed = time.perf_counter() - t_start
rtf = elapsed / max(total_audio_seconds, 1e-6)
print(f"Total {elapsed * 1000:.1f} ms for {total_audio_seconds:.2f} s of audio (RTF={rtf:.3f})")
async def main_streaming(args) -> None:
os.makedirs(args.output_dir, exist_ok=True)
model_name = args.model or DEFAULT_MODEL
prompt = build_prompt(args.text, args.ref_audio, args.ref_text, model_name)
omni = AsyncOmni(**_omni_kwargs(args, model_name))
request_id = "0"
chunks: list[torch.Tensor] = []
sample_rate = None
t_start = time.perf_counter()
t_prev = t_start
chunk_index = 0
async for stage_output in omni.generate(prompt, request_id=request_id):
multimodal_output = stage_output.outputs[0].multimodal_output
if not multimodal_output:
continue
audio = multimodal_output.get("model_outputs")
if audio is None:
audio = multimodal_output.get("audio")
if audio is not None:
if isinstance(audio, list):
chunks.extend(torch.as_tensor(chunk).reshape(-1) for chunk in audio)
else:
chunks.append(torch.as_tensor(audio).reshape(-1))
sr_raw = multimodal_output.get("sr")
if isinstance(sr_raw, list):
sr_raw = sr_raw[-1] if sr_raw else None
if sr_raw is not None:
sample_rate = int(sr_raw.item()) if hasattr(sr_raw, "item") else int(sr_raw)
if stage_output.finished:
break
now = time.perf_counter()
if chunk_index == 0:
print(f"chunk 0: TTFA={(now - t_start) * 1000:.1f} ms")
else:
print(f"chunk {chunk_index}: inter_chunk={(now - t_prev) * 1000:.1f} ms")
t_prev = now
chunk_index += 1
if not chunks or sample_rate is None:
raise RuntimeError("Audio8 TTS streaming produced no audio")
_save_wav(args.output_dir, request_id, torch.cat(chunks, dim=0), sample_rate)
print(f"Streaming done: total={(time.perf_counter() - t_start) * 1000:.1f} ms chunks={chunk_index}")
def parse_args():
parser = TrackingArgumentParser(description="Audio8 TTS Preview offline inference with vLLM Omni")
parser.add_argument("--model", type=str, default=DEFAULT_MODEL, help="Model path or HuggingFace repo ID.")
parser.add_argument(
"--text",
type=str,
default="Welcome to Audio8 TTS, a compact multilingual text to speech model.",
help="Text to synthesize.",
)
parser.add_argument("--ref-audio", type=str, default=None, help="Reference audio path for voice cloning.")
parser.add_argument(
"--ref-text",
type=str,
default=None,
help="Transcript of --ref-audio; required for voice cloning and must match the recording.",
)
parser.add_argument("--output-dir", type=str, default="output_audio", help="Directory for output WAV files.")
parser.add_argument(
"--num-prompts",
type=int,
default=1,
help="Duplicate the prompt N times to smoke-test concurrent requests.",
)
parser.add_argument(
"--deploy-config",
type=str,
default=None,
help="Override the deploy config path; defaults to vllm_omni/deploy/audio8_tts.yaml via model_type.",
)
parser.add_argument("--streaming", action="store_true", help="Use AsyncOmni and report per-chunk latency.")
parser.add_argument("--log-stats", action="store_true", help="Enable engine stats logging.")
parser.add_argument("--stage-init-timeout", type=int, default=600, help="Per-stage init timeout in seconds.")
return parser.parse_args()
if __name__ == "__main__":
parsed_args = parse_args()
if parsed_args.streaming:
asyncio.run(main_streaming(parsed_args))
else:
main(parsed_args)
examples/offline_inference/text_to_speech/cosyvoice3/end2end.py
+27 -3
View File
@@ -1,9 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import argparse
import functools
import json
import os
import urllib.request
from pathlib import Path
from typing import Any
import numpy as np
import soundfile as sf
@@ -29,6 +32,20 @@ def _default_ref_audio() -> str:
return str(dest)
def parse_json_object(value: str, flag_name: str = "argument") -> dict[str, Any]:
"""Parse a CLI value as a JSON object, attributing errors to ``flag_name``."""
try:
config = json.loads(value)
except json.JSONDecodeError as e:
raise argparse.ArgumentTypeError(f"{flag_name} must be valid JSON: {e}") from e
if not isinstance(config, dict):
raise argparse.ArgumentTypeError(f"{flag_name} must be a JSON object")
return config
parse_profiler_config = functools.partial(parse_json_object, flag_name="--profiler-config")
def run_e2e():
parser = argparse.ArgumentParser()
# ""FunAudioLLM/Fun-CosyVoice3-0.5B-2512
@@ -64,6 +81,12 @@ def run_e2e():
required=True,
help="Path to tokenizer directory (e.g., <model_path>/CosyVoice-BlankEN).",
)
parser.add_argument(
"--profiler-config",
type=parse_profiler_config,
default=None,
help='JSON profiler config for torch/cuda profiling, e.g. \'{"profiler":"torch","torch_profiler_dir":"./perf"}\'.',
)
args = parser.parse_args()
# Ensure tokenizer directory exists
if not os.path.exists(args.tokenizer):
@@ -79,6 +102,7 @@ def run_e2e():
deploy_config=args.deploy_config,
tokenizer=args.tokenizer,
log_stats=True,
profiler_config=args.profiler_config,
)
sampling_cfg = {"top_p": 0.8, "top_k": 25, "eos_token_id": 6561 + 1}
@@ -149,8 +173,8 @@ def run_e2e():
sampling_params_list = [gpt_sampling, s2mel_sampling]
# Start profiling (requires VLLM_TORCH_PROFILER_DIR env var)
if os.environ.get("VLLM_TORCH_PROFILER_DIR"):
profiler_enabled = args.profiler_config is not None
if profiler_enabled:
print("Starting profiler...")
omni.start_profile()
@@ -158,7 +182,7 @@ def run_e2e():
outputs = list(omni.generate(prompts, sampling_params_list=sampling_params_list[:2]))
# Stop profiling and get results
if os.environ.get("VLLM_TORCH_PROFILER_DIR"):
if profiler_enabled:
print("Stopping profiler...")
profile_results = omni.stop_profile()
print(f"Profile traces saved to: {profile_results}")
examples/online_serving/qwen3_omni/README.md
+21 -3
View File
@@ -264,7 +264,7 @@ python examples/online_serving/openai_chat_completion_client_for_multimodal_gene
#### Realtime WebSocket client (`openai_realtime_client.py`)
[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://<host>:<port>/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks (OpenAI-style `input_audio_buffer.append` / `commit`), and receives **`response.audio.delta`** (incremental PCM for the reply) plus **`transcription.*`** events. By default it concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging.
[`openai_realtime_client.py`](./openai_realtime_client.py) connects to **`ws://<host>:<port>/v1/realtime`**, streams a local WAV as **PCM16 mono @ 16 kHz** in fixed-size chunks, and receives **`response.audio.*`**, **`response.output_text.*`**, and transcript events. By default it uses explicit `input_audio_buffer.commit`; pass **`--server-vad`** to let the server detect speech endpoints and commit the turn automatically. The client concatenates audio deltas and writes **`--output-wav`** (model output is typically **24 kHz**). Optional **`--delta-dump-dir`** saves each delta as `delta_000001.wav`, … for debugging.
Streaming input works well for translation-style use cases; if the Thinker runs while input is still incomplete, consider limiting **`max_tokens`** in your session / server defaults to avoid over-generation.
@@ -282,6 +282,7 @@ python openai_realtime_client.py \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
--input-wav /path/to/input_16k_mono.wav \
--output-wav realtime_output.wav \
--server-vad \
--delta-dump-dir ./rt_delta_wavs
```
@@ -297,15 +298,32 @@ python openai_realtime_client.py \
| `--chunk-ms` | `200` | Size of each uploaded audio chunk (milliseconds of audio) |
| `--send-delay-ms` | `0` | Delay between chunk sends (simulate realtime upload) |
| `--delta-dump-dir` | *(optional)* | Directory to write per-`response.audio.delta` WAV files |
| `--server-vad` | disabled | Enable server-side Silero VAD instead of sending explicit commits |
| `--num-requests` | `1` | Number of sequential sessions (see `--concurrency`) |
| `--concurrency` | `1` | Max concurrent WebSocket sessions when `--num-requests` > 1 |
Ensure the server is running, for example:
Server VAD uses the pinned Silero v6.2 ONNX artifact from `istupakov/silero-vad-onnx` and never downloads model files while processing a session. Make the artifact available in the Hugging Face cache before startup, or configure a local artifact in the deployment YAML:
```yaml
duplex_session:
server_vad_model_path: /models/silero_vad.onnx
```
Start from the bundled Qwen3-Omni deployment YAML, add the fields above at the top level, and serve that complete configuration:
```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
cp vllm_omni/deploy/qwen3_omni_moe.yaml /path/to/qwen3_omni_server_vad.yaml
# Edit /path/to/qwen3_omni_server_vad.yaml and add duplex_session.
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
--omni \
--port 8091 \
--deploy-config /path/to/qwen3_omni_server_vad.yaml
```
Keep Qwen's default `session_mode: turn`; `duplex_session` enables the Realtime handler without changing scheduler
semantics. Bare `/v1/realtime` selects that handler. The client uses `?duplex=0` only for the existing non-Server-VAD
wire flow; `?duplex=1` remains a supported compatibility alias.
The Python client supports the following command-line arguments:
- `--query-type` (or `-q`): Query type (default: `use_video`). Options: `text`, `use_audio`, `use_image`, `use_video`
examples/online_serving/qwen3_omni/openai_realtime_client.py
+81 -16
View File
@@ -1,9 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Realtime client for vLLM-Omni /v1/realtime (audio + text events).
This client:
1) Reads a local WAV file (must be mono, 16-bit PCM, 16kHz),
2) Streams PCM16 chunks to /v1/realtime with OpenAI-style events,
3) Receives response.audio.* and transcription.* events,
3) Receives response.audio.*, response.output_text.*, and transcript events,
4) Saves synthesized audio to an output WAV file and optional text file.
By default each ``response.audio.delta`` is treated as an **incremental PCM**
@@ -18,6 +21,7 @@ Usage:
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
--input-wav input_16k_mono.wav \
--output-wav realtime_output.wav \
--server-vad \
--delta-dump-dir ./rt_delta_wavs
Dependencies:
@@ -32,6 +36,7 @@ import base64
import json
import wave
from pathlib import Path
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
try:
import websockets
@@ -70,6 +75,14 @@ def _write_wav_pcm16(path: Path, pcm16_bytes: bytes, sample_rate_hz: int) -> Non
wf.writeframes(pcm16_bytes)
def _with_realtime_route(url: str, *, server_vad: bool) -> str:
parts = urlsplit(url)
query = [(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key != "duplex"]
if not server_vad:
query.append(("duplex", "0"))
return urlunsplit(parts._replace(query=urlencode(query)))
async def run_client(
url: str,
model: str,
@@ -79,6 +92,7 @@ async def run_client(
chunk_ms: int,
send_delay_ms: int,
delta_dump_dir: Path | None,
server_vad: bool,
request_idx: int = 1,
total_requests: int = 1,
) -> None:
@@ -92,23 +106,44 @@ async def run_client(
delta_index = 0
text_chunks: list[str] = []
final_text: str = ""
final_audio_transcript: str = ""
if delta_dump_dir is not None:
delta_dump_dir.mkdir(parents=True, exist_ok=True)
# Keep the CLI mode authoritative even if the input URL already contains
# an explicit duplex query parameter.
url = _with_realtime_route(url, server_vad=server_vad)
async with websockets.connect(url, max_size=64 * 1024 * 1024) as ws:
# 1) Validate model.
await ws.send(
json.dumps(
{
"type": "session.update",
if server_vad:
session_update = {
"type": "session.update",
"session": {
"model": model,
}
)
)
"audio": {
"input": {
"format": {"type": "audio/pcm", "rate": 16_000},
"turn_detection": {
"type": "server_vad",
"threshold": 0.5,
"prefix_padding_ms": 300,
"silence_duration_ms": 500,
"create_response": True,
"interrupt_response": False,
},
}
},
},
}
else:
# Preserve the existing non-duplex wire format.
session_update = {"type": "session.update", "model": model}
await ws.send(json.dumps(session_update))
# 2) Start generation once (non-final commit).
await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": False}))
# 2) Preserve the legacy explicit-commit flow unless Server VAD owns endpointing.
if not server_vad:
await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": False}))
# 3) Stream audio chunks.
for i in range(0, len(pcm16), chunk_bytes):
@@ -124,8 +159,20 @@ async def run_client(
if send_delay_ms > 0:
await asyncio.sleep(send_delay_ms / 1000.0)
# 4) Final commit closes input stream.
await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True}))
# 4) Finalize the turn explicitly, or append silence for Server VAD endpointing.
if server_vad:
trailing_silence = bytes(16_000 * 2)
for i in range(0, len(trailing_silence), chunk_bytes):
await ws.send(
json.dumps(
{
"type": "input_audio_buffer.append",
"audio": base64.b64encode(trailing_silence[i : i + chunk_bytes]).decode("utf-8"),
}
)
)
else:
await ws.send(json.dumps({"type": "input_audio_buffer.commit", "final": True}))
# 5) Receive server events until audio done.
while True:
@@ -144,7 +191,7 @@ async def run_client(
sr = event.get("sample_rate_hz")
if isinstance(sr, int) and sr > 0:
output_sample_rate = sr
audio_b64 = event.get("audio", "")
audio_b64 = event.get("delta" if server_vad else "audio", "")
if audio_b64:
pcm_delta = base64.b64decode(audio_b64)
incremental_pcm_parts.append(pcm_delta)
@@ -158,14 +205,14 @@ async def run_client(
)
continue
if event_type == "transcription.delta":
if event_type in {"response.output_text.delta", "transcription.delta"}:
delta = event.get("delta", "")
if delta:
text_chunks.append(delta)
print(delta, end="", flush=True)
continue
if event_type == "transcription.done":
if event_type in {"response.output_text.done", "transcription.done"}:
final_text = event.get("text", "") or "".join(text_chunks)
usage = event.get("usage")
final_text_with_tag = f"Final transcription: {final_text}"
@@ -176,7 +223,16 @@ async def run_client(
print(f"{log_prefix}text usage: {usage}")
continue
if event_type == "response.audio_transcript.done":
final_audio_transcript = event.get("transcript", "")
continue
if event_type == "response.audio.done":
if not server_vad:
break
continue
if event_type == "response.done" and server_vad:
break
if event_type == "error":
@@ -191,7 +247,7 @@ async def run_client(
print(f"{log_prefix}Saved realtime audio to: {output_wav} (incremental chunks joined)")
if output_text is not None:
text_to_save = final_text if final_text else "".join(text_chunks)
text_to_save = final_text or "".join(text_chunks) or final_audio_transcript
output_text.parent.mkdir(parents=True, exist_ok=True)
output_text.write_text(text_to_save, encoding="utf-8")
print(f"{log_prefix}Saved realtime text to: {output_text}")
@@ -213,6 +269,7 @@ async def run_clients_concurrent(
chunk_ms: int,
send_delay_ms: int,
delta_dump_dir: Path | None,
server_vad: bool,
num_requests: int,
concurrency: int,
) -> None:
@@ -235,6 +292,7 @@ async def run_clients_concurrent(
chunk_ms=chunk_ms,
send_delay_ms=send_delay_ms,
delta_dump_dir=per_delta_dir,
server_vad=server_vad,
request_idx=index,
total_requests=num_requests,
)
@@ -283,6 +341,11 @@ def main() -> None:
default=None,
help="If set, each response.audio.delta is saved as delta_NNNNNN.wav under this directory",
)
parser.add_argument(
"--server-vad",
action="store_true",
help="Let the server detect speech endpoints instead of sending input_audio_buffer.commit",
)
parser.add_argument("--num-requests", type=int, default=1, help="Total number of requests to send")
parser.add_argument(
"--concurrency",
@@ -309,6 +372,7 @@ def main() -> None:
chunk_ms=args.chunk_ms,
send_delay_ms=args.send_delay_ms,
delta_dump_dir=args.delta_dump_dir,
server_vad=args.server_vad,
)
)
else:
@@ -322,6 +386,7 @@ def main() -> None:
chunk_ms=args.chunk_ms,
send_delay_ms=args.send_delay_ms,
delta_dump_dir=args.delta_dump_dir,
server_vad=args.server_vad,
num_requests=args.num_requests,
concurrency=concurrency,
)
examples/online_serving/text_to_speech/README.md
+189 -4
View File
@@ -14,6 +14,7 @@ For the full list of supported architectures across all modalities, see
| Model | HuggingFace repo | Voice cloning | Streaming | Voice presets / upload | Gradio demo |
| --- | --- | --- | --- | --- | --- |
| Audio8 TTS Preview | `Audio8/Audio8-TTS-Preview-0.6b` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | uploaded audio voice only; no presets | ✓ |
| Fish Speech S2 Pro | `fishaudio/s2-pro` | ✓ (`ref_audio`+`ref_text`) | ✓ (PCM stream) | — | ✓ |
| GLM-TTS | `zai-org/GLM-TTS` | ✓ (`ref_audio`+`ref_text`, required) | ✓ (PCM stream) | — | ✓ |
| IndexTTS-2 | `IndexTeam/IndexTTS-2` | ✓ (`ref_audio` or uploaded `voice`) | `stream=true` response, non-chunk | uploaded audio voice only; no presets | — |
@@ -253,16 +254,152 @@ python examples/online_serving/text_to_speech/indextts2/speech_client.py \
---
## Fish Speech S2 Pro
## Audio8 TTS Preview
4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec.
0.6B DualAR TTS at 44.1 kHz, 11 languages, zero-shot voice cloning.
### Prerequisites
None beyond the base install: the neural audio codec is implemented in tree and
its weights (`codec.pth`) ship with the checkpoint.
### Launch
```bash
pip install fish-speech
vllm serve Audio8/Audio8-TTS-Preview-0.6b --omni --port 8092
# or:
./audio8_tts/run_server.sh
```
The deploy config auto-loads from `vllm_omni/deploy/audio8_tts.yaml` (HF
`model_type` is `arktts`). Do **not** pass `--trust-remote-code`: vllm-omni
registers its own `arktts` config, and transformers would otherwise prefer the
checkpoint's remote code and bypass it.
### Text-only synthesis
```bash
curl -X POST http://localhost:8092/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Welcome to Audio8 TTS.",
"response_format": "wav"
}' --output output.wav
```
### Voice cloning
```bash
curl -X POST http://localhost:8092/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Welcome to Audio8 TTS.",
"ref_audio": "https://example.com/reference.wav",
"ref_text": "The exact transcript of the reference recording."
}' --output cloned.wav
```
`ref_audio` also accepts a `data:audio/wav;base64,...` URL. `ref_text` is
mandatory whenever `ref_audio` is present and must match the recording.
Uploading a voice via `POST /v1/audio/voices` lets the server reuse the encoded
reference codes across requests (`voice: "<name>"`).
### Streaming
```bash
python audio8_tts/speech_client.py --text "Welcome to Audio8 TTS." --stream --output out.pcm
ffplay -f s16le -ar 44100 -ac 1 out.pcm
```
### Gradio demo
```bash
./audio8_tts/run_gradio_demo.sh # server + demo
python audio8_tts/gradio_demo.py --api-base http://localhost:8092 # demo only
```
### Benchmark
```bash
export VLLM_OMNI_BENCH_AUDIO_SAMPLE_RATE=44100 # required: harness defaults to 24 kHz
vllm bench serve --omni --host 127.0.0.1 --port 8092 \
--model Audio8/Audio8-TTS-Preview-0.6b \
--backend openai-audio-speech --endpoint /v1/audio/speech \
--dataset-name seed-tts-text --dataset-path benchmarks/build_dataset/seed_tts_smoke \
--seed-tts-locale en --num-prompts 20 --num-warmups 2 \
--max-concurrency 1 --request-rate inf \
--percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun
```
Measured on 1x H20 **shared with an unrelated job holding ~80% SM**, so treat
these as a lower bound. 20 prompts from `seed_tts_smoke/en`, mean audio 4.5 s,
deploy defaults:
| | HF reference (batch=1) | vllm-omni c=1 | c=4 | c=8 |
| --- | --- | --- | --- | --- |
| RTF (mean) | 1.008 | **0.19** | 0.30 | 0.54 |
| Throughput (req/s) | 0.23 | 1.18 | 2.84 | **2.93** |
| E2E latency (mean, ms) | 4375 | 850 | 1349 | 2361 |
| Time to first packet (mean, ms) | n/a (no streaming) | 63 | 105 | 1124 |
| Underrun p99 / continuity OK | n/a | 0 s / 100% | 0 s / 100% | 0 s / 100% |
~5.3x lower per-request RTF than the upstream reference implementation at
concurrency 1, ~12.8x aggregate throughput at concurrency 8.
#### Stage-0 `max_num_seqs`: first-packet vs throughput
This subsection is operator tuning guidance, not a claim this PR optimizes
anything: the shipped deploy default is unchanged (`max_num_seqs: 4`). The
numbers below are a single run on the same contended H20 as above (one unrelated
job holding ~80% SM, no repeats), so read them as directional, not as measured
speedups. Same setup, varying stage 0's `max_num_seqs` (stage 1 stays at 1):
| stage-0 `max_num_seqs` | client concurrency | req/s | RTF | TTFP mean / p99 (ms) | underrun p99 (s) |
| --- | --- | --- | --- | --- | --- |
| 4 (default) | 1 | 1.18 | 0.19 | 63 / 67 | 0.00 |
| 4 | 4 | 2.84 | 0.30 | 105 / 148 | 0.00 |
| 4 | 8 | 2.93 | 0.54 | 1124 / 1642 | 0.00 |
| 8 | 1 | 1.16 | 0.19 | 64 / 70 | 0.00 |
| 8 | 4 | 2.88 | 0.30 | 107 / 148 | 0.00 |
| 8 | 8 | **3.72** | 0.40 | **247** / 733 | 0.22 |
| 8 | 16 | 3.91 | 0.65 | 1366 / 2530 | 0.29 |
Reading of this:
- Below the limit (`concurrency <= max_num_seqs`) the setting is irrelevant, as
expected — c=1 and c=4 are identical for both.
- At c=8, raising the limit to 8 admits every request instead of queueing half
of them: in this single run, roughly +27% throughput and ~4.5x lower
first-packet latency.
- Throughput saturates around 3.7-3.9 req/s (~17 s of audio per second). Past
c=8 you only buy queueing delay.
- The cost of admitting more streams is a 0.22-0.29 s worst-case buffer deficit
in 1 of 20 requests (`Streaming continuity OK rate` 95%), because stage 1 runs
`max_num_seqs: 1` and the codec window round-robins across streams. Client-side
prebuffering (the bundled Gradio demo holds 2 chunks) absorbs that; raising
stage 1's `max_num_seqs` is the untested knob if you need both.
The shipped default stays at 4 because every measured point there had zero
underrun. Raise it to 8 if first-packet latency under load matters more than a
rare sub-300 ms gap.
### Notes
- Output: 44.1 kHz mono; ~21.5 codec frames per second.
- No built-in speaker presets. Omit `voice` for a random timbre, or clone one.
- `max_new_tokens` caps generated codec frames (1 frame ~= 46 ms).
- Context is 2048 packed text+audio positions (`max_model_len: 2048`).
- Do not send `temperature: 0`: greedy decoding is degenerate for this
checkpoint and truncates the utterance. The deploy defaults
(0.7 / top_k 50 / top_p 0.9) mirror `generation_config.json`.
---
## Fish Speech S2 Pro
4B dual-AR TTS at 44.1 kHz. Server uses the DAC codec, which is vendored in
vLLM-Omni (no extra packages required).
### Kvcache attention fast path
Fish Speech S2 Pro uses a Triton decode-only kvcache attention fast path by
@@ -425,6 +562,46 @@ python examples/online_serving/text_to_speech/ming_flash_omni_tts/speech_client.
---
## MOSS-TTS Local Transformer v1.5
For a single H200, the optional
[`moss_tts_local_h200.yaml`](../../../vllm_omni/deploy/moss_tts_local_h200.yaml)
deployment places both the talker and codec on logical GPU 0:
```bash
CUDA_VISIBLE_DEVICES=0 VLLM_OMNI_EVENT_DRIVEN_ORCH=1 \
vllm serve OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
--omni --trust-remote-code \
--deploy-config vllm_omni/deploy/moss_tts_local_h200.yaml \
--disable-log-stats
```
Run from the repository root and select an available physical GPU with
`CUDA_VISIBLE_DEVICES`. The command enables event-driven orchestration and
disables detailed per-request statistics logging to reduce CPU overhead at
high concurrency. Remove `--disable-log-stats` when those statistics are needed;
keep these settings identical when comparing performance.
This preset configures 256 request slots per stage,
a 32 GiB talker KV cache, talker CUDA Graph buckets through 512 scheduled tokens,
and codec batch buckets through 256. It requires more than 80 GiB of GPU memory;
request capacity also depends on input and generated lengths. The default
[`moss_tts_local.yaml`](../../../vllm_omni/deploy/moss_tts_local.yaml) remains
available for smaller deployments.
The preset enables the optional codec backend with
`hf_overrides.codec_attention_backend: triton` on stage 1. It preserves the
streaming ring-cache mask and uses BF16 attention with 64-dimensional heads;
other attention shapes use PyTorch SDPA. Set the backend to `sdpa` to use the
default attention implementation. Codec terminal tails share execution only
when they already map to the same padded CUDA Graph; returned audio retains
each request's actual length.
Voice cloning requests use `ref_audio` and `ref_text`. For streaming output,
set `stream: true`, `stream_format: "audio"`, and `response_format: "pcm"`;
the native PCM format is 48 kHz, stereo, signed 16-bit little-endian. Compute
audio throughput as `PCM bytes / (48000 * 2 * 2) / elapsed seconds`, and compare
the same requests, concurrency, and reference-cache state.
## MOSS-TTS-Nano
Single-stage 0.1B AR LM + MOSS-Audio-Tokenizer-Nano codec at 48 kHz mono. Every request must include `ref_audio`; there are no built-in speaker presets.
@@ -668,13 +845,21 @@ Raw PCM streaming requires `stream_format="audio"`, `response_format="pcm"`, and
### Streaming WebSocket
The `/v1/audio/speech/stream` endpoint accepts text incrementally and synthesizes the buffered text as one continuous request on `input.done`:
The `/v1/audio/speech/stream` endpoint accepts text incrementally and, by default, synthesizes the buffered text as one continuous request on `input.done`:
```bash
python qwen3_tts/streaming_speech_client.py --text "Hello world. How are you? I am fine."
python qwen3_tts/streaming_speech_client.py --text "..." --simulate-stt --stt-delay 0.1
```
For per-sentence audio (including Indic danda `।`) before `input.done`:
```bash
python qwen3_tts/streaming_speech_client.py \
--text "नमस्ते। कैसे हो?" \
--split-granularity sentence
```
`input.done` flushes without closing, so repeating `--text` synthesizes several utterances over one connection:
```bash
examples/online_serving/text_to_speech/audio8_tts/gradio_demo.py
+258
View File
@@ -0,0 +1,258 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Gradio demo for Audio8 TTS Preview via the /v1/audio/speech API.
Supports:
- text-to-speech synthesis in 11 languages
- zero-shot voice cloning from reference audio (upload, microphone, or URL)
- streaming (progressive PCM chunks) and non-streaming modes
Usage:
# Start the server first (see run_server.sh), then:
python gradio_demo.py --api-base http://localhost:8092
"""
import argparse
import base64
import io
try:
import gradio as gr
except ImportError:
raise ImportError("gradio is required to run this demo. Install it with: pip install 'vllm-omni[demo]'") from None
import httpx
import numpy as np
import soundfile as sf
#: Audio8 TTS decodes at 44.1 kHz; raw PCM must be interpreted at that rate.
PCM_SAMPLE_RATE = 44100
DEFAULT_API_BASE = "http://localhost:8092"
def encode_audio_to_base64(audio_data: tuple) -> str:
"""Encode a Gradio ``(sample_rate, numpy_array)`` input as a data URL."""
sample_rate, audio_np = audio_data
if audio_np.dtype != np.int16:
if audio_np.dtype in (np.float32, np.float64):
audio_np = (np.clip(audio_np, -1.0, 1.0) * 32767).astype(np.int16)
else:
audio_np = audio_np.astype(np.int16)
buf = io.BytesIO()
sf.write(buf, audio_np, sample_rate, format="WAV")
return f"data:audio/wav;base64,{base64.b64encode(buf.getvalue()).decode('utf-8')}"
def build_payload(
text: str,
ref_audio: tuple | None,
ref_audio_url: str,
ref_text: str,
response_format: str = "wav",
stream: bool = False,
) -> dict:
"""Build the ``/v1/audio/speech`` payload.
Audio8 TTS has no built-in speakers: omit ``voice`` for a random timbre, or
supply ``ref_audio`` + ``ref_text`` to clone a voice.
"""
if not text or not text.strip():
raise gr.Error("Please enter text to synthesize.")
payload: dict = {
"input": text.strip(),
"response_format": "pcm" if stream else response_format,
"stream": stream,
}
if stream:
payload["stream_format"] = "audio"
ref_url = ref_audio_url.strip() if ref_audio_url else ""
if ref_audio is not None:
payload["ref_audio"] = encode_audio_to_base64(ref_audio)
elif ref_url:
payload["ref_audio"] = ref_url
if "ref_audio" in payload:
if not ref_text or not ref_text.strip():
raise gr.Error("Voice cloning requires the transcript of the reference audio.")
payload["ref_text"] = ref_text.strip()
return payload
def generate_speech(api_base: str, text: str, ref_audio, ref_audio_url, ref_text, response_format):
"""Non-streaming request; returns the whole waveform."""
payload = build_payload(text, ref_audio, ref_audio_url, ref_text, response_format, stream=False)
try:
with httpx.Client(timeout=300.0) as client:
resp = client.post(
f"{api_base}/v1/audio/speech",
json=payload,
headers={"Content-Type": "application/json", "Authorization": "Bearer EMPTY"},
)
except httpx.TimeoutException:
raise gr.Error("Request timed out. The server may be busy.") from None
except httpx.ConnectError:
raise gr.Error(f"Cannot connect to server at {api_base}. Is the server running?") from None
if resp.status_code != 200:
raise gr.Error(f"Server error ({resp.status_code}): {resp.text}")
if "application/json" in resp.headers.get("content-type", ""):
raise gr.Error(f"Server error: {resp.text}")
try:
if response_format == "pcm":
audio_np = np.frombuffer(resp.content, dtype=np.int16).astype(np.float32) / 32767.0
return (PCM_SAMPLE_RATE, audio_np)
audio_np, sample_rate = sf.read(io.BytesIO(resp.content))
if audio_np.ndim > 1:
audio_np = audio_np[:, 0]
return (sample_rate, audio_np.astype(np.float32))
except Exception as e:
raise gr.Error(f"Failed to decode audio: {e}") from None
def generate_speech_stream(api_base, text, ref_audio, ref_audio_url, ref_text, response_format, chunk_seconds=0.5):
"""Streaming request; yields PCM chunks for progressive playback."""
payload = build_payload(text, ref_audio, ref_audio_url, ref_text, response_format, stream=True)
min_chunk_samples = int(PCM_SAMPLE_RATE * chunk_seconds)
prebuffer_chunks = 2
pending: list[np.ndarray] = []
pending_len = 0
chunks_yielded = 0
prebuffer: list[np.ndarray] = []
leftover = b""
try:
with (
httpx.Client(timeout=300.0) as client,
client.stream(
"POST",
f"{api_base}/v1/audio/speech",
json=payload,
headers={"Content-Type": "application/json", "Authorization": "Bearer EMPTY"},
) as resp,
):
if resp.status_code != 200:
resp.read()
raise gr.Error(f"Server error ({resp.status_code}): {resp.text}")
for chunk in resp.iter_bytes():
if not chunk:
continue
raw = leftover + chunk
# s16le: never split a sample across chunk boundaries.
usable = len(raw) - (len(raw) % 2)
leftover = raw[usable:]
if usable == 0:
continue
samples = np.frombuffer(raw[:usable], dtype=np.int16).copy().astype(np.float32) / 32767.0
pending.append(samples)
pending_len += len(samples)
if pending_len < min_chunk_samples:
continue
audio_chunk = np.concatenate(pending)
pending.clear()
pending_len = 0
if chunks_yielded < prebuffer_chunks:
prebuffer.append(audio_chunk)
chunks_yielded += 1
if chunks_yielded == prebuffer_chunks:
yield (PCM_SAMPLE_RATE, np.concatenate(prebuffer))
prebuffer.clear()
else:
yield (PCM_SAMPLE_RATE, audio_chunk)
remaining = prebuffer + pending
if remaining:
yield (PCM_SAMPLE_RATE, np.concatenate(remaining))
except httpx.TimeoutException:
raise gr.Error("Request timed out. The server may be busy.") from None
except httpx.ConnectError:
raise gr.Error(f"Cannot connect to server at {api_base}. Is the server running?") from None
def on_stream_change(stream: bool):
"""Streaming only makes sense with raw PCM."""
if stream:
return gr.update(value="pcm", interactive=False)
return gr.update(value="wav", interactive=True)
def build_interface(api_base: str, stream_chunk_seconds: float = 0.5):
with gr.Blocks(title="Audio8 TTS Preview Demo") as demo:
gr.Markdown("# Audio8 TTS Preview 0.6B - Text to Speech")
gr.Markdown(f"**Server:** `{api_base}` | **Model:** Audio8/Audio8-TTS-Preview-0.6b | **Output:** 44.1 kHz")
with gr.Row():
with gr.Column(scale=3):
text_input = gr.Textbox(label="Text to Synthesize", placeholder="Enter text here...", lines=4)
with gr.Accordion("Voice Cloning (optional)", open=False):
gr.Markdown(
"Upload or link a short reference recording (5-30 s) and paste its **exact** transcript. "
"A transcript that does not match the audio degrades both stability and speaker similarity."
)
ref_audio = gr.Audio(label="Reference Audio", type="numpy", sources=["upload", "microphone"])
ref_audio_url = gr.Textbox(
label="Reference Audio URL (alternative to upload)",
placeholder="https://example.com/reference.wav",
lines=1,
)
ref_text = gr.Textbox(
label="Reference Audio Transcript (required for cloning)",
placeholder="Exact transcript of the reference audio...",
lines=2,
)
with gr.Row():
response_format = gr.Dropdown(
choices=["wav", "mp3", "flac", "pcm"], value="wav", label="Audio Format", scale=1
)
stream_checkbox = gr.Checkbox(
label="Stream output", value=False, info="Progressive PCM streaming", scale=1
)
generate_btn = gr.Button("Generate Speech", variant="primary", size="lg")
with gr.Column(scale=2):
audio_output = gr.Audio(label="Generated Audio", interactive=False, streaming=True, autoplay=True)
gr.Markdown(
"### About\n"
"- **Audio8 TTS Preview** — 0.6B DualAR model (slow AR + fast AR codebook decoder)\n"
"- **Languages**: Cantonese, Chinese, Dutch, English, French, German, "
"Italian, Japanese, Korean, Polish, Spanish\n"
"- **Voice cloning**: zero-shot from a reference clip + transcript\n"
"- **44.1 kHz** output, ~21.5 codec frames per second"
)
stream_checkbox.change(fn=on_stream_change, inputs=[stream_checkbox], outputs=[response_format])
all_inputs = [text_input, ref_audio, ref_audio_url, ref_text, response_format]
def dispatch(stream_enabled, *args):
if stream_enabled:
yield from generate_speech_stream(api_base, *args, chunk_seconds=stream_chunk_seconds)
else:
yield generate_speech(api_base, *args)
generate_btn.click(fn=dispatch, inputs=[stream_checkbox] + all_inputs, outputs=[audio_output])
demo.queue()
return demo
def main() -> None:
parser = argparse.ArgumentParser(description="Gradio demo for Audio8 TTS Preview")
parser.add_argument("--api-base", default=DEFAULT_API_BASE, help=f"API base URL (default: {DEFAULT_API_BASE})")
parser.add_argument("--host", default="0.0.0.0", help="Gradio host")
parser.add_argument("--port", type=int, default=7861, help="Gradio port")
parser.add_argument("--share", action="store_true", help="Create a public share link")
parser.add_argument("--stream-chunk-seconds", type=float, default=0.5, help="Seconds of audio per yielded chunk")
args = parser.parse_args()
print(f"Connecting to vLLM server at: {args.api_base}")
build_interface(args.api_base, stream_chunk_seconds=args.stream_chunk_seconds).launch(
server_name=args.host, server_port=args.port, share=args.share
)
if __name__ == "__main__":
main()
examples/online_serving/text_to_speech/audio8_tts/run_gradio_demo.sh
+44
View File
@@ -0,0 +1,44 @@
#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
# Launch the Audio8 TTS Preview server + Gradio demo together.
#
# Usage:
# ./run_gradio_demo.sh
# CUDA_VISIBLE_DEVICES=0 PORT=8092 GRADIO_PORT=7861 ./run_gradio_demo.sh
set -e
MODEL="${MODEL:-Audio8/Audio8-TTS-Preview-0.6b}"
PORT="${PORT:-8092}"
GRADIO_PORT="${GRADIO_PORT:-7861}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
echo "Starting Audio8 TTS Preview server (port $PORT)..."
vllm serve "$MODEL" \
--omni \
--host 0.0.0.0 \
--port "$PORT" &
SERVER_PID=$!
cleanup() {
echo "Stopping server (PID $SERVER_PID)..."
kill $SERVER_PID 2>/dev/null
wait $SERVER_PID 2>/dev/null
}
trap cleanup EXIT
echo "Waiting for server to start..."
for _ in $(seq 1 120); do
if curl -s "http://localhost:$PORT/health" > /dev/null 2>&1; then
echo "Server ready."
break
fi
sleep 2
done
echo "Starting Gradio demo (port $GRADIO_PORT)..."
python "$SCRIPT_DIR/gradio_demo.py" \
--api-base "http://localhost:$PORT" \
--port "$GRADIO_PORT"
examples/online_serving/text_to_speech/audio8_tts/run_server.sh
+24
View File
@@ -0,0 +1,24 @@
#!/bin/bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
# Launch a vLLM-Omni server for Audio8 TTS Preview 0.6B.
#
# Usage:
# ./run_server.sh
# CUDA_VISIBLE_DEVICES=0 PORT=8092 ./run_server.sh
set -e
MODEL="${MODEL:-Audio8/Audio8-TTS-Preview-0.6b}"
PORT="${PORT:-8092}"
echo "Starting Audio8 TTS Preview server with model: $MODEL"
# --trust-remote-code is deliberately NOT passed: vllm-omni registers its own
# `arktts` config, and transformers would otherwise prefer the checkpoint's
# remote code and bypass it.
vllm serve "$MODEL" \
--omni \
--host 0.0.0.0 \
--port "$PORT"
examples/online_serving/text_to_speech/audio8_tts/speech_client.py
+131
View File
@@ -0,0 +1,131 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Client for Audio8 TTS Preview via the /v1/audio/speech endpoint.
Examples:
# Basic TTS
python speech_client.py --text "Welcome to Audio8 TTS."
# Zero-shot voice cloning (the transcript must match the reference audio)
python speech_client.py --text "Welcome to Audio8 TTS." \
--ref-audio ref.wav --ref-text "The exact transcript of the reference recording."
# Streaming PCM output
python speech_client.py --text "Welcome to Audio8 TTS." --stream --output output.pcm
"""
import argparse
import base64
import os
import httpx
DEFAULT_API_BASE = "http://localhost:8092"
DEFAULT_API_KEY = "EMPTY"
DEFAULT_MODEL = "Audio8/Audio8-TTS-Preview-0.6b"
#: The codec runs at 44.1 kHz, so raw PCM must be played back at that rate.
PCM_SAMPLE_RATE = 44100
def encode_audio_to_base64(audio_path: str) -> str:
"""Encode a local audio file as a base64 data URL."""
if not os.path.exists(audio_path):
raise FileNotFoundError(f"Audio file not found: {audio_path}")
ext = audio_path.lower().rsplit(".", 1)[-1]
mime_type = {"wav": "audio/wav", "mp3": "audio/mpeg", "flac": "audio/flac", "ogg": "audio/ogg"}.get(
ext, "audio/wav"
)
with open(audio_path, "rb") as handle:
audio_b64 = base64.b64encode(handle.read()).decode("utf-8")
return f"data:{mime_type};base64,{audio_b64}"
def build_payload(args) -> dict:
payload = {
"model": args.model,
"input": args.text,
"response_format": args.response_format,
}
if args.ref_audio:
if args.ref_audio.startswith(("http://", "https://")):
payload["ref_audio"] = args.ref_audio
else:
payload["ref_audio"] = encode_audio_to_base64(args.ref_audio)
if args.ref_text:
payload["ref_text"] = args.ref_text
if args.max_new_tokens is not None:
payload["max_new_tokens"] = args.max_new_tokens
if args.stream:
payload["stream"] = True
payload["stream_format"] = "audio"
payload["response_format"] = "pcm"
return payload
def run_tts(args) -> None:
payload = build_payload(args)
api_url = f"{args.api_base}/v1/audio/speech"
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {args.api_key}"}
print(f"Model: {args.model}")
print(f"Text: {args.text}")
if args.ref_audio:
print(f"Voice cloning: ref_audio={args.ref_audio!r} ref_text={args.ref_text!r}")
if args.stream:
output_path = args.output or "output.pcm"
with (
httpx.Client(timeout=300.0) as client,
client.stream("POST", api_url, json=payload, headers=headers) as resp,
):
if resp.status_code != 200:
print(f"Error {resp.status_code}: {resp.read().decode()}")
return
total_bytes = 0
with open(output_path, "wb") as handle:
for chunk in resp.iter_bytes():
handle.write(chunk)
total_bytes += len(chunk)
seconds = total_bytes / 2 / PCM_SAMPLE_RATE
print(f"Streamed {total_bytes} bytes (~{seconds:.2f}s of s16le @ {PCM_SAMPLE_RATE} Hz) to {output_path}")
print(f"Play with: ffplay -f s16le -ar {PCM_SAMPLE_RATE} -ac 1 {output_path}")
return
with httpx.Client(timeout=300.0) as client:
response = client.post(api_url, json=payload, headers=headers)
if response.status_code != 200:
print(f"Error {response.status_code}: {response.text}")
return
if response.headers.get("content-type", "").startswith("application/json"):
print(f"Error: {response.text}")
return
output_path = args.output or f"output.{args.response_format}"
with open(output_path, "wb") as handle:
handle.write(response.content)
print(f"Audio saved to: {output_path} ({len(response.content)} bytes)")
def main() -> None:
parser = argparse.ArgumentParser(description="Audio8 TTS Preview client")
parser.add_argument("--api-base", default=DEFAULT_API_BASE, help="API base URL")
parser.add_argument("--api-key", default=DEFAULT_API_KEY, help="API key")
parser.add_argument("--model", "-m", default=DEFAULT_MODEL, help="Model name")
parser.add_argument("--text", required=True, help="Text to synthesize")
parser.add_argument("--ref-audio", default=None, help="Reference audio for voice cloning (path or URL)")
parser.add_argument("--ref-text", default=None, help="Transcript of the reference audio")
parser.add_argument("--max-new-tokens", type=int, default=None, help="Cap the number of generated codec frames")
parser.add_argument("--stream", action="store_true", help="Stream PCM instead of returning a whole file")
parser.add_argument(
"--response-format",
default="wav",
choices=["wav", "mp3", "flac", "pcm", "aac", "opus"],
help="Audio format (default: wav)",
)
parser.add_argument("--output", "-o", default=None, help="Output file path")
run_tts(parser.parse_args())
if __name__ == "__main__":
main()
examples/online_serving/text_to_speech/qwen3_tts/streaming_speech_client.py
+17 -2
View File
@@ -24,6 +24,11 @@ Usage:
--text "Hello world. How are you? I am fine." \
--simulate-stt --stt-delay 0.1
# Opt into per-sentence synthesis (Indic danda, CJK, Latin)
python streaming_speech_client.py \
--text "नमस्ते। कैसे हो?" \
--split-granularity sentence
# Receive JSON sidecar chunks with word-level timestamps
python streaming_speech_client.py \
--text "Hello world. How are you?" \
@@ -73,8 +78,9 @@ except ImportError:
def frame_basename(msg: dict) -> str:
"""Name a file after the utterance and sentence a frame belongs to.
Every utterance is sentence 0, so the utterance index is what keeps the
files of one connection from overwriting each other.
Every utterance uses `sentence_index` for units inside that flush.
With default `split_granularity=none` that index stays 0; sentence mode
uses 0, 1, ... so files from one connection do not overwrite each other.
"""
return f"utterance_{msg['utterance_index']:03d}_sentence_{msg['sentence_index']:03d}"
@@ -368,6 +374,13 @@ def main():
)
parser.add_argument("--speed", type=float, default=1.0, help="Playback speed (0.25-4.0)")
parser.add_argument("--max-new-tokens", type=int, default=None, help="Max tokens")
parser.add_argument("--seed", type=int, default=None, help="Sampling seed forwarded to the server")
parser.add_argument(
"--split-granularity",
default=None,
choices=["none", "sentence", "clause"],
help="Text split mode (default: server none = one request per input.done)",
)
# Base task options
parser.add_argument("--ref-audio", default=None, help="Reference audio")
@@ -409,6 +422,8 @@ def main():
"max_new_tokens",
"ref_audio",
"ref_text",
"seed",
"split_granularity",
]:
val = getattr(args, key.replace("-", "_"), None)
if val is not None:
pyproject.toml
+7 -5
View File
@@ -83,7 +83,6 @@ dev = [
# Provides top-level cosyvoice2; do not co-install stepaudio2-minicpmo,
# whose top-level stepaudio2 package conflicts with this distribution.
"step-audio2==1.0.0", # Step-Audio2 / MiniCPM-o 4.5 Token2Wav (flashcosyvoice / hyperpyyaml)
"silero-vad==6.1", # Realtime server-VAD duplex E2E
]
demo = [
@@ -93,9 +92,6 @@ demo = [
"cn2an>=0.5.22", # Chinese number normalization for TTS (e.g. "1次" -> "一次")
]
server-vad = [
"silero-vad==6.1",
]
forced-aligner = [
"qwen-asr>=0.0.6", # faithful Japanese/Korean word segmentation for the TTS forced aligner
@@ -171,7 +167,7 @@ where = ["."]
include = ["vllm_omni*"]
[tool.setuptools.package-data]
"vllm_omni" = ["_version.py", "py.typed"]
"vllm_omni" = ["_version.py", "py.typed", "transformers_utils/chat_templates/*.jinja"]
[tool.setuptools_scm]
# no extra settings needed, presence enables setuptools-scm
@@ -293,6 +289,8 @@ markers = [
"S5000: [hardware-resource] [musa] Tests that require S5000 GPU (Moore Threads/MUSA)",
"A2: [hardware-resource] [npu] Tests that require A2 NPU",
"A3: [hardware-resource] [npu] Tests that require A3 NPU",
"A5: [hardware-resource] [npu] Tests that require A5 NPU",
"310P: [hardware-resource] [npu] Tests that require 310P NPU",
# Tag with [hardware-cards] for tests.helpers.mark.get_supported_card_counts().
"cards_1: [hardware-cards] Tests that require 1 accelerator card",
"cards_2: [hardware-cards] Tests that require 2 accelerator cards",
@@ -384,3 +382,7 @@ ba = "ba"
SER = "SER"
# NumPy ndarray.flags.writeable (canonical flag name, not "writable").
writeable = "writeable"
# "convnet" abbreviates convolutional network. It appears in the vendored Fish
# Speech DAC codec identifiers (convnet_type, transconvnet_type) and is not a
# typo of "convent".
convnet = "convnet"
recipes/Boogu/Boogu-Image.md
+50 -15
View File
@@ -1,12 +1,13 @@
# Boogu-Image
> Text-to-image and image-editing online serving
(Boogu-Image-0.1-Base / -Edit / -Edit-Turbo)
(Boogu-Image-0.1-Base / -Turbo / -Edit / -Edit-Turbo)
## Summary
- Vendor: Boogu
- Model: `Boogu/Boogu-Image-0.1-Base` (text-to-image),
`Boogu/Boogu-Image-0.1-Turbo` (four-step text-to-image),
`Boogu/Boogu-Image-0.1-Edit` (image editing), and
`Boogu/Boogu-Image-0.1-Edit-Turbo` (four-step image editing)
- Task: Text-to-image generation and text-guided image editing (TI2I)
@@ -16,8 +17,9 @@
## When to use this recipe
Use this recipe when you want a known-good starting point for serving
`Boogu/Boogu-Image-0.1-Base`, `Boogu/Boogu-Image-0.1-Edit`, or
`Boogu/Boogu-Image-0.1-Edit-Turbo` with vLLM-Omni's native pipeline (no
`Boogu/Boogu-Image-0.1-Base`, `Boogu/Boogu-Image-0.1-Turbo`,
`Boogu/Boogu-Image-0.1-Edit`, or `Boogu/Boogu-Image-0.1-Edit-Turbo` with
vLLM-Omni's native pipeline (no
`--diffusion-load-format diffusers`, and the upstream `boogu` package is not
required).
@@ -25,12 +27,14 @@ Boogu-Image-0.1 is an Apache-2.0 unified image generation and editing model
family. The Base text-to-image checkpoint pairs a Qwen3-VL multimodal encoder
with a Diffusion Transformer (DiT) and a flow-match Euler scheduler with
time-shift. It handles photorealistic generation and Chinese/English text
rendering. The Edit and Edit-Turbo checkpoints use the same native
`BooguImagePipeline` for image editing.
rendering. The Edit checkpoint uses the native `BooguImagePipeline`, while the
distilled Turbo and Edit-Turbo checkpoints run the four-step DMD path through
`BooguImageTurboPipeline`.
## References
- Upstream model card: <https://huggingface.co/Boogu/Boogu-Image-0.1-Base>
- Turbo model card: <https://huggingface.co/Boogu/Boogu-Image-0.1-Turbo>
- Edit model card: <https://huggingface.co/Boogu/Boogu-Image-0.1-Edit>
- Edit-Turbo 1K hotfix: <https://huggingface.co/Boogu/Boogu-Image-0.1-Edit-Turbo/tree/hotfix-1k-20260708>
- Project page: <https://boogu.org>
@@ -148,7 +152,7 @@ The following configuration was validated:
- vLLM-Omni version or commit: Use a commit that contains TorchAO FP8
checkpoint loading for diffusion models
- TorchAO version: 0.17.0
- `kernels` / `kernels-data`: 0.15.2 / 0.16.1
- `kernels` / `kernels-data`: 0.16.1 / matching kernels-data
#### Command
@@ -186,6 +190,19 @@ ID passed to `vllm serve`. For example, use
corresponding Edit model.
Base text-to-image:
## Fast text-to-image (Boogu-Image-0.1-Turbo)
Turbo is the distilled text-to-image checkpoint. Its `model_index.json`
declares `BooguImageTurboPipeline`, which runs the four-step DMD path instead
of the scheduler-driven path used by Base.
### Command
```bash
vllm serve Boogu/Boogu-Image-0.1-Turbo --omni --port 8091
```
### Verification
```bash
curl -X POST http://localhost:8091/v1/images/generations \
@@ -230,6 +247,23 @@ curl -X POST http://localhost:8091/v1/images/edits \
checkpoint-declared FP8 configuration. With the regular checkpoints used for
online FP8, the MLLM remains in BF16. The VAE and scheduler are outside this
quantization routing.
"model": "Boogu/Boogu-Image-0.1-Turbo",
"prompt": "A mountain lake at sunset, photorealistic, cinematic lighting",
"size": "1024x1024",
"seed": 42
}' | jq -r '.data[0].b64_json' | base64 -d > output-turbo.png
```
### Notes
- **Required settings:** `guidance_scale=1.0` and `guidance_scale_2=1.0`; any
other value is rejected. `num_inference_steps` defaults to 4. All three are
pipeline defaults, so the request above omits them.
- **Resolution:** upstream constrains Turbo to 1K, so `1024x1024` is the
supported working resolution. The shared pipeline still clamps at 2K; larger
sizes are not validated for this checkpoint.
- **Known limitations:** the same single-GPU limitations as Base apply; CPU
offload, Cache-DiT, and multi-GPU parallelism are not yet validated.
## Image editing (Boogu-Image-0.1-Edit)
@@ -354,16 +388,18 @@ curl -s http://localhost:8091/v1/chat/completions \
## Fast image editing (Boogu-Image-0.1-Edit-Turbo)
Edit-Turbo is the distilled image-editing checkpoint. Its `model_index.json`
declares `BooguImagePipeline`, so it uses the same native TI2I path as Edit.
The upstream 1K hotfix is recommended for stable results and is pinned below
to avoid silently loading the older checkpoint from the repository's default
revision.
declares `BooguImagePipeline`, so `--model-class-name BooguImageTurboPipeline`
is required to reach the four-step DMD path; without it the checkpoint is
served on the dense Edit path. The upstream 1K hotfix is recommended for
stable results and is pinned below to avoid silently loading the older
checkpoint from the repository's default revision.
### Command
```bash
vllm serve Boogu/Boogu-Image-0.1-Edit-Turbo \
--omni \
--model-class-name BooguImageTurboPipeline \
--revision hotfix-1k-20260708 \
--port 8091
```
@@ -407,11 +443,10 @@ curl -s http://localhost:8091/v1/chat/completions \
- **Pinned revision:** use `hotfix-1k-20260708`; upstream recommends the 1K
hotfix over the 1.5K variant for more stable results.
- **Recommended settings:** `num_inference_steps=4` and
`guidance_scale=1.0`. Edit-Turbo is guidance-distilled, meaning the guidance
behavior is already baked into the distilled checkpoint. A scale of `1.0`
avoids applying additional classifier-free guidance; unlike the regular Edit
checkpoint, it should not be increased to `5.0`.
- **Required settings:** `guidance_scale=1.0` and `guidance_scale_2=1.0`; the
Turbo pipeline rejects any other value, so the `guidance_scale_2=2.0` shown
for the regular Edit checkpoint does not apply here. `num_inference_steps`
defaults to 4.
- **Reference images:** the same single-reference and `align_res` behavior as
the regular Edit checkpoint applies.
- **Known limitations:** the same single-GPU limitations as Base and Edit apply;
recipes/JD/JoyAI-VL-Interaction.md
+48
View File
@@ -219,3 +219,51 @@ the audio-track caveat.
5 chunks, `key_frames=0` = summarize all chunk frames) are aligned to the JoyVL
reference adapter so per-tick behavior matches the released model; the framework only
supplies the serving structure.
## Native multi-stage pipeline (experimental)
An experimental native speech path is available alongside the existing Day-0 serving path:
```text
one request with image/video and text
-> JoyAI complete action text
-> Qwen3-TTS Talker
-> Code2Wav
-> 24 kHz audio
```
It runs behind one `vllm serve --omni` endpoint:
```bash
vllm serve jdopensource/JoyAI-VL-Interaction-Preview --omni \
--deploy-config vllm_omni/deploy/joyai_vl_interaction.yaml \
--port 8091
```
With `modalities: ["text", "audio"]`, its behavior is:
| JoyAI action | Native output |
|---|---|
| `</response> <text>` | action text + speech for `<text>` |
| `</silence>` | action text + empty audio output (zero samples) |
| `</response> <note> </delegation> <question>` | action text + speech for `<note>` only |
Native speech supports the Qwen3-TTS `CustomVoice` task. The request-level `voice` and
`language` fields select the speaker and language; when omitted, they default to
`Vivian` and `Auto`. Values supplied as `tts_speaker` and `tts_language` inside
`additional_information` take priority.
This experimental native multi-stage pipeline is request-scoped, stateless, and
all-sync. It currently does not include:
- continuous frame sessions, persistent standing instructions, or session memory;
- audio input / ASR, full-duplex interaction, or barge-in;
- background Agent execution for delegated questions;
- async-chunk support from Talker to Code2Wav;
- integration with the existing WebUI.
For actions with spoken text, the Talker starts only after the complete JoyAI action is
available, and Code2Wav starts only after the Talker completes. Silence actions skip both
TTS stages and return an empty audio output with zero samples. All three stages are placed
on GPU 0 by the provided
[`deploy config`](../../vllm_omni/deploy/joyai_vl_interaction.yaml).
recipes/MiniMaxAI/MiniMax-H3-Spark-GB10.md
+46 -8
View File
@@ -7,13 +7,14 @@ recipes it uses **no offload of any kind** — see the capacity note below.
Validated on:
- Host: DGX Spark (GB10), aarch64
- GPUs: 1 (unified memory)
- Driver: <FILL_DRIVER_VERSION>
- GPUs: 1, or 2 hosts with one unified-memory GPU each
- Driver: 580.173.02 for the two-host qualification
- vLLM: 0.26.0
- vLLM-Omni: `main` at `e1aa6eae75c460cd1893bc320546e81e66973831`
- Workloads: T2VA and Ref2VA, 960x576, `duration=8.0`, 50 steps, `flow_shift=12`,
`seed=1101`
- One-host workloads: T2VA and Ref2VA, 960x576, `duration=8.0`, 50 steps
- Two-host workload: T2VA, 1344x768, `duration=5.0` and `duration=10.0`,
50 steps, text-encoder TP2, and DiT USP2
- Common sampling settings: `flow_shift=12`, `seed=1101`
## Capacity requirements
@@ -24,6 +25,7 @@ Validated on:
| Checkpoint storage | 135 GiB per partition |
| Measured peak allocator high-water, T2VA (FP8) | 97.7 GiB |
| Measured peak allocator high-water, Ref2VA (FP8) | 102.8 GiB |
| Measured peak per rank, two-host T2VA 5 s / 10 s (FP8) | 75.45 / 87.17 GiB |
`FL2VA` and `Ref2VA` are separate 135 GiB checkpoint partitions. Start one server
at a time.
@@ -164,6 +166,41 @@ curl --fail-with-body -sS -X POST http://127.0.0.1:8000/v1/videos/sync \
The response is `200 OK` with an MP4 body: H.264 at the requested geometry plus a
32 kHz stereo AAC track. `ffprobe` reports 8.032 s for `duration=8.0`.
## Two GB10 hosts: T2VA at 1344x768
The two-host qualification uses one GB10 GPU per DGX Spark, connected through a
single ConnectX-7 RoCE interface. The FL2VA checkpoint and software revision are
identical on both hosts. The distributed configuration is:
- world size 2;
- text-encoder tensor parallel size 2;
- DiT Ulysses sequence parallel size 2 and ring degree 1;
- VAE patch parallel size 1;
- online FP8, eager execution, cuDNN attention, and tiled VAE decode.
The current diffusion multiprocess executor starts all ranks on one host and sets
`MASTER_ADDR=localhost`. These measurements therefore used an external `env://`
launcher with one rank on each host and a benchmark-only adaptation that maps
both hosts' local worker to `cuda:0`. This qualifies the two-rank MiniMax-H3
pipeline and its output, but it is not yet a supported cross-host `vllm serve`
command. Do not infer that the single-host launch command above works unchanged
across two machines.
One 5-second, 2-step request warmed both workers before the measured runs. The
reported pipeline elapsed time covers distributed model execution through GPU
completion; MP4 encoding happened afterward and is not included.
| Duration | Steps | Text encode | Denoise | Per step | VAE decode | Pipeline elapsed | Max peak per rank |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 5 s | 50 | 0.21 s | 1570.19 s | 31.40 s | 79.45 s | 1649.87 s (27 min 30 s) | 75.45 GiB |
| 10 s | 50 | 0.15 s | 4448.12 s | 88.96 s | 160.71 s | 4609.00 s (76 min 49 s) | 87.17 GiB |
Both ranks reported stage times and allocator peaks within measurement noise.
The generated outputs were H.264 at 1344x768 and 24 FPS with 32 kHz stereo AAC:
124 frames (5.167 s) and 243 frames (10.125 s). Both files decoded successfully,
and neither run encountered an OOM or NCCL error. These are single qualification
runs (`n=1`), not throughput guarantees.
## Measured latency
Single GB10, FP8, eager, cuDNN attention, tiled VAE, one request at a time:
@@ -238,9 +275,11 @@ shorter than a 50-step Ref2VA run.
## Known limitations
- One GPU means no Ulysses, ring, TP, or VAE patch parallelism. Pass
- With one GPU there is no Ulysses, ring, TP, or VAE patch parallelism. Pass
`--usp 1 --ring 1 --vae-patch-parallel-size 1` explicitly if a profile or
script would otherwise supply higher degrees.
- The two-host results require the external launch adaptation described above;
cross-host diffusion worker launch is not exposed by the documented server CLI.
- `--enforce-eager` is used because regional compile adds startup time without a
parallelism win on a single rank.
- `t2va` rejects requests without an explicit `aspect_ratio`. When `width` and
@@ -255,8 +294,7 @@ shorter than a 50-step Ref2VA run.
adding references or raising the output shape.
- Ref2VA requires a vLLM-Omni build newer than `v0.26.0`. See the version note
under **Validated on**: FP8 weight loading and image-only Ref2VA are both
broken on `release/v0.26.0`.suggestion is `v0.26.1`.
broken on `release/v0.26.0`. The suggested version is `v0.26.1`.
- Online FP8 is incompatible with layerwise offload — the offload path produces a
weight stride the Cutlass FP8 kernel rejects. This is not a practical
restriction here since offload is unusable on GB10 anyway.
```
recipes/OpenBMB/MiniCPM-o-4_5.md
+17 -4
View File
@@ -28,13 +28,13 @@ also provided.
- Default deploy configs (auto-loaded by HF `model_type=minicpmo` +
`hf_config.version="4.5"`):
- Default single-GPU compatibility layout (auto-loaded):
- Default single-GPU compatibility layout (auto-loaded):
[`vllm_omni/deploy/minicpmo_4_5.yaml`](../../vllm_omni/deploy/minicpmo_4_5.yaml)
- Recommended 2-GPU continuous-batching layout:
- Recommended 2-GPU continuous-batching layout:
[`vllm_omni/deploy/minicpmo_4_5_2gpu.yaml`](../../vllm_omni/deploy/minicpmo_4_5_2gpu.yaml),
- 3-GPU layout:
- 3-GPU layout:
[`vllm_omni/deploy/minicpmo_4_5_3gpu.yaml`](../../vllm_omni/deploy/minicpmo_4_5_3gpu.yaml)
- 8x RTX 4090 layout:
- 8x RTX 4090 layout:
[`vllm_omni/deploy/minicpmo_4_5_8x4090.yaml`](../../vllm_omni/deploy/minicpmo_4_5_8x4090.yaml)
- Online example + Gradio demo:
[`examples/online_serving/minicpmo/`](../../examples/online_serving/minicpmo/)
@@ -126,9 +126,20 @@ video. Use the two-GPU profile for production throughput.
```bash
vllm serve openbmb/MiniCPM-o-4_5 --omni \
--trust-remote-code \
--chat-template vllm_omni/transformers_utils/chat_templates/minicpmo45_native.jinja \
--chat-template-content-format openai \
--host 0.0.0.0 --port 8099
```
Run these commands from the repository root. The bundled chat template
matches the Hugging Face `chat(omni_mode=True)` content layout: media and text
parts are concatenated without adding separators, while whitespace explicitly
included in a text part is preserved. This matters for speech generation because
an inserted newline changes the Thinker condition passed to Talker. For image
questions, place the image before the question, as in the native image-chat
layout. Continue to request `use_tts_template: true` and
`enable_thinking: false` for spoken answers.
The deploy config is auto-loaded by the model registry — no
`--deploy-config` flag needed for this default single-GPU layout.
@@ -271,6 +282,8 @@ uses GPU 5. GPUs 6–7 are left free.
vllm serve openbmb/MiniCPM-o-4_5 --omni \
--deploy-config vllm_omni/deploy/minicpmo_4_5_8x4090.yaml \
--trust-remote-code \
--chat-template vllm_omni/transformers_utils/chat_templates/minicpmo45_native.jinja \
--chat-template-content-format openai \
--host 0.0.0.0 --port 8099
```
recipes/OpenMOSS/MOSS-TTS.md
+10 -2
View File
@@ -16,7 +16,7 @@ Use this recipe for 24 kHz multilingual TTS with voice cloning (20 languages inc
Chinese and English). Choose a variant based on your latency and quality requirements:
| Model | Params | Use case |
|---|---|---|
| --- | --- | --- |
| MOSS-TTS | 8B | General TTS, highest quality |
| MOSS-TTS-v1.5 | 8B | General TTS upgrade of 1.0: 31 languages, steadier cloning, `[pause Xs]` markers (set `language` for best results); same `MossTTSDelay` API |
| MOSS-TTS-Realtime | 1.7B | Lowest latency (TTFB ~180 ms), streaming-first |
@@ -24,9 +24,17 @@ Chinese and English). Choose a variant based on your latency and quality require
| MOSS-SoundEffect | 8B | Sound effect synthesis from text description |
| MOSS-VoiceGenerator | 1.7B | Zero-shot voice design |
All variants share the same codec (`OpenMOSS-Team/MOSS-Audio-Tokenizer`, ~7 GB) and
The variants above share the same codec (`OpenMOSS-Team/MOSS-Audio-Tokenizer`, ~7 GB) and
output 24 kHz mono audio.
MOSS-TTS-Local-Transformer-v1.5 uses MOSS-Audio-Tokenizer-v2 and outputs 48 kHz
stereo audio. For Local voice cloning through `/v1/audio/speech`, provide an
accurate `ref_text` transcript alongside `ref_audio`. The adapter uses the
reference transcript and audio as a continuation prefix, then generates only
the requested `input` speech. Without a nonblank `ref_text`, Local uses
audio-reference generation. Reference transcripts must match the reference
audio; they are not style instructions.
## References
- Offline inference example: [`examples/offline_inference/text_to_speech/moss_tts/`](../../examples/offline_inference/text_to_speech/moss_tts/)
recipes/Qwen/Qwen3-TTS.md
+88
View File
@@ -263,6 +263,94 @@ curl -X POST http://localhost:8091/v1/audio/speech \
gpu_memory_utilization: 0.15
```
### 1x A100 40GB (0.6B CustomVoice)
#### Environment
- OS: Linux 4.18.0-553 (RHEL 8.10), x86_64
- Python: 3.12.14
- PyTorch: 2.13.0+cu129
- Driver / runtime: NVIDIA 570.124.06 / CUDA 12.8 (`nvidia-smi`); PyTorch CUDA 12.9
- GPU: NVIDIA A100-SXM4-40GB, 40960 MiB
- vLLM version: 0.28.0+cu129
- vLLM-Omni version or commit: `3204a0b2ade0f05424b7f11e3bcc03cb3542e0c6`
#### Command
```bash
vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice \
--deploy-config vllm_omni/deploy/qwen3_tts.yaml \
--omni --port 8091
```
The default deploy config (`qwen3_tts.yaml`) works without modification on this
A100 40GB. Both stages (talker + code2wav) share GPU 0 with
`gpu_memory_utilization: 0.3` each.
#### Verification
**English synthesis:**
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "Hello, this is Qwen3-TTS running on A100 40GB.",
"voice": "vivian",
"language": "English"
}' --output test_english.wav
```
**Chinese synthesis:**
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "你好，这是在A100 40GB上运行的语音合成测试。",
"voice": "vivian",
"language": "Chinese"
}' --output test_chinese.wav
```
**With emotion instruction:**
```bash
curl -X POST http://localhost:8091/v1/audio/speech \
-H "Content-Type: application/json" \
-d '{
"input": "I am so excited about this!",
"voice": "vivian",
"language": "English",
"instructions": "Speak with great enthusiasm"
}' --output test_emotion.wav
```
#### Notes
- Memory usage: **16283 MiB / 40960 MiB** peak (`nvidia-smi`) during the three
verification requests with the default deploy config
(`gpu_memory_utilization: 0.3` per stage). Stage 0 used 12404 MiB and stage 1
used 3862 MiB. The 0.6B weights occupy only ~2.4 GiB (Stage 0: 1.91 GiB,
Stage 1: 0.45 GiB); the rest is KV cache reserved as a fraction of device
memory, so the same `0.3` setting reads higher in GiB on 40GB than the
~13.5 GiB idle figure on the 24GB 4090 profile above. To shrink the
reservation, use the `gpu_memory_utilization: 0.15` override documented in
that 4090 section.
- vLLM 0.28.0's default wheel targets CUDA 13.0. Driver 570 cannot load that
variant. This qualification used the official CUDA 12.9 wheel (same pin as
the A100 Wan2.2 recipe):
```bash
uv pip install \
'https://github.com/vllm-project/vllm/releases/download/v0.28.0/vllm-0.28.0%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl' \
--torch-backend=cu129
```
Confirm `torch.__version__` contains `cu129`. If a CPU build is already
installed, `uv pip install --torch-backend=cu129` is a no-op; use
`--reinstall`.
### 1x AMD MI300X, 1.7B checkpoints
#### Environment
recipes/README.md
+2
View File
@@ -42,6 +42,7 @@ recipes/
| -------- | ------ | ---------- |
| [`Baidu/ERNIE-Image.md`](./Baidu/ERNIE-Image.md) | Text-to-image online serving (ERNIE-Image 8B) | 1x or 2x RTX 4090 24GB |
| [`Bagel/BAGEL-7B-MoT.md`](./Bagel/BAGEL-7B-MoT.md) | Text-to-image with shared online/offline examples | 1x A100 80GB / 2x CUDA GPUs |
| [`black-forest-labs/FLUX.1-schnell.md`](./black-forest-labs/FLUX.1-schnell.md) | Offline text-to-image generation with CPU offload | 1x NVIDIA GeForce RTX 5090 32GB |
| [`Boogu/Boogu-Image.md`](./Boogu/Boogu-Image.md) | Text-to-image online serving (Boogu-Image-0.1-Base) | 1x A100/H100 40GB+ |
| [`BosonAI/Higgs-Audio-V3-TTS.md`](./BosonAI/Higgs-Audio-V3-TTS.md) | Online + offline multilingual TTS with voice cloning | 1x H100 80GB |
| [`ByteDance/Lance.md`](./ByteDance/Lance.md) | Unified AR+diffusion: text/img/video gen + understanding (Lance 3B) | 1x B300 / A100 80GB |
@@ -54,6 +55,7 @@ recipes/
| [`IndexTeam/IndexTTS-2.md`](./IndexTeam/IndexTTS-2.md) | Online serving for voice-cloned TTS with optional emotion control | 1x L4 24GB or larger CUDA GPU |
| [`IndexTeam/IndexTTS-2_5.md`](./IndexTeam/IndexTTS-2_5.md) | Online + offline multilingual voice-cloned TTS with native speed and emotion control | 1x NVIDIA H20 96GB |
| [`MiniMaxAI/MiniMax-H3.md`](./MiniMaxAI/MiniMax-H3.md) | T2VA, FL2VA, image+audio Ref2VA, and multi-video Ref2VA serving | 1x GPU with CPU offload / 4x B300 |
| [`MiniMaxAI/MiniMax-H3-Spark-GB10.md`](./MiniMaxAI/MiniMax-H3-Spark-GB10.md) | MiniMax H3 T2VA and Ref2VA on DGX Spark | 1x or 2x DGX Spark (GB10) |
| [`MiniMaxAI/MiniMax-H3-MUSA.md`](./MiniMaxAI/MiniMax-H3-MUSA.md) | MiniMax H3 T2VA, FL2VA, and Ref2VA on MUSA | 4x MTT S5000 (TP4 Ref2VA profile) |
| [`krea/Krea-2.md`](./krea/Krea-2.md) | Text-to-image (Turbo + Raw), offline + online, with LoRA | 1x H100 80GB |
| [`LTX/LTX-2.md`](./LTX/LTX-2.md) | LTX-2/LTX-2.3 text-to-video and image-to-video with synchronized audio | H200 141GB / 96GB-class GPU |
recipes/black-forest-labs/FLUX.1-schnell.md
+160
View File
@@ -0,0 +1,160 @@
# FLUX.1-schnell
> Offline text-to-image generation on one NVIDIA GeForce RTX 5090 32 GB
## Summary
- Vendor: Black Forest Labs
- Model: `black-forest-labs/FLUX.1-schnell`
- Task: Text-to-image generation
- Mode: Offline inference with the shared text-to-image runner
- Maintainer: Community
## When to use this recipe
Use this recipe as a personally validated starting point for generating a
1024x1024 image with FLUX.1-schnell on one NVIDIA GeForce RTX 5090. The tested
configuration uses BF16, four denoising steps and model-level CPU offload.
This recipe covers single-image generation. It does not attempt to optimize
throughput with quantization or cache acceleration.
## References
- Model card: <https://huggingface.co/black-forest-labs/FLUX.1-schnell>
- Canonical offline text-to-image guide:
[`docs/user_guide/examples/offline_inference/text_to_image.md`](../../docs/user_guide/examples/offline_inference/text_to_image.md)
- Shared runnable example:
[`examples/offline_inference/text_to_image`](../../examples/offline_inference/text_to_image)
- CPU offload guide:
[`docs/user_guide/diffusion/cpu_offload.md`](../../docs/user_guide/diffusion/cpu_offload.md)
- Community recipe tracker:
[#2645](https://github.com/vllm-project/vllm-omni/issues/2645)
## Hardware Support
## GPU
### 1x NVIDIA GeForce RTX 5090 32 GB
This configuration was personally validated end to end on September 9, 2026.
#### Environment
- OS: Ubuntu 22.04.3 x86_64
- Python: 3.12.7
- GPU: 1x NVIDIA GeForce RTX 5090, 32,607 MiB reported by `nvidia-smi`
- Host RAM: approximately 62 GiB (64 GB class)
- NVIDIA driver: 595.71.05; CUDA 13.2 reported by `nvidia-smi`
- PyTorch: 2.13.0+cu130; PyTorch CUDA runtime: 13.0
- vLLM: 0.28.0
- vLLM-Omni: `0.28.1.dev112+ga31690041`
- Diffusers: 0.40.0
- Transformers: 5.14.1
#### Command
Run the shared example from the vLLM-Omni repository root in an environment
with vLLM-Omni installed:
```bash
python examples/offline_inference/text_to_image/text_to_image.py \
--model black-forest-labs/FLUX.1-schnell \
--prompt 'A red fox standing in fresh snow, cinematic photography' \
--height 1024 --width 1024 \
--num-inference-steps 4 --guidance-scale 0 --seed 42 \
--enable-cpu-offload \
--output flux-schnell-rtx5090.png
```
The command uses the public model ID for portability. The measured runs used
already downloaded FLUX.1-schnell weights; initial Hub download and
authentication were not part of the inference validation.
#### Verification
Verify that the runner succeeded and inspect the saved image:
```bash
python - <<'PY'
import hashlib
from pathlib import Path
from PIL import Image, ImageStat
path = Path("flux-schnell-rtx5090.png")
with Image.open(path) as image:
image.load()
assert image.format == "PNG"
assert image.size == (1024, 1024)
rgb = image.convert("RGB")
stats = ImageStat.Stat(rgb)
assert max(stats.stddev) > 1, "Image is nearly uniform"
print(f"format={image.format}")
print(f"size={image.width}x{image.height}")
print(f"mode={image.mode}")
print(f"bytes={path.stat().st_size}")
print("rgb_mean=" + ",".join(f"{value:.3f}" for value in stats.mean))
print(f"rgb_extrema={rgb.getextrema()}")
print(f"sha256={hashlib.sha256(path.read_bytes()).hexdigest()}")
PY
```
The repeated experiment completed 36 image requests (6 additional warmups and
30 measured outputs) with successful process exits. Every PNG passed decoding,
1024x1024 dimension and non-uniform pixel checks. SHA-256 hashes were recorded
for the artifacts; matching those hashes is not required across environments.
These checks establish file integrity, not a quantitative image-quality score.
#### Performance
The experiment used three independent processes. Each process initialized the
model once, generated two warmup images, then generated ten measured images
sequentially. All 30 measured images used BF16, CPU offload, 1024x1024, four
steps, guidance scale 0 and TP=1, without concurrent requests.
| Generation measurement | Result |
| --- | ---: |
| Measured images | 30 |
| Mean latency | 25.8411 seconds |
| P95 latency (nearest-rank) | 26.1446 seconds |
| Startup / resource measurement | Maximum |
| --- | ---: |
| Model initialization (seconds) | 64.73 |
| Full-lifetime RAM PSS peak (MiB) | 49306 |
| Full-lifetime device GPU memory peak (MiB) | 28121 |
| Measured PyTorch reserved GPU memory peak (MiB) | 27360 |
Latency measures only `Omni.generate()`, excluding image saving and
verification. The two warmup images per process are excluded. P95 uses the
nearest-rank method over the 30 measured images. Model initialization includes
the engine's built-in dummy warmup; weights were already downloaded and OS file
caches were not cleared.
RAM and whole-device GPU memory were sampled throughout each process lifetime
at a target interval of 200 ms. RAM is the maximum simultaneous sum of PSS for
the inference process and its workers, read from Linux `smaps_rollup`.
Whole-device GPU memory is the maximum reported by `nvidia-smi`. Both are
sampled peaks and may miss shorter spikes.
The PyTorch value is the maximum worker-reported `peak_memory_mb` across the
30 measured requests. Despite the field name, its unit is MiB; it measures the
PyTorch allocator's reserved-memory high-water mark rather than whole-device
usage.
#### Notes
- `--enable-cpu-offload` swaps the transformer and text encoders between host
and device memory at phase boundaries. It reduces GPU residency but adds
transfer overhead and requires sufficient host RAM. The minimum host RAM
requirement was not measured.
- CPU offload does not imply CPU-only computation. The tested model-level
offload backend keeps the VAE on the GPU.
- Only offline single-image generation on one GPU with BF16, 1024x1024,
four steps and CPU offload was validated. Online serving, FP8, other
resolutions, concurrency, execution without offload and smaller host RAM
configurations are outside this recipe's validated scope.
- The upstream revision of the downloaded weights was not recorded, so the
measurements do not establish exact checkpoint reproducibility.
recipes/fishaudio/Fish-Speech-S2-Pro.md
+3 -10
View File
@@ -36,19 +36,13 @@ Other hardware and configurations are welcome as community validation lands.
- OS: Linux
- Python: 3.10+
- Required package: `fish-speech` (for DAC codec)
- CUDA 12.8
- vLLM version: 0.19.0
- vLLM-Omni version or commit: c93359bb354a6aa5c14d062430cb85b2c4db251e
```bash
# Install PortAudio dependency if missing (required by pyaudio which is a dependency of fish-speech)
# For Ubuntu/Debian:
if ! dpkg -l | grep -q libportaudio2; then
sudo apt-get update && sudo apt-get install -y libportaudio2 portaudio19-dev
fi
pip install fish-speech
```
No extra packages are required: the DAC codec architecture is vendored in
vLLM-Omni (`vllm_omni/model_executor/models/fish_speech/dac_modules/`), so the
model runs out of the box, including in the official Docker image.
#### Kvcache attention fast path
@@ -129,7 +123,6 @@ curl -X POST http://localhost:8091/v1/audio/speech \
- OS: Linux
- Python: 3.10+
- Required package: `fish-speech` (for DAC codec)
- CUDA 12.8
- vLLM version: 0.19.0+
requirements/common.txt
+3
View File
@@ -1,3 +1,6 @@
# Common dependencies for all platforms
# Split (edge WP6): core.txt = AR/speech runtime, media.txt = diffusion/video/extra
# audio. The v0.29.0rc1 pin changes (transformers >= 5.13.0, kernels==0.16.1) are
# applied inside those two files.
# -r core.txt
-r media.txt
requirements/core.txt
+6 -1
View File
@@ -1,6 +1,11 @@
# Core runtime deps for AR / generation (TTS, speech) pipelines. Media /
# diffusion / video deps live in media.txt; common.txt pulls both.
transformers >= 5.10.1, < 5.15
# transformers: required unified multimodal model classes are unavailable
# before 5.10.1; 5.10.0 is yanked, and 5.15 has a known construction
# regression. Floor is 5.13.0 so transformers.integrations.hub_kernels
# LayerRepository entries always pass version=/revision= required by
# kernels>=0.16 (#6971). Keep in sync with kernels in media.txt.
transformers >= 5.13.0, < 5.15
av>=14.0.0
omegaconf>=2.3.0
safetensors>=0.8.0
requirements/media.txt
+3 -1
View File
@@ -1,6 +1,8 @@
# Diffusion / image / video / extra-audio deps (not needed by AR-only edge builds).
diffusers==0.40.0
kernels==0.15.2
# Hub kernels, including LTX-2.5 NATTEN. 0.16+ resolves torch-stable-abi*
# FA3 builds on torch 2.13 / cu130; keep in sync with transformers in core.txt.
kernels==0.16.1
cache-dit==1.5.0
torchsde>=0.2.6
openai-whisper>=20250625
tests/attention/test_fish_kvcache_attn.py
+13 -4
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
import torch
@@ -214,11 +214,20 @@ def test_fish_kvcache_backend_wraps_only_model_instance(monkeypatch):
assert fish_kvcache_backend.get_fish_kvcache_attn_stats()["small_hit_count"] == 1
def _vllm_kv_cache_shape(num_blocks, block_size, num_kv_heads, head_size):
def _vllm_kv_cache_shape(num_blocks, block_size, num_kv_heads, head_size, dtype=torch.float16):
# Source of truth for the paged KV cache layout the backend must unpack.
from vllm.v1.attention.backends.flash_attn import FlashAttentionBackend
# vLLM 0.29 removed AttentionBackend.get_kv_cache_shape; the per-layer shape
# now comes from the spec, with the content dim expressed in bytes.
from vllm.v1.kv_cache_interface import FullAttentionSpec, compute_layer_kv_cache_shape_bytes
return FlashAttentionBackend.get_kv_cache_shape(num_blocks, block_size, num_kv_heads, head_size)
spec = FullAttentionSpec(
block_size=block_size,
num_kv_heads=num_kv_heads,
head_size=head_size,
dtype=dtype,
)
num_pages, num_heads, num_states, content_bytes = compute_layer_kv_cache_shape_bytes(spec, num_blocks)
return (num_pages, num_heads, num_states, content_bytes // dtype.itemsize)
def test_fish_kvcache_backend_unbinds_kv_on_vllm_cache_layout(monkeypatch):
tests/benchmarks/metrics/test_metrics.py
+131
View File
@@ -457,5 +457,136 @@ def test_text_benchmark_still_reports_ttft(capsys):
assert "Time to First Token" in out, "text benchmarks must keep TTFT"
# ============================================================================
# Suppress text metrics for pure image / video endpoints
# ============================================================================
def _make_image_output(prompt_len: int) -> MixRequestFuncOutput:
output = MixRequestFuncOutput()
output.success = True
output.prompt_len = prompt_len
output.output_tokens = 0
output.generated_text = ""
output.ttft = 0.0
output.text_latency = 0.0
output.latency = 2.5
output.start_time = 0.0
output.itl = []
output.image_count = 1
output.image_generation_time_ms = 2000.0
output.image_pixels = 1024 * 1024
output.input_audio_duration = 0.0
output.error = ""
return output
def _make_video_output(prompt_len: int) -> MixRequestFuncOutput:
output = MixRequestFuncOutput()
output.success = True
output.prompt_len = prompt_len
output.output_tokens = 0
output.generated_text = ""
output.ttft = 0.0
output.text_latency = 0.0
output.latency = 5.0
output.start_time = 0.0
output.itl = []
output.video_duration = 2.0
output.video_frames = 48
output.video_generation_time_ms = 4000.0
output.video_rtf = 2.0
output.input_audio_duration = 0.0
output.error = ""
return output
_MEDIA_PERCENTILE_METRICS = ["e2el", "ttft"]
def test_pure_image_benchmark_omits_text_result(capsys):
"""Pure image generation must not print Text Result or invent peak tok/s."""
outputs = [_make_image_output(64), _make_image_output(64)]
metrics, actual_output_lens = calculate_metrics(
input_requests=[],
outputs=outputs,
dur_s=5.0,
tokenizer=None,
selected_percentiles=[99.0],
goodput_config_dict={},
task_type=TaskType.GENERATION,
selected_percentile_metrics=_MEDIA_PERCENTILE_METRICS,
max_concurrency=None,
request_rate=float("inf"),
benchmark_duration=5.0,
)
out = capsys.readouterr().out
assert actual_output_lens == [0, 0]
assert metrics.total_output == 0
assert metrics.max_output_tokens_per_s == 0.0
assert " Text Result " not in out
assert "Peak output token throughput" not in out
assert "Time to First Token" not in out
assert " Image Result " in out
def test_pure_video_benchmark_omits_text_result(capsys):
"""Pure video generation must not print Text Result or invent peak tok/s."""
outputs = [_make_video_output(63), _make_video_output(63)]
metrics, actual_output_lens = calculate_metrics(
input_requests=[],
outputs=outputs,
dur_s=10.0,
tokenizer=None,
selected_percentiles=[99.0],
goodput_config_dict={},
task_type=TaskType.GENERATION,
selected_percentile_metrics=_MEDIA_PERCENTILE_METRICS,
max_concurrency=None,
request_rate=float("inf"),
benchmark_duration=10.0,
)
out = capsys.readouterr().out
assert actual_output_lens == [0, 0]
assert metrics.total_output == 0
assert metrics.max_output_tokens_per_s == 0.0
assert " Text Result " not in out
assert "Peak output token throughput" not in out
assert "Time to First Token" not in out
assert " Video Result " in out
def test_image_with_generated_text_still_reports_text_result(capsys):
"""Image edits / AR text alongside images must keep Text Result."""
output = _make_image_output(64)
output.output_tokens = 8
output.generated_text = "revised caption"
output.ttft = 0.05
output.itl = [0.01] * 7
calculate_metrics(
input_requests=[],
outputs=[output],
dur_s=3.0,
tokenizer=_EmptyAwareTokenizer(),
selected_percentiles=[99.0],
goodput_config_dict={},
task_type=TaskType.GENERATION,
selected_percentile_metrics=_MEDIA_PERCENTILE_METRICS,
max_concurrency=None,
request_rate=float("inf"),
benchmark_duration=3.0,
)
out = capsys.readouterr().out
assert " Text Result " in out
assert "Time to First Token" in out
assert " Image Result " in out
if __name__ == "__main__":
pytest.main([__file__, "-v", "-s"])
tests/benchmarks/patch/test_patch.py
+339
View File
@@ -22,9 +22,16 @@ from vllm_omni.benchmarks.data_modules.seed_tts_dataset import (
)
from vllm_omni.benchmarks.patch.patch import (
MixRequestFuncOutput,
_add_video_extra_body_to_form,
_add_video_reference_to_form,
_apply_stage0_token_timings,
_apply_video_metrics_from_payload,
_attach_seed_tts_to_request_func_input,
_build_benchmark_session,
_omni_request_timeout_s,
async_request_openai_chat_omni_completions,
async_request_openai_image_edits_omni,
async_request_openai_image_generations_omni,
async_request_openai_realtime_duplex,
should_request_stage_metrics,
)
@@ -1196,5 +1203,337 @@ async def test_prompt_len_assigned_from_usage(mocker: MockerFixture):
)
class TestOmniRequestTimeout:
"""``--omni-request-timeout-s`` precedence: explicit value > 900 s default."""
_OVERRIDE = "vllm_omni.benchmarks.patch.patch._REQUEST_TIMEOUT_OVERRIDE_S"
def test_default_timeout_is_900s_when_not_set(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(self._OVERRIDE, None)
assert _omni_request_timeout_s() == 900.0
def test_explicit_value_wins_over_default(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(self._OVERRIDE, 123.5)
assert _omni_request_timeout_s() == 123.5
def test_non_positive_restores_legacy_6h_cap(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(self._OVERRIDE, 0)
assert _omni_request_timeout_s() == 6 * 60 * 60.0
async def test_benchmark_session_uses_configured_timeout(self, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(self._OVERRIDE, 42.0)
session = _build_benchmark_session(max_concurrency=8, ssl_setting=False)
try:
assert session.timeout.total == 42.0
assert session.connector.limit == 8
assert session.connector.limit_per_host == 8
finally:
await session.close()
async def test_hung_server_request_times_out_as_failed(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""A server that accepts the request but never responds must surface as ``failed``."""
async def handler(reader, writer):
# Drain the request but never respond — the original hang failure mode.
try:
while await reader.read(4096):
pass
except ConnectionResetError:
pass
finally:
# Close our side of the socket: the client aborts after the
# timeout and only sends FIN, so without writer.close() the
# half-closed connection keeps Server.wait_closed() (which on
# Python 3.12+ waits for every accepted connection to drop)
# blocked forever.
writer.close()
try:
await writer.wait_closed()
except (ConnectionResetError, BrokenPipeError):
pass
monkeypatch.setattr(self._OVERRIDE, 1.0)
server = await asyncio.start_server(handler, "127.0.0.1", 0)
port = server.sockets[0].getsockname()[1]
session = _build_benchmark_session(max_concurrency=1, ssl_setting=False)
try:
request_input = _seed_tts_request_func_input()
request_input.api_url = f"http://127.0.0.1:{port}/v1/chat/completions"
output = await async_request_openai_chat_omni_completions(request_input, session)
assert output.success is False
assert output.error
finally:
await session.close()
server.close()
await server.wait_closed()
def test_video_rtf_prefers_generation_time_over_poll_latency():
"""RTF must not grow with client poll_interval overshoot in output.latency."""
payload = {
"duration_s": 2.0,
"num_frames": 48,
"fps": 24.0,
"stage_durations": {"stage_0_gen_ms": 4000.0},
}
request_body: dict[str, object] = {}
short_poll = MixRequestFuncOutput()
short_poll.latency = 4.2 # ~generation + small poll overshoot
_apply_video_metrics_from_payload(short_poll, payload, request_body)
long_poll = MixRequestFuncOutput()
long_poll.latency = 8.0 # same job, larger poll_interval_s overshoot
_apply_video_metrics_from_payload(long_poll, payload, request_body)
assert short_poll.video_generation_time_ms == pytest.approx(4000.0)
assert long_poll.video_generation_time_ms == pytest.approx(4000.0)
assert short_poll.video_rtf == pytest.approx(2.0)
assert long_poll.video_rtf == pytest.approx(2.0)
assert short_poll.video_rtf == long_poll.video_rtf
def test_video_rtf_falls_back_to_e2e_latency_without_generation_time():
output = MixRequestFuncOutput()
output.latency = 6.0
_apply_video_metrics_from_payload(
output,
{"duration_s": 2.0, "num_frames": 48, "fps": 24.0},
{},
)
assert output.video_generation_time_ms == 0.0
assert output.video_rtf == pytest.approx(3.0)
# 1x1 PNG used as a valid b64_json image payload in edit-client tests.
_MIN_PNG_B64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
@pytest.mark.asyncio
async def test_image_edits_defaults_to_non_streaming_json(mocker: MockerFixture) -> None:
"""Single-stage servers reject stream=true; default path must send stream=false + JSON."""
class MockJsonResponse:
status = 200
def __init__(self):
self._payload = {
"created": 1,
"data": [
{
"b64_json": _MIN_PNG_B64,
"stage_durations": {"stage_0_gen_ms": 12.5},
}
],
}
async def json(self):
return self._payload
async def text(self):
return json.dumps(self._payload)
async def __aenter__(self):
return self
async def __aexit__(self, *_args):
return None
captured_stream: list[str] = []
real_add_field = None
def tracking_add_field(self, name, value=None, **kwargs):
if name == "stream":
captured_stream.append(str(value))
return real_add_field(self, name, value, **kwargs)
import aiohttp
real_add_field = aiohttp.FormData.add_field
mocker.patch.object(aiohttp.FormData, "add_field", tracking_add_field)
mock_session = mocker.AsyncMock()
mock_session.post = mocker.MagicMock(return_value=MockJsonResponse())
request = RequestFuncInput(
model="single-stage-edit",
model_name="single-stage-edit",
prompt="make it sunny",
api_url="http://test.com/v1/images/edits",
prompt_len=4,
output_len=1,
multi_modal_content=[{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{_MIN_PNG_B64}"}}],
extra_body={"num_inference_steps": 4},
)
output = await async_request_openai_image_edits_omni(request, mock_session, pbar=None)
assert captured_stream == ["false"]
assert output.success is True
assert not output.error
assert output.image_count == 1
assert output.image_generation_time_ms == pytest.approx(12.5)
@pytest.mark.asyncio
async def test_image_edits_stream_true_uses_sse_path(mocker: MockerFixture) -> None:
"""Explicit stream=true keeps the multi-stage SSE client path."""
sse_chunk = (
b'data: {"type":"image","data":[{"b64_json":"' + _MIN_PNG_B64.encode() + b'"}]}\n\n' + b"data: [DONE]\n\n"
)
mock_response = MockResponse(200, [sse_chunk])
captured_stream: list[str] = []
import aiohttp
real_add_field = aiohttp.FormData.add_field
def tracking_add_field(self, name, value=None, **kwargs):
if name == "stream":
captured_stream.append(str(value))
return real_add_field(self, name, value, **kwargs)
mocker.patch.object(aiohttp.FormData, "add_field", tracking_add_field)
mock_session = mocker.AsyncMock()
mock_session.post = mocker.MagicMock(return_value=mock_response)
request = RequestFuncInput(
model="multi-stage-edit",
model_name="multi-stage-edit",
prompt="edit",
api_url="http://test.com/v1/images/edits",
prompt_len=2,
output_len=1,
multi_modal_content=[{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{_MIN_PNG_B64}"}}],
extra_body={"stream": True},
)
output = await async_request_openai_image_edits_omni(request, mock_session, pbar=None)
assert captured_stream == ["true"]
assert output.success is True
assert output.image_count == 1
@pytest.mark.asyncio
async def test_image_generations_e2el_includes_json_body_consume(mocker: MockerFixture) -> None:
"""E2EL must include body transfer/decode, not stop at HTTP headers."""
class SlowJsonResponse:
status = 200
def __init__(self):
self._payload = {
"created": 1,
"data": [{"b64_json": _MIN_PNG_B64, "stage_durations": {"stage_0_gen_ms": 1.0}}],
}
async def json(self):
await asyncio.sleep(0.05)
return self._payload
async def text(self):
return json.dumps(self._payload)
async def __aenter__(self):
return self
async def __aexit__(self, *_args):
return None
mock_session = mocker.AsyncMock()
mock_session.post = mocker.MagicMock(return_value=SlowJsonResponse())
request = RequestFuncInput(
model="img-gen",
model_name="img-gen",
prompt="a cat",
api_url="http://test.com/v1/images/generations",
prompt_len=2,
output_len=1,
extra_body={},
)
output = await async_request_openai_image_generations_omni(request, mock_session, pbar=None)
assert output.success is True
assert output.latency >= 0.05
def test_video_local_image_reference_not_forwarded_as_raw_extra_field(tmp_path, mocker: MockerFixture) -> None:
"""Local image_reference must only be uploaded via dedicated serializer."""
import aiohttp
ref_path = tmp_path / "ref.png"
ref_path.write_bytes(base64.b64decode(_MIN_PNG_B64))
captured: list[tuple[str, object]] = []
real_add_field = aiohttp.FormData.add_field
def tracking_add_field(self, name, value=None, **kwargs):
captured.append((str(name), value))
return real_add_field(self, name, value, **kwargs)
mocker.patch.object(aiohttp.FormData, "add_field", tracking_add_field)
form = aiohttp.FormData()
extra_body = {
"num_inference_steps": 2,
"image_reference": str(ref_path),
}
request_body = {"model": "vid", "prompt": "p", **extra_body}
_add_video_extra_body_to_form(form, extra_body, request_body)
assert _add_video_reference_to_form(form, extra_body["image_reference"]) is True
field_names = [name for name, _ in captured]
assert "image_reference" not in field_names
assert field_names.count("input_reference") == 1
# Dedicated uploader sends file bytes, not the raw local path string.
uploaded = next(value for name, value in captured if name == "input_reference")
assert uploaded == base64.b64decode(_MIN_PNG_B64)
@pytest.mark.parametrize(
"reference",
[
{"image_url": "https://example.com/ref.png"},
{"file_id": "file-abc"},
[{"image_url": "https://example.com/a.png"}, {"file_id": "file-xyz"}],
],
)
def test_video_structured_image_reference_serialized_to_form(reference: object, mocker: MockerFixture) -> None:
"""Object-form image_reference must be JSON-serialized, not silently dropped."""
import aiohttp
captured: list[tuple[str, object]] = []
real_add_field = aiohttp.FormData.add_field
def tracking_add_field(self, name, value=None, **kwargs):
captured.append((str(name), value))
return real_add_field(self, name, value, **kwargs)
mocker.patch.object(aiohttp.FormData, "add_field", tracking_add_field)
form = aiohttp.FormData()
extra_body = {"image_reference": reference}
request_body = {"model": "vid", "prompt": "p", **extra_body}
_add_video_extra_body_to_form(form, extra_body, request_body)
assert _add_video_reference_to_form(form, reference) is True
field_names = [name for name, _ in captured]
# Reserved key must not be double-forwarded as a generic extra field;
# exactly one dedicated image_reference field.
assert field_names.count("image_reference") == 1
assert "input_reference" not in field_names
payload = next(value for name, value in captured if name == "image_reference")
assert json.loads(payload) == reference
def test_video_unsupported_image_reference_raises() -> None:
import aiohttp
form = aiohttp.FormData()
with pytest.raises(ValueError, match="Unsupported image_reference"):
_add_video_reference_to_form(form, {"not_a_supported_key": "x"})
with pytest.raises(ValueError, match="Unsupported image_reference"):
_add_video_reference_to_form(form, "/tmp/does-not-exist-ref.png")
if __name__ == "__main__":
pytest.main([__file__, "-v", "-s"])
tests/benchmarks/test_accuracy_bench_utils.py
+6 -6
View File
@@ -1,9 +1,11 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
# ruff: noqa: E402, I001
import argparse
import math
import os
import sys
import types
from pathlib import Path
import pytest
@@ -106,11 +108,9 @@ def test_resolve_seed_tts_root_downloads_only_requested_locale(monkeypatch, tmp_
captured["allow_patterns"] = allow_patterns
return str(downloaded_root)
monkeypatch.setitem(
sys.modules,
"huggingface_hub",
types.SimpleNamespace(snapshot_download=fake_snapshot_download),
)
from vllm_omni.transformers_utils import repo_utils
monkeypatch.setattr(repo_utils.hf_api(), "snapshot_download", fake_snapshot_download)
resolved = resolve_seed_tts_root(
"zhaochenyang20/seed-tts-eval",
tests/benchmarks/test_seed_tts_dataset_variants.py
+16 -4
View File
@@ -26,6 +26,7 @@ _MODULE_NAME = "vllm_omni.benchmarks.data_modules.seed_tts_dataset"
if _MODULE_NAME not in sys.modules:
_spec = importlib.util.spec_from_file_location(_MODULE_NAME, _MODULE_PATH)
assert _spec is not None and _spec.loader is not None
_mod = importlib.util.module_from_spec(_spec)
sys.modules[_MODULE_NAME] = _mod
_spec.loader.exec_module(_mod)
@@ -237,7 +238,8 @@ def test_seed_tts_design_dataset_rejects_missing_description(seed_tts_design_roo
assert req.seed_tts_utterance_id == "ok"
def test_seed_tts_whisper_transcribe_passes_attention_mask(monkeypatch):
@pytest.mark.parametrize("audio_seconds", [0.1, 61])
def test_seed_tts_whisper_transcribe_passes_attention_mask(monkeypatch, audio_seconds):
from vllm_omni.benchmarks.data_modules import seed_tts_eval
calls = {}
@@ -252,8 +254,9 @@ def test_seed_tts_whisper_transcribe_passes_attention_mask(monkeypatch):
return self
class FakeProcessor:
def __call__(self, wav, *, sampling_rate, return_tensors, return_attention_mask=False):
def __call__(self, wav, *, sampling_rate, return_tensors, return_attention_mask=False, **kwargs):
calls["return_attention_mask"] = return_attention_mask
calls["processor_kwargs"] = kwargs
assert sampling_rate == 16000
assert return_tensors == "pt"
assert len(wav) > 0
@@ -262,9 +265,10 @@ def test_seed_tts_whisper_transcribe_passes_attention_mask(monkeypatch):
attention_mask=FakeTensor("mask") if return_attention_mask else None,
)
def get_decoder_prompt_ids(self, *, language, task):
def get_decoder_prompt_ids(self, *, language, task, no_timestamps=True):
assert language == "english"
assert task == "transcribe"
calls["no_timestamps"] = no_timestamps
return [(1, 2)]
def batch_decode(self, predicted_ids, *, skip_special_tokens):
@@ -283,10 +287,18 @@ def test_seed_tts_whisper_transcribe_passes_attention_mask(monkeypatch):
monkeypatch.setattr(seed_tts_eval, "_en_model", FakeModel())
monkeypatch.setattr(seed_tts_eval, "_device", "cuda:1")
text = seed_tts_eval._transcribe_en_f32_16k(np.ones(1600, dtype=np.float32))
text = seed_tts_eval._transcribe_en_f32_16k(np.ones(int(audio_seconds * 16000), dtype=np.float32))
assert text == "hello"
assert calls["return_attention_mask"] is True
assert calls["input_features"].device == "cuda:1"
assert calls["generate_kwargs"]["attention_mask"].device == "cuda:1"
assert calls["generate_kwargs"]["forced_decoder_ids"] == [(1, 2)]
if audio_seconds > 30:
assert calls["processor_kwargs"] == {"truncation": False, "padding": "longest"}
assert calls["generate_kwargs"]["return_timestamps"] is True
assert calls["no_timestamps"] is False
else:
assert calls["processor_kwargs"] == {}
assert "return_timestamps" not in calls["generate_kwargs"]
assert calls["no_timestamps"] is True
tests/benchmarks/test_serve_cli.py
+35
View File
@@ -167,6 +167,26 @@ def test_preprocess_serve_args_applies_safe_omniinteract_prompt_default(
{"bot_task": "recaption"},
{"backend", "print_stage", "bot_task"},
),
(
[
"--endpoint",
"/v1/images/edits",
"--image-edits-bot-task",
"think",
],
{"bot_task": "think"},
{"endpoint", "bot_task"},
),
(
[
"--backend",
"/v1/images/edits",
"--image-edits-bot-task",
"recaption",
],
{"bot_task": "recaption"},
{"backend", "bot_task"},
),
(
["--extra-body", '{"bot_task":"vanilla"}'],
{"bot_task": "vanilla"},
@@ -182,6 +202,7 @@ def test_omni_args_parse_and_preprocess(
parser = TrackingArgumentParser()
parser.add_argument("--extra-body", type=json.loads, default=None)
parser.add_argument("--backend", default="openai-chat-omni")
parser.add_argument("--endpoint", default="/v1/chat/completions")
add_omni_args(parser)
args = parser.parse_args(argv)
@@ -358,3 +379,17 @@ def test_bench_serve_cli_mocks_http_request(tmp_path: Path):
)
assert bench_requests
assert all(url == expected_url for url in bench_requests), f"Unexpected target URLs: {bench_requests}"
def test_omni_request_timeout_s_flag_defaults_and_parses() -> None:
parser = TrackingArgumentParser()
add_omni_args(parser)
args = parser.parse_args([])
assert args.omni_request_timeout_s is None
args = parser.parse_args(["--omni-request-timeout-s", "0"])
assert args.omni_request_timeout_s == 0.0
args = parser.parse_args(["--omni-request-timeout-s", "3600"])
assert args.omni_request_timeout_s == 3600.0
tests/buildkite/test_amd_bootstrap.py
+156
View File
@@ -0,0 +1,156 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import os
import shutil
import subprocess
import sys
from pathlib import Path
import pytest
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
REPO_ROOT = Path(__file__).resolve().parents[2]
BOOTSTRAP_PATH = REPO_ROOT / ".buildkite/amd/scripts/bootstrap-amd-omni.sh"
def _run_bootstrap(
tmp_path: Path,
*,
debug_test_yaml: str | None = None,
curl_status: int = 0,
jq_status: int = 0,
labels: tuple[str, ...] = (),
pull_request: bool = True,
) -> subprocess.CompletedProcess[str]:
for relative_path in (
".buildkite/common/scripts/resolve_skip_ci.sh",
".buildkite/amd/scripts/select_test_suites.py",
):
destination = tmp_path / relative_path
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(REPO_ROOT / relative_path, destination)
# Stop after real suite selection, before rendering or uploading a pipeline.
(tmp_path / ".buildkite/common/scripts/skip_ci.py").write_text("print('test-skip')\n", encoding="utf-8")
env = {
"PATH": os.environ["PATH"],
"BUILDKITE_PULL_REQUEST": "123" if pull_request else "false",
"BUILDKITE_BRANCH": "feature" if pull_request else "main",
"TEST_PYTHON": sys.executable,
"TEST_CURL_STATUS": str(curl_status),
"TEST_JQ_STATUS": str(jq_status),
"TEST_PR_LABELS": "\n".join(labels),
}
if debug_test_yaml is not None:
env["DEBUG_TEST_YAML"] = debug_test_yaml
# Source the unmodified bootstrap with subprocess-local command doubles.
script = r"""
curl() {
echo 'curl called' >&2
return "$TEST_CURL_STATUS"
}
jq() {
echo 'jq called' >&2
printf '%s\n' "$TEST_PR_LABELS"
return "$TEST_JQ_STATUS"
}
git() { :; }
python3() { "$TEST_PYTHON" "$@"; }
source "$1"
"""
return subprocess.run(
["bash", "-c", script, "test-amd-bootstrap", str(BOOTSTRAP_PATH)],
cwd=tmp_path,
env=env,
capture_output=True,
text=True,
timeout=10,
check=False,
)
@pytest.mark.parametrize(("curl_status", "jq_status"), [(22, 0), (0, 4)], ids=["request-failure", "parse-failure"])
@pytest.mark.parametrize(
("debug_test_yaml", "expected_levels"),
[
("ready", ["l2"]),
("merge", ["l3"]),
("ready,merge", ["l2", "l3"]),
(" MERGE, ready,", ["l3", "l2"]),
],
)
def test_debug_override_skips_label_lookup(tmp_path, debug_test_yaml, expected_levels, curl_status, jq_status):
result = _run_bootstrap(
tmp_path,
debug_test_yaml=debug_test_yaml,
curl_status=curl_status,
jq_status=jq_status,
)
assert result.returncode == 0, result.stderr
assert "curl called" not in result.stderr
assert "jq called" not in result.stderr
assert "legacy ready-suite fallback" not in result.stderr
assert [
line.split()[2] for line in result.stdout.splitlines() if line.startswith("Skipping AMD ")
] == expected_levels
@pytest.mark.parametrize(("curl_status", "jq_status"), [(22, 0), (0, 4)], ids=["request-failure", "parse-failure"])
@pytest.mark.parametrize("debug_test_yaml", [None, ""], ids=["unset", "empty"])
def test_label_lookup_failure_without_override(tmp_path, debug_test_yaml, curl_status, jq_status):
result = _run_bootstrap(
tmp_path,
debug_test_yaml=debug_test_yaml,
curl_status=curl_status,
jq_status=jq_status,
)
assert result.returncode == 1
assert "Could not read PR labels" in result.stderr
assert "Skipping AMD " not in result.stdout
@pytest.mark.parametrize(
("labels", "expected_levels"),
[
(("ready",), ["l2"]),
(("merge-test",), ["l3"]),
(("ready", "merge-test"), ["l2", "l3"]),
((), ["l2"]),
],
)
def test_label_lookup_selects_tiers(tmp_path, labels, expected_levels):
result = _run_bootstrap(tmp_path, labels=labels)
assert result.returncode == 0, result.stderr
assert "curl called" in result.stderr
assert "jq called" in result.stderr
assert [
line.split()[2] for line in result.stdout.splitlines() if line.startswith("Skipping AMD ")
] == expected_levels
@pytest.mark.parametrize("debug_test_yaml", ["ready,ready", "nightly", ", ,", " \t"])
def test_invalid_debug_override_fails_without_label_lookup(tmp_path, debug_test_yaml):
result = _run_bootstrap(tmp_path, debug_test_yaml=debug_test_yaml, curl_status=22)
assert result.returncode == 1
assert "DEBUG_TEST_YAML" in result.stderr
assert "curl called" not in result.stderr
assert "jq called" not in result.stderr
assert "Skipping AMD " not in result.stdout
def test_main_does_not_lookup_labels(tmp_path):
result = _run_bootstrap(tmp_path, pull_request=False, curl_status=22, jq_status=4)
assert result.returncode == 0, result.stderr
assert "curl called" not in result.stderr
assert "jq called" not in result.stderr
assert "Skipping AMD l3 suite" in result.stdout
tests/buildkite/test_amd_suite_selection.py
+61
View File
@@ -0,0 +1,61 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
import importlib.util
from pathlib import Path
import pytest
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
SELECTOR_PATH = Path(".buildkite/amd/scripts/select_test_suites.py")
SPEC = importlib.util.spec_from_file_location("select_test_suites", SELECTOR_PATH)
assert SPEC is not None and SPEC.loader is not None
SELECTOR = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(SELECTOR)
@pytest.mark.parametrize(
("branch", "labels", "expected"),
[
("main", (), ("merge",)),
("feature", ("ready",), ("ready",)),
("feature", ("merge-test",), ("merge",)),
("feature", ("merge-test", "ready"), ("ready", "merge")),
("feature", ("amd-test",), ("ready",)),
("feature", ("amd-test", "merge-test"), ("merge",)),
("feature", ("amd-test", "ready", "merge-test"), ("ready", "merge")),
("feature", ("nightly-test",), ("ready",)),
("feature", ("not-ready", "merge-test-extra"), ("ready",)),
],
)
def test_label_suite_selection(branch, labels, expected):
assert SELECTOR.select_amd_test_suites(branch=branch, labels=labels) == expected
def test_debug_override_takes_precedence_and_normalizes_input():
assert SELECTOR.select_amd_test_suites(
branch="main",
labels=("ready",),
debug_test_yaml=" MERGE, ready,",
) == ("merge", "ready")
def test_empty_debug_override_uses_normal_selection():
assert SELECTOR.select_amd_test_suites(
branch="feature",
labels=("merge-test",),
debug_test_yaml="",
) == ("merge",)
@pytest.mark.parametrize("value", ["ready,ready", "nightly", ", ,", " \t"])
def test_invalid_debug_override(value):
with pytest.raises(ValueError):
SELECTOR.select_amd_test_suites(
branch="feature",
labels=(),
debug_test_yaml=value,
)
tests/buildkite/test_upload_pipeline.py
+32
View File
@@ -12,6 +12,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[2] / ".buildkite" / "com
from skip_ci import resolve_ci_decision # noqa: E402
from upload_pipeline import ( # noqa: E402
CUDA_HF_TOKEN_EXPORT,
_expand_mirror_hardwares,
_get_mirror_hw_selector,
_load_bootstrap_steps,
@@ -143,6 +144,10 @@ def test_mirror_hardwares_l4_1_expands_to_agents_and_plugins(monkeypatch: pytest
container = step["plugins"][0]["kubernetes"]["podSpec"]["containers"][0]
assert container["image"].endswith("$BUILDKITE_COMMIT")
assert container["resources"]["limits"]["nvidia.com/gpu"] == 1
env_names = {item["name"] for item in container["env"]}
assert "VLLM_CI_HF_TOKEN" in env_names
assert "HF_TOKEN" not in env_names
assert step["commands"] == [CUDA_HF_TOKEN_EXPORT, "pytest -sv tests/example"]
def test_mirror_hardwares_l4_preserves_explicit_retry() -> None:
@@ -182,6 +187,33 @@ def test_mirror_hardwares_a2b3_npu_4_expands_agents_image_and_plugins() -> None:
assert step["plugins"][0]["kubernetes"]["podSpecPatch"]["imagePullSecrets"] == [
{"name": "swr-secret"},
]
assert step["commands"] == ["pytest -sv tests/example"]
def test_all_cuda_mirror_hardwares_restore_hf_token_at_runtime() -> None:
for name in (
"l4_1",
"l4_2",
"l4_3",
"l4_4",
"h100_1",
"h100_2",
"h100_3",
"h100_4",
"b200_1",
"b200_2",
"b200_3",
"b200_4",
):
step = _expand_mirror_hardwares(
{"label": name, "mirror_hardwares": name, "commands": ["pytest -sv tests/example"]},
)
assert step is not None
container = step["plugins"][0]["kubernetes"]["podSpec"]["containers"][0]
env_names = {item["name"] for item in container["env"]}
assert "VLLM_CI_HF_TOKEN" in env_names
assert "HF_TOKEN" not in env_names
assert step["commands"][0] == CUDA_HF_TOKEN_EXPORT
def _gpu_limit(step: dict) -> int:
tests/config/test_config_factory.py
+41 -8
View File
@@ -6,7 +6,6 @@ Unit tests for StageConfigFactory and related classes.
import importlib
import warnings
from dataclasses import dataclass
from pathlib import Path
from unittest.mock import patch
@@ -37,6 +36,7 @@ from vllm_omni.config.stage_config import (
normalize_pipeline_cli_overrides,
pipeline_cfg_resolver,
)
from vllm_omni.diffusion.data import DiffusionParallelConfig
from vllm_omni.engine.arg_utils import SHARED_FIELDS, internal_blacklist_keys
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
@@ -451,13 +451,8 @@ class TestStageConfigFactory:
def test_default_diffusion_with_parallel_config(self):
"""Test diffusion config calculates devices from parallel_config."""
@dataclass
class MockParallelConfig:
world_size: int = 4
kwargs = {
"parallel_config": MockParallelConfig(),
"parallel_config": DiffusionParallelConfig(tensor_parallel_size=4),
"cache_backend": "tea_cache",
}
configs = StageConfigFactory.create_default_diffusion(kwargs)
@@ -937,6 +932,44 @@ class TestPipelineRegistration:
assert isinstance(omni_config, VllmOmniConfig)
assert omni_config.stage_by_id(0).diffusion_config.model == "fake/model"
def test_cosyvoice3_deploy_resolves_hash_snapshot_without_hf_metadata(self, tmp_path):
snapshot = tmp_path / "models--FunAudioLLM--Fun-CosyVoice3-0.5B-2512" / "snapshots" / ("a" * 40)
snapshot.mkdir(parents=True)
(snapshot / "config.json").write_text("{}\n", encoding="utf-8")
model = str(snapshot)
deploy_path = get_deploy_config_path("cosyvoice3.yaml")
assert StageConfigFactory.try_infer_model_type(model, trust_remote_code=False) is None
assert StageConfigFactory.get_pipeline_config(model, trust_remote_code=False) is None
omni_config = StageConfigFactory.create_from_model(
model,
trust_remote_code=False,
cli_overrides={},
deploy_config_path=deploy_path,
)
legacy_configs, _ = StageConfigFactory.create_legacy_stage_configs_from_model(
model,
trust_remote_code=False,
cli_overrides={},
deploy_config_path=deploy_path,
)
assert omni_config is not None
assert omni_config.pipeline_config.model_type == "cosyvoice3"
assert [
(stage.stage_id, stage.stage_type, stage.worker_type, stage.model_stage)
for stage in omni_config.stage_configs
] == [
(0, StageType.LLM, "ar", "cosyvoice3_talker"),
(1, StageType.LLM, "generation", "cosyvoice3_code2wav"),
]
assert all(stage.model_config.model == model for stage in omni_config.stage_configs)
assert all(stage.stage_type is not StageType.DIFFUSION for stage in omni_config.stage_configs)
assert legacy_configs is not None
assert [stage.stage_type for stage in legacy_configs] == [StageType.LLM, StageType.LLM]
def test_pipeline_registration(self, clean_pipeline_registry):
"""Ensure that we can register and create a custom pipeline config."""
new_model_type = "new_model_type"
@@ -1602,7 +1635,7 @@ stages:
assert stage.max_model_len == 1024
assert stage.max_num_batched_tokens == 1024
assert stage.max_num_seqs == 1
assert stage.gpu_memory_utilization == 0.7
assert stage.gpu_memory_utilization == 0.8
assert stage.skip_mm_profiling is True
assert stage.enforce_eager is True
assert stage.async_scheduling is False
tests/config/test_environment_variables.py
+2 -2
View File
@@ -170,7 +170,7 @@ def test_inventory_matches_reviewed_snapshot_counts():
EnvironmentVariableCategory.PUBLIC_OMNI: 32,
EnvironmentVariableCategory.INHERITED_VLLM: 20,
EnvironmentVariableCategory.PLATFORM_EXTERNAL: 27,
EnvironmentVariableCategory.MODEL_SPECIFIC: 62,
EnvironmentVariableCategory.MODEL_SPECIFIC: 61,
EnvironmentVariableCategory.BENCHMARK_TRANSITIONAL: 20,
EnvironmentVariableCategory.INTERNAL: 4,
}
@@ -182,7 +182,7 @@ def test_inventory_matches_reviewed_snapshot_counts():
)
assert {disposition: disposition_counts[disposition] for disposition in ModelEnvironmentVariableDisposition} == {
ModelEnvironmentVariableDisposition.PROMOTE: 37,
ModelEnvironmentVariableDisposition.REQUEST_SCOPE: 7,
ModelEnvironmentVariableDisposition.REQUEST_SCOPE: 6,
ModelEnvironmentVariableDisposition.EXTERNAL: 0,
ModelEnvironmentVariableDisposition.INTERNALIZE: 13,
ModelEnvironmentVariableDisposition.DEPRECATE_REMOVE: 5,
tests/config/test_omni_config.py
+137
View File
@@ -215,6 +215,16 @@ def test_from_pipeline_config_normalizes_stage_engine_extras_without_expanding_s
assert stage.diffusion_config.model_config["default_robot_embodiment"] == "roboarena"
@pytest.mark.parametrize("disabled", [True, False])
def test_frontend_log_stats_flag_is_not_an_unowned_stage_argument(disabled):
from vllm_omni.engine.stage_init_utils import build_engine_args_dict_from_omni_stage_config
config = _from_pipeline_key("dots_tts", cli_overrides={"disable_log_stats": disabled})
assert config.stage_configs
engine_args = build_engine_args_dict_from_omni_stage_config(config.stage_by_id(0), model="test-model")
assert "disable_log_stats" not in engine_args
def test_from_pipeline_config_applies_cli_overrides_without_stage_config_runtime_bridge():
omni_config = _from_pipeline_key(
"qwen3_tts",
@@ -261,6 +271,57 @@ def test_from_pipeline_config_rejects_unowned_deploy_engine_extras(engine_extras
VllmOmniConfig.from_pipeline_config(pipeline, user_deploy_config=deploy)
@pytest.mark.parametrize("stage_id", [0, 2], ids=["ar", "generation"])
def test_llm_additional_config_roundtrip_and_isolation(stage_id, monkeypatch):
from vllm_omni.engine import stage_init_utils
from vllm_omni.engine.stage_init_utils import build_engine_args_dict_from_omni_stage_config
# Worker discovery requires hardware support; this test covers config transport.
monkeypatch.setattr(stage_init_utils, "resolve_worker_cls", lambda engine_args: None)
additional_config = {"backend_options": {"enabled": True}}
pipeline = _resolve_pipeline_or_skip("minicpmo_4_5")
deploy = DeployConfig(
stages=[StageDeployConfig(stage_id=i, engine_extras={"additional_config": additional_config}) for i in (0, 2)]
)
config = VllmOmniConfig.from_pipeline_config(pipeline, user_deploy_config=deploy)
stage = config.stage_by_id(stage_id)
assert stage.runtime_config.additional_config == additional_config
engine_args = build_engine_args_dict_from_omni_stage_config(stage, model="test-model")
assert engine_args["additional_config"] == additional_config
engine_args["additional_config"]["backend_options"]["enabled"] = False
assert stage.runtime_config.additional_config["backend_options"]["enabled"] is True
stage.runtime_config.additional_config["backend_options"]["enabled"] = False
assert config.stage_by_id(2 if stage_id == 0 else 0).runtime_config.additional_config == additional_config
assert additional_config["backend_options"]["enabled"] is True
@pytest.mark.parametrize("deploy_name", ["minicpmo_4_5", "minicpmo_4_5_2gpu", "minicpmo_4_5_3gpu"])
def test_minicpmo_npu_additional_config_reaches_engine_args(monkeypatch, deploy_name):
from vllm_omni.engine import stage_init_utils
from vllm_omni.engine.stage_init_utils import build_engine_args_dict_from_omni_stage_config
from vllm_omni.platforms import current_omni_platform
monkeypatch.setattr(current_omni_platform, "device_name", "npu")
monkeypatch.setattr(stage_init_utils, "resolve_worker_cls", lambda engine_args: None)
stage = _from_pipeline_key("minicpmo_4_5", deploy_config_path=deploy_name).stage_by_id(2)
expected = {"code2wav_enable_npu_graph": True, "code2wav_max_npu_graphs": 32}
assert stage.runtime_config.additional_config == expected
engine_args = build_engine_args_dict_from_omni_stage_config(stage, model="test-model")
assert engine_args["additional_config"] == expected
def test_diffusion_additional_config_keeps_diffusion_owner():
from vllm_omni.engine.stage_init_utils import build_engine_args_dict_from_omni_stage_config
additional_config = {"torchair_graph_config": {"enabled": True}}
stage = _from_pipeline_key("dreamzero", cli_overrides={"additional_config": additional_config}).stage_by_id(0)
assert stage.runtime_config.additional_config is None
assert stage.diffusion_config.additional_config == additional_config
engine_args = build_engine_args_dict_from_omni_stage_config(stage, model="test-model")
assert engine_args["additional_config"] == additional_config
@pytest.mark.parametrize(
("engine_extras", "cli_overrides"),
[
@@ -532,6 +593,17 @@ def test_from_pipeline_config_dispatches_async_chunk_processors_without_mutating
assert pipeline.get_stage(1).custom_process_input_func is None
def test_joyai_code2wav_waits_for_full_payload():
config = _from_pipeline_key("joyai_vl_interaction")
talker = config.stage_by_id(1)
code2wav = config.stage_by_id(2)
assert talker.custom_process_next_stage_input_func.endswith("talker2code2wav_full_payload")
assert code2wav.custom_process_input_func.endswith("talker2code2wav_token_only")
assert code2wav.connector_config.async_chunk is False
assert code2wav.model_config.requires_full_payload_input is True
def test_vllm_omni_stage_config_public_fields_use_typed_stage_realizations():
assert not hasattr(BaseVllmOmniStageConfig, "from_stage_config")
assert not hasattr(BaseVllmOmniStageConfig, "to_legacy_stage_config")
@@ -559,6 +631,7 @@ def test_vllm_omni_stage_config_public_fields_use_typed_stage_realizations():
def test_runtime_config_fields_match_structured_runtime_scope():
assert {f.name for f in fields(OmniStageRuntimeConfig)} == {
"additional_config",
"distributed_executor_backend",
"worker_cls",
"devices",
@@ -1253,6 +1326,70 @@ def test_diffusion_config_from_kwargs_reuses_legacy_normalization(monkeypatch):
assert cfg.diffusers_call_kwargs == {}
def test_diffusion_config_none_values_preserve_dataclass_defaults():
from vllm_omni.diffusion.data import OmniDiffusionConfig
normalized = OmniDiffusionConfig.normalize_init_kwargs(
{
"lora_scale": None,
"enable_sleep_mode": None,
"diffusers_load_kwargs": None,
}
)
assert "lora_scale" not in normalized
assert "enable_sleep_mode" not in normalized
assert normalized["diffusers_load_kwargs"] == {}
config = OmniDiffusionConfig.from_kwargs(
lora_scale=None,
enable_sleep_mode=None,
)
assert config.lora_scale == 1.0
assert config.enable_sleep_mode is False
@pytest.mark.parametrize(
("canonical_key", "alias_key", "canonical_value", "alias_value"),
[
("lora_scale", "static_lora_scale", 0.75, 0.25),
(
"quantization_config",
"diffusion_quantization_config",
{"method": "canonical"},
{"method": "diffusion-alias"},
),
(
"quantization_config",
"quantization",
{"method": "canonical"},
"legacy-alias",
),
("diffusion_kv_cache_dtype", "kv_cache_dtype", "fp8", "fp16"),
("diffusion_kv_cache_skip_steps", "kv_cache_skip_steps", "0-1", "2-3"),
("diffusion_kv_cache_skip_layers", "kv_cache_skip_layers", "1-2", "3-4"),
("streaming_output", "diffusion_streaming_output", False, True),
],
)
def test_diffusion_alias_conflicts_prefer_canonical_key(
canonical_key,
alias_key,
canonical_value,
alias_value,
):
from vllm_omni.diffusion.data import normalize_omni_diffusion_kwargs
normalized = normalize_omni_diffusion_kwargs(
{
canonical_key: canonical_value,
alias_key: alias_value,
}
)
assert normalized[canonical_key] == canonical_value
assert alias_key not in normalized
def test_from_pipeline_config_normalizes_diffusion_config_aliases_from_engine_args(tmp_path, monkeypatch):
from vllm_omni.platforms import current_omni_platform
tests/core/sched/test_omni_scheduler_forces_v1_model_runner.py
+59
View File
@@ -0,0 +1,59 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""The omni scheduler must stay on the v1 model runner, like every omni worker.
vLLM 0.28 resolved this config to v1; 0.29 ends its fallback chain with v2. When
only the workers forced v1, the scheduler took its v2 fast path -- which omits
resumed-request token ids -- while the v1 runner still read them, so resuming a
request raised ``KeyError: <req_id>`` in ``_update_states`` and killed the engine
core (Multi-Replica Startup Test).
The default is upstream's to change, so this pins Omni's side of the contract
rather than the default itself.
"""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from vllm_omni.core.sched.omni_scheduler_mixin import OmniSchedulerMixin
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
class _Scheduler(OmniSchedulerMixin):
"""Minimal carrier for the shared init hook both omni schedulers call."""
def __init__(self, *, use_v2_model_runner: bool) -> None:
self.use_v2_model_runner = use_v2_model_runner
self.vllm_config = SimpleNamespace(
model_config=SimpleNamespace(
async_chunk=False,
stage_id=0,
pooling_output_decoder=None,
)
)
self._init_omni_io_scheduling_state()
@pytest.mark.parametrize("initial", [True, False])
def test_scheduler_ends_on_v1_model_runner(initial: bool) -> None:
"""Whatever upstream resolves, the omni scheduler runs v1."""
assert _Scheduler(use_v2_model_runner=initial).use_v2_model_runner is False
def test_v2_default_does_not_reach_the_resumed_request_fast_path() -> None:
"""The v2 fast path is what dropped resumed-request token ids.
``OmniGenerationScheduler`` branches on ``use_v2_model_runner`` when it
assembles SchedulerOutput; forcing it false keeps the v1 assembly that
carries ``all_token_ids`` for resumed requests, which the v1 runner reads.
"""
scheduler = _Scheduler(use_v2_model_runner=True)
assert not scheduler.use_v2_model_runner, (
"scheduler would take the v2 fast path while omni workers run v1; "
"resumed requests then KeyError in _update_states"
)
tests/dfx/perf/scripts/run_diffusion_benchmark.py
+7 -6
View File
@@ -1,3 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""
Performance benchmark CI runner for diffusion models.
@@ -356,6 +359,8 @@ def _resolve_offline_model(model: str) -> str:
"""
import huggingface_hub
from vllm_omni.transformers_utils.repo_utils import hf_api
if not model or os.path.isdir(model):
return model
@@ -380,9 +385,7 @@ def _resolve_offline_model(model: str) -> str:
if len(parts) >= 3:
repo_id = "/".join(parts[:2])
subfolder = "/".join(parts[2:])
from huggingface_hub import snapshot_download
snapshot_root = snapshot_download(
snapshot_root = hf_api().snapshot_download(
repo_id,
allow_patterns=[f"{subfolder}/**"],
local_files_only=huggingface_hub.constants.HF_HUB_OFFLINE,
@@ -391,9 +394,7 @@ def _resolve_offline_model(model: str) -> str:
if not huggingface_hub.constants.HF_HUB_OFFLINE:
return model
from huggingface_hub import snapshot_download
return snapshot_download(model, local_files_only=True)
return hf_api().snapshot_download(model, local_files_only=True)
class DiffusionServer:
tests/dfx/reliability/invalid_param_test/test_invalid_audio_diffusion.py
+15 -18
View File
@@ -13,9 +13,6 @@ from tests.helpers.runtime import OmniServer, OmniServerParams, OnlineOmniClient
pytestmark = [pytest.mark.slow, pytest.mark.diffusion]
# https://github.com/vllm-project/vllm-omni/issues/3649 — invalid inputs still HTTP 200 / timeout / missing validation.
_SKIP_ISSUE_3649 = pytest.mark.skip(reason="https://github.com/vllm-project/vllm-omni/issues/3649")
_PARAMS = [
pytest.param(
OmniServerParams(
@@ -36,10 +33,8 @@ _PARAMS = [
@pytest.mark.parametrize(
"overrides, err_message",
[
pytest.param({"input": ""}, ("cannot be empty", "input"), id="input_empty", marks=_SKIP_ISSUE_3649),
pytest.param(
{"input": " "}, ("cannot be empty", "input"), id="input_whitespace_only", marks=_SKIP_ISSUE_3649
),
pytest.param({"input": ""}, ("cannot be empty", "input"), id="input_empty"),
pytest.param({"input": " "}, ("cannot be empty", "input"), id="input_whitespace_only"),
pytest.param({"input": 123}, ("input", "a valid string"), id="input_wrong_type"),
pytest.param(
{"response_format": "mpeg"},
@@ -54,10 +49,8 @@ _PARAMS = [
("sse", "stream_format", "value_error", "not a supported stream_format"),
id="stream_format_sse_blocked",
),
pytest.param(
{"audio_length": -1.0}, ("audio_end_in_s", "higher than", "audio_start_in_s"), id="audio_length_negative"
),
pytest.param({"audio_length": 0}, "audio_length", id="audio_length_zero", marks=_SKIP_ISSUE_3649),
pytest.param({"audio_length": -1.0}, ("audio_length", "greater_than", "0"), id="audio_length_negative"),
pytest.param({"audio_length": 0}, ("audio_length", "greater_than", "0"), id="audio_length_zero"),
# ``OmniOpenAIServingAudioGenerate`` only forwards ``audio_start`` when ``audio_length`` is set.
pytest.param(
{"audio_start": -0.5, "audio_length": 5.0},
@@ -67,26 +60,30 @@ _PARAMS = [
pytest.param(
{"negative_prompt": ["noise"]}, ("negative_prompt", "a valid string"), id="negative_prompt_wrong_type"
),
pytest.param({"guidance_scale": -1.0}, "guidance_scale", id="guidance_scale_negative", marks=_SKIP_ISSUE_3649),
pytest.param({"guidance_scale": 0}, "guidance_scale", id="guidance_scale_zero", marks=_SKIP_ISSUE_3649),
pytest.param(
{"guidance_scale": -1.0},
("guidance_scale", "greater_than_equal", "0"),
id="guidance_scale_negative",
),
pytest.param(
{"num_inference_steps": 0},
("num_inference_steps", "greater than or equal to 1"),
("num_inference_steps", "greater_than_equal", "1"),
id="num_inference_steps_zero",
),
pytest.param(
{"num_inference_steps": -1},
("num_inference_steps", "greater than or equal to 1"),
("num_inference_steps", "greater_than_equal", "1"),
id="num_inference_steps_negative",
),
pytest.param(
{"num_inference_steps": 6000},
"num_inference_steps",
("num_inference_steps", "less_than_equal", "1000"),
id="num_inference_steps_above_max",
marks=_SKIP_ISSUE_3649,
),
pytest.param(
{"guidance_scale": 1001.0}, "guidance_scale", id="guidance_scale_above_max", marks=_SKIP_ISSUE_3649
{"guidance_scale": 1001.0},
("guidance_scale", "less_than_equal", "1000"),
id="guidance_scale_above_max",
),
pytest.param(
{"audio_length": 86401.0}, ("Requested audio length", "exceeds maximum"), id="audio_length_above_max"
tests/diffusion/attention/test_kernels_hub.py
+74
View File
@@ -13,6 +13,12 @@ from vllm_omni.diffusion.attention.backends.sdpa import SDPAImpl
from vllm_omni.platforms import current_omni_platform
def _clear_hub_module_cache(monkeypatch: pytest.MonkeyPatch) -> None:
from vllm_omni.diffusion.attention.backends import flash_attn_hub
monkeypatch.setattr(flash_attn_hub, "_hub_modules", {})
@pytest.mark.core_model
@pytest.mark.cpu
def test_explicit_kernels_hub_selection_does_not_fallback(monkeypatch: pytest.MonkeyPatch):
@@ -54,6 +60,74 @@ def test_explicit_kernels_hub_selection_does_not_fallback(monkeypatch: pytest.Mo
CudaOmniPlatform.get_diffusion_attn_backend_cls("FLASH_ATTN_3_HUB", head_size=64)
@pytest.mark.core_model
@pytest.mark.cpu
def test_explicit_kernels_hub_variant_miss_does_not_fallback(monkeypatch: pytest.MonkeyPatch):
"""Explicit Hub backends must fail loudly when no build variant resolves (#6971)."""
from vllm.platforms.interface import DeviceCapability
from vllm_omni.diffusion.envs import PACKAGES_CHECKER
from vllm_omni.platforms.cuda.platform import CudaOmniPlatform
monkeypatch.setattr(
CudaOmniPlatform,
"get_device_capability",
classmethod(lambda cls, device_id=0: DeviceCapability(9, 0)),
)
monkeypatch.setattr(PACKAGES_CHECKER, "get_packages_info", lambda: {"has_flash_attn": True})
_clear_hub_module_cache(monkeypatch)
kernels_module = types.ModuleType("kernels")
def _missing_variant(repo_id, version=None, **kwargs):
raise FileNotFoundError(f"Cannot find a build variant for this system in {repo_id}")
kernels_module.get_kernel = _missing_variant # type: ignore[attr-defined]
monkeypatch.setitem(sys.modules, "kernels", kernels_module)
with pytest.raises(RuntimeError, match="no compatible build"):
CudaOmniPlatform.get_diffusion_attn_backend_cls("FLASH_ATTN_3_HUB", head_size=64)
with pytest.raises(RuntimeError, match="no compatible build"):
CudaOmniPlatform.get_diffusion_attn_backend_cls("FLASH_ATTN_HUB", head_size=64)
@pytest.mark.core_model
@pytest.mark.cpu
def test_explicit_kernels_hub_preflight_uses_v2_when_v1_misses(monkeypatch: pytest.MonkeyPatch):
"""Preflight must share _load_hub_module's (1, 2) policy, not pin version=1."""
from vllm.platforms.interface import DeviceCapability
from vllm_omni.diffusion.attention.backends.registry import DiffusionAttentionBackendEnum
from vllm_omni.diffusion.envs import PACKAGES_CHECKER
from vllm_omni.platforms.cuda.platform import CudaOmniPlatform
monkeypatch.setattr(
CudaOmniPlatform,
"get_device_capability",
classmethod(lambda cls, device_id=0: DeviceCapability(9, 0)),
)
monkeypatch.setattr(PACKAGES_CHECKER, "get_packages_info", lambda: {"has_flash_attn": True})
_clear_hub_module_cache(monkeypatch)
kernels_module = types.ModuleType("kernels")
versions_tried: list[int | None] = []
def _v1_miss_v2_hit(repo_id, version=None, **kwargs):
versions_tried.append(version)
if version == 1:
raise FileNotFoundError(f"Cannot find a build variant for this system in {repo_id}")
return types.SimpleNamespace(name=f"{repo_id}@v{version}")
kernels_module.get_kernel = _v1_miss_v2_hit # type: ignore[attr-defined]
monkeypatch.setitem(sys.modules, "kernels", kernels_module)
fa3_path = CudaOmniPlatform.get_diffusion_attn_backend_cls("FLASH_ATTN_3_HUB", head_size=64)
assert fa3_path == DiffusionAttentionBackendEnum.FLASH_ATTN_3_HUB.get_path()
fa2_path = CudaOmniPlatform.get_diffusion_attn_backend_cls("FLASH_ATTN_HUB", head_size=64)
assert fa2_path == DiffusionAttentionBackendEnum.FLASH_ATTN_HUB.get_path()
assert versions_tried == [1, 2, 1, 2]
@pytest.mark.core_model
@pytest.mark.cpu
def test_explicit_flash_attention_unavailable_does_not_fallback(monkeypatch: pytest.MonkeyPatch):
tests/diffusion/diffusion_kv/test_block_tables.py
+3 -3
View File
@@ -279,17 +279,17 @@ def _install_native_modules(monkeypatch: pytest.MonkeyPatch, events: list[tuple]
group = SimpleNamespace(get_metadata_builder=lambda _index: SimpleNamespace(reorder_batch_threshold=None))
return [[group], [group]], "cg-support", [4, 8]
# vLLM 0.29 dropped attn_groups and cache_dtype from init_kv_cache: the
# layout now rides on the resolved CacheConfig.kv_cache_layout instead.
def init_cache(
runner_kv_caches,
forward_context,
kv_cache_config,
attn_groups,
device,
cache_dtype,
kernel_block_sizes,
vllm_config,
):
events.append(("cache", kv_cache_config, attn_groups, kernel_block_sizes))
events.append(("cache", kv_cache_config, kernel_block_sizes))
runner_kv_caches.extend(["cache-0", "cache-1"])
return {"layer-0": "tensor-0", "layer-1": "tensor-1"}
tests/diffusion/diffusion_kv/test_initialization.py
+5 -3
View File
@@ -299,8 +299,10 @@ def test_minus_one_max_model_len_is_auto_fitted_by_native_cache_sizing() -> None
)
assert vllm_config.model_config.original_max_model_len == -1
# Sixteen physical pages provide ``16 * block_size`` tokens. The native
# backend chooses the block geometry, so keep this assertion backend
# Sixteen physical pages provide ``(16 - 1) * block_size`` tokens: vLLM 0.29
# subtracts the null block that BlockPool permanently holds back before
# sizing (check_enough_kv_cache_memory), so only 15 pages are usable. The
# native backend chooses the block geometry, so keep this assertion backend
# independent instead of assuming the generic 16-token block size.
expected_max_model_len = min(model_limit, spec.block_size * 16)
expected_max_model_len = min(model_limit, spec.block_size * (16 - 1))
assert vllm_config.model_config.max_model_len == expected_max_model_len
tests/diffusion/diffusion_kv/test_layout.py
+147
View File
@@ -0,0 +1,147 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
"""Physical KV cache layout selection and propagation for the Diffusion cache."""
from __future__ import annotations
from types import SimpleNamespace
import pytest
import torch
from vllm.config import CacheConfig
from vllm.v1.kv_cache_interface import FullAttentionSpec
from vllm.v1.kv_cache_layout import KVCacheLayout
from vllm_omni.diffusion.diffusion_kv.layout import (
DEFAULT_DIFFUSION_KV_CACHE_LAYOUT,
adopt_kv_cache_layout,
assert_backend_layout_supported,
build_kv_cache_tensor,
resolve_diffusion_kv_cache_layout,
)
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
def _config():
return SimpleNamespace(cache_config=CacheConfig())
class _DenseBackend:
@classmethod
def indexes_kv_by_block_stride(cls) -> bool:
return False
class _BlockStrideBackend:
@classmethod
def indexes_kv_by_block_stride(cls) -> bool:
return True
def test_default_layout_is_not_block_outermost() -> None:
"""The default reproduces the pre-0.29 indexes_kv_by_block_stride=False."""
vllm_config = _config()
layout = resolve_diffusion_kv_cache_layout(vllm_config)
assert layout is DEFAULT_DIFFUSION_KV_CACHE_LAYOUT
assert layout.is_block_outermost is False
assert vllm_config.cache_config.get_resolved_kv_cache_layout() is layout
def test_block_stride_backend_selects_block_outermost_layout() -> None:
layout = resolve_diffusion_kv_cache_layout(_config(), indexes_kv_by_block_stride=True)
assert layout.is_block_outermost is True
def test_resolution_is_idempotent() -> None:
vllm_config = _config()
assert resolve_diffusion_kv_cache_layout(vllm_config) is resolve_diffusion_kv_cache_layout(vllm_config)
def test_contradicting_requirement_raises_instead_of_misaddressing() -> None:
vllm_config = _config()
resolve_diffusion_kv_cache_layout(vllm_config)
with pytest.raises(ValueError, match="contradicts the attention"):
resolve_diffusion_kv_cache_layout(vllm_config, indexes_kv_by_block_stride=True)
def test_preset_layout_is_honoured() -> None:
vllm_config = _config()
vllm_config.cache_config.kv_cache_layout = KVCacheLayout.BLNHC.name
layout = resolve_diffusion_kv_cache_layout(vllm_config, indexes_kv_by_block_stride=True)
assert layout is KVCacheLayout.BLNHC
def test_backend_assertion_is_quiet_until_the_layout_is_resolved() -> None:
"""Workers collect specs before the control plane hands them the layout."""
vllm_config = _config()
assert_backend_layout_supported(vllm_config, _BlockStrideBackend) # unresolved: nothing to check
resolve_diffusion_kv_cache_layout(vllm_config)
assert_backend_layout_supported(vllm_config, _DenseBackend)
assert_backend_layout_supported(vllm_config, None)
with pytest.raises(ValueError, match="contradicts the attention"):
assert_backend_layout_supported(vllm_config, _BlockStrideBackend)
def test_worker_adopts_the_layout_carried_on_the_kv_cache_config() -> None:
"""Diffusion workers receive only a KVCacheConfig, so the name rides on it."""
worker = _config()
shipped = SimpleNamespace(kv_cache_layout=KVCacheLayout.BLNHC.name)
assert adopt_kv_cache_layout(worker, shipped) is KVCacheLayout.BLNHC
assert worker.cache_config.get_resolved_kv_cache_layout() is KVCacheLayout.BLNHC
def test_worker_without_a_carried_layout_falls_back_to_the_same_default() -> None:
worker = _config()
assert adopt_kv_cache_layout(worker, SimpleNamespace(kv_cache_layout=None)) is DEFAULT_DIFFUSION_KV_CACHE_LAYOUT
def test_worker_disagreeing_with_the_control_plane_raises() -> None:
worker = _config()
resolve_diffusion_kv_cache_layout(worker)
with pytest.raises(ValueError, match="disagrees with the layout"):
adopt_kv_cache_layout(worker, SimpleNamespace(kv_cache_layout=KVCacheLayout.BLNHC.name))
def test_block_stride_requirement_propagates_control_plane_to_worker() -> None:
"""A block-stride backend must reach the worker as a block-outermost layout.
Covers the whole seam: the control plane resolves with the requirement, the
name rides on each rank-local KVCacheConfig, the worker adopts it, and the
backend contract check then passes instead of raising.
"""
control_plane = _config()
layout = resolve_diffusion_kv_cache_layout(control_plane, indexes_kv_by_block_stride=True)
assert layout.is_block_outermost is True
shipped = SimpleNamespace(kv_cache_layout=layout.name)
worker = _config()
assert adopt_kv_cache_layout(worker, shipped) is layout
# The worker's own backend requirement now agrees with what it was handed.
assert_backend_layout_supported(worker, _BlockStrideBackend)
with pytest.raises(ValueError, match="contradicts the attention"):
assert_backend_layout_supported(worker, _DenseBackend)
def test_tensor_strides_come_from_the_upstream_derivation() -> None:
spec = FullAttentionSpec(block_size=16, num_kv_heads=2, head_size=8, dtype=torch.bfloat16)
tensor = build_kv_cache_tensor(spec, 4, ["layer0", "layer1"])
assert tensor.layers == ["layer0", "layer1"]
assert tensor.size == spec.page_size_bytes * 2 * 4
# Layer-outermost: each layer owns a contiguous run of its own blocks.
assert tensor.layer_stride == spec.page_size_bytes * 4
assert tensor.block_stride == spec.page_size_bytes
tests/diffusion/diffusion_kv/test_manager.py
+4 -3
View File
@@ -1,11 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
import torch
import vllm.v1.core.single_type_kv_cache_manager as native_kv_managers
from vllm.v1.kv_cache_interface import FullAttentionSpec, KVCacheConfig, KVCacheGroupSpec, KVCacheTensor
from vllm.v1.kv_cache_interface import FullAttentionSpec, KVCacheConfig, KVCacheGroupSpec
from tests.helpers.kv_layout import build_kv_cache_tensor
from vllm_omni.diffusion.diffusion_kv.manager import DiffusionKVAdmissionError, DiffusionKVCacheManager
from vllm_omni.diffusion.diffusion_kv.request import DiffusionKVContext, DiffusionKVRequest
@@ -24,7 +25,7 @@ def _config(num_blocks: int) -> KVCacheConfig:
)
return KVCacheConfig(
num_blocks=num_blocks,
kv_cache_tensors=[KVCacheTensor(size=spec.page_size_bytes * num_blocks, shared_by=["layer0"])],
kv_cache_tensors=[build_kv_cache_tensor(spec, num_blocks, ["layer0"])],
kv_cache_groups=[KVCacheGroupSpec(layer_names=["layer0"], kv_cache_spec=spec)],
)
tests/diffusion/diffusion_kv/test_paged_attention_adapter.py
+3 -1
View File
@@ -15,6 +15,7 @@ import torch
from torch import nn
from vllm.v1.kv_cache_interface import FullAttentionSpec
from tests.helpers.kv_layout import layout_for_backend
from vllm_omni.diffusion.attention.backends.flash_attn import FlashAttentionImpl
from vllm_omni.diffusion.attention.layer import Attention
from vllm_omni.diffusion.diffusion_kv import paged_attention_adapter as adapter_module
@@ -1239,7 +1240,8 @@ def test_layer_adapter_accepts_platform_native_backend_and_uses_rank_local_heads
assert native_layer.num_heads == 4
assert native_layer.num_kv_heads == 2
assert native_layer.spec.num_kv_heads == 2
assert native_layer.spec.indexes_kv_by_block_stride is True
# 0.29: the flag no longer rides on the spec, it selects the layout.
assert layout_for_backend(native_layer.attn_backend).is_block_outermost is True
assert native_layer._q_scale_float == 1.0
assert native_layer._k_scale_float == 1.0
assert native_layer._v_scale_float == 1.0
tests/diffusion/diffusion_kv/test_paged_attention_adapter_gpu.py
+10 -10
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
from __future__ import annotations
@@ -10,12 +10,14 @@ import torch
from torch import nn
from torch.nn import functional as F
from vllm.config import AttentionConfig, CacheConfig, VllmConfig, set_current_vllm_config
from vllm.v1.kv_cache_interface import FullAttentionSpec, KVCacheConfig, KVCacheGroupSpec, KVCacheTensor
from vllm.v1.kv_cache_interface import FullAttentionSpec, KVCacheConfig, KVCacheGroupSpec
from vllm.v1.worker.gpu.attn_utils import init_attn_backend, init_kv_cache
from vllm.v1.worker.gpu.block_table import BlockTables
from tests.helpers.kv_layout import build_kv_cache_tensor
from vllm_omni.diffusion.attention.backends.abstract import AttentionMetadata
from vllm_omni.diffusion.attention.backends.flash_attn import FlashAttentionImpl
from vllm_omni.diffusion.diffusion_kv.layout import resolve_diffusion_kv_cache_layout
from vllm_omni.diffusion.diffusion_kv.paged_attention_adapter import (
DiffusionPagedAttentionAdapter,
DiffusionPagedAttentionLayerAdapter,
@@ -88,12 +90,7 @@ def test_adapter_executes_native_paged_attention_on_non_contiguous_blocks() -> N
canonical_spec = native_layer.spec
kv_cache_config = KVCacheConfig(
num_blocks=4,
kv_cache_tensors=[
KVCacheTensor(
size=canonical_spec.page_size_bytes * 4,
shared_by=[_LAYER_NAME],
)
],
kv_cache_tensors=[build_kv_cache_tensor(canonical_spec, 4, [_LAYER_NAME])],
kv_cache_groups=[
KVCacheGroupSpec(
layer_names=[_LAYER_NAME],
@@ -101,6 +98,9 @@ def test_adapter_executes_native_paged_attention_on_non_contiguous_blocks() -> N
)
],
)
# 0.29 reads the resolved physical layout when allocating; the diffusion
# control plane pins it, so this standalone harness must too.
resolve_diffusion_kv_cache_layout(vllm_config)
with set_current_vllm_config(vllm_config):
attn_groups, _, kernel_block_sizes = init_attn_backend(
kv_cache_config,
@@ -108,13 +108,13 @@ def test_adapter_executes_native_paged_attention_on_non_contiguous_blocks() -> N
device,
)
runner_kv_caches: list[torch.Tensor | list[torch.Tensor]] = []
# vLLM 0.29 dropped attn_groups and cache_dtype here: the layout now
# comes from the resolved CacheConfig.kv_cache_layout inside the call.
init_kv_cache(
runner_kv_caches,
vllm_config.compilation_config.static_forward_context,
kv_cache_config,
attn_groups,
device,
vllm_config.cache_config.cache_dtype,
kernel_block_sizes,
vllm_config,
)
tests/diffusion/diffusion_kv/test_request.py
+3 -3
View File
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import pytest
import torch
@@ -10,10 +10,10 @@ from vllm.v1.kv_cache_interface import (
FullAttentionSpec,
KVCacheConfig,
KVCacheGroupSpec,
KVCacheTensor,
)
from vllm.v1.request import RequestStatus
from tests.helpers.kv_layout import build_kv_cache_tensor
from vllm_omni.diffusion.diffusion_kv.request import DiffusionKVContext, DiffusionKVRequest
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
@@ -119,7 +119,7 @@ def _manager(*, enable_caching: bool) -> KVCacheManager:
group = KVCacheGroupSpec(layer_names=["layer0"], kv_cache_spec=spec)
config = KVCacheConfig(
num_blocks=8,
kv_cache_tensors=[KVCacheTensor(size=spec.page_size_bytes * 8, shared_by=["layer0"])],
kv_cache_tensors=[build_kv_cache_tensor(spec, 8, ["layer0"])],
kv_cache_groups=[group],
)
return KVCacheManager(
tests/diffusion/diffusion_kv/test_spec_discovery.py
+7 -4
View File
@@ -8,9 +8,10 @@ from unittest.mock import Mock
import pytest
import torch
from torch import nn
from vllm.v1.kv_cache_interface import FullAttentionSpec, KVCacheConfig, KVCacheGroupSpec, KVCacheTensor
from vllm.v1.kv_cache_interface import FullAttentionSpec, KVCacheConfig, KVCacheGroupSpec
import vllm_omni.diffusion.worker.diffusion_worker as diffusion_worker_module
from tests.helpers.kv_layout import build_kv_cache_tensor, layout_for_backend
from vllm_omni.diffusion.attention.layer import Attention
from vllm_omni.diffusion.diffusion_kv.config import DiffusionKVCacheMode
from vllm_omni.diffusion.worker.diffusion_model_runner import DiffusionModelRunner
@@ -79,7 +80,9 @@ def test_runner_discovers_native_spec_from_loaded_attention() -> None:
assert spec.num_kv_heads == 2
assert spec.head_size == 8
assert spec.dtype is torch.bfloat16
assert spec.indexes_kv_by_block_stride is True
# vLLM 0.29 moved the block-stride decision off the spec onto the resolved
# physical layout; the backend's declaration must still select one.
assert layout_for_backend(_Backend).is_block_outermost is True
assert spec.non_causal is True
@@ -112,7 +115,7 @@ def test_runner_retains_matching_rank_local_config() -> None:
spec = runner.get_kv_cache_spec()["image_attention"]
config = KVCacheConfig(
num_blocks=8,
kv_cache_tensors=[KVCacheTensor(size=spec.page_size_bytes * 8, shared_by=["image_attention"])],
kv_cache_tensors=[build_kv_cache_tensor(spec, 8, ["image_attention"])],
kv_cache_groups=[KVCacheGroupSpec(layer_names=["image_attention"], kv_cache_spec=spec)],
)
@@ -126,7 +129,7 @@ def test_runner_rejects_rank_local_config_for_different_layers() -> None:
spec = runner.get_kv_cache_spec()["image_attention"]
config = KVCacheConfig(
num_blocks=8,
kv_cache_tensors=[KVCacheTensor(size=spec.page_size_bytes * 8, shared_by=["other_attention"])],
kv_cache_tensors=[build_kv_cache_tensor(spec, 8, ["other_attention"])],
kv_cache_groups=[KVCacheGroupSpec(layer_names=["other_attention"], kv_cache_spec=spec)],
)
tests/diffusion/distributed/test_wan_chunked_decode.py
+174
View File
@@ -0,0 +1,174 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
import weakref
from types import SimpleNamespace
from unittest.mock import Mock
import pytest
import torch
from diffusers.models.autoencoders import AutoencoderKLWan
from vllm_omni.diffusion.distributed.autoencoders import wan_spatial_shard
from vllm_omni.diffusion.distributed.autoencoders.autoencoder_kl_wan import (
DistributedAutoencoderKLWan,
OmniAutoencoderKLWan,
)
from vllm_omni.diffusion.models.interface import supports_chunked_vae_decode
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
@pytest.fixture
def vae():
# Real causal convolutions and caches, with small random weights for CPU CI.
return OmniAutoencoderKLWan(
base_dim=4,
z_dim=2,
dim_mult=[1, 2],
num_res_blocks=1,
temperal_downsample=[True],
scale_factor_spatial=2,
scale_factor_temporal=2,
).eval()
@pytest.mark.parametrize("patch_size", [None, 2])
@pytest.mark.parametrize("return_dict", [False, True])
@torch.no_grad()
def test_chunked_decode_matches_diffusers(vae, patch_size, return_dict):
if patch_size is not None:
vae = OmniAutoencoderKLWan.from_config(vae.config, patch_size=patch_size, out_channels=12).eval()
z = torch.randn(1, 2, 3, 2, 2)
expected = AutoencoderKLWan._decode(vae, z).sample
actual = vae.decode(z, return_dict=return_dict)
torch.testing.assert_close(actual.sample if return_dict else actual[0], expected, rtol=0, atol=0)
chunks: list[torch.Tensor] = []
assert supports_chunked_vae_decode(vae)
assert not supports_chunked_vae_decode(object())
assert vae.decode_with_chunks(z, on_chunk=chunks.append) is None
assert [chunk.shape[2] for chunk in chunks] == [1, 2, 2]
torch.testing.assert_close(torch.cat(chunks, dim=2), expected, rtol=0, atol=0)
assert all(entry is None for entry in vae._feat_map)
@torch.no_grad()
def test_slicing_and_forward_hooks(vae):
z = torch.randn(2, 2, 2, 2, 2)
vae.enable_slicing()
batches = []
handle = vae.post_quant_conv.register_forward_pre_hook(lambda module, args: batches.append(args[0].shape[0]))
vae._hf_hook = SimpleNamespace(pre_forward=Mock())
try:
assert vae.decode(z).sample.shape[0] == 2
assert batches == [1, 1]
vae._hf_hook.pre_forward.assert_called_once_with(vae)
batches.clear()
with pytest.raises(ValueError, match="use_slicing"):
vae.decode_with_chunks(z, on_chunk=Mock())
assert batches == []
vae._hf_hook.pre_forward.reset_mock()
vae.decode_with_chunks(z[:1], on_chunk=Mock())
assert batches == [1]
vae._hf_hook.pre_forward.assert_called_once_with(vae)
finally:
handle.remove()
@torch.no_grad()
def test_callback_failure_drains_and_clears_cache(vae):
z = torch.randn(1, 2, 4, 2, 2)
decoded = Mock(return_value=None)
handle = vae.decoder.register_forward_hook(lambda *args: decoded())
error = RuntimeError("consumer failed")
callback = Mock(side_effect=error)
try:
with pytest.raises(RuntimeError, match="consumer failed") as exc:
vae.decode_with_chunks(z, on_chunk=callback)
assert exc.value is error
assert callback.call_count == 1
assert decoded.call_count == z.shape[2]
assert all(entry is None for entry in vae._feat_map)
finally:
handle.remove()
# The same instance can decode again after the failure.
chunks: list[torch.Tensor] = []
vae.decode_with_chunks(z, on_chunk=chunks.append)
torch.testing.assert_close(torch.cat(chunks, 2), vae.decode(z).sample, rtol=0, atol=0)
def test_chunked_decode_rejects_spatial_tiling(vae):
vae.enable_tiling(tile_sample_min_height=2, tile_sample_min_width=2)
with pytest.raises(ValueError, match="spatial tiling"):
vae.decode_with_chunks(torch.zeros(1, 2, 2, 2, 2), on_chunk=Mock())
@pytest.mark.parametrize("rank", [0, 1])
@pytest.mark.parametrize("fail_callback", [False, True])
def test_spatial_callback_ownership_and_bounded_retention(monkeypatch, rank, fail_callback):
# Isolate the temporal loop from halo kernels; weakrefs detect retention of
# decoded GPU-sized tensors even after the consumer has failed.
monkeypatch.setattr(wan_spatial_shard, "install_wan_spatial_shard_decode", lambda *args, **kwargs: None)
monkeypatch.setattr(wan_spatial_shard, "_rank_world", lambda group: (rank, 2))
references: list[weakref.ReferenceType[torch.Tensor]] = []
def decode(x, **kwargs):
assert sum(ref() is not None for ref in references) <= 1
chunk = x.clone()
references.append(weakref.ref(chunk))
return chunk
vae = SimpleNamespace(
config=SimpleNamespace(patch_size=None),
clear_cache=Mock(),
post_quant_conv=lambda z: z,
decoder=decode,
_feat_map=[],
)
callback = Mock(side_effect=RuntimeError("consumer failed") if fail_callback else None)
z = torch.zeros(1, 3, 6, 2, 2)
if rank == 0 and fail_callback:
with pytest.raises(RuntimeError, match="consumer failed"):
wan_spatial_shard.spatial_shard_decode(vae, z, group=None, on_chunk=callback)
else:
assert wan_spatial_shard.spatial_shard_decode(vae, z, group=None, on_chunk=callback) is None
assert len(references) == z.shape[2]
assert callback.call_count == (0 if rank else 1 if fail_callback else z.shape[2])
assert vae.clear_cache.call_count == 2
@pytest.mark.parametrize("rank", [0, 1])
@torch.no_grad()
def test_distributed_small_latent_callback_ownership(vae, monkeypatch, rank):
vae.is_distributed_enabled = lambda: True
vae.distributed_executor = SimpleNamespace(group=None)
monkeypatch.setattr(torch.distributed, "get_rank", lambda **kwargs: rank)
callback = Mock()
vae.decode_with_chunks(torch.zeros(1, 2, 2, 2, 2), on_chunk=callback)
assert callback.call_count == (2 if rank == 0 else 0)
@pytest.mark.parametrize("mode", ["spatial_shard_height", "spatial_shard_width", "tile"])
def test_distributed_chunked_routing(monkeypatch, mode):
vae = DistributedAutoencoderKLWan.__new__(DistributedAutoencoderKLWan)
torch.nn.Module.__init__(vae)
vae.use_tiling = True
vae.use_slicing = False
vae.tile_sample_min_height = vae.tile_sample_min_width = 2
vae.spatial_compression_ratio = 2
vae.distributed_executor = SimpleNamespace(group=None, parallel_mode=mode, parallel_size=2)
monkeypatch.setattr(vae, "is_distributed_enabled", lambda: True)
monkeypatch.setattr(torch.distributed, "get_world_size", lambda **kwargs: 2)
decode = Mock()
monkeypatch.setattr(wan_spatial_shard, "spatial_shard_decode", decode)
callback = Mock()
z = torch.zeros(1, 2, 2, 2, 2)
if mode == "tile":
with pytest.raises(ValueError, match="require spatial-shard"):
vae.decode_with_chunks(z, on_chunk=callback)
decode.assert_not_called()
else:
assert vae.decode_with_chunks(z, on_chunk=callback) is None
decode.assert_called_once_with(
vae, z, group=None, return_dict=True, split_dim=mode.removeprefix("spatial_shard_"), on_chunk=callback
)
tests/diffusion/layers/test_activation.py
+12 -4
View File
@@ -5,7 +5,7 @@ import pytest
import torch
import torch.nn.functional as F
from tests.helpers.mark import hardware_test
from tests.helpers.mark import hardware_marks
from vllm_omni.diffusion.layers.activation import SiluAndMul
pytestmark = [pytest.mark.core_model, pytest.mark.diffusion]
@@ -19,9 +19,17 @@ def test_silu_and_mul_native_matches_packed_reference() -> None:
torch.testing.assert_close(SiluAndMul().forward_native(packed), F.silu(gate) * up)
@hardware_test(res={"npu": "A3"}, num_cards=1)
def test_silu_and_mul_npu_matches_packed_reference() -> None:
packed = torch.randn(257, 256, device="npu", dtype=torch.bfloat16)
@pytest.mark.parametrize(
"dtype",
[
pytest.param(torch.bfloat16, marks=hardware_marks(res={"npu": "A2"}, num_cards=1), id="A2-bf16"),
pytest.param(torch.bfloat16, marks=hardware_marks(res={"npu": "A3"}, num_cards=1), id="A3-bf16"),
pytest.param(torch.bfloat16, marks=hardware_marks(res={"npu": "A5"}, num_cards=1), id="A5-bf16"),
pytest.param(torch.float16, marks=hardware_marks(res={"npu": "310P"}, num_cards=1), id="310P-fp16"),
],
)
def test_silu_and_mul_npu_matches_packed_reference(dtype: torch.dtype) -> None:
packed = torch.randn(257, 256, device="npu", dtype=dtype)
gate, up = packed.chunk(2, dim=-1)
torch.testing.assert_close(
Loaded 100 of 513 files, more files were not shown because too many files have changed in this diff. Show more