[Doc] Standardize serving names and identifiers (#5081)
Signed-off-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com> Co-authored-by: hsliu_ustc <hsliu_ustc@noreply.gitcode.com>
This commit is contained in:
1 parent
66b9ad287a
commit
625892032b
45 files changed
+223
-87
No files matched your search
+2
-2
@@ -23,7 +23,7 @@ nav:
|
||||
- Image-To-Video: user_guide/examples/offline_inference/image_to_video.md
|
||||
- InternVLA-A1: user_guide/examples/offline_inference/internvla_a1.md
|
||||
- MammothModa2-Preview: user_guide/examples/offline_inference/mammothmodal2_preview.md
|
||||
- MiMo-Audio Offline Inference: user_guide/examples/offline_inference/mimo_audio.md
|
||||
- MiMo-Audio: user_guide/examples/offline_inference/mimo_audio.md
|
||||
- Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md
|
||||
- Qwen3-Omni: user_guide/examples/offline_inference/qwen3_omni.md
|
||||
- Text-To-Audio: user_guide/examples/offline_inference/text_to_audio.md
|
||||
@@ -38,7 +38,7 @@ nav:
|
||||
- GLM-Image Online Serving: user_guide/examples/online_serving/glm_image.md
|
||||
- Image-To-Image: user_guide/examples/online_serving/image_to_image.md
|
||||
- Image-To-Video: user_guide/examples/online_serving/image_to_video.md
|
||||
- Online serving Example of vLLM-Omni for MiMo-Audio: user_guide/examples/online_serving/mimo_audio.md
|
||||
- MiMo-Audio: user_guide/examples/online_serving/mimo_audio.md
|
||||
- Qwen2.5-Omni: user_guide/examples/online_serving/qwen2_5_omni.md
|
||||
- Qwen3-Omni: user_guide/examples/online_serving/qwen3_omni.md
|
||||
- Text-To-Speech: user_guide/examples/online_serving/text_to_speech.md
|
||||
|
||||
+8
-2
@@ -18,11 +18,17 @@ Specify the port:
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
|
||||
```
|
||||
|
||||
If you have custom stage configs file, launch the server with command below
|
||||
For a migrated model, load a custom deploy configuration with `--deploy-config`:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --stage-configs-path /path/to/stage_configs_file
|
||||
vllm serve Qwen/Qwen2.5-Omni-7B --omni --deploy-config /path/to/deploy_config.yaml
|
||||
```
|
||||
|
||||
The deprecated `--stage-configs-path` flag is retained for models that still use the legacy `stage_args` schema:
|
||||
|
||||
```bash
|
||||
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --stage-configs-path /path/to/legacy_stage_config.yaml
|
||||
```
|
||||
|
||||
## bench
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
The vllm bench command launches the vLLM-Omni benchmark to evaluate the performance of multimodal models.
|
||||
|
||||
## Notes
|
||||
We currently only support using the "openai-chat-omni" and "openai-image-edits-omni" backend.
|
||||
vLLM-Omni registers the `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, and `daily-omni` serving benchmark backends.
|
||||
|
||||
## Basic Parameter Description
|
||||
You can use `vllm bench serve --omni --help=all` to get descriptions of all parameters. The commonly used parameters are described below:
|
||||
@@ -10,7 +10,7 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para
|
||||
Enable Omni (multimodal) mode, supporting multimodal inputs and outputs such as images, videos, and audio.
|
||||
|
||||
- `--backend`
|
||||
Specify the backend adapter as openai-chat-omni, using OpenAI Chat compatible API behavior as the protocol. Currently only openai-chat-omni is supported.
|
||||
Specify the backend adapter. vLLM-Omni adds `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, and `daily-omni` to the upstream vLLM backend choices.
|
||||
|
||||
- `--model`
|
||||
The model identifier to load, filled according to the models supported by vLLM-Omni.
|
||||
@@ -55,7 +55,7 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para
|
||||
'Allowed metric names are "ttft", "tpot", "itl", "ttfc", "tpoc", "icl", '
|
||||
'"tpop", "e2el", "audio_ttfp", "audio_rtf", "audio_duration". '
|
||||
|
||||
- `-print-stage`
|
||||
- `--print-stage`
|
||||
Print per-stage benchmark metrics for --omni serving when stage metrics are returned by the server. Disabled by default.
|
||||
|
||||
- `--save-result`
|
||||
@@ -312,7 +312,7 @@ vllm bench serve \
|
||||
--ignore-eos \
|
||||
--percentile-metrics ttft,tpot,itl \
|
||||
--random-output-len 2 \
|
||||
--extra_body '{"modalities": ["text"]}'
|
||||
--extra-body '{"modalities": ["text"]}'
|
||||
```
|
||||
|
||||
If successful, you will see the following output:
|
||||
@@ -390,7 +390,7 @@ vllm bench serve --omni \
|
||||
--ignore-eos \
|
||||
--percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf,ttfc,tpoc,icl \
|
||||
--random-output-len 900 \
|
||||
--extra_body '{"modalities": ["text","audio"]}' \
|
||||
--extra-body '{"modalities": ["text","audio"]}' \
|
||||
--print-stage
|
||||
```
|
||||
|
||||
|
||||
@@ -92,12 +92,12 @@ Full pattern hierarchy: `TakeRank`, `Union`, `GatherDim`, `AllGather`, `StitchSp
|
||||
|
||||
From highest to lowest, when the same axis is touched by multiple layers:
|
||||
|
||||
1. **CLI overrides** (`--stage_N_tensor_parallel_size`, `--stage-overrides`, `--omni-lb-policy`, etc.). A CLI value always wins; a warning is emitted whenever it overrides a strategy-declared axis (see `_reconcile_strategy_with_cli` in `vllm_omni/config/config_factory.py`).
|
||||
1. **CLI overrides** (`--tensor-parallel-size`, `--stage-overrides`, `--omni-lb-policy`, etc.). A CLI value always wins; a warning is emitted whenever it overrides a strategy-declared axis (see `_reconcile_strategy_with_cli` in `vllm_omni/config/config_factory.py`).
|
||||
2. **Strategy YAML** (`--strategy-config`). Derived sizing is written onto the merged stages with *conflict-on-explicit* semantics — if the deploy YAML already set a value to something different, application raises `StrategyApplyError`.
|
||||
3. **Deploy YAML** (`--deploy-config` or the bundled `vllm_omni/deploy/<model_type>.yaml`).
|
||||
4. **Parser defaults.**
|
||||
|
||||
After CLI overrides land, the device-layout guard re-runs (`check_device_layout`, called from `_reconcile_strategy_with_cli`) against the *effective* `tp * dp * pp * num_replicas` and the resolved `devices` string, so a `--devices` or `--tensor-parallel-size` that conflicts with the strategy's world size cannot slip past the pre-spawn check.
|
||||
After CLI overrides land, the device-layout guard re-runs (`check_device_layout`, called from `_reconcile_strategy_with_cli`) against the *effective* `tp * dp * pp * num_replicas` and the resolved `devices` string, so a `devices` value passed through `--stage-overrides` or a `--tensor-parallel-size` value that conflicts with the strategy's world size cannot slip past the pre-spawn check.
|
||||
|
||||
## Worked example
|
||||
|
||||
@@ -214,7 +214,7 @@ Strategy keys MUST be `model_stage` *names* (strings), the same labels stage con
|
||||
Use the `model_stage` name (e.g. `thinker`) instead.
|
||||
|
||||
!!! warning "Mixing CLI overrides with a strategy on the same axis"
|
||||
A strategy is meant to be the single writer for the axes it declares. If a CLI override (`--stage_0_tensor_parallel_size`, `--stage-overrides '{"0": {...}}'`) sets the same axis, the **CLI value wins** and a warning is emitted from `_reconcile_strategy_with_cli`. The device-layout guard then re-runs against the effective layout, so a CLI override that breaks `tp * dp * pp * num_replicas` will still fail fast at config time.
|
||||
A strategy is meant to be the single writer for the axes it declares. If a CLI override (`--tensor-parallel-size 2`, `--stage-overrides '{"0": {"tensor_parallel_size": 2}}'`) sets the same axis, the **CLI value wins** and a warning is emitted from `_reconcile_strategy_with_cli`. The device-layout guard then re-runs against the effective layout, so a CLI override that breaks `tp * dp * pp * num_replicas` will still fail fast at config time.
|
||||
|
||||
!!! warning "Passing `--strategy-config` against the legacy `--stage-configs-path`"
|
||||
Composable parallel only applies on the registry-based deploy path. If a model still resolves through the legacy `stage_args` YAML (loaded via `--stage-configs-path`), `--strategy-config` is silently inapplicable and emits:
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# Stage configs for vLLM-Omni
|
||||
# Deploy and legacy stage configurations
|
||||
|
||||
In vLLM-Omni, the target model is separated into multiple stages, which are processed by different LLMEngines, DiffusionEngines or other types of engines. Depending on different types of stages, such as Autoregressive (AR) stage or Diffusion transformer (DiT) stage, each can choose corresponding schedulers, model workers to load with the Engines in a plug-in fashion.
|
||||
In vLLM-Omni, a model's `PipelineConfig` defines its fixed stage topology, while a deploy configuration controls how those stages run. Models that have not migrated to this split still use legacy stage configurations with a `stage_args` schema.
|
||||
|
||||
!!! note
|
||||
Default deploy config YAMLs (for example, `vllm_omni/deploy/qwen2_5_omni.yaml`, `vllm_omni/deploy/qwen3_omni_moe.yaml`, and `vllm_omni/deploy/qwen3_tts.yaml`) are bundled and loaded automatically when neither `--stage-configs-path` nor `--deploy-config` is provided — the model registry resolves the right pipeline + deploy YAML by `model_type`. The bundled defaults have been verified on 1xH100 for Qwen2.5-Omni and 2xH100 for Qwen3-Omni. Models that have not yet migrated to the new schema continue to use the legacy `vllm_omni/model_executor/stage_configs/<model>.yaml` files via `--stage-configs-path`.
|
||||
|
||||
## New deploy schema reference
|
||||
## Deploy configuration schema
|
||||
|
||||
The new deploy schema lives under `vllm_omni/deploy/` and is paired with a frozen `PipelineConfig` registered by the model's `pipeline.py`. Each deploy YAML has these top-level fields:
|
||||
|
||||
@@ -82,14 +82,14 @@ stages:
|
||||
input_connectors: {from_stage_0: shm}
|
||||
```
|
||||
|
||||
### CLI flags introduced in this refactor
|
||||
### CLI flags
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--deploy-config PATH` | Load a new-schema deploy YAML. Takes precedence over `--stage-configs-path`. **Optional** — when omitted, the bundled `vllm_omni/deploy/<model_type>.yaml` is auto-loaded by the model registry. |
|
||||
| `--deploy-config PATH` | Load a new-schema deploy YAML. It is mutually exclusive with `--stage-configs-path`. **Optional** — when omitted, the bundled `vllm_omni/deploy/<model_type>.yaml` is auto-loaded by the model registry. |
|
||||
| `--stage-overrides JSON` | Per-stage JSON overrides, e.g. `'{"0":{"gpu_memory_utilization":0.5}}'`. Per-stage values always win over global flags. |
|
||||
| `--async-chunk` / `--no-async-chunk` | Flip the deploy YAML's `async_chunk:` bool. Unset (default) leaves the YAML value in force. |
|
||||
| `--stage-configs-path` | **Deprecated.** Accepts legacy `stage_args` yamls and (auto-detected) new deploy yamls; emits a deprecation warning. Migrate to `--deploy-config`. To be removed in a follow-up PR. |
|
||||
| `--stage-configs-path` | **Deprecated.** Load a legacy `stage_args` YAML. New-schema YAMLs passed to this flag are auto-detected for compatibility, but should use `--deploy-config`. It is mutually exclusive with `--deploy-config` and will be removed in a future release. |
|
||||
|
||||
### Stage-Based CLI Paradigm
|
||||
|
||||
@@ -146,7 +146,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
|
||||
|
||||
From highest to lowest:
|
||||
|
||||
1. Per-stage flags (`--stage-overrides` JSON, `--stage-<id>-<key>` if registered)
|
||||
1. Per-stage overrides (`--stage-overrides` JSON)
|
||||
2. Explicit global CLI flags (`--gpu-memory-utilization 0.85`, etc.)
|
||||
3. Platform section (`platforms.npu.stages`, etc.) on top of the base `stages:`
|
||||
4. Overlay YAML (via `base_config:`) on top of the base YAML
|
||||
@@ -210,7 +210,7 @@ Effective config per stage after the merge:
|
||||
| 1 | `max_model_len` | `16384` | global CLI |
|
||||
| 2 | (all defaults) | — | base YAML (no overrides apply) |
|
||||
|
||||
Therefore, as a core part of vLLM-Omni, the stage configs for a model have several main functions:
|
||||
Therefore, as a core part of vLLM-Omni, a model's pipeline and deployment configurations have several main functions:
|
||||
|
||||
- Claim partition of stages and their corresponding class implementation in `model_executor/models`.
|
||||
- The disaggregated configuration for each stage and the communication topology among them.
|
||||
@@ -219,16 +219,16 @@ Therefore, as a core part of vLLM-Omni, the stage configs for a model have sever
|
||||
- Default input parameters.
|
||||
|
||||
To override specific parameters, explicitly inject the customized configuration schema
|
||||
in both online and offline instantiation flows. Prioritize the `--deploy-config` flag
|
||||
when loading the new-schema deploy YAML schemas, reserving the `--stage-configs-path` parameter
|
||||
in both online and offline instantiation flows. Use the `--deploy-config` flag
|
||||
when loading a deploy configuration, reserving the `--stage-configs-path` parameter
|
||||
exclusively to maintain compatibility with legacy `stage_args` YAML constructs.
|
||||
|
||||
Examples:
|
||||
|
||||
For offline (Assume necessary dependencies have been imported):
|
||||
For offline inference (assuming the necessary dependencies have been imported):
|
||||
```python
|
||||
model_name = "Qwen/Qwen2.5-Omni-7B"
|
||||
omni = Omni(model=model_name, stage_configs_path="/path/to/custom_stage_configs.yaml")
|
||||
omni = Omni(model=model_name, deploy_config="/path/to/deploy_config.yaml")
|
||||
```
|
||||
|
||||
For online serving:
|
||||
@@ -242,11 +242,14 @@ Legacy online serving:
|
||||
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
|
||||
```
|
||||
!!! important
|
||||
We are actively iterating on the definition of stage configs, and we welcome all feedbacks from both community users and developers to help us shape the development!
|
||||
We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers.
|
||||
|
||||
## Legacy stage configuration reference
|
||||
|
||||
The remainder of this page documents the deprecated `stage_args` schema for models that have not migrated to deploy configurations. The following historical Qwen2.5-Omni configuration illustrates that schema; current Qwen2.5-Omni deployments should use `vllm_omni/deploy/qwen2_5_omni.yaml` instead.
|
||||
|
||||
Below is a specific example of stage_configs.yaml in Qwen2.5-omni.
|
||||
```python
|
||||
# stage config for running qwen2.5-omni with AsyncOmniEngine + Orchestrator runtime.
|
||||
# Historical stage configuration for Qwen2.5-Omni.
|
||||
stage_args:
|
||||
- stage_id: 0 # mark the unique id for each stage
|
||||
runtime: # The disaggregated configuration
|
||||
@@ -338,7 +341,7 @@ runtime:
|
||||
|
||||
```
|
||||
|
||||
## Stage Configuration Arguments
|
||||
## Legacy stage configuration arguments
|
||||
|
||||
Each stage in the `stage_args` list contains the following configuration options:
|
||||
|
||||
|
||||
@@ -67,6 +67,21 @@ docs/
|
||||
└── stylesheets/ # Custom CSS
|
||||
```
|
||||
|
||||
## Naming Model Examples
|
||||
|
||||
Offline inference and online serving examples for the same model use the same
|
||||
directory name and the shared display name in
|
||||
`examples/model_display_names.yml`. Use these title forms:
|
||||
|
||||
- `# <Model>: Offline inference`
|
||||
- `# <Model>: Online serving`
|
||||
|
||||
The documentation generator uses the full title for the page H1 and the shared
|
||||
display name alone for navigation, and fails the build if a mapped README uses
|
||||
a different H1. Keep checkpoint identifiers in commands and prose rather than
|
||||
in the display name, and do not rename an existing example directory solely to
|
||||
adjust its title because the directory defines its public documentation URL.
|
||||
|
||||
## Publishing Documentation
|
||||
|
||||
### GitHub Pages (Recommended)
|
||||
|
||||
@@ -638,7 +638,7 @@ mkdir -p examples/online_serving/your_model_name
|
||||
|
||||
- Script: `examples/offline_inference/your_model_name/end2end.py`
|
||||
- Parse args like BAGEL (`--model`, `--modality`, optional `--image-path`, `--steps`, etc.)
|
||||
- Use `from vllm_omni.entrypoints.omni import Omni` (or `OmniDiffusion` if your model is diffusion-only)
|
||||
- Use `from vllm_omni.entrypoints.omni import Omni` for both multi-stage and diffusion-only models
|
||||
- Save outputs (images/audio/video/text) with deterministic filenames (e.g., `output_0_0.png`)
|
||||
- Doc: `examples/offline_inference/your_model_name/README.md`
|
||||
- Include at least one runnable command, e.g.:
|
||||
|
||||
@@ -470,7 +470,7 @@ class Attention(nn.Module):
|
||||
role="self", role_category=None, ...):
|
||||
# Resolve backend for this role from the active OmniDiffusionConfig
|
||||
config = get_current_diffusion_config_or_none()
|
||||
attention_config = config.attention_config if config is not None else None
|
||||
attention_config = config.diffusion_attention_config if config is not None else None
|
||||
attn_backend_cls, spec = get_attn_backend_for_role(
|
||||
role=role,
|
||||
head_size=head_size,
|
||||
@@ -493,7 +493,7 @@ These backends provide the **kernel implementations** for attention computation.
|
||||
|
||||
#### Backend Selection Mechanism
|
||||
|
||||
Selection is driven by `AttentionConfig` on `OmniDiffusionConfig`, which carries a global `default` plus a `per_role` map of `AttentionSpec` entries. A role string is resolved in this order:
|
||||
Selection is driven by `OmniDiffusionConfig.diffusion_attention_config`, which carries a global `default` plus a `per_role` map of `AttentionSpec` entries. A role string is resolved in this order:
|
||||
|
||||
```python
|
||||
def get_attn_backend_for_role(role, head_size, attention_config=None, role_category=None):
|
||||
@@ -909,7 +909,7 @@ def initialize_model_parallel(
|
||||
|
||||
```
|
||||
1. User Request
|
||||
└─> OmniDiffusion.generate(prompt)
|
||||
└─> Omni.generate(prompt)
|
||||
└─> Prepare OmniDiffusionRequest
|
||||
└─> DiffusionEngine.step(requests)
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@ This guide demonstrates how to use the newly added features for extending vLLM-O
|
||||
Three main features enable custom pipeline extension:
|
||||
|
||||
1. **`WorkerWrapperBase`**: A wrapper class that enables dynamic worker extension with custom functionality
|
||||
2. **`load_format`**: A parameter that controls how diffusion models are loaded, including support for custom pipelines
|
||||
2. **`diffusion_load_format`**: A parameter that controls how diffusion models are loaded, including support for custom pipelines
|
||||
3. **`CustomPipelineWorkerExtension`**: An extension class that enables pipeline re-initialization with custom implementations
|
||||
|
||||
## Features
|
||||
@@ -27,15 +27,15 @@ Three main features enable custom pipeline extension:
|
||||
|
||||
**Location:** `vllm_omni/diffusion/worker/diffusion_worker.py`
|
||||
|
||||
### load_format Parameter
|
||||
### diffusion_load_format parameter
|
||||
|
||||
The `load_format` parameter controls how diffusion models are loaded. It supports the following values:
|
||||
The `diffusion_load_format` parameter controls the initial diffusion model load. The relevant values are:
|
||||
|
||||
- **`"default"`**: Standard model loading using the model registry (default behavior)
|
||||
- **`"custom_pipeline"`**: Load a custom pipeline class specified by `custom_pipeline_name`
|
||||
- **`"dummy"`**: Skip model loading (useful for testing or when pipeline will be initialized separately)
|
||||
- **`"dummy"`**: Skip the initial model load; use this with `custom_pipeline_args["pipeline_class"]` for the custom pipeline workflow below
|
||||
- **`"diffusers"`**: Load through the Hugging Face Diffusers adapter
|
||||
|
||||
**Location:** `vllm_omni/diffusion/model_loader/diffusers_loader.py`
|
||||
**Location:** `vllm_omni/diffusion/data.py` (`OmniDiffusionConfig`)
|
||||
|
||||
### CustomPipelineWorkerExtension
|
||||
|
||||
|
||||
@@ -16,8 +16,35 @@ ROOT_DIR_RELATIVE = "../../../../.."
|
||||
EXAMPLE_DIR = ROOT_DIR / "examples"
|
||||
EXAMPLE_DOC_DIR = ROOT_DIR / "docs/user_guide/examples"
|
||||
NAV_FILE = ROOT_DIR / "docs/.nav.yml"
|
||||
MODEL_DISPLAY_NAMES_FILE = EXAMPLE_DIR / "model_display_names.yml"
|
||||
MAX_INLINE_MATERIAL_SIZE = 8 * 1024
|
||||
|
||||
SERVING_MODE_TITLES = {
|
||||
"offline_inference": "Offline inference",
|
||||
"online_serving": "Online serving",
|
||||
}
|
||||
|
||||
|
||||
def load_model_display_names() -> dict[str, str]:
|
||||
try:
|
||||
with open(MODEL_DISPLAY_NAMES_FILE, encoding="utf-8") as f:
|
||||
model_display_names = yaml.safe_load(f)
|
||||
except (OSError, yaml.YAMLError) as exc:
|
||||
raise ValueError(f"Failed to load MODEL_DISPLAY_NAMES_FILE at {MODEL_DISPLAY_NAMES_FILE}: {exc}") from exc
|
||||
|
||||
if not isinstance(model_display_names, dict) or not all(
|
||||
isinstance(key, str) and isinstance(value, str) for key, value in model_display_names.items()
|
||||
):
|
||||
raise ValueError(
|
||||
"MODEL_DISPLAY_NAMES_FILE "
|
||||
f"at {MODEL_DISPLAY_NAMES_FILE} must contain a YAML mapping "
|
||||
"of string model IDs to string display names"
|
||||
)
|
||||
return model_display_names
|
||||
|
||||
|
||||
MODEL_DISPLAY_NAMES = load_model_display_names()
|
||||
|
||||
|
||||
def fix_case(text: str) -> str:
|
||||
subs = {
|
||||
@@ -54,6 +81,7 @@ class Example:
|
||||
main_file (Path): The main file in the directory.
|
||||
other_files (list[Path]): list of other files in the directory.
|
||||
title (str): The title of the document.
|
||||
nav_title (str): The concise title used in navigation.
|
||||
|
||||
Methods:
|
||||
__post_init__(): Initializes the main_file, other_files, and title attributes.
|
||||
@@ -78,6 +106,16 @@ class Example:
|
||||
def is_code(self) -> bool:
|
||||
return self.main_file.suffix != ".md"
|
||||
|
||||
@property
|
||||
def model_display_name(self) -> str | None:
|
||||
if self.category not in SERVING_MODE_TITLES:
|
||||
return None
|
||||
return MODEL_DISPLAY_NAMES.get(self.path.stem)
|
||||
|
||||
@property
|
||||
def nav_title(self) -> str:
|
||||
return self.model_display_name or self.title
|
||||
|
||||
def determine_main_file(self) -> Path:
|
||||
"""
|
||||
Determines the main file in the given path.
|
||||
@@ -151,13 +189,25 @@ class Example:
|
||||
return [file for file in self.path.rglob("*") if is_other_file(file)]
|
||||
|
||||
def determine_title(self) -> str:
|
||||
model_display_name = self.model_display_name
|
||||
if not self.is_code:
|
||||
# Specify encoding for building on Windows
|
||||
with open(self.main_file, encoding="utf-8") as f:
|
||||
first_line = f.readline().strip()
|
||||
match = re.match(r"^#\s+(?P<title>.+)$", first_line)
|
||||
if model_display_name:
|
||||
expected_title = f"{model_display_name}: {SERVING_MODE_TITLES[self.category]}"
|
||||
actual_title = match.group("title") if match else first_line
|
||||
if actual_title != expected_title:
|
||||
raise ValueError(
|
||||
f"Model example title mismatch in {self.main_file}: "
|
||||
f"expected '# {expected_title}', got {first_line!r}"
|
||||
)
|
||||
return expected_title
|
||||
if match:
|
||||
return match.group("title")
|
||||
elif model_display_name:
|
||||
raise ValueError(f"Mapped model example must use a Markdown README: {self.path}")
|
||||
return fix_case(self.path.stem.replace("_", " ").title())
|
||||
|
||||
def fix_relative_links(self, content: str) -> str:
|
||||
@@ -300,7 +350,7 @@ def update_nav_file(examples: list[Example]):
|
||||
for example in category_examples:
|
||||
doc_path = EXAMPLE_DOC_DIR / example.category / f"{example.path.stem}.md"
|
||||
rel_path = doc_path.relative_to(ROOT_DIR / "docs")
|
||||
category_items.append({example.title: rel_path.as_posix()})
|
||||
category_items.append({example.nav_title: rel_path.as_posix()})
|
||||
|
||||
if category_items:
|
||||
# Format category name (e.g., "offline_inference" -> "Offline Inference")
|
||||
|
||||
@@ -252,7 +252,7 @@ Validation errors (missing required fields):
|
||||
|
||||
```bash
|
||||
# Check if server is responding
|
||||
curl -X http://localhost:8000/v1/images/edit \
|
||||
curl http://localhost:8000/v1/images/edits \
|
||||
-F "prompt='test'"
|
||||
```
|
||||
|
||||
|
||||
@@ -138,7 +138,7 @@ Content-Type: application/json
|
||||
| `model` | string | server's model | Model to use (optional, should match server if specified) |
|
||||
| `n` | integer | 1 | Number of images to generate (1-10) |
|
||||
| `size` | string | model defaults | Image dimensions in WxH format (e.g., "1024x1024", "512x512") |
|
||||
| `response_format` | string | "b64_json" | Response format (only "b64_json" supported) |
|
||||
| `response_format` | string | "b64_json" | Response format (`"b64_json"` or `"file"`) |
|
||||
| `user` | string | null | User identifier for tracking |
|
||||
|
||||
#### vllm-omni Extension Parameters
|
||||
|
||||
@@ -83,7 +83,7 @@ When constructing `OmniDiffusionConfig` directly:
|
||||
from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, OmniDiffusionConfig
|
||||
|
||||
config = OmniDiffusionConfig(
|
||||
attention_config=AttentionConfig(
|
||||
diffusion_attention_config=AttentionConfig(
|
||||
default=AttentionSpec(backend="FLASH_ATTN"),
|
||||
per_role={
|
||||
"cross": AttentionSpec(backend="TORCH_SDPA"),
|
||||
|
||||
@@ -33,7 +33,7 @@ m = Omni(model="Wan-AI/Wan2.2-T2V-A14B-Diffusers", enable_cpu_offload=True)
|
||||
|
||||
**CLI:**
|
||||
```bash
|
||||
vllm-omni serve diffusion Wan-AI/Wan2.2-T2V-A14B-Diffusers --enable-cpu-offload
|
||||
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --enable-cpu-offload
|
||||
```
|
||||
|
||||
### To Support a Model
|
||||
@@ -146,10 +146,10 @@ m = Omni(model="Wan-AI/Wan2.2-I2V-A14B-Diffusers", enable_layerwise_offload=True
|
||||
**CLI:**
|
||||
```bash
|
||||
# Text-to-video
|
||||
vllm-omni serve diffusion Wan-AI/Wan2.2-T2V-A14B-Diffusers --enable-layerwise-offload
|
||||
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --enable-layerwise-offload
|
||||
|
||||
# Or image-to-video
|
||||
vllm-omni serve diffusion Wan-AI/Wan2.2-I2V-A14B-Diffusers --enable-layerwise-offload
|
||||
vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --enable-layerwise-offload
|
||||
```
|
||||
|
||||
### To Support a Model
|
||||
|
||||
@@ -257,7 +257,7 @@ python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \
|
||||
| :------- | :--- | :------ | :---------- |
|
||||
| `--deploy-config` | string | `None` | Path to deploy YAML (auto-detected if omitted) |
|
||||
| `--stage-configs-path` | string | `None` | [Deprecated] Legacy path to `stage_args` YAML; prefer `--deploy-config` |
|
||||
| `--worker-backend` | choice | `process` | `process` or `ray` |
|
||||
| `--worker-backend` | choice | `multi_process` | `multi_process` or `ray` |
|
||||
| `--ray-address` | string | `None` | Ray cluster address |
|
||||
| `--quantization` | string | `None` | Quantization method (e.g. `fp8`) |
|
||||
| `--log-stats` | flag | `False` | Enable statistics logging |
|
||||
|
||||
@@ -26,18 +26,22 @@ Stage 0 (AR Model) Stage 1 (Diffusion)
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
if __name__ == "__main__":
|
||||
omni = Omni(model="zai-org/GLM-Image")
|
||||
outputs = omni.generate(
|
||||
"A photorealistic mountain landscape at sunset",
|
||||
sampling_params={
|
||||
"height": 1024,
|
||||
"width": 1024,
|
||||
"num_inference_steps": 50,
|
||||
"guidance_scale": 1.5,
|
||||
"seed": 42,
|
||||
},
|
||||
sampling_params_list=[
|
||||
omni.default_sampling_params_list[0],
|
||||
OmniDiffusionSamplingParams(
|
||||
height=1024,
|
||||
width=1024,
|
||||
num_inference_steps=50,
|
||||
guidance_scale=1.5,
|
||||
seed=42,
|
||||
),
|
||||
],
|
||||
)
|
||||
outputs[0].request_output.images[0].save("output.png")
|
||||
```
|
||||
@@ -46,6 +50,7 @@ if __name__ == "__main__":
|
||||
|
||||
```python
|
||||
from vllm_omni.entrypoints.omni import Omni
|
||||
from vllm_omni.inputs.data import OmniDiffusionSamplingParams
|
||||
|
||||
if __name__ == "__main__":
|
||||
omni = Omni(model="zai-org/GLM-Image")
|
||||
@@ -56,13 +61,16 @@ if __name__ == "__main__":
|
||||
"image": "input.png",
|
||||
},
|
||||
},
|
||||
sampling_params={
|
||||
"height": 1024,
|
||||
"width": 1024,
|
||||
"num_inference_steps": 50,
|
||||
"guidance_scale": 1.5,
|
||||
"seed": 42,
|
||||
},
|
||||
sampling_params_list=[
|
||||
omni.default_sampling_params_list[0],
|
||||
OmniDiffusionSamplingParams(
|
||||
height=1024,
|
||||
width=1024,
|
||||
num_inference_steps=50,
|
||||
guidance_scale=1.5,
|
||||
seed=42,
|
||||
),
|
||||
],
|
||||
)
|
||||
outputs[0].request_output.images[0].save("output.png")
|
||||
```
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# MiMo-Audio Offline Inference
|
||||
# MiMo-Audio: Offline inference
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/mimo_audio>.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen2.5-Omni
|
||||
# Qwen2.5-Omni: Offline inference
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen2_5_omni>.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen3-Omni
|
||||
# Qwen3-Omni: Offline inference
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen3_omni>.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# AURA Omni Pipeline
|
||||
# AURA Omni: Online serving
|
||||
|
||||
`aura_omni` wires ASR, AURA, and Qwen3-TTS into one vLLM-Omni pipeline:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Online serving Example of vLLM-Omni for MiMo-Audio
|
||||
# MiMo-Audio: Online serving
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/mimo_audio>.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen2.5-Omni
|
||||
# Qwen2.5-Omni: Online serving
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen2_5_omni>.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen3-Omni
|
||||
# Qwen3-Omni: Online serving
|
||||
|
||||
Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni>.
|
||||
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
# Canonical display names for model examples that have both offline and online pages.
|
||||
# The docs generator combines these names with the serving mode for each page H1.
|
||||
aura_omni: AURA Omni
|
||||
covo_audio: Covo-Audio-Chat
|
||||
dynin_omni: Dynin-Omni
|
||||
lance: Lance
|
||||
mimo_audio: MiMo-Audio
|
||||
ming_flash_omni: Ming-flash-omni 2.0
|
||||
qwen2_5_omni: Qwen2.5-Omni
|
||||
qwen3_omni: Qwen3-Omni
|
||||
sensenova_u1: SenseNova-U1
|
||||
step_audio2: Step-Audio2
|
||||
@@ -1,4 +1,4 @@
|
||||
# AURA Omni
|
||||
# AURA Omni: Offline inference
|
||||
|
||||
This example runs the native AURA Omni pipeline offline:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Covo-Audio-Chat (Offline Inference)
|
||||
# Covo-Audio-Chat: Offline inference
|
||||
|
||||
## Setup
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Dynin-Omni Offline End2End Example
|
||||
# Dynin-Omni: Offline inference
|
||||
|
||||
This folder contains a unified offline inference entrypoint:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Lance (ByteDance) — offline inference
|
||||
# Lance: Offline inference
|
||||
|
||||
[Lance](https://huggingface.co/bytedance-research/Lance) is a 3B unified
|
||||
autoregressive + diffusion multimodal model on a Qwen2.5-VL backbone. It is
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# MiMo-Audio Offline Inference
|
||||
# MiMo-Audio: Offline inference
|
||||
|
||||
This directory contains an offline demo for running MiMo-Audio models with vLLM Omni. It builds task-specific inputs and generates WAV files or text outputs locally.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Ming-flash-omni 2.0
|
||||
# Ming-flash-omni 2.0: Offline inference
|
||||
|
||||
[Ming-flash-omni-2.0](https://github.com/inclusionAI/Ming) is an omni-modal model supporting text, image, video, and audio understanding, with text and speech outputs.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen2.5-Omni
|
||||
# Qwen2.5-Omni: Offline inference
|
||||
|
||||
## Setup
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen3-Omni
|
||||
# Qwen3-Omni: Offline inference
|
||||
|
||||
## Setup
|
||||
Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# SenseNova-U1-8B-MoT
|
||||
# SenseNova-U1: Offline inference
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Step-Audio2 Offline Inference Examples
|
||||
# Step-Audio2: Offline inference
|
||||
|
||||
This directory contains examples for running offline inference with Step-Audio2 using vLLM-Omni.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# AURA Omni Native Pipeline
|
||||
# AURA Omni: Online serving
|
||||
|
||||
`aura_omni` serves AURA as a native multi-stage vLLM-Omni pipeline:
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Covo-Audio-Chat
|
||||
# Covo-Audio-Chat: Online serving
|
||||
|
||||
## Installation
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Dynin-Omni Online Serving Example
|
||||
# Dynin-Omni: Online serving
|
||||
|
||||
## Installation
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Lance — online serving
|
||||
# Lance: Online serving
|
||||
|
||||
OpenAI-compatible chat completions API for Lance, served by `vllm-omni`.
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Online serving Example of vLLM-Omni for MiMo-Audio
|
||||
# MiMo-Audio: Online serving
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Ming-flash-omni 2.0
|
||||
# Ming-flash-omni 2.0: Online serving
|
||||
|
||||
## Installation
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen2.5-Omni
|
||||
# Qwen2.5-Omni: Online serving
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Qwen3-Omni
|
||||
# Qwen3-Omni: Online serving
|
||||
|
||||
## 🛠️ Installation
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# SenseNova-U1 Online Serving
|
||||
# SenseNova-U1: Online serving
|
||||
|
||||
## Launch the Server
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Step-Audio2 Online Serving
|
||||
# Step-Audio2: Online serving
|
||||
|
||||
This directory contains examples for running Step-Audio2 with vLLM-Omni's online serving API.
|
||||
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT_DIR = Path(__file__).parents[2]
|
||||
HOOK_PATH = ROOT_DIR / "docs/mkdocs/hooks/generate_examples.py"
|
||||
SPEC = importlib.util.spec_from_file_location("generate_examples", HOOK_PATH)
|
||||
assert SPEC and SPEC.loader
|
||||
generate_examples = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(generate_examples)
|
||||
|
||||
|
||||
def test_model_example_titles_match_shared_display_names():
|
||||
for slug, display_name in generate_examples.MODEL_DISPLAY_NAMES.items():
|
||||
for category, mode_title in generate_examples.SERVING_MODE_TITLES.items():
|
||||
example_path = ROOT_DIR / "examples" / category / slug
|
||||
example = generate_examples.Example(example_path, category)
|
||||
|
||||
assert example.title == f"{display_name}: {mode_title}"
|
||||
assert example.nav_title == display_name
|
||||
|
||||
|
||||
def test_model_example_title_mismatch_is_rejected(tmp_path):
|
||||
example_path = tmp_path / "mimo_audio"
|
||||
example_path.mkdir()
|
||||
(example_path / "README.md").write_text("# A different title\n", encoding="utf-8")
|
||||
|
||||
with pytest.raises(ValueError, match="Model example title mismatch"):
|
||||
generate_examples.Example(example_path, "offline_inference")
|
||||
|
||||
|
||||
def test_model_display_names_require_string_mapping(tmp_path, monkeypatch):
|
||||
display_names_path = tmp_path / "model_display_names.yml"
|
||||
display_names_path.write_text("- not a mapping\n", encoding="utf-8")
|
||||
monkeypatch.setattr(generate_examples, "MODEL_DISPLAY_NAMES_FILE", display_names_path)
|
||||
|
||||
with pytest.raises(ValueError, match="MODEL_DISPLAY_NAMES_FILE"):
|
||||
generate_examples.load_model_display_names()
|
||||
Reference in new issue
Block a user