diff --git a/docs/.nav.yml b/docs/.nav.yml index 2e7097efb..f94907a67 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -23,7 +23,7 @@ nav: - Image-To-Video: user_guide/examples/offline_inference/image_to_video.md - InternVLA-A1: user_guide/examples/offline_inference/internvla_a1.md - MammothModa2-Preview: user_guide/examples/offline_inference/mammothmodal2_preview.md - - MiMo-Audio Offline Inference: user_guide/examples/offline_inference/mimo_audio.md + - MiMo-Audio: user_guide/examples/offline_inference/mimo_audio.md - Qwen2.5-Omni: user_guide/examples/offline_inference/qwen2_5_omni.md - Qwen3-Omni: user_guide/examples/offline_inference/qwen3_omni.md - Text-To-Audio: user_guide/examples/offline_inference/text_to_audio.md @@ -38,7 +38,7 @@ nav: - GLM-Image Online Serving: user_guide/examples/online_serving/glm_image.md - Image-To-Image: user_guide/examples/online_serving/image_to_image.md - Image-To-Video: user_guide/examples/online_serving/image_to_video.md - - Online serving Example of vLLM-Omni for MiMo-Audio: user_guide/examples/online_serving/mimo_audio.md + - MiMo-Audio: user_guide/examples/online_serving/mimo_audio.md - Qwen2.5-Omni: user_guide/examples/online_serving/qwen2_5_omni.md - Qwen3-Omni: user_guide/examples/online_serving/qwen3_omni.md - Text-To-Speech: user_guide/examples/online_serving/text_to_speech.md diff --git a/docs/cli/README.md b/docs/cli/README.md index 1fcfdb14e..9e3240de4 100644 --- a/docs/cli/README.md +++ b/docs/cli/README.md @@ -18,11 +18,17 @@ Specify the port: vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 ``` -If you have custom stage configs file, launch the server with command below +For a migrated model, load a custom deploy configuration with `--deploy-config`: + ```bash -vllm serve Qwen/Qwen2.5-Omni-7B --omni --stage-configs-path /path/to/stage_configs_file +vllm serve Qwen/Qwen2.5-Omni-7B --omni --deploy-config /path/to/deploy_config.yaml ``` +The deprecated `--stage-configs-path` flag is retained for models that still use the legacy `stage_args` schema: + +```bash +vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --stage-configs-path /path/to/legacy_stage_config.yaml +``` ## bench diff --git a/docs/cli/bench/serve.md b/docs/cli/bench/serve.md index d6b3ef393..2a1acd1f0 100644 --- a/docs/cli/bench/serve.md +++ b/docs/cli/bench/serve.md @@ -2,7 +2,7 @@ The vllm bench command launches the vLLM-Omni benchmark to evaluate the performance of multimodal models. ## Notes -We currently only support using the "openai-chat-omni" and "openai-image-edits-omni" backend. +vLLM-Omni registers the `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, and `daily-omni` serving benchmark backends. ## Basic Parameter Description You can use `vllm bench serve --omni --help=all` to get descriptions of all parameters. The commonly used parameters are described below: @@ -10,7 +10,7 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para Enable Omni (multimodal) mode, supporting multimodal inputs and outputs such as images, videos, and audio. - `--backend` - Specify the backend adapter as openai-chat-omni, using OpenAI Chat compatible API behavior as the protocol. Currently only openai-chat-omni is supported. + Specify the backend adapter. vLLM-Omni adds `openai-chat-omni`, `openai-audio-speech`, `openai-image-edits-omni`, and `daily-omni` to the upstream vLLM backend choices. - `--model` The model identifier to load, filled according to the models supported by vLLM-Omni. @@ -55,7 +55,7 @@ You can use `vllm bench serve --omni --help=all` to get descriptions of all para 'Allowed metric names are "ttft", "tpot", "itl", "ttfc", "tpoc", "icl", ' '"tpop", "e2el", "audio_ttfp", "audio_rtf", "audio_duration". ' -- `-print-stage` +- `--print-stage` Print per-stage benchmark metrics for --omni serving when stage metrics are returned by the server. Disabled by default. - `--save-result` @@ -312,7 +312,7 @@ vllm bench serve \ --ignore-eos \ --percentile-metrics ttft,tpot,itl \ --random-output-len 2 \ - --extra_body '{"modalities": ["text"]}' + --extra-body '{"modalities": ["text"]}' ``` If successful, you will see the following output: @@ -390,7 +390,7 @@ vllm bench serve --omni \ --ignore-eos \ --percentile-metrics ttft,tpot,itl,e2el,audio_ttfp,audio_rtf,ttfc,tpoc,icl \ --random-output-len 900 \ - --extra_body '{"modalities": ["text","audio"]}' \ + --extra-body '{"modalities": ["text","audio"]}' \ --print-stage ``` diff --git a/docs/configuration/composable_parallel.md b/docs/configuration/composable_parallel.md index 400fbab38..a456de843 100644 --- a/docs/configuration/composable_parallel.md +++ b/docs/configuration/composable_parallel.md @@ -92,12 +92,12 @@ Full pattern hierarchy: `TakeRank`, `Union`, `GatherDim`, `AllGather`, `StitchSp From highest to lowest, when the same axis is touched by multiple layers: -1. **CLI overrides** (`--stage_N_tensor_parallel_size`, `--stage-overrides`, `--omni-lb-policy`, etc.). A CLI value always wins; a warning is emitted whenever it overrides a strategy-declared axis (see `_reconcile_strategy_with_cli` in `vllm_omni/config/config_factory.py`). +1. **CLI overrides** (`--tensor-parallel-size`, `--stage-overrides`, `--omni-lb-policy`, etc.). A CLI value always wins; a warning is emitted whenever it overrides a strategy-declared axis (see `_reconcile_strategy_with_cli` in `vllm_omni/config/config_factory.py`). 2. **Strategy YAML** (`--strategy-config`). Derived sizing is written onto the merged stages with *conflict-on-explicit* semantics — if the deploy YAML already set a value to something different, application raises `StrategyApplyError`. 3. **Deploy YAML** (`--deploy-config` or the bundled `vllm_omni/deploy/.yaml`). 4. **Parser defaults.** -After CLI overrides land, the device-layout guard re-runs (`check_device_layout`, called from `_reconcile_strategy_with_cli`) against the *effective* `tp * dp * pp * num_replicas` and the resolved `devices` string, so a `--devices` or `--tensor-parallel-size` that conflicts with the strategy's world size cannot slip past the pre-spawn check. +After CLI overrides land, the device-layout guard re-runs (`check_device_layout`, called from `_reconcile_strategy_with_cli`) against the *effective* `tp * dp * pp * num_replicas` and the resolved `devices` string, so a `devices` value passed through `--stage-overrides` or a `--tensor-parallel-size` value that conflicts with the strategy's world size cannot slip past the pre-spawn check. ## Worked example @@ -214,7 +214,7 @@ Strategy keys MUST be `model_stage` *names* (strings), the same labels stage con Use the `model_stage` name (e.g. `thinker`) instead. !!! warning "Mixing CLI overrides with a strategy on the same axis" - A strategy is meant to be the single writer for the axes it declares. If a CLI override (`--stage_0_tensor_parallel_size`, `--stage-overrides '{"0": {...}}'`) sets the same axis, the **CLI value wins** and a warning is emitted from `_reconcile_strategy_with_cli`. The device-layout guard then re-runs against the effective layout, so a CLI override that breaks `tp * dp * pp * num_replicas` will still fail fast at config time. + A strategy is meant to be the single writer for the axes it declares. If a CLI override (`--tensor-parallel-size 2`, `--stage-overrides '{"0": {"tensor_parallel_size": 2}}'`) sets the same axis, the **CLI value wins** and a warning is emitted from `_reconcile_strategy_with_cli`. The device-layout guard then re-runs against the effective layout, so a CLI override that breaks `tp * dp * pp * num_replicas` will still fail fast at config time. !!! warning "Passing `--strategy-config` against the legacy `--stage-configs-path`" Composable parallel only applies on the registry-based deploy path. If a model still resolves through the legacy `stage_args` YAML (loaded via `--stage-configs-path`), `--strategy-config` is silently inapplicable and emits: diff --git a/docs/configuration/stage_configs.md b/docs/configuration/stage_configs.md index 803c74e0d..1f14d21fd 100644 --- a/docs/configuration/stage_configs.md +++ b/docs/configuration/stage_configs.md @@ -1,11 +1,11 @@ -# Stage configs for vLLM-Omni +# Deploy and legacy stage configurations -In vLLM-Omni, the target model is separated into multiple stages, which are processed by different LLMEngines, DiffusionEngines or other types of engines. Depending on different types of stages, such as Autoregressive (AR) stage or Diffusion transformer (DiT) stage, each can choose corresponding schedulers, model workers to load with the Engines in a plug-in fashion. +In vLLM-Omni, a model's `PipelineConfig` defines its fixed stage topology, while a deploy configuration controls how those stages run. Models that have not migrated to this split still use legacy stage configurations with a `stage_args` schema. !!! note Default deploy config YAMLs (for example, `vllm_omni/deploy/qwen2_5_omni.yaml`, `vllm_omni/deploy/qwen3_omni_moe.yaml`, and `vllm_omni/deploy/qwen3_tts.yaml`) are bundled and loaded automatically when neither `--stage-configs-path` nor `--deploy-config` is provided — the model registry resolves the right pipeline + deploy YAML by `model_type`. The bundled defaults have been verified on 1xH100 for Qwen2.5-Omni and 2xH100 for Qwen3-Omni. Models that have not yet migrated to the new schema continue to use the legacy `vllm_omni/model_executor/stage_configs/.yaml` files via `--stage-configs-path`. -## New deploy schema reference +## Deploy configuration schema The new deploy schema lives under `vllm_omni/deploy/` and is paired with a frozen `PipelineConfig` registered by the model's `pipeline.py`. Each deploy YAML has these top-level fields: @@ -82,14 +82,14 @@ stages: input_connectors: {from_stage_0: shm} ``` -### CLI flags introduced in this refactor +### CLI flags | Flag | Description | |------|-------------| -| `--deploy-config PATH` | Load a new-schema deploy YAML. Takes precedence over `--stage-configs-path`. **Optional** — when omitted, the bundled `vllm_omni/deploy/.yaml` is auto-loaded by the model registry. | +| `--deploy-config PATH` | Load a new-schema deploy YAML. It is mutually exclusive with `--stage-configs-path`. **Optional** — when omitted, the bundled `vllm_omni/deploy/.yaml` is auto-loaded by the model registry. | | `--stage-overrides JSON` | Per-stage JSON overrides, e.g. `'{"0":{"gpu_memory_utilization":0.5}}'`. Per-stage values always win over global flags. | | `--async-chunk` / `--no-async-chunk` | Flip the deploy YAML's `async_chunk:` bool. Unset (default) leaves the YAML value in force. | -| `--stage-configs-path` | **Deprecated.** Accepts legacy `stage_args` yamls and (auto-detected) new deploy yamls; emits a deprecation warning. Migrate to `--deploy-config`. To be removed in a follow-up PR. | +| `--stage-configs-path` | **Deprecated.** Load a legacy `stage_args` YAML. New-schema YAMLs passed to this flag are auto-detected for compatibility, but should use `--deploy-config`. It is mutually exclusive with `--deploy-config` and will be removed in a future release. | ### Stage-Based CLI Paradigm @@ -146,7 +146,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \ From highest to lowest: -1. Per-stage flags (`--stage-overrides` JSON, `--stage--` if registered) +1. Per-stage overrides (`--stage-overrides` JSON) 2. Explicit global CLI flags (`--gpu-memory-utilization 0.85`, etc.) 3. Platform section (`platforms.npu.stages`, etc.) on top of the base `stages:` 4. Overlay YAML (via `base_config:`) on top of the base YAML @@ -210,7 +210,7 @@ Effective config per stage after the merge: | 1 | `max_model_len` | `16384` | global CLI | | 2 | (all defaults) | — | base YAML (no overrides apply) | -Therefore, as a core part of vLLM-Omni, the stage configs for a model have several main functions: +Therefore, as a core part of vLLM-Omni, a model's pipeline and deployment configurations have several main functions: - Claim partition of stages and their corresponding class implementation in `model_executor/models`. - The disaggregated configuration for each stage and the communication topology among them. @@ -219,16 +219,16 @@ Therefore, as a core part of vLLM-Omni, the stage configs for a model have sever - Default input parameters. To override specific parameters, explicitly inject the customized configuration schema -in both online and offline instantiation flows. Prioritize the `--deploy-config` flag -when loading the new-schema deploy YAML schemas, reserving the `--stage-configs-path` parameter +in both online and offline instantiation flows. Use the `--deploy-config` flag +when loading a deploy configuration, reserving the `--stage-configs-path` parameter exclusively to maintain compatibility with legacy `stage_args` YAML constructs. Examples: -For offline (Assume necessary dependencies have been imported): +For offline inference (assuming the necessary dependencies have been imported): ```python model_name = "Qwen/Qwen2.5-Omni-7B" -omni = Omni(model=model_name, stage_configs_path="/path/to/custom_stage_configs.yaml") +omni = Omni(model=model_name, deploy_config="/path/to/deploy_config.yaml") ``` For online serving: @@ -242,11 +242,14 @@ Legacy online serving: vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/stage_configs_file ``` !!! important - We are actively iterating on the definition of stage configs, and we welcome all feedbacks from both community users and developers to help us shape the development! + We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers. + +## Legacy stage configuration reference + +The remainder of this page documents the deprecated `stage_args` schema for models that have not migrated to deploy configurations. The following historical Qwen2.5-Omni configuration illustrates that schema; current Qwen2.5-Omni deployments should use `vllm_omni/deploy/qwen2_5_omni.yaml` instead. -Below is a specific example of stage_configs.yaml in Qwen2.5-omni. ```python -# stage config for running qwen2.5-omni with AsyncOmniEngine + Orchestrator runtime. +# Historical stage configuration for Qwen2.5-Omni. stage_args: - stage_id: 0 # mark the unique id for each stage runtime: # The disaggregated configuration @@ -338,7 +341,7 @@ runtime: ``` -## Stage Configuration Arguments +## Legacy stage configuration arguments Each stage in the `stage_args` list contains the following configuration options: diff --git a/docs/contributing/DOCS_GUIDE.md b/docs/contributing/DOCS_GUIDE.md index 100bac674..916f5e60f 100644 --- a/docs/contributing/DOCS_GUIDE.md +++ b/docs/contributing/DOCS_GUIDE.md @@ -67,6 +67,21 @@ docs/ └── stylesheets/ # Custom CSS ``` +## Naming Model Examples + +Offline inference and online serving examples for the same model use the same +directory name and the shared display name in +`examples/model_display_names.yml`. Use these title forms: + +- `# : Offline inference` +- `# : Online serving` + +The documentation generator uses the full title for the page H1 and the shared +display name alone for navigation, and fails the build if a mapped README uses +a different H1. Keep checkpoint identifiers in commands and prose rather than +in the display name, and do not rename an existing example directory solely to +adjust its title because the directory defines its public documentation URL. + ## Publishing Documentation ### GitHub Pages (Recommended) diff --git a/docs/contributing/model/adding_diffusion_model.md b/docs/contributing/model/adding_diffusion_model.md index 6703dd9c7..b38805112 100644 --- a/docs/contributing/model/adding_diffusion_model.md +++ b/docs/contributing/model/adding_diffusion_model.md @@ -638,7 +638,7 @@ mkdir -p examples/online_serving/your_model_name - Script: `examples/offline_inference/your_model_name/end2end.py` - Parse args like BAGEL (`--model`, `--modality`, optional `--image-path`, `--steps`, etc.) - - Use `from vllm_omni.entrypoints.omni import Omni` (or `OmniDiffusion` if your model is diffusion-only) + - Use `from vllm_omni.entrypoints.omni import Omni` for both multi-stage and diffusion-only models - Save outputs (images/audio/video/text) with deterministic filenames (e.g., `output_0_0.png`) - Doc: `examples/offline_inference/your_model_name/README.md` - Include at least one runnable command, e.g.: diff --git a/docs/design/module/dit_module.md b/docs/design/module/dit_module.md index 953e11f7f..02d50181b 100644 --- a/docs/design/module/dit_module.md +++ b/docs/design/module/dit_module.md @@ -470,7 +470,7 @@ class Attention(nn.Module): role="self", role_category=None, ...): # Resolve backend for this role from the active OmniDiffusionConfig config = get_current_diffusion_config_or_none() - attention_config = config.attention_config if config is not None else None + attention_config = config.diffusion_attention_config if config is not None else None attn_backend_cls, spec = get_attn_backend_for_role( role=role, head_size=head_size, @@ -493,7 +493,7 @@ These backends provide the **kernel implementations** for attention computation. #### Backend Selection Mechanism -Selection is driven by `AttentionConfig` on `OmniDiffusionConfig`, which carries a global `default` plus a `per_role` map of `AttentionSpec` entries. A role string is resolved in this order: +Selection is driven by `OmniDiffusionConfig.diffusion_attention_config`, which carries a global `default` plus a `per_role` map of `AttentionSpec` entries. A role string is resolved in this order: ```python def get_attn_backend_for_role(role, head_size, attention_config=None, role_category=None): @@ -909,7 +909,7 @@ def initialize_model_parallel( ``` 1. User Request - └─> OmniDiffusion.generate(prompt) + └─> Omni.generate(prompt) └─> Prepare OmniDiffusionRequest └─> DiffusionEngine.step(requests) diff --git a/docs/features/custom_pipeline.md b/docs/features/custom_pipeline.md index 31334526a..78d680f26 100644 --- a/docs/features/custom_pipeline.md +++ b/docs/features/custom_pipeline.md @@ -10,7 +10,7 @@ This guide demonstrates how to use the newly added features for extending vLLM-O Three main features enable custom pipeline extension: 1. **`WorkerWrapperBase`**: A wrapper class that enables dynamic worker extension with custom functionality -2. **`load_format`**: A parameter that controls how diffusion models are loaded, including support for custom pipelines +2. **`diffusion_load_format`**: A parameter that controls how diffusion models are loaded, including support for custom pipelines 3. **`CustomPipelineWorkerExtension`**: An extension class that enables pipeline re-initialization with custom implementations ## Features @@ -27,15 +27,15 @@ Three main features enable custom pipeline extension: **Location:** `vllm_omni/diffusion/worker/diffusion_worker.py` -### load_format Parameter +### diffusion_load_format parameter -The `load_format` parameter controls how diffusion models are loaded. It supports the following values: +The `diffusion_load_format` parameter controls the initial diffusion model load. The relevant values are: - **`"default"`**: Standard model loading using the model registry (default behavior) -- **`"custom_pipeline"`**: Load a custom pipeline class specified by `custom_pipeline_name` -- **`"dummy"`**: Skip model loading (useful for testing or when pipeline will be initialized separately) +- **`"dummy"`**: Skip the initial model load; use this with `custom_pipeline_args["pipeline_class"]` for the custom pipeline workflow below +- **`"diffusers"`**: Load through the Hugging Face Diffusers adapter -**Location:** `vllm_omni/diffusion/model_loader/diffusers_loader.py` +**Location:** `vllm_omni/diffusion/data.py` (`OmniDiffusionConfig`) ### CustomPipelineWorkerExtension diff --git a/docs/mkdocs/hooks/generate_examples.py b/docs/mkdocs/hooks/generate_examples.py index 81f2bac24..6cb601b96 100644 --- a/docs/mkdocs/hooks/generate_examples.py +++ b/docs/mkdocs/hooks/generate_examples.py @@ -16,8 +16,35 @@ ROOT_DIR_RELATIVE = "../../../../.." EXAMPLE_DIR = ROOT_DIR / "examples" EXAMPLE_DOC_DIR = ROOT_DIR / "docs/user_guide/examples" NAV_FILE = ROOT_DIR / "docs/.nav.yml" +MODEL_DISPLAY_NAMES_FILE = EXAMPLE_DIR / "model_display_names.yml" MAX_INLINE_MATERIAL_SIZE = 8 * 1024 +SERVING_MODE_TITLES = { + "offline_inference": "Offline inference", + "online_serving": "Online serving", +} + + +def load_model_display_names() -> dict[str, str]: + try: + with open(MODEL_DISPLAY_NAMES_FILE, encoding="utf-8") as f: + model_display_names = yaml.safe_load(f) + except (OSError, yaml.YAMLError) as exc: + raise ValueError(f"Failed to load MODEL_DISPLAY_NAMES_FILE at {MODEL_DISPLAY_NAMES_FILE}: {exc}") from exc + + if not isinstance(model_display_names, dict) or not all( + isinstance(key, str) and isinstance(value, str) for key, value in model_display_names.items() + ): + raise ValueError( + "MODEL_DISPLAY_NAMES_FILE " + f"at {MODEL_DISPLAY_NAMES_FILE} must contain a YAML mapping " + "of string model IDs to string display names" + ) + return model_display_names + + +MODEL_DISPLAY_NAMES = load_model_display_names() + def fix_case(text: str) -> str: subs = { @@ -54,6 +81,7 @@ class Example: main_file (Path): The main file in the directory. other_files (list[Path]): list of other files in the directory. title (str): The title of the document. + nav_title (str): The concise title used in navigation. Methods: __post_init__(): Initializes the main_file, other_files, and title attributes. @@ -78,6 +106,16 @@ class Example: def is_code(self) -> bool: return self.main_file.suffix != ".md" + @property + def model_display_name(self) -> str | None: + if self.category not in SERVING_MODE_TITLES: + return None + return MODEL_DISPLAY_NAMES.get(self.path.stem) + + @property + def nav_title(self) -> str: + return self.model_display_name or self.title + def determine_main_file(self) -> Path: """ Determines the main file in the given path. @@ -151,13 +189,25 @@ class Example: return [file for file in self.path.rglob("*") if is_other_file(file)] def determine_title(self) -> str: + model_display_name = self.model_display_name if not self.is_code: # Specify encoding for building on Windows with open(self.main_file, encoding="utf-8") as f: first_line = f.readline().strip() match = re.match(r"^#\s+(?P.+)$", first_line) + if model_display_name: + expected_title = f"{model_display_name}: {SERVING_MODE_TITLES[self.category]}" + actual_title = match.group("title") if match else first_line + if actual_title != expected_title: + raise ValueError( + f"Model example title mismatch in {self.main_file}: " + f"expected '# {expected_title}', got {first_line!r}" + ) + return expected_title if match: return match.group("title") + elif model_display_name: + raise ValueError(f"Mapped model example must use a Markdown README: {self.path}") return fix_case(self.path.stem.replace("_", " ").title()) def fix_relative_links(self, content: str) -> str: @@ -300,7 +350,7 @@ def update_nav_file(examples: list[Example]): for example in category_examples: doc_path = EXAMPLE_DOC_DIR / example.category / f"{example.path.stem}.md" rel_path = doc_path.relative_to(ROOT_DIR / "docs") - category_items.append({example.title: rel_path.as_posix()}) + category_items.append({example.nav_title: rel_path.as_posix()}) if category_items: # Format category name (e.g., "offline_inference" -> "Offline Inference") diff --git a/docs/serving/image_edit_api.md b/docs/serving/image_edit_api.md index f64d37f44..fef28142e 100644 --- a/docs/serving/image_edit_api.md +++ b/docs/serving/image_edit_api.md @@ -252,7 +252,7 @@ Validation errors (missing required fields): ```bash # Check if server is responding -curl -X http://localhost:8000/v1/images/edit \ +curl http://localhost:8000/v1/images/edits \ -F "prompt='test'" ``` diff --git a/docs/serving/image_generation_api.md b/docs/serving/image_generation_api.md index 4702a19c1..12e2935ea 100644 --- a/docs/serving/image_generation_api.md +++ b/docs/serving/image_generation_api.md @@ -138,7 +138,7 @@ Content-Type: application/json | `model` | string | server's model | Model to use (optional, should match server if specified) | | `n` | integer | 1 | Number of images to generate (1-10) | | `size` | string | model defaults | Image dimensions in WxH format (e.g., "1024x1024", "512x512") | -| `response_format` | string | "b64_json" | Response format (only "b64_json" supported) | +| `response_format` | string | "b64_json" | Response format (`"b64_json"` or `"file"`) | | `user` | string | null | User identifier for tracking | #### vllm-omni Extension Parameters diff --git a/docs/user_guide/diffusion/attention_backends.md b/docs/user_guide/diffusion/attention_backends.md index 12446f3fe..50e02971f 100644 --- a/docs/user_guide/diffusion/attention_backends.md +++ b/docs/user_guide/diffusion/attention_backends.md @@ -83,7 +83,7 @@ When constructing `OmniDiffusionConfig` directly: from vllm_omni.diffusion.data import AttentionConfig, AttentionSpec, OmniDiffusionConfig config = OmniDiffusionConfig( - attention_config=AttentionConfig( + diffusion_attention_config=AttentionConfig( default=AttentionSpec(backend="FLASH_ATTN"), per_role={ "cross": AttentionSpec(backend="TORCH_SDPA"), diff --git a/docs/user_guide/diffusion/cpu_offload_diffusion.md b/docs/user_guide/diffusion/cpu_offload_diffusion.md index 765fb9535..d6fcd8944 100644 --- a/docs/user_guide/diffusion/cpu_offload_diffusion.md +++ b/docs/user_guide/diffusion/cpu_offload_diffusion.md @@ -33,7 +33,7 @@ m = Omni(model="Wan-AI/Wan2.2-T2V-A14B-Diffusers", enable_cpu_offload=True) **CLI:** ```bash -vllm-omni serve diffusion Wan-AI/Wan2.2-T2V-A14B-Diffusers --enable-cpu-offload +vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --enable-cpu-offload ``` ### To Support a Model @@ -146,10 +146,10 @@ m = Omni(model="Wan-AI/Wan2.2-I2V-A14B-Diffusers", enable_layerwise_offload=True **CLI:** ```bash # Text-to-video -vllm-omni serve diffusion Wan-AI/Wan2.2-T2V-A14B-Diffusers --enable-layerwise-offload +vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --enable-layerwise-offload # Or image-to-video -vllm-omni serve diffusion Wan-AI/Wan2.2-I2V-A14B-Diffusers --enable-layerwise-offload +vllm serve Wan-AI/Wan2.2-I2V-A14B-Diffusers --omni --enable-layerwise-offload ``` ### To Support a Model diff --git a/docs/user_guide/examples/offline_inference/bagel.md b/docs/user_guide/examples/offline_inference/bagel.md index 0d3498b28..391c41b35 100644 --- a/docs/user_guide/examples/offline_inference/bagel.md +++ b/docs/user_guide/examples/offline_inference/bagel.md @@ -257,7 +257,7 @@ python end2end.py --model ByteDance-Seed/BAGEL-7B-MoT \ | :------- | :--- | :------ | :---------- | | `--deploy-config` | string | `None` | Path to deploy YAML (auto-detected if omitted) | | `--stage-configs-path` | string | `None` | [Deprecated] Legacy path to `stage_args` YAML; prefer `--deploy-config` | -| `--worker-backend` | choice | `process` | `process` or `ray` | +| `--worker-backend` | choice | `multi_process` | `multi_process` or `ray` | | `--ray-address` | string | `None` | Ray cluster address | | `--quantization` | string | `None` | Quantization method (e.g. `fp8`) | | `--log-stats` | flag | `False` | Enable statistics logging | diff --git a/docs/user_guide/examples/offline_inference/glm_image.md b/docs/user_guide/examples/offline_inference/glm_image.md index c6ac6e33f..09d927ff7 100644 --- a/docs/user_guide/examples/offline_inference/glm_image.md +++ b/docs/user_guide/examples/offline_inference/glm_image.md @@ -26,18 +26,22 @@ Stage 0 (AR Model) Stage 1 (Diffusion) ```python from vllm_omni.entrypoints.omni import Omni +from vllm_omni.inputs.data import OmniDiffusionSamplingParams if __name__ == "__main__": omni = Omni(model="zai-org/GLM-Image") outputs = omni.generate( "A photorealistic mountain landscape at sunset", - sampling_params={ - "height": 1024, - "width": 1024, - "num_inference_steps": 50, - "guidance_scale": 1.5, - "seed": 42, - }, + sampling_params_list=[ + omni.default_sampling_params_list[0], + OmniDiffusionSamplingParams( + height=1024, + width=1024, + num_inference_steps=50, + guidance_scale=1.5, + seed=42, + ), + ], ) outputs[0].request_output.images[0].save("output.png") ``` @@ -46,6 +50,7 @@ if __name__ == "__main__": ```python from vllm_omni.entrypoints.omni import Omni +from vllm_omni.inputs.data import OmniDiffusionSamplingParams if __name__ == "__main__": omni = Omni(model="zai-org/GLM-Image") @@ -56,13 +61,16 @@ if __name__ == "__main__": "image": "input.png", }, }, - sampling_params={ - "height": 1024, - "width": 1024, - "num_inference_steps": 50, - "guidance_scale": 1.5, - "seed": 42, - }, + sampling_params_list=[ + omni.default_sampling_params_list[0], + OmniDiffusionSamplingParams( + height=1024, + width=1024, + num_inference_steps=50, + guidance_scale=1.5, + seed=42, + ), + ], ) outputs[0].request_output.images[0].save("output.png") ``` diff --git a/docs/user_guide/examples/offline_inference/mimo_audio.md b/docs/user_guide/examples/offline_inference/mimo_audio.md index 1cba2f77d..a43d8e408 100644 --- a/docs/user_guide/examples/offline_inference/mimo_audio.md +++ b/docs/user_guide/examples/offline_inference/mimo_audio.md @@ -1,4 +1,4 @@ -# MiMo-Audio Offline Inference +# MiMo-Audio: Offline inference Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/mimo_audio>. diff --git a/docs/user_guide/examples/offline_inference/qwen2_5_omni.md b/docs/user_guide/examples/offline_inference/qwen2_5_omni.md index c54976b54..efd14bc1c 100644 --- a/docs/user_guide/examples/offline_inference/qwen2_5_omni.md +++ b/docs/user_guide/examples/offline_inference/qwen2_5_omni.md @@ -1,4 +1,4 @@ -# Qwen2.5-Omni +# Qwen2.5-Omni: Offline inference Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen2_5_omni>. diff --git a/docs/user_guide/examples/offline_inference/qwen3_omni.md b/docs/user_guide/examples/offline_inference/qwen3_omni.md index 2d856f738..3781c1cc8 100644 --- a/docs/user_guide/examples/offline_inference/qwen3_omni.md +++ b/docs/user_guide/examples/offline_inference/qwen3_omni.md @@ -1,4 +1,4 @@ -# Qwen3-Omni +# Qwen3-Omni: Offline inference Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen3_omni>. diff --git a/docs/user_guide/examples/online_serving/aura_omni.md b/docs/user_guide/examples/online_serving/aura_omni.md index 9c66d0df9..8b7f56ac5 100644 --- a/docs/user_guide/examples/online_serving/aura_omni.md +++ b/docs/user_guide/examples/online_serving/aura_omni.md @@ -1,4 +1,4 @@ -# AURA Omni Pipeline +# AURA Omni: Online serving `aura_omni` wires ASR, AURA, and Qwen3-TTS into one vLLM-Omni pipeline: diff --git a/docs/user_guide/examples/online_serving/mimo_audio.md b/docs/user_guide/examples/online_serving/mimo_audio.md index c8752f578..9db59f9e8 100644 --- a/docs/user_guide/examples/online_serving/mimo_audio.md +++ b/docs/user_guide/examples/online_serving/mimo_audio.md @@ -1,4 +1,4 @@ -# Online serving Example of vLLM-Omni for MiMo-Audio +# MiMo-Audio: Online serving Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/mimo_audio>. diff --git a/docs/user_guide/examples/online_serving/qwen2_5_omni.md b/docs/user_guide/examples/online_serving/qwen2_5_omni.md index b3a2c9f2a..d8c48a403 100644 --- a/docs/user_guide/examples/online_serving/qwen2_5_omni.md +++ b/docs/user_guide/examples/online_serving/qwen2_5_omni.md @@ -1,4 +1,4 @@ -# Qwen2.5-Omni +# Qwen2.5-Omni: Online serving Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen2_5_omni>. diff --git a/docs/user_guide/examples/online_serving/qwen3_omni.md b/docs/user_guide/examples/online_serving/qwen3_omni.md index 1973b198b..ec509d597 100644 --- a/docs/user_guide/examples/online_serving/qwen3_omni.md +++ b/docs/user_guide/examples/online_serving/qwen3_omni.md @@ -1,4 +1,4 @@ -# Qwen3-Omni +# Qwen3-Omni: Online serving Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/qwen3_omni>. diff --git a/examples/model_display_names.yml b/examples/model_display_names.yml new file mode 100644 index 000000000..be3f8e381 --- /dev/null +++ b/examples/model_display_names.yml @@ -0,0 +1,12 @@ +# Canonical display names for model examples that have both offline and online pages. +# The docs generator combines these names with the serving mode for each page H1. +aura_omni: AURA Omni +covo_audio: Covo-Audio-Chat +dynin_omni: Dynin-Omni +lance: Lance +mimo_audio: MiMo-Audio +ming_flash_omni: Ming-flash-omni 2.0 +qwen2_5_omni: Qwen2.5-Omni +qwen3_omni: Qwen3-Omni +sensenova_u1: SenseNova-U1 +step_audio2: Step-Audio2 diff --git a/examples/offline_inference/aura_omni/README.md b/examples/offline_inference/aura_omni/README.md index 37fabf1ba..b73976a15 100644 --- a/examples/offline_inference/aura_omni/README.md +++ b/examples/offline_inference/aura_omni/README.md @@ -1,4 +1,4 @@ -# AURA Omni +# AURA Omni: Offline inference This example runs the native AURA Omni pipeline offline: diff --git a/examples/offline_inference/covo_audio/README.md b/examples/offline_inference/covo_audio/README.md index d13783d39..80a822fa2 100644 --- a/examples/offline_inference/covo_audio/README.md +++ b/examples/offline_inference/covo_audio/README.md @@ -1,4 +1,4 @@ -# Covo-Audio-Chat (Offline Inference) +# Covo-Audio-Chat: Offline inference ## Setup diff --git a/examples/offline_inference/dynin_omni/README.md b/examples/offline_inference/dynin_omni/README.md index 0b1cc22f2..19fa8c69e 100644 --- a/examples/offline_inference/dynin_omni/README.md +++ b/examples/offline_inference/dynin_omni/README.md @@ -1,4 +1,4 @@ -# Dynin-Omni Offline End2End Example +# Dynin-Omni: Offline inference This folder contains a unified offline inference entrypoint: diff --git a/examples/offline_inference/lance/README.md b/examples/offline_inference/lance/README.md index d029abdea..f3a422a3b 100644 --- a/examples/offline_inference/lance/README.md +++ b/examples/offline_inference/lance/README.md @@ -1,4 +1,4 @@ -# Lance (ByteDance) — offline inference +# Lance: Offline inference [Lance](https://huggingface.co/bytedance-research/Lance) is a 3B unified autoregressive + diffusion multimodal model on a Qwen2.5-VL backbone. It is diff --git a/examples/offline_inference/mimo_audio/README.md b/examples/offline_inference/mimo_audio/README.md index 5615dea51..60f433a4f 100644 --- a/examples/offline_inference/mimo_audio/README.md +++ b/examples/offline_inference/mimo_audio/README.md @@ -1,4 +1,4 @@ -# MiMo-Audio Offline Inference +# MiMo-Audio: Offline inference This directory contains an offline demo for running MiMo-Audio models with vLLM Omni. It builds task-specific inputs and generates WAV files or text outputs locally. diff --git a/examples/offline_inference/ming_flash_omni/README.md b/examples/offline_inference/ming_flash_omni/README.md index e60efe88b..ee7e93fd0 100644 --- a/examples/offline_inference/ming_flash_omni/README.md +++ b/examples/offline_inference/ming_flash_omni/README.md @@ -1,4 +1,4 @@ -# Ming-flash-omni 2.0 +# Ming-flash-omni 2.0: Offline inference [Ming-flash-omni-2.0](https://github.com/inclusionAI/Ming) is an omni-modal model supporting text, image, video, and audio understanding, with text and speech outputs. diff --git a/examples/offline_inference/qwen2_5_omni/README.md b/examples/offline_inference/qwen2_5_omni/README.md index 292045033..059c10226 100644 --- a/examples/offline_inference/qwen2_5_omni/README.md +++ b/examples/offline_inference/qwen2_5_omni/README.md @@ -1,4 +1,4 @@ -# Qwen2.5-Omni +# Qwen2.5-Omni: Offline inference ## Setup Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. diff --git a/examples/offline_inference/qwen3_omni/README.md b/examples/offline_inference/qwen3_omni/README.md index 0710faa13..2f4ee11cc 100644 --- a/examples/offline_inference/qwen3_omni/README.md +++ b/examples/offline_inference/qwen3_omni/README.md @@ -1,4 +1,4 @@ -# Qwen3-Omni +# Qwen3-Omni: Offline inference ## Setup Please refer to the [stage configuration documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/configuration/stage_configs/) to configure memory allocation appropriately for your hardware setup. diff --git a/examples/offline_inference/sensenova_u1/README.md b/examples/offline_inference/sensenova_u1/README.md index 1e656c22d..9fdc23a00 100644 --- a/examples/offline_inference/sensenova_u1/README.md +++ b/examples/offline_inference/sensenova_u1/README.md @@ -1,4 +1,4 @@ -# SenseNova-U1-8B-MoT +# SenseNova-U1: Offline inference ## Architecture diff --git a/examples/offline_inference/step_audio2/README.md b/examples/offline_inference/step_audio2/README.md index 2ba69ceb5..84d0ecdde 100644 --- a/examples/offline_inference/step_audio2/README.md +++ b/examples/offline_inference/step_audio2/README.md @@ -1,4 +1,4 @@ -# Step-Audio2 Offline Inference Examples +# Step-Audio2: Offline inference This directory contains examples for running offline inference with Step-Audio2 using vLLM-Omni. diff --git a/examples/online_serving/aura_omni/README.md b/examples/online_serving/aura_omni/README.md index 6abd0ef47..5f246c336 100644 --- a/examples/online_serving/aura_omni/README.md +++ b/examples/online_serving/aura_omni/README.md @@ -1,4 +1,4 @@ -# AURA Omni Native Pipeline +# AURA Omni: Online serving `aura_omni` serves AURA as a native multi-stage vLLM-Omni pipeline: diff --git a/examples/online_serving/covo_audio/README.md b/examples/online_serving/covo_audio/README.md index 6894507ef..b6a4ad74a 100644 --- a/examples/online_serving/covo_audio/README.md +++ b/examples/online_serving/covo_audio/README.md @@ -1,4 +1,4 @@ -# Covo-Audio-Chat +# Covo-Audio-Chat: Online serving ## Installation diff --git a/examples/online_serving/dynin_omni/README.md b/examples/online_serving/dynin_omni/README.md index 50e2f3f80..6e78a5c1f 100644 --- a/examples/online_serving/dynin_omni/README.md +++ b/examples/online_serving/dynin_omni/README.md @@ -1,4 +1,4 @@ -# Dynin-Omni Online Serving Example +# Dynin-Omni: Online serving ## Installation diff --git a/examples/online_serving/lance/README.md b/examples/online_serving/lance/README.md index 5f2febce9..614f4990c 100644 --- a/examples/online_serving/lance/README.md +++ b/examples/online_serving/lance/README.md @@ -1,4 +1,4 @@ -# Lance — online serving +# Lance: Online serving OpenAI-compatible chat completions API for Lance, served by `vllm-omni`. diff --git a/examples/online_serving/mimo_audio/README.md b/examples/online_serving/mimo_audio/README.md index 9c1be7f21..7c5e2db0f 100644 --- a/examples/online_serving/mimo_audio/README.md +++ b/examples/online_serving/mimo_audio/README.md @@ -1,4 +1,4 @@ -# Online serving Example of vLLM-Omni for MiMo-Audio +# MiMo-Audio: Online serving ## 🛠️ Installation diff --git a/examples/online_serving/ming_flash_omni/README.md b/examples/online_serving/ming_flash_omni/README.md index 9810841db..377acc566 100644 --- a/examples/online_serving/ming_flash_omni/README.md +++ b/examples/online_serving/ming_flash_omni/README.md @@ -1,4 +1,4 @@ -# Ming-flash-omni 2.0 +# Ming-flash-omni 2.0: Online serving ## Installation diff --git a/examples/online_serving/qwen2_5_omni/README.md b/examples/online_serving/qwen2_5_omni/README.md index c52873206..48406003d 100644 --- a/examples/online_serving/qwen2_5_omni/README.md +++ b/examples/online_serving/qwen2_5_omni/README.md @@ -1,4 +1,4 @@ -# Qwen2.5-Omni +# Qwen2.5-Omni: Online serving ## 🛠️ Installation diff --git a/examples/online_serving/qwen3_omni/README.md b/examples/online_serving/qwen3_omni/README.md index 5e2e24a1f..3c3ad2b16 100644 --- a/examples/online_serving/qwen3_omni/README.md +++ b/examples/online_serving/qwen3_omni/README.md @@ -1,4 +1,4 @@ -# Qwen3-Omni +# Qwen3-Omni: Online serving ## 🛠️ Installation diff --git a/examples/online_serving/sensenova_u1/README.md b/examples/online_serving/sensenova_u1/README.md index d66a3506a..d529eedad 100644 --- a/examples/online_serving/sensenova_u1/README.md +++ b/examples/online_serving/sensenova_u1/README.md @@ -1,4 +1,4 @@ -# SenseNova-U1 Online Serving +# SenseNova-U1: Online serving ## Launch the Server diff --git a/examples/online_serving/step_audio2/README.md b/examples/online_serving/step_audio2/README.md index 3a04f4565..e93f58ccc 100644 --- a/examples/online_serving/step_audio2/README.md +++ b/examples/online_serving/step_audio2/README.md @@ -1,4 +1,4 @@ -# Step-Audio2 Online Serving +# Step-Audio2: Online serving This directory contains examples for running Step-Audio2 with vLLM-Omni's online serving API. diff --git a/tests/docs/test_generate_examples.py b/tests/docs/test_generate_examples.py new file mode 100644 index 000000000..0b29837a9 --- /dev/null +++ b/tests/docs/test_generate_examples.py @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright contributors to the vLLM project + +import importlib.util +from pathlib import Path + +import pytest + +ROOT_DIR = Path(__file__).parents[2] +HOOK_PATH = ROOT_DIR / "docs/mkdocs/hooks/generate_examples.py" +SPEC = importlib.util.spec_from_file_location("generate_examples", HOOK_PATH) +assert SPEC and SPEC.loader +generate_examples = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(generate_examples) + + +def test_model_example_titles_match_shared_display_names(): + for slug, display_name in generate_examples.MODEL_DISPLAY_NAMES.items(): + for category, mode_title in generate_examples.SERVING_MODE_TITLES.items(): + example_path = ROOT_DIR / "examples" / category / slug + example = generate_examples.Example(example_path, category) + + assert example.title == f"{display_name}: {mode_title}" + assert example.nav_title == display_name + + +def test_model_example_title_mismatch_is_rejected(tmp_path): + example_path = tmp_path / "mimo_audio" + example_path.mkdir() + (example_path / "README.md").write_text("# A different title\n", encoding="utf-8") + + with pytest.raises(ValueError, match="Model example title mismatch"): + generate_examples.Example(example_path, "offline_inference") + + +def test_model_display_names_require_string_mapping(tmp_path, monkeypatch): + display_names_path = tmp_path / "model_display_names.yml" + display_names_path.write_text("- not a mapping\n", encoding="utf-8") + monkeypatch.setattr(generate_examples, "MODEL_DISPLAY_NAMES_FILE", display_names_path) + + with pytest.raises(ValueError, match="MODEL_DISPLAY_NAMES_FILE"): + generate_examples.load_model_display_names()