Compare commits

...
Author SHA1 Message Date
hsliuustc0106 765caa2a12 [Doc] Address README review feedback
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
2026-08-03 23:46:43 +08:00
hsliuustc0106 75fdab6b35 [Doc] Align v0.26 release documentation
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
2026-08-03 23:46:43 +08:00
hsliuustc0106 e261962661 [Doc] Update README for v0.26.0
Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>
2026-08-03 23:46:43 +08:00
Yukim1 7a2007cc17 [Refactor] Remove legacy --stage-configs-path from the serve CLI (#5647)
Signed-off-by: zwhzzz0821 <2831474076@qq.com>
2026-08-03 23:41:52 +08:00
WeiQing Chen 76de642c54 Update WeChat community QR code (#5701)
Signed-off-by: David Chen <530634352@qq.com>
2026-08-03 17:56:16 +08:00
36 changed files with 135 additions and 125 deletions

No files matched your search

+5 -8
View File
@@ -16,14 +16,10 @@ Easy, fast, and cheap omni-modality model serving for everyone
--- ---
*Latest News* 🔥 *Latest News* 🔥
- [2026/08] We released [0.26.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.26.0) - aligned with the vLLM 0.26 release line, featuring [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) joint video/audio generation, an experimental full-duplex realtime runtime for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md), distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
- [2026/07] We released [0.24.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.24.0) - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support. - [2026/07] We released [0.24.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.24.0) - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
- [2026/06] We released [0.22.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.22.0) - an **omnimodal world-model** release aligned with vLLM 0.22, featuring [Nvidia Cosmos3](recipes/cosmos3/Cosmos3-Nano.md)/DreamZero world model support, expanded quantization coverage across Blackwell/NPU/XPU, TTS production improvements, new models including MiniCPM-o 4.5, MOSS-TTS, and Lance, plus RL integration with [VeRL-Omni](https://github.com/verl-project/verl-omni). - [2026/06] Starting with [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0), vLLM-Omni publishes a stable release aligned with every even-numbered upstream vLLM minor version. [0.16.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.16.0), [0.18.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.18.0), [0.20.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.20.0), and [0.22.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.22.0) continued this cadence, expanding omni and world-model support with [NVIDIA Cosmos3](recipes/cosmos3/Cosmos3-Nano.md) and DreamZero, adding models such as MiniCPM-o 4.5, MOSS-TTS, and Lance, and advancing TTS, diffusion, distributed execution, quantization, RL integration through [VeRL-Omni](https://github.com/verl-project/verl-omni), and CUDA/ROCm/MUSA/NPU/XPU coverage.
- [2026/05] We released [0.20.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.20.0) - refreshes the serving/runtime stack for large-scale omni workloads, and improves diffusion model performance, quantization, and hardware readiness across CUDA, ROCm, MUSA, NPU, and XPU backends.
- [2026/03] We released [0.18.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.18.0) - strengthens the core runtime through a large entrypoint refactor and scheduler/runtime cleanups, expands unified quantization and diffusion execution, broadens multimodal model coverage, and improves production readiness across audio, omni, image, video, RL, and multi-platform deployments.
- [2026/03] Check out our first public [project deepdive](https://youtu.be/sgwNfsNnR9I) at the vLLM Hong Kong Meetup! - [2026/03] Check out our first public [project deepdive](https://youtu.be/sgwNfsNnR9I) at the vLLM Hong Kong Meetup!
- [2026/03] **[vllm-omni-skills](https://github.com/hsliuustc0106/vllm-omni-skills)** is a community-driven collection of AI assistant skills that help developers work with vLLM-Omni more effectively. These skills can be used with popular agentic AI coding assistants like **Cursor IDE**, **Claude**, **Codex**, and more.
- [2026/02] We released [0.16.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.16.0) - A major alignment + capability release that rebases onto **upstream vLLM v0.16.0** and significantly expands performance, distributed execution, and production readiness across **Qwen3-Omni / Qwen3-TTS**, **Bagel**, **MiMo-Audio**, **GLM-Image** and the **Diffusion (DiT) image/video stack**—while also improving platform coverage (CUDA / ROCm / NPU / XPU), CI quality, and documentation.
- [2026/02] We released [0.14.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.14.0) - This is the first **stable release** of vLLM-Omni that expands Omni’s diffusion / image-video generation and audio / TTS stack, improves distributed execution and memory efficiency, and broadens platform/backend coverage (GPU/ROCm/NPU/XPU). It also brings meaningful upgrades to serving APIs, profiling & benchmarking, and overall stability. Please check our latest [paper](https://arxiv.org/abs/2602.02204) for architecture design and performance results.
- [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving. - [2025/11] vLLM community officially released [vllm-project/vllm-omni](https://github.com/vllm-project/vllm-omni) in order to support omni-modality models serving.
--- ---
@@ -55,12 +51,13 @@ vLLM-Omni is flexible and easy to use with:
- Tensor, pipeline, data and expert parallelism support for distributed inference - Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs - Streaming outputs
- OpenAI-compatible API server - OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output (experimental)
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including: vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- **Omni-modality models** (e.g. Qwen3-Omni, Cosmos3, HunyuanImage, BAGEL) - **Omni-modality models** (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- **TTS models** (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3) - **TTS models** (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
- **Diffusion models** — image, video, and audio generation (e.g. Qwen-Image, Wan2.2, FLUX) - **Diffusion models** — image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX)
- **Robot-policy and action models** (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy) - **Robot-policy and action models** (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
## Getting Started ## Getting Started
+3 -2
View File
@@ -52,12 +52,13 @@ vLLM-Omni is flexible and easy to use with:
- Tensor, pipeline, data and expert parallelism support for distributed inference - Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs - Streaming outputs
- OpenAI-compatible API server - OpenAI-compatible API server
- Full-duplex realtime serving with streaming audio input and output (experimental)
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including: vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
- Omni-modality models (e.g. Qwen3-Omni, Cosmos3, HunyuanImage, BAGEL) - Omni-modality models (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
- TTS models (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3) - TTS models (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
- Diffusion models — image, video, and audio generation (e.g. Qwen-Image, Wan2.2, FLUX) - Diffusion models — image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX)
- Robot-policy and action models (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy) - Robot-policy and action models (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
For more information, checkout the following: For more information, checkout the following:
Binary file not shown.

Before

Width:  |  Height:  |  Size: 281 KiB

After

Width:  |  Height:  |  Size: 274 KiB

+1 -7
View File
@@ -18,18 +18,12 @@ Specify the port:
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
``` ```
For a migrated model, load a custom deploy configuration with `--deploy-config`: Load a custom deploy configuration with `--deploy-config`:
```bash ```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --deploy-config /path/to/deploy_config.yaml vllm serve Qwen/Qwen2.5-Omni-7B --omni --deploy-config /path/to/deploy_config.yaml
``` ```
The deprecated `--stage-configs-path` flag is retained for models that still use the legacy `stage_args` schema:
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --stage-configs-path /path/to/legacy_stage_config.yaml
```
## bench ## bench
Run benchmark tests for online serving throughput. Run benchmark tests for online serving throughput.
+1 -2
View File
@@ -8,7 +8,6 @@ The stage-based CLI is designed for deployments that require launching each pipe
- For **migrated models** that utilize the bundled deployment YAML configurations located in - For **migrated models** that utilize the bundled deployment YAML configurations located in
`vllm_omni/deploy/`, the `--deploy-config` flag is only required to override the default configuration. By default, executing `vllm serve MODEL --omni ...` `vllm_omni/deploy/`, the `--deploy-config` flag is only required to override the default configuration. By default, executing `vllm serve MODEL --omni ...`
automatically loads the bundled deployment configuration. automatically loads the bundled deployment configuration.
- For custom legacy `stage_args` YAMLs, pass the file with `--stage-configs-path`.
Example: Initializing Stage 0 (Orchestrator and API Server): Example: Initializing Stage 0 (Orchestrator and API Server):
The commands below show a common device mapping where Stage 0 uses GPU 0 and The commands below show a common device mapping where Stage 0 uses GPU 0 and
@@ -32,7 +31,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--omni-master-port 26000 --omni-master-port 26000
``` ```
When utilizing a custom deployment YAML based on the new schema, append `--deploy-config /path/to/override.yaml` to each command execution. Conversely, for legacy models, substitute this parameter with `--stage-configs-path /path/to/stage_configs.yaml`. When utilizing a custom deployment YAML, append `--deploy-config /path/to/override.yaml` to each command execution.
In the standard execution paradigm, the `--stage-overrides` argument is utilized to apply stage-specific configurations from a single CLI command. In the standard execution paradigm, the `--stage-overrides` argument is utilized to apply stage-specific configurations from a single CLI command.
However, under the **stage-based CLI** paradigm, where each process strictly encapsulates a single stage, it is recommended to specify tuning parameters directly via discrete command-line flags for the respective stage, rather than constructing a composite `--stage-overrides` JSON string. However, under the **stage-based CLI** paradigm, where each process strictly encapsulates a single stage, it is recommended to specify tuning parameters directly via discrete command-line flags for the respective stage, rather than constructing a composite `--stage-overrides` JSON string.
+1 -1
View File
@@ -2,7 +2,7 @@
This section lists the most common options for running vLLM-Omni. This section lists the most common options for running vLLM-Omni.
For options within a vLLM Engine. Please refer to [vLLM Configuration](https://docs.vllm.ai/en/v0.16.0/configuration/index.html) For options within a vLLM Engine, please refer to the [vLLM 0.26 configuration guide](https://docs.vllm.ai/en/v0.26.0/configuration/index.html).
Currently, the main options are maintained by stage configs for each model. Currently, the main options are maintained by stage configs for each model.
-15
View File
@@ -34,12 +34,8 @@ Two additional `StrategySpec` slots — `layer_hook_specs` and `kernel_specs`
| Flag | Description | | Flag | Description |
|------|-------------| |------|-------------|
| `--strategy-config PATH` | Path to a `strategy.yaml`. Only takes effect on registry-based deploy paths (`--deploy-config` or the bundled default deploy YAML). Its derived sizing is overlaid onto the merged stages *before* CLI overrides. | | `--strategy-config PATH` | Path to a `strategy.yaml`. Only takes effect on registry-based deploy paths (`--deploy-config` or the bundled default deploy YAML). Its derived sizing is overlaid onto the merged stages *before* CLI overrides. |
| `--stage-configs-path PATH` | **Legacy.** Loads the old `stage_args` YAML schema for un-migrated models. Passing `--strategy-config` together with this path is silently inapplicable — vLLM-Omni emits a warning and the strategy is ignored (see `vllm_omni/entrypoints/utils.py`, lines 372–380). |
| `--omni-lb-policy POLICY` | Orchestrator-wide load-balancer policy. Strategy-derived from any `stage_replica` axis; an explicit CLI value wins and must match the derived one or `AsyncOmniEngine` raises `Conflicting load-balancer policy` at construction. | | `--omni-lb-policy POLICY` | Orchestrator-wide load-balancer policy. Strategy-derived from any `stage_replica` axis; an explicit CLI value wins and must match the derived one or `AsyncOmniEngine` raises `Conflicting load-balancer policy` at construction. |
!!! warning
`--strategy-config` cannot be used with `--stage-configs-path` (the legacy deploy path). If your model has not been migrated to the registry-based pipeline yet, the strategy is silently dropped with a warning. Migrate to a registry-based model or use `--deploy-config` to apply a strategy.
## Sub-schemas ## Sub-schemas
### `axis` ### `axis`
@@ -216,17 +212,6 @@ Strategy keys MUST be `model_stage` *names* (strings), the same labels stage con
!!! warning "Mixing CLI overrides with a strategy on the same axis" !!! warning "Mixing CLI overrides with a strategy on the same axis"
A strategy is meant to be the single writer for the axes it declares. If a CLI override (`--tensor-parallel-size 2`, `--stage-overrides '{"0": {"tensor_parallel_size": 2}}'`) sets the same axis, the **CLI value wins** and a warning is emitted from `_reconcile_strategy_with_cli`. The device-layout guard then re-runs against the effective layout, so a CLI override that breaks `tp * dp * pp * num_replicas` will still fail fast at config time. A strategy is meant to be the single writer for the axes it declares. If a CLI override (`--tensor-parallel-size 2`, `--stage-overrides '{"0": {"tensor_parallel_size": 2}}'`) sets the same axis, the **CLI value wins** and a warning is emitted from `_reconcile_strategy_with_cli`. The device-layout guard then re-runs against the effective layout, so a CLI override that breaks `tp * dp * pp * num_replicas` will still fail fast at config time.
!!! warning "Passing `--strategy-config` against the legacy `--stage-configs-path`"
Composable parallel only applies on the registry-based deploy path. If a model still resolves through the legacy `stage_args` YAML (loaded via `--stage-configs-path`), `--strategy-config` is silently inapplicable and emits:
```text
--strategy-config (<path>) was provided but model '<model>' resolves via the
legacy stage_configs YAML path, which does not support composable-parallel
strategies; the strategy is ignored. Use a registry-based model to apply it.
```
Migrate the model to the new deploy schema (see `configuration/stage_configs.md`) before applying a strategy.
!!! warning "Conflicting `omni_lb_policy` across stages" !!! warning "Conflicting `omni_lb_policy` across stages"
`omni_lb_policy` is a single pipeline-wide knob. If two `stage_replica` axes in different stages derive different policies, `apply_strategy_specs` raises: `omni_lb_policy` is a single pipeline-wide knob. If two `stage_replica` axes in different stages derive different policies, `apply_strategy_specs` raises:
+1 -1
View File
@@ -158,7 +158,7 @@ runtime:
```bash ```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091 \
--stage-configs-path /path/to/qwen3_omni_pd.yaml --deploy-config /path/to/qwen3_omni_pd.yaml
``` ```
## Operational Notes ## Operational Notes
+6 -14
View File
@@ -1,9 +1,9 @@
# Deploy and legacy stage configurations # Pipeline and deploy configurations
In vLLM-Omni, a model's `PipelineConfig` defines its fixed stage topology, while a deploy configuration controls how those stages run. Models that have not migrated to this split still use legacy stage configurations with a `stage_args` schema. In vLLM-Omni, a model's `PipelineConfig` defines its fixed stage topology, while a deploy configuration controls how those stages run.
!!! note !!! note
Default deploy config YAMLs (for example, `vllm_omni/deploy/qwen2_5_omni.yaml`, `vllm_omni/deploy/qwen3_omni_moe.yaml`, and `vllm_omni/deploy/qwen3_tts.yaml`) are bundled and loaded automatically when neither `--stage-configs-path` nor `--deploy-config` is provided — the model registry resolves the right pipeline + deploy YAML by `model_type`. Custom legacy `stage_args` YAMLs are still accepted via `--stage-configs-path`. Default deploy config YAMLs (for example, `vllm_omni/deploy/qwen2_5_omni.yaml`, `vllm_omni/deploy/qwen3_omni_moe.yaml`, and `vllm_omni/deploy/qwen3_tts.yaml`) are bundled and loaded automatically when `--deploy-config` is omitted. The model registry resolves the right pipeline and deploy YAML by `model_type`.
## Deploy configuration schema ## Deploy configuration schema
@@ -86,10 +86,9 @@ stages:
| Flag | Description | | Flag | Description |
|------|-------------| |------|-------------|
| `--deploy-config PATH` | Load a new-schema deploy YAML. It is mutually exclusive with `--stage-configs-path`. **Optional** — when omitted, the bundled `vllm_omni/deploy/<model_type>.yaml` is auto-loaded by the model registry. | | `--deploy-config PATH` | Load a deploy YAML. **Optional** — when omitted, the bundled `vllm_omni/deploy/<model_type>.yaml` is auto-loaded by the model registry. |
| `--stage-overrides JSON` | Per-stage JSON overrides, e.g. `'{"0":{"gpu_memory_utilization":0.5}}'`. Per-stage values always win over global flags. | | `--stage-overrides JSON` | Per-stage JSON overrides, e.g. `'{"0":{"gpu_memory_utilization":0.5}}'`. Per-stage values always win over global flags. |
| `--async-chunk` / `--no-async-chunk` | Flip the deploy YAML's `async_chunk:` bool. Unset (default) leaves the YAML value in force. | | `--async-chunk` / `--no-async-chunk` | Flip the deploy YAML's `async_chunk:` bool. Unset (default) leaves the YAML value in force. |
| `--stage-configs-path` | **Deprecated.** Load a legacy `stage_args` YAML. New-schema YAMLs passed to this flag are auto-detected for compatibility, but should use `--deploy-config`. It is mutually exclusive with `--deploy-config` and will be removed in a future release. |
### Stage-Based CLI Paradigm ### Stage-Based CLI Paradigm
@@ -119,8 +118,7 @@ CUDA_VISIBLE_DEVICES=1 vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --omni \
--omni-master-port 26000 --omni-master-port 26000
``` ```
When instantiating a custom deployment YAML conforming to the updated schema, append the `--deploy-config /path/to/override.yaml` directive When instantiating a custom deployment YAML, append the `--deploy-config /path/to/override.yaml` directive to all node invocations.
to all node invocations. For legacy architectures (e.g., BAGEL) configured via deprecated `stage_args:` schemas, continue to specify the relevant configuration via `--stage-configs-path /path/to/config.yaml`.
In the context of standard initialization architectures, utilizing the `--stage-overrides` parameter operates as the optimal methodology In the context of standard initialization architectures, utilizing the `--stage-overrides` parameter operates as the optimal methodology
for delineating stage-specific tuning from the CLI interface: for delineating stage-specific tuning from the CLI interface:
@@ -220,8 +218,7 @@ Therefore, as a core part of vLLM-Omni, a model's pipeline and deployment config
To override specific parameters, explicitly inject the customized configuration schema To override specific parameters, explicitly inject the customized configuration schema
in both online and offline instantiation flows. Use the `--deploy-config` flag in both online and offline instantiation flows. Use the `--deploy-config` flag
when loading a deploy configuration, reserving the `--stage-configs-path` parameter when loading a deploy configuration.
exclusively to maintain compatibility with legacy `stage_args` YAML constructs.
Examples: Examples:
@@ -236,11 +233,6 @@ For online serving:
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml
``` ```
Legacy online serving:
```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 --stage-configs-path /path/to/stage_configs_file
```
!!! important !!! important
We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers. We are actively iterating on the definition of deployment configurations, and we welcome feedback from users and developers.
@@ -653,7 +653,7 @@ python end2end.py --model your-org/your-model-name --modality text2img --prompts
Mirror BAGEL’s online serving setup: Mirror BAGEL’s online serving setup:
- Server launcher: `examples/online_serving/your_model_name/run_server.sh` - Server launcher: `examples/online_serving/your_model_name/run_server.sh`
- Wrap `vllm serve ... --omni --port ...` (and `--stage-configs-path ...` if needed) - Wrap `vllm serve ... --omni --port ...` (and `--deploy-config ...` if needed)
- Client: `examples/online_serving/your_model_name/openai_chat_client.py` - Client: `examples/online_serving/your_model_name/openai_chat_client.py`
- Send requests to `POST /v1/chat/completions` - Send requests to `POST /v1/chat/completions`
- Support multimodal inputs (e.g., base64 image) if your model needs it - Support multimodal inputs (e.g., base64 image) if your model needs it
+2 -2
View File
@@ -48,8 +48,8 @@ vllm_omni/deploy/
``` ```
New in-tree models should register their `PipelineConfig` in New in-tree models should register their `PipelineConfig` in
`vllm_omni/config/pipeline_registry.py`; use `--stage-configs-path` only for `vllm_omni/config/pipeline_registry.py`; use `--deploy-config` for custom
custom legacy `stage_args` YAMLs. deployment YAMLs.
## Step-by-Step Implementation ## Step-by-Step Implementation
+2 -3
View File
@@ -449,9 +449,8 @@ Key configuration fields:
!!! note !!! note
New in-tree models should define frozen topology in `models/<model>/pipeline.py`, New in-tree models should define frozen topology in `models/<model>/pipeline.py`,
register it in `vllm_omni/config/pipeline_registry.py`, and put deployment register it in `vllm_omni/config/pipeline_registry.py`, and put deployment
knobs in `vllm_omni/deploy/<model>.yaml`. Legacy `stage_args` YAMLs are still knobs in `vllm_omni/deploy/<model>.yaml`. Use `--deploy-config` to load a
accepted for custom configs via `--stage-configs-path`, but should not be used custom deployment YAML.
for new bundled defaults.
### Batch mode ### Batch mode
@@ -419,8 +419,7 @@ Notes:
```bash ```bash
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \ vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct \
--omni \ --omni \
--port 8091 \ --port 8091
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_omni_moe_async_chunk.yaml
``` ```
#### 3) Key config knobs #### 3) Key config knobs
@@ -494,7 +493,7 @@ The default config (`qwen3_tts.yaml`) enables the full optimization stack:
vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \ vllm-omni serve Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
--omni \ --omni \
--port 8000 \ --port 8000 \
--stage-configs-path vllm_omni/model_executor/stage_configs/qwen3_tts_no_async_chunk.yaml --no-async-chunk
``` ```
#### 3) Key config knobs #### 3) Key config knobs
@@ -1,5 +1,8 @@
# Installation # Installation
!!! important
vLLM-Omni is released against the matching upstream vLLM major/minor version. vLLM-Omni 0.26.x requires vLLM 0.26.x; the stable v0.26.0 instructions pin vLLM 0.26.0.
vLLM-Omni supports the following hardware platforms: vLLM-Omni supports the following hardware platforms:
- [GPU](gpu.md) - [GPU](gpu.md)
@@ -5,7 +5,7 @@
# --8<-- [end:requirements] # --8<-- [end:requirements]
# --8<-- [start:set-up-using-python] # --8<-- [start:set-up-using-python]
vLLM-Omni depends vLLM. So please follow instructions below mainly for vLLM. vLLM-Omni depends on the matching major/minor release of vLLM. The vLLM-Omni 0.26.x release line uses vLLM 0.26.x.
!!! note !!! note
PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details. PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <gh-issue:8420> for more details.
@@ -124,7 +124,7 @@ If you want to specify the base vLLM version:
```bash ```bash
DOCKER_BUILDKIT=1 docker build \ DOCKER_BUILDKIT=1 docker build \
-f docker/Dockerfile.cuda \ -f docker/Dockerfile.cuda \
--build-arg BASE_IMAGE=vllm/vllm-openai:v0.22.1 \ --build-arg BASE_IMAGE=vllm/vllm-openai:v0.26.0 \
-t vllm-omni-cuda . -t vllm-omni-cuda .
``` ```
@@ -53,16 +53,20 @@ We are keeping [issue #886](https://github.com/vllm-project/vllm-omni/issues/886
You can also build vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. (But sometimes it will break for a while. You can check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest commit of vLLM-Omni main branch on NPU.) You can also build vLLM-Omni from the latest main branch if you want to use the latest features or bug fixes. (But sometimes it will break for a while. You can check [issue #886](https://github.com/vllm-project/vllm-omni/issues/886) for the status of the latest commit of vLLM-Omni main branch on NPU.)
```bash ```bash
# Pin vLLM version to 0.18.0 # Pin vLLM and vLLM-Ascend to the v0.26 release line
git clone -b v0.18.0 https://github.com/vllm-project/vllm.git git clone -b v0.26.0 https://github.com/vllm-project/vllm.git
cd vllm
VLLM_TARGET_DEVICE=empty pip install -v -e . VLLM_TARGET_DEVICE=empty pip install -v -e .
cd ..
git clone -b v0.18.0rc1 https://github.com/vllm-project/vllm-ascend.git git clone -b releases/v0.26.0rc https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
pip install -v -e . pip install -v -e .
cd ..
# Install vLLM-Omni from the latest main branch # Install vLLM-Omni from the latest main branch
git clone https://github.com/vllm-project/vllm-omni.git git clone https://github.com/vllm-project/vllm-omni.git
cd /vllm-workspace/vllm-omni cd vllm-omni
pip install -v -e . --no-build-isolation pip install -v -e . --no-build-isolation
# or VLLM_OMNI_TARGET_DEVICE=npu pip install -v -e . # or VLLM_OMNI_TARGET_DEVICE=npu pip install -v -e .
export VLLM_WORKER_MULTIPROC_METHOD=spawn export VLLM_WORKER_MULTIPROC_METHOD=spawn
@@ -40,7 +40,7 @@ bash run_server_stage_cli.sh --stage 0
bash run_server_stage_cli.sh --stage 1 bash run_server_stage_cli.sh --stage 1
``` ```
To use a custom deploy YAML (note: `--stage-configs-path` is deprecated in favor of `--deploy-config`): To use a custom deploy YAML:
```bash ```bash
vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 \ vllm serve ByteDance-Seed/BAGEL-7B-MoT --omni --port 8091 \
@@ -11,8 +11,8 @@ memory, sampling params) live in `vllm_omni/deploy/glm_image.yaml`.
vllm serve zai-org/GLM-Image --omni --port 8091 vllm serve zai-org/GLM-Image --omni --port 8091
``` ```
The config system auto-detects the pipeline from the model's `model_index.json` — no The config system auto-detects the pipeline from the model's `model_index.json`, so
manual `--stage-configs-path` or `--deploy-config` needed. no manual `--deploy-config` is needed.
By default, stage 0 (AR) runs on GPU 0 and stage 1 (Diffusion) on GPU 1. To colocate By default, stage 0 (AR) runs on GPU 0 and stage 1 (Diffusion) on GPU 1. To colocate
both stages on a single GPU, override per stage: both stages on a single GPU, override per stage:
@@ -15,9 +15,9 @@ Please refer to [README.md](https://github.com/vllm-project/vllm-omni/tree/main/
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
``` ```
If you have custom stage configs file, launch the server with command below To use a custom deploy config, launch the server with the command below:
```bash ```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml
``` ```
### Send Multi-modal Request ### Send Multi-modal Request
@@ -182,7 +182,7 @@ The script supports the following arguments:
- `--model`: Model name/path (default: Qwen/Qwen2.5-Omni-7B) - `--model`: Model name/path (default: Qwen/Qwen2.5-Omni-7B)
- `--server-port`: Port for vLLM server (default: 8091) - `--server-port`: Port for vLLM server (default: 8091)
- `--gradio-port`: Port for Gradio demo (default: 7861) - `--gradio-port`: Port for Gradio demo (default: 7861)
- `--stage-configs-path`: Path to custom stage configs YAML file (optional) - `--deploy-config`: Path to a custom deploy config YAML file (optional)
- `--server-host`: Host for vLLM server (default: 0.0.0.0) - `--server-host`: Host for vLLM server (default: 0.0.0.0)
- `--gradio-ip`: IP for Gradio demo (default: 127.0.0.1) - `--gradio-ip`: IP for Gradio demo (default: 127.0.0.1)
- `--share`: Share Gradio demo publicly (creates a public link) - `--share`: Share Gradio demo publicly (creates a public link)
@@ -195,9 +195,9 @@ The script supports the following arguments:
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
``` ```
If you have custom stage configs file: To use a custom deploy config:
```bash ```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml
``` ```
**Step 2: Run the Gradio demo** **Step 2: Run the Gradio demo**
@@ -69,19 +69,19 @@ The behaviors and conclusions above are summarized from current fault injection
### Example: `SIGTERM` fault injection log (Qwen3-Omni) ### Example: `SIGTERM` fault injection log (Qwen3-Omni)
The following full log is from a real run where a `SIGTERM` signal was sent to the serve root process. The following log is based on a real run where a `SIGTERM` signal was sent to the serve root process. Its command line has been normalized to the current `--deploy-config` option.
```text ```text
Launching OmniServer with: /workspace/.venv/bin/python3 -m vllm_omni.entrypoints.cli.main serve Qwen/Qwen3-Omni-30B-A3B-Instruct --host 127.0.0.1 --port 60675 --omni --async-chunk --stage-init-timeout 600 --init-timeout 900 --log-stats --stage-configs-path vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml Launching OmniServer with: /workspace/.venv/bin/python3 -m vllm_omni.entrypoints.cli.main serve Qwen/Qwen3-Omni-30B-A3B-Instruct --host 127.0.0.1 --port 60675 --omni --async-chunk --stage-init-timeout 600 --init-timeout 900 --log-stats --deploy-config vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml
Server ready on 127.0.0.1:60675 (OmniServer startup took 206.023s) Server ready on 127.0.0.1:60675 (OmniServer startup took 206.023s)
OmniServer started successfully OmniServer started successfully
[reliability][process-kill] current_server_proc pid=556855 name=python3 cmdline=/workspace/.venv/bin/python3 -m vllm_omni.entrypoints.cli.main serve Qwen/Qwen3-Omni-30B-A3B-Instruct --host 127.0.0.1 --port 60675 --omni --async-chunk --stage-init-timeout 600 --init-timeout 900 --log-stats --stage-configs-path vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml [reliability][process-kill] current_server_proc pid=556855 name=python3 cmdline=/workspace/.venv/bin/python3 -m vllm_omni.entrypoints.cli.main serve Qwen/Qwen3-Omni-30B-A3B-Instruct --host 127.0.0.1 --port 60675 --omni --async-chunk --stage-init-timeout 600 --init-timeout 900 --log-stats --deploy-config vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml
[reliability][process-kill] current_server_proc pid=557118 name=python3 cmdline=/workspace/.venv/bin/python3 -c from multiprocessing.resource_tracker import main;main(57) [reliability][process-kill] current_server_proc pid=557118 name=python3 cmdline=/workspace/.venv/bin/python3 -c from multiprocessing.resource_tracker import main;main(57)
[reliability][process-kill] current_server_proc pid=557119 name=VLLM::StageEngineCoreProc_noid_replica0_DP0 cmdline=VLLM::StageEngineCoreProc_noid_replica0_DP0 [reliability][process-kill] current_server_proc pid=557119 name=VLLM::StageEngineCoreProc_noid_replica0_DP0 cmdline=VLLM::StageEngineCoreProc_noid_replica0_DP0
[reliability][process-kill] current_server_proc pid=557122 name=VLLM::StageEngineCoreProc_noid_replica0_DP0 cmdline=VLLM::StageEngineCoreProc_noid_replica0_DP0 [reliability][process-kill] current_server_proc pid=557122 name=VLLM::StageEngineCoreProc_noid_replica0_DP0 cmdline=VLLM::StageEngineCoreProc_noid_replica0_DP0
[reliability][process-kill] current_server_proc pid=558201 name=VLLM::StageEngineCoreProc_noid_replica0_DP0 cmdline=VLLM::StageEngineCoreProc_noid_replica0_DP0 [reliability][process-kill] current_server_proc pid=558201 name=VLLM::StageEngineCoreProc_noid_replica0_DP0 cmdline=VLLM::StageEngineCoreProc_noid_replica0_DP0
[reliability][process-kill] root-kill pid=556855 name=python3 signal=SIGTERM cmdline=/workspace/.venv/bin/python3 -m vllm_omni.entrypoints.cli.main serve Qwen/Qwen3-Omni-30B-A3B-Instruct --host 127.0.0.1 --port 60675 --omni --async-chunk --stage-init-timeout 600 --init-timeout 900 --log-stats --stage-configs-path vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml [reliability][process-kill] root-kill pid=556855 name=python3 signal=SIGTERM cmdline=/workspace/.venv/bin/python3 -m vllm_omni.entrypoints.cli.main serve Qwen/Qwen3-Omni-30B-A3B-Instruct --host 127.0.0.1 --port 60675 --omni --async-chunk --stage-init-timeout 600 --init-timeout 900 --log-stats --deploy-config vllm-omni/vllm_omni/deploy/qwen3_omni_moe.yaml
[0;36m(APIServer pid=556855)[0;0m INFO 05-19 12:24:51 [omni_base.py:463] [AsyncOmni] Shutting down [0;36m(APIServer pid=556855)[0;0m INFO 05-19 12:24:51 [omni_base.py:463] [AsyncOmni] Shutting down
[0;36m(APIServer pid=556855)[0;0m INFO 05-19 12:24:51 [async_omni_engine.py:2397] [AsyncOmniEngine] Shutting down Orchestrator [0;36m(APIServer pid=556855)[0;0m INFO 05-19 12:24:51 [async_omni_engine.py:2397] [AsyncOmniEngine] Shutting down Orchestrator
[0;36m(APIServer pid=556855)[0;0m INFO 05-19 12:24:51 [orchestrator.py:351] [Orchestrator] Received shutdown signal [0;36m(APIServer pid=556855)[0;0m INFO 05-19 12:24:51 [orchestrator.py:351] [Orchestrator] Received shutdown signal
+1 -1
View File
@@ -53,5 +53,5 @@ vllm-omni serve "$MODEL" \
--host 0.0.0.0 \ --host 0.0.0.0 \
--port "$PORT" \ --port "$PORT" \
--trust-remote-code \ --trust-remote-code \
--stage-configs-path "$DEPLOY_YAML" \ --deploy-config "$DEPLOY_YAML" \
--omni --omni
@@ -28,8 +28,11 @@ If image.command is set, uses that as a full override.
- {{ .Values.omniArgs.numGpus | quote }} - {{ .Values.omniArgs.numGpus | quote }}
{{- end }} {{- end }}
{{- if .Values.omniArgs.stageConfigsPath }} {{- if .Values.omniArgs.stageConfigsPath }}
- "--stage-configs-path" {{- fail "omniArgs.stageConfigsPath has been removed; use omniArgs.deployConfigPath" }}
- {{ .Values.omniArgs.stageConfigsPath | quote }} {{- end }}
{{- if .Values.omniArgs.deployConfigPath }}
- "--deploy-config"
- {{ .Values.omniArgs.deployConfigPath | quote }}
{{- end }} {{- end }}
{{- if and .Values.omniArgs.cacheBackend (ne .Values.omniArgs.cacheBackend "none") }} {{- if and .Values.omniArgs.cacheBackend (ne .Values.omniArgs.cacheBackend "none") }}
- "--cache-backend" - "--cache-backend"
@@ -78,6 +78,26 @@ tests:
path: spec.template.spec.containers[0].command path: spec.template.spec.containers[0].command
content: "--enable-layerwise-offload" content: "--enable-layerwise-offload"
- it: should include deploy config in command
set:
omniArgs:
deployConfigPath: "/configs/deploy.yaml"
asserts:
- equal:
path: spec.template.spec.containers[0].command[8]
value: "--deploy-config"
- equal:
path: spec.template.spec.containers[0].command[9]
value: "/configs/deploy.yaml"
- it: should reject removed stage config value
set:
omniArgs:
stageConfigsPath: "/configs/legacy.yaml"
asserts:
- failedTemplate:
errorMessage: "omniArgs.stageConfigsPath has been removed; use omniArgs.deployConfigPath"
- it: should use full command override when image.command is set - it: should use full command override when image.command is set
set: set:
image: image:
@@ -67,8 +67,8 @@ omniArgs:
enableCpuOffload: false enableCpuOffload: false
# -- Number of GPUs for diffusion inference (empty = auto) # -- Number of GPUs for diffusion inference (empty = auto)
numGpus: numGpus:
# -- Path to stage configs file (empty = auto-detected from model) # -- Path to deploy config file (empty = auto-detected from model)
stageConfigsPath: deployConfigPath:
# -- Cache backend: none, tea_cache, or cache_dit # -- Cache backend: none, tea_cache, or cache_dit
cacheBackend: "none" cacheBackend: "none"
# -- Default sampling params as a JSON string # -- Default sampling params as a JSON string
@@ -3,7 +3,7 @@
# #
# Prereq: # Prereq:
# Start a vllm-omni OpenAI server for MiniCPM-o 4.5 on :8099 (see the # Start a vllm-omni OpenAI server for MiniCPM-o 4.5 on :8099 (see the
# 8x4090 stage config under vllm_omni/model_executor/stage_configs). # 8x4090 deploy config at vllm_omni/deploy/minicpmo_4_5_8x4090.yaml).
set -e set -e
HERE="$(cd "$(dirname "$0")" && pwd)" HERE="$(cd "$(dirname "$0")" && pwd)"
@@ -12,9 +12,9 @@ Please refer to [README.md](../../../README.md)
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
``` ```
If you have custom stage configs file, launch the server with command below To use a custom deploy config, launch the server with the command below:
```bash ```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml
``` ```
### Send Multi-modal Request ### Send Multi-modal Request
@@ -173,7 +173,7 @@ The script supports the following arguments:
- `--model`: Model name/path (default: Qwen/Qwen2.5-Omni-7B) - `--model`: Model name/path (default: Qwen/Qwen2.5-Omni-7B)
- `--server-port`: Port for vLLM server (default: 8091) - `--server-port`: Port for vLLM server (default: 8091)
- `--gradio-port`: Port for Gradio demo (default: 7861) - `--gradio-port`: Port for Gradio demo (default: 7861)
- `--stage-configs-path`: Path to custom stage configs YAML file (optional) - `--deploy-config`: Path to a custom deploy config YAML file (optional)
- `--server-host`: Host for vLLM server (default: 0.0.0.0) - `--server-host`: Host for vLLM server (default: 0.0.0.0)
- `--gradio-ip`: IP for Gradio demo (default: 127.0.0.1) - `--gradio-ip`: IP for Gradio demo (default: 127.0.0.1)
- `--share`: Share Gradio demo publicly (creates a public link) - `--share`: Share Gradio demo publicly (creates a public link)
@@ -186,9 +186,9 @@ The script supports the following arguments:
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091
``` ```
If you have custom stage configs file: To use a custom deploy config:
```bash ```bash
vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --stage-configs-path /path/to/stage_configs_file vllm serve Qwen/Qwen2.5-Omni-7B --omni --port 8091 --deploy-config /path/to/deploy_config.yaml
``` ```
**Step 2: Run the Gradio demo** **Step 2: Run the Gradio demo**
@@ -13,7 +13,7 @@ set -e
MODEL="Qwen/Qwen2.5-Omni-7B" MODEL="Qwen/Qwen2.5-Omni-7B"
SERVER_PORT=8091 SERVER_PORT=8091
GRADIO_PORT=7861 GRADIO_PORT=7861
STAGE_CONFIGS_PATH="" DEPLOY_CONFIG_PATH=""
SERVER_HOST="0.0.0.0" SERVER_HOST="0.0.0.0"
GRADIO_IP="127.0.0.1" GRADIO_IP="127.0.0.1"
GRADIO_SHARE=false GRADIO_SHARE=false
@@ -33,8 +33,8 @@ while [[ $# -gt 0 ]]; do
GRADIO_PORT="$2" GRADIO_PORT="$2"
shift 2 shift 2
;; ;;
--stage-configs-path) --deploy-config)
STAGE_CONFIGS_PATH="$2" DEPLOY_CONFIG_PATH="$2"
shift 2 shift 2
;; ;;
--server-host) --server-host)
@@ -56,7 +56,7 @@ while [[ $# -gt 0 ]]; do
echo " --model MODEL Model name/path (default: Qwen/Qwen2.5-Omni-7B)" echo " --model MODEL Model name/path (default: Qwen/Qwen2.5-Omni-7B)"
echo " --server-port PORT Port for vLLM server (default: 8091)" echo " --server-port PORT Port for vLLM server (default: 8091)"
echo " --gradio-port PORT Port for Gradio demo (default: 7861)" echo " --gradio-port PORT Port for Gradio demo (default: 7861)"
echo " --stage-configs-path PATH Path to custom stage configs YAML file (optional)" echo " --deploy-config PATH Path to custom deploy config YAML file (optional)"
echo " --server-host HOST Host for vLLM server (default: 0.0.0.0)" echo " --server-host HOST Host for vLLM server (default: 0.0.0.0)"
echo " --gradio-ip IP IP for Gradio demo (default: 127.0.0.1)" echo " --gradio-ip IP IP for Gradio demo (default: 127.0.0.1)"
echo " --share Share Gradio demo publicly" echo " --share Share Gradio demo publicly"
@@ -87,8 +87,8 @@ echo "=========================================="
# Build vLLM server command # Build vLLM server command
SERVER_CMD=("vllm" "serve" "$MODEL" "--omni" "--port" "$SERVER_PORT" "--host" "$SERVER_HOST") SERVER_CMD=("vllm" "serve" "$MODEL" "--omni" "--port" "$SERVER_PORT" "--host" "$SERVER_HOST")
if [ -n "$STAGE_CONFIGS_PATH" ]; then if [ -n "$DEPLOY_CONFIG_PATH" ]; then
SERVER_CMD+=("--stage-configs-path" "$STAGE_CONFIGS_PATH") SERVER_CMD+=("--deploy-config" "$DEPLOY_CONFIG_PATH")
fi fi
# Function to cleanup on exit # Function to cleanup on exit
@@ -336,7 +336,7 @@ vllm serve OpenMOSS-Team/MOSS-TTS-Nano --omni --port 8091
# or: # or:
./moss_tts_nano/run_server.sh ./moss_tts_nano/run_server.sh
``` ```
The deploy config at `vllm_omni/deploy/moss_tts_nano.yaml` auto-loads; no `--stage-configs-path`, `--trust-remote-code`, or `--enforce-eager` flags are needed. The deploy config at `vllm_omni/deploy/moss_tts_nano.yaml` auto-loads; no `--deploy-config`, `--trust-remote-code`, or `--enforce-eager` flags are needed.
### Sending requests ### Sending requests
```bash ```bash
@@ -241,12 +241,12 @@ def _fault_keywords_match_response(*, status: int, body: bytes) -> bool:
return any(key in text for key in body_fault_hints) return any(key in text for key in body_fault_hints)
def _stage_config_path_from_omni_server(omni_server: _HasServeArgs) -> str | None: def _deploy_config_path_from_omni_server(omni_server: _HasServeArgs) -> str | None:
args: list[str] = omni_server.serve_args args: list[str] = omni_server.serve_args
for i, arg in enumerate(args): for i, arg in enumerate(args):
if arg == "--stage-configs-path" and i + 1 < len(args): if arg == "--deploy-config" and i + 1 < len(args):
return args[i + 1] return args[i + 1]
if arg.startswith("--stage-configs-path="): if arg.startswith("--deploy-config="):
return arg.split("=", 1)[1] return arg.split("=", 1)[1]
return None return None
@@ -331,7 +331,7 @@ def test_reliability_fault_gpu_oom_error_contract_consistent_chat_speech(
""" """
device_spec = resolve_oom_device_spec( device_spec = resolve_oom_device_spec(
OOM_INJECTION_CONFIG, OOM_INJECTION_CONFIG,
_stage_config_path_from_omni_server(omni_server_function), _deploy_config_path_from_omni_server(omni_server_function),
) )
handle = inject_gpu_oom( handle = inject_gpu_oom(
device=device_spec, device=device_spec,
@@ -394,7 +394,7 @@ def test_reliability_fault_gpu_oom_error_contract_consistent_chat_speech(
def test_reliability_fault_gpu_oom_chat_large_payload_failure(omni_server_function, openai_client_function) -> None: def test_reliability_fault_gpu_oom_chat_large_payload_failure(omni_server_function, openai_client_function) -> None:
device_spec = resolve_oom_device_spec( device_spec = resolve_oom_device_spec(
OOM_INJECTION_CONFIG, OOM_INJECTION_CONFIG,
_stage_config_path_from_omni_server(omni_server_function), _deploy_config_path_from_omni_server(omni_server_function),
) )
handle = inject_gpu_oom( handle = inject_gpu_oom(
device=device_spec, device=device_spec,
@@ -437,7 +437,7 @@ def test_reliability_fault_gpu_oom_chat_large_payload_failure(omni_server_functi
def test_reliability_fault_gpu_oom_concurrent_pressure_failure(omni_server_function, openai_client_function) -> None: def test_reliability_fault_gpu_oom_concurrent_pressure_failure(omni_server_function, openai_client_function) -> None:
device_spec = resolve_oom_device_spec( device_spec = resolve_oom_device_spec(
OOM_INJECTION_CONFIG, OOM_INJECTION_CONFIG,
_stage_config_path_from_omni_server(omni_server_function), _deploy_config_path_from_omni_server(omni_server_function),
) )
handle = inject_gpu_oom( handle = inject_gpu_oom(
device=device_spec, device=device_spec,
@@ -791,7 +791,7 @@ def test_reliability_fault_gpu_oom_state_converges_after_fault_removed(
""" """
device_spec = resolve_oom_device_spec( device_spec = resolve_oom_device_spec(
OOM_RECOVER_INJECTION_CONFIG, OOM_RECOVER_INJECTION_CONFIG,
_stage_config_path_from_omni_server(omni_server_function), _deploy_config_path_from_omni_server(omni_server_function),
) )
handle = inject_gpu_oom( handle = inject_gpu_oom(
device=device_spec, device=device_spec,
+1 -1
View File
@@ -696,7 +696,7 @@ def _dreamzero_omni_server_from_params(params):
server_args = list(params.server_args or []) server_args = list(params.server_args or [])
if params.stage_config_path is not None: if params.stage_config_path is not None:
server_args += ["--stage-configs-path", params.stage_config_path] server_args += ["--deploy-config", params.stage_config_path]
return OmniServer( return OmniServer(
params.model, params.model,
server_args, server_args,
+3 -3
View File
@@ -359,12 +359,12 @@ def _run_offline(deploy_config_path: str, output_path: Path) -> tuple[Image.Imag
return image, cot_text, elapsed return image, cot_text, elapsed
def _run_online(stage_configs_path: str, output_path: Path) -> tuple[Image.Image, str, float]: def _run_online(deploy_config_path: str, output_path: Path) -> tuple[Image.Image, str, float]:
from benchmarks.accuracy.common import decode_base64_image, pil_to_png_bytes from benchmarks.accuracy.common import decode_base64_image, pil_to_png_bytes
server_args = [ server_args = [
"--stage-configs-path", "--deploy-config",
stage_configs_path, deploy_config_path,
"--stage-init-timeout", "--stage-init-timeout",
"300", "300",
"--init-timeout", "--init-timeout",
+24 -1
View File
@@ -46,6 +46,26 @@ def test_serve_parser_accepts_strategy_config() -> None:
assert args.get_explicit_kwargs_dict()["strategy_config"] == "/tmp/strategy.yaml" assert args.get_explicit_kwargs_dict()["strategy_config"] == "/tmp/strategy.yaml"
def test_serve_parser_accepts_deploy_config() -> None:
parser = TrackingArgumentParser()
subparsers = parser.add_subparsers(dest="subcommand")
OmniServeCommand().subparser_init(subparsers)
args = parser.parse_args(["serve", "fake-model", "--omni", "--deploy-config", "/tmp/deploy.yaml"])
assert args.deploy_config == "/tmp/deploy.yaml"
assert args.get_explicit_kwargs_dict()["deploy_config"] == "/tmp/deploy.yaml"
def test_serve_parser_rejects_stage_configs_path() -> None:
parser = TrackingArgumentParser()
subparsers = parser.add_subparsers(dest="subcommand")
OmniServeCommand().subparser_init(subparsers)
with pytest.raises(SystemExit):
parser.parse_args(["serve", "fake-model", "--omni", "--stage-configs-path", "/tmp/stages.yaml"])
def test_serve_parser_accepts_four_way_cfg_parallelism() -> None: def test_serve_parser_accepts_four_way_cfg_parallelism() -> None:
parser = TrackingArgumentParser() parser = TrackingArgumentParser()
subparsers = parser.add_subparsers(dest="subcommand") subparsers = parser.add_subparsers(dest="subcommand")
@@ -66,7 +86,6 @@ def _make_headless_args(**kwargs) -> TrackingNamespace:
"omni_replica_address": None, "omni_replica_address": None,
"omni_dp_size_local": 1, "omni_dp_size_local": 1,
"worker_backend": "multi_process", "worker_backend": "multi_process",
"stage_configs_path": None,
"deploy_config": None, "deploy_config": None,
"log_stats": False, "log_stats": False,
"disable_log_stats": False, "disable_log_stats": False,
@@ -154,6 +173,7 @@ def test_run_headless_parses_and_forwards_stage_overrides(mocker: MockerFixture)
captured: dict = {} captured: dict = {}
def _fake_resolve(*args, **kwargs): def _fake_resolve(*args, **kwargs):
captured["args"] = args
captured.update(kwargs) captured.update(kwargs)
# Return a stage that does NOT match stage_id=0 so run_headless stops # Return a stage that does NOT match stage_id=0 so run_headless stops
# right after the resolver call (we only care about how it was called). # right after the resolver call (we only care about how it was called).
@@ -166,12 +186,15 @@ def test_run_headless_parses_and_forwards_stage_overrides(mocker: MockerFixture)
args = _make_headless_args( args = _make_headless_args(
stage_id=0, stage_id=0,
deploy_config="/tmp/deploy.yaml",
strategy_config="/tmp/strategy.yaml", strategy_config="/tmp/strategy.yaml",
stage_overrides='{"0": {"devices": "0,1"}, "1": {"devices": "2"}}', stage_overrides='{"0": {"devices": "0,1"}, "1": {"devices": "2"}}',
) )
with pytest.raises(ValueError, match="No stage config found for stage_id=0"): with pytest.raises(ValueError, match="No stage config found for stage_id=0"):
run_headless(args) run_headless(args)
assert captured["args"][1] is None
assert captured["deploy_config_path"] == "/tmp/deploy.yaml"
assert captured["stage_overrides"] == {"0": {"devices": "0,1"}, "1": {"devices": "2"}} assert captured["stage_overrides"] == {"0": {"devices": "0,1"}, "1": {"devices": "2"}}
assert captured["strategy_config_path"] == "/tmp/strategy.yaml" assert captured["strategy_config_path"] == "/tmp/strategy.yaml"
+3 -3
View File
@@ -598,7 +598,7 @@ class OmniServerStageCli(OmniServer):
"serve", "serve",
self.model, self.model,
"--omni", "--omni",
"--stage-configs-path", "--deploy-config",
self.stage_config_path, self.stage_config_path,
"--stage-id", "--stage-id",
str(stage_id), str(stage_id),
@@ -3135,7 +3135,7 @@ def iter_omni_server(
raise ValueError("omni_server with use_stage_cli=True requires use_omni=True") raise ValueError("omni_server with use_stage_cli=True requires use_omni=True")
if stage_config_path is None: if stage_config_path is None:
raise ValueError("omni_server with use_stage_cli=True requires a stage_config_path") raise ValueError("omni_server with use_stage_cli=True requires a stage_config_path")
server_args += ["--stage-configs-path", stage_config_path] server_args += ["--deploy-config", stage_config_path]
with OmniServerStageCli( with OmniServerStageCli(
model, model,
@@ -3151,7 +3151,7 @@ def iter_omni_server(
print("OmniServer stopping...") print("OmniServer stopping...")
else: else:
if stage_config_path is not None: if stage_config_path is not None:
server_args += ["--stage-configs-path", stage_config_path] server_args += ["--deploy-config", stage_config_path]
with ( with (
OmniServer( OmniServer(
@@ -35,7 +35,7 @@ def test_stage_cli_builds_headless_replica_cmd(tmp_path: Path) -> None:
assert "--headless" in cmd assert "--headless" in cmd
assert cmd[cmd.index("--stage-id") + 1] == "1" assert cmd[cmd.index("--stage-id") + 1] == "1"
assert cmd[cmd.index("--replica-id") + 1] == "2" assert cmd[cmd.index("--replica-id") + 1] == "2"
assert cmd[cmd.index("--stage-configs-path") + 1] == server.stage_config_path assert cmd[cmd.index("--deploy-config") + 1] == server.stage_config_path
def test_stage_cli_splits_devices_per_replica(tmp_path: Path) -> None: def test_stage_cli_splits_devices_per_replica(tmp_path: Path) -> None:
+1 -1
View File
@@ -17,7 +17,7 @@
# measured 0.79% here. The serving layer currently injects 0.05 for # measured 0.79% here. The serving layer currently injects 0.05 for
# guided requests on all sizes; acceptable (still within the anchor # guided requests on all sizes; acceptable (still within the anchor
# +2pp standing gate) but a size-aware refinement is a known follow-up. # +2pp standing gate) but a size-aware refinement is a known follow-up.
# ALWAYS pass this yaml explicitly (--stage-configs-path): bare serve of the # ALWAYS pass this yaml explicitly (--deploy-config): bare serve of the
# 30B repo auto-detects the shared model_type and lands on the 2B-tuned yaml. # 30B repo auto-detects the shared model_type and lands on the 2B-tuned yaml.
pipeline: audex_tts pipeline: audex_tts
async_chunk: true async_chunk: true
+4 -13
View File
@@ -147,7 +147,7 @@ class OmniServeCommand(CLISubcommand):
raise ValueError( raise ValueError(
"The following CLI args are not supported under --omni: " "The following CLI args are not supported under --omni: "
f"{', '.join(offenders)}. Configure parallelism through the " f"{', '.join(offenders)}. Configure parallelism through the "
"per-stage YAML (`--deploy-config` / `--stage-configs-path`) " "per-stage YAML (`--deploy-config`) "
"and replica count via the per-stage `num_replicas` config field " "and replica count via the per-stage `num_replicas` config field "
"(single-runtime) or `--omni-dp-size-local` (headless / multi-runtime)." "(single-runtime) or `--omni-dp-size-local` (headless / multi-runtime)."
) )
@@ -240,20 +240,11 @@ class OmniServeCommand(CLISubcommand):
"flag, when set, overrides the YAML model field." "flag, when set, overrides the YAML model field."
), ),
) )
# TODO(@lishunyang12): deprecate once all models migrate to --deploy-config
omni_config_group.add_argument(
"--stage-configs-path",
type=str,
default=None,
help="[Deprecated — will be removed in a future release] Path to a legacy "
"stage configs YAML (stage_args format). Prefer --deploy-config for new-format deploy YAMLs.",
)
omni_config_group.add_argument( omni_config_group.add_argument(
"--deploy-config", "--deploy-config",
type=str, type=str,
default=None, default=None,
help="Path to a deploy config YAML (new format with stages/engine_args). " help="Path to a deploy config YAML (new format with stages/engine_args).",
"Mutually exclusive with --stage-configs-path.",
) )
omni_config_group.add_argument( omni_config_group.add_argument(
"--strategy-config", "--strategy-config",
@@ -846,7 +837,6 @@ def run_headless(args: TrackingNamespace) -> None:
omni_master_address: str | None = args.omni_master_address omni_master_address: str | None = args.omni_master_address
omni_master_port: int | None = args.omni_master_port omni_master_port: int | None = args.omni_master_port
worker_backend: str | None = args.worker_backend worker_backend: str | None = args.worker_backend
stage_configs_path: str | None = args.stage_configs_path
omni_replica_address: str | None = getattr(args, "omni_replica_address", None) omni_replica_address: str | None = getattr(args, "omni_replica_address", None)
omni_dp_size_local: int = max(1, int(getattr(args, "omni_dp_size_local", 1) or 1)) omni_dp_size_local: int = max(1, int(getattr(args, "omni_dp_size_local", 1) or 1))
@@ -882,7 +872,8 @@ def run_headless(args: TrackingNamespace) -> None:
config_path, stage_configs, _ = load_and_resolve_stage_configs( config_path, stage_configs, _ = load_and_resolve_stage_configs(
model, model,
stage_configs_path, # The serve CLI no longer accepts legacy stage_args YAMLs.
None,
args_dict, args_dict,
# store_true cannot express an explicit False: absent maps to None # store_true cannot express an explicit False: absent maps to None
# ("not specified") so the deploy yaml's per-stage value applies. # ("not specified") so the deploy yaml's per-stage value applies.