Signed-off-by: Kushan A. <kahmadian@nvidia.com> Signed-off-by: kushanam <kushanam@users.noreply.github.com> Signed-off-by: Kushan A <kahmadian@nvidia.com> Signed-off-by: kahmadian <kahmadian@users.noreply.github.com> Co-authored-by: kushanam <kushanam@users.noreply.github.com> Co-authored-by: kahmadian <kahmadian@users.noreply.github.com>
This commit is contained in:
30 files changed
+2722
-51
No files matched your search
@@ -0,0 +1,239 @@
|
|||||||
|
# Composable parallel strategies
|
||||||
|
|
||||||
|
In vLLM-Omni, the `composable_parallel` system layers a declarative *parallel strategy* on top of the existing stage configs. A strategy YAML maps each pipeline stage (by its `model_stage` name) to a stack of axis declarations — one per parallelism dimension — that the engine translates into per-stage parallel sizing (`tensor_parallel_size`, `data_parallel_size`, `pipeline_parallel_size`, expert-parallel toggle, replica count) and, when relevant, an orchestrator-wide load-balancer policy.
|
||||||
|
|
||||||
|
Strategies are intentionally narrow: each one only writes the axes it declares, leaving every other engine arg untouched. This makes a strategy file a small, reusable overlay you can swap independently of the deploy YAML (see `configuration/stage_configs.md`).
|
||||||
|
|
||||||
|
!!! note
|
||||||
|
Composable parallel is **opt-in**. Passing `--strategy-config <path/to/strategy.yaml>` overlays the derived sizing onto the registry-merged stages *before* any CLI overrides are applied. When `--strategy-config` is omitted, stages remain exactly as their deploy YAML produces them.
|
||||||
|
|
||||||
|
## Top-level schema reference
|
||||||
|
|
||||||
|
A strategy file has a single top-level mapping, `strategies`, keyed by the `model_stage` name of each stage you want to overlay. Each value is a list of axis declarations (see [StrategySpec fields](#strategyspec-fields) below).
|
||||||
|
|
||||||
|
| Field | Type | Required | Default | Description |
|
||||||
|
|-------|------|----------|---------|-------------|
|
||||||
|
| `strategies` | `dict[str, list]` | required | — | Maps `model_stage` name (e.g. `thinker`, `talker`, `code2wav`, `dit`) to a list of axis declarations. The key MUST be the stage name, not the integer `stage_id`. See `vllm_omni/config/composable_parallel/strategy_loader.py`. |
|
||||||
|
|
||||||
|
### StrategySpec fields
|
||||||
|
|
||||||
|
Each entry in a stage's list becomes one `StrategySpec` (see `vllm_omni/config/composable_parallel/spec.py`). The loader fills in `routing` and `aggregation` from per-kind defaults, so the file stays compact.
|
||||||
|
|
||||||
|
| Field | Type | Required | Default | Description |
|
||||||
|
|-------|------|----------|---------|-------------|
|
||||||
|
| `axis` | str | required | — | Mesh-axis kind. One of `tp`, `dp`, `pp`, `ep`, `stage_replica` (wired today) or `sp_ulysses`, `sp_ring`, `cfg`, `vae_pp`, `hsdp`, `stage_pp`, `cp` (reserved; declaring one raises `AxisTranslationError` at translate time). See `MeshAxisKind` in `vllm_omni/config/composable_parallel/spec.py`. |
|
||||||
|
| `size` | int | required | — | Axis degree. Must be `> 0`. |
|
||||||
|
| `routing` | str | optional | per-kind default | Stateless routing policy. Accepted only for `dp` / `stage_replica`; other kinds reject it. Values: `random`, `round_robin`, `least_queue`. (`hash` is recognised but currently raises `NotImplementedError`.) See `RouteByStage` in `vllm_omni/config/composable_parallel/routing.py`. |
|
||||||
|
| `l1_owner` | str | optional | per-kind default | Overrides which layer owns request routing for this axis. `engine` for `tp`/`dp`/`pp`/`ep`, `delegated` for `stage_replica`. Recorded into `shard_extension`. |
|
||||||
|
| `name` | str | optional | axis kind | Human-readable label for this spec, used in error messages. |
|
||||||
|
|
||||||
|
Two additional `StrategySpec` slots — `layer_hook_specs` and `kernel_specs` — let advanced authors bind layer-walker hook IDs (`LayerHookSpec`) or kernel IDs (`KernelSpec`) onto an axis. These are not exposed by the YAML loader today and are intended for programmatic construction. See `vllm_omni/config/composable_parallel/spec.py` for the dataclasses.
|
||||||
|
|
||||||
|
### CLI flags
|
||||||
|
|
||||||
|
| Flag | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `--strategy-config PATH` | Path to a `strategy.yaml`. Only takes effect on registry-based deploy paths (`--deploy-config` or the bundled default deploy YAML). Its derived sizing is overlaid onto the merged stages *before* CLI overrides. |
|
||||||
|
| `--stage-configs-path PATH` | **Legacy.** Loads the old `stage_args` YAML schema for un-migrated models. Passing `--strategy-config` together with this path is silently inapplicable — vLLM-Omni emits a warning and the strategy is ignored (see `vllm_omni/entrypoints/utils.py`, lines 372–380). |
|
||||||
|
| `--omni-lb-policy POLICY` | Orchestrator-wide load-balancer policy. Strategy-derived from any `stage_replica` axis; an explicit CLI value wins and must match the derived one or `AsyncOmniEngine` raises `Conflicting load-balancer policy` at construction. |
|
||||||
|
|
||||||
|
!!! warning
|
||||||
|
`--strategy-config` cannot be used with `--stage-configs-path` (the legacy deploy path). If your model has not been migrated to the registry-based pipeline yet, the strategy is silently dropped with a warning. Migrate to a registry-based model or use `--deploy-config` to apply a strategy.
|
||||||
|
|
||||||
|
## Sub-schemas
|
||||||
|
|
||||||
|
### `axis`
|
||||||
|
|
||||||
|
The set of recognised axis kinds is the `MeshAxisKind` enum in `vllm_omni/config/composable_parallel/spec.py`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Translatable today (see `_SUPPORTED_KINDS` in translator.py).
|
||||||
|
axis: tp # tensor parallel — sets tensor_parallel_size
|
||||||
|
axis: dp # engine data parallel — sets data_parallel_size (vLLM's intra-engine DP LB)
|
||||||
|
axis: pp # pipeline parallel — sets pipeline_parallel_size
|
||||||
|
axis: ep # dense expert parallel — sets enable_expert_parallel=True; size must equal tp*dp
|
||||||
|
axis: stage_replica # omni-coordinator-level replica fan-out — sets num_replicas + omni_lb_policy
|
||||||
|
|
||||||
|
# Reserved (raise AxisTranslationError today; declared in the type system for forward compat).
|
||||||
|
axis: sp_ulysses
|
||||||
|
axis: sp_ring
|
||||||
|
axis: cfg
|
||||||
|
axis: vae_pp
|
||||||
|
axis: hsdp
|
||||||
|
axis: stage_pp
|
||||||
|
axis: cp
|
||||||
|
```
|
||||||
|
|
||||||
|
### `routing`
|
||||||
|
|
||||||
|
The full closed hierarchy lives in `vllm_omni/config/composable_parallel/routing.py` (`Broadcast`, `PartitionByHash`, `ShardSequence`, `ShardSpatial`, `RouteByExpert`, `PipelineMicrobatch`, `DuplicateWithCondTag`, `RouteByStage`). For YAML authors only `RouteByStage` is exposed as a `routing:` string, and only for `dp` / `stage_replica` axes (other axes have fixed routing — `tp`/`ep` broadcast, `pp` microbatch). Recognised policies:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
routing: random # uniform-random across replicas/DP ranks
|
||||||
|
routing: round_robin # cycle through replicas/DP ranks (the default for dp/stage_replica)
|
||||||
|
routing: least_queue # send to the replica with the smallest backlog
|
||||||
|
# routing: hash # recognised but raises NotImplementedError; no key-stable balancer yet
|
||||||
|
```
|
||||||
|
|
||||||
|
### `aggregation`
|
||||||
|
|
||||||
|
Aggregation is filled in automatically by the loader (`_default_aggregation` in `vllm_omni/config/composable_parallel/strategy_loader.py`) from the axis kind, so users do not author it:
|
||||||
|
|
||||||
|
| Axis kind | Aggregation pattern |
|
||||||
|
|-----------|---------------------|
|
||||||
|
| `tp`, `ep` | `TakeRank` (all ranks agree; pick one) |
|
||||||
|
| `dp`, ep (in some flows) | `Union` (disjoint replica results) |
|
||||||
|
| `pp` | `StitchPipeline` |
|
||||||
|
| `stage_replica` | `FanInByStage` |
|
||||||
|
|
||||||
|
Full pattern hierarchy: `TakeRank`, `Union`, `GatherDim`, `AllGather`, `StitchSpatial`, `StitchPipeline`, `Combine`, `FanInByStage` (see `vllm_omni/config/composable_parallel/aggregation.py`).
|
||||||
|
|
||||||
|
## Precedence
|
||||||
|
|
||||||
|
From highest to lowest, when the same axis is touched by multiple layers:
|
||||||
|
|
||||||
|
1. **CLI overrides** (`--stage_N_tensor_parallel_size`, `--stage-overrides`, `--omni-lb-policy`, etc.). A CLI value always wins; a warning is emitted whenever it overrides a strategy-declared axis (see `_reconcile_strategy_with_cli` in `vllm_omni/config/config_factory.py`).
|
||||||
|
2. **Strategy YAML** (`--strategy-config`). Derived sizing is written onto the merged stages with *conflict-on-explicit* semantics — if the deploy YAML already set a value to something different, application raises `StrategyApplyError`.
|
||||||
|
3. **Deploy YAML** (`--deploy-config` or the bundled `vllm_omni/deploy/<model_type>.yaml`).
|
||||||
|
4. **Parser defaults.**
|
||||||
|
|
||||||
|
After CLI overrides land, the device-layout guard re-runs (`check_device_layout`, called from `_reconcile_strategy_with_cli`) against the *effective* `tp * dp * pp * num_replicas` and the resolved `devices` string, so a `--devices` or `--tensor-parallel-size` that conflicts with the strategy's world size cannot slip past the pre-spawn check.
|
||||||
|
|
||||||
|
## Worked example
|
||||||
|
|
||||||
|
### Example 1 — `strategy_tp2.yaml`
|
||||||
|
|
||||||
|
Shard the `thinker` stage with TP=2 on top of the bundled Qwen2.5-Omni deploy YAML. Routing is fixed (broadcast) for TP, so no `routing:` field is needed.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# examples/offline_inference/qwen2_5_omni/strategy_tp2.yaml
|
||||||
|
strategies:
|
||||||
|
thinker:
|
||||||
|
- axis: tp
|
||||||
|
size: 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Launch:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 end2end.py --query-type text --num-prompts 6 \
|
||||||
|
--strategy-config strategy_tp2.yaml \
|
||||||
|
--stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Effective `OmniParallelConfig` per stage after the overlay (see `OmniParallelConfig` in `vllm_omni/config/composable_parallel/translator.py`):
|
||||||
|
|
||||||
|
| Stage (`model_stage`) | `tensor_parallel_size` | `data_parallel_size` | `pipeline_parallel_size` | `stage_replica_size` | `omni_lb_policy` |
|
||||||
|
|-----------------------|------------------------|----------------------|--------------------------|----------------------|------------------|
|
||||||
|
| `thinker` | **2** (from strategy) | 1 | 1 | 1 | — |
|
||||||
|
| `talker` | 1 | 1 | 1 | 1 | — |
|
||||||
|
| `code2wav` | 1 | 1 | 1 | 1 | — |
|
||||||
|
|
||||||
|
### Example 2 — `strategy_stage_replica.yaml`
|
||||||
|
|
||||||
|
Replicate the `thinker` stage across two independent engines and let omni's StagePool round-robin requests across them.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# examples/offline_inference/qwen2_5_omni/strategy_stage_replica.yaml
|
||||||
|
strategies:
|
||||||
|
thinker:
|
||||||
|
- axis: stage_replica
|
||||||
|
size: 2
|
||||||
|
routing: round_robin
|
||||||
|
```
|
||||||
|
|
||||||
|
Launch:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 end2end.py --query-type text --num-prompts 6 \
|
||||||
|
--strategy-config strategy_stage_replica.yaml \
|
||||||
|
--stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
|
||||||
|
```
|
||||||
|
|
||||||
|
Effective per-stage config and the derived orchestrator knob:
|
||||||
|
|
||||||
|
| Stage (`model_stage`) | `tensor_parallel_size` | `stage_replica_size` | `omni_lb_policy` (derived) |
|
||||||
|
|-----------------------|------------------------|----------------------|----------------------------|
|
||||||
|
| `thinker` | 1 | **2** | **`round-robin`** |
|
||||||
|
| `talker` | 1 | 1 | — |
|
||||||
|
| `code2wav` | 1 | 1 | — |
|
||||||
|
|
||||||
|
The derived `omni_lb_policy = round-robin` is **not** written into any stage's engine args. Instead it is surfaced on `StrategyApplyResult.omni_lb_policy` and applied once at orchestrator construction by `AsyncOmniEngine._apply_strategy_lb_policy` (see `vllm_omni/engine/async_omni_engine.py`, lines 1107–1134). If you also pass `--omni-lb-policy round-robin` the values must match; passing a different value raises `Conflicting load-balancer policy`.
|
||||||
|
|
||||||
|
## Per-field deep-dive
|
||||||
|
|
||||||
|
### `name`
|
||||||
|
|
||||||
|
Default: the axis kind (e.g. `"tp"`).
|
||||||
|
|
||||||
|
Human-readable label that appears in `AxisTranslationError` and `StrategyApplyError` messages. Set it to disambiguate two stages declaring the same axis kind when scanning logs.
|
||||||
|
|
||||||
|
### `mesh_axis` (`axis` + `size`)
|
||||||
|
|
||||||
|
Default: no default — both fields are required.
|
||||||
|
|
||||||
|
`axis` chooses the parallelism dimension; `size` is its degree. Five kinds are wired end-to-end today (`tp`, `dp`, `pp`, `ep`, `stage_replica`); the remaining kinds in `MeshAxisKind` are reserved and raise `AxisTranslationError` at translate time so a typo cannot silently become a no-op (see `_SUPPORTED_KINDS` in `vllm_omni/config/composable_parallel/translator.py`). For `ep`, the size MUST equal `tensor_parallel_size * data_parallel_size`.
|
||||||
|
|
||||||
|
### `routing`
|
||||||
|
|
||||||
|
Default: per-kind — `RouteByStage(round_robin)` for `dp` and `stage_replica`; `PipelineMicrobatch()` for `pp`; `Broadcast()` for `tp`/`ep` and any other kind.
|
||||||
|
|
||||||
|
User-facing string only on `dp` and `stage_replica` axes. The four recognised values are `random`, `round_robin`, `least_queue`, `hash`; `hash` is rejected today with `NotImplementedError`. See `RouteByStage` in `vllm_omni/config/composable_parallel/routing.py`.
|
||||||
|
|
||||||
|
### `l1_owner`
|
||||||
|
|
||||||
|
Default: per-kind — `engine` for `tp`/`dp`/`pp`/`ep`, `delegated` for `stage_replica`.
|
||||||
|
|
||||||
|
Overrides which layer owns request routing along this axis. `engine` means vLLM realizes routing intra-engine; `delegated` means the omni coordinator owns routing across independent replicas. Set this only when overriding the per-kind default in `_DEFAULT_L1_OWNER` (`vllm_omni/config/composable_parallel/translator.py`); invalid values fail fast in `_resolve_l1_owner`.
|
||||||
|
|
||||||
|
### `stage_replica` and `omni_lb_policy`
|
||||||
|
|
||||||
|
`stage_replica` is *not* a vLLM world dimension; it spins up `size` independent engine replicas of one stage, coordinated by the omni `StagePool` load balancer. The axis writes both:
|
||||||
|
|
||||||
|
- The stage's `num_replicas` (a per-stage deploy knob).
|
||||||
|
- The pipeline-wide `omni_lb_policy` string (an orchestrator-level knob).
|
||||||
|
|
||||||
|
Because `omni_lb_policy` is read once at `AsyncOmniEngine` construction — not from per-stage engine args — only one value is allowed across the whole pipeline. If two stages declare conflicting policies, application raises (see `apply_strategy_specs` in `vllm_omni/config/composable_parallel/apply.py`, lines 253–260). To set this knob manually instead, use `--omni-lb-policy` on the CLI; see [the model stage docs](stage_configs.md) for how stage replicas plug into the orchestrator at runtime.
|
||||||
|
|
||||||
|
### `model_stage` keys
|
||||||
|
|
||||||
|
Default: no default — required for every entry under `strategies:`.
|
||||||
|
|
||||||
|
Strategy keys MUST be `model_stage` *names* (strings), the same labels stage configs use for `engine_args.model_stage` (see [`model_stage` in `stage_configs.md`](stage_configs.md#engine_argsmodel_stage)). Integer `stage_id` values are not accepted; the resolver matches on `model_stage` exactly (`_resolve_stage` in `vllm_omni/config/composable_parallel/apply.py`).
|
||||||
|
|
||||||
|
## Common pitfalls
|
||||||
|
|
||||||
|
!!! warning "Using `stage_id` integers as keys"
|
||||||
|
Strategy keys are `model_stage` names (str), not integer stage IDs. A file that uses an integer index will raise at apply time:
|
||||||
|
|
||||||
|
```text
|
||||||
|
StrategyApplyError: strategy role model_stage=0 did not match any stage;
|
||||||
|
available stages: 'thinker'(id=0), 'talker'(id=1), 'code2wav'(id=2)
|
||||||
|
```
|
||||||
|
|
||||||
|
Use the `model_stage` name (e.g. `thinker`) instead.
|
||||||
|
|
||||||
|
!!! warning "Mixing CLI overrides with a strategy on the same axis"
|
||||||
|
A strategy is meant to be the single writer for the axes it declares. If a CLI override (`--stage_0_tensor_parallel_size`, `--stage-overrides '{"0": {...}}'`) sets the same axis, the **CLI value wins** and a warning is emitted from `_reconcile_strategy_with_cli`. The device-layout guard then re-runs against the effective layout, so a CLI override that breaks `tp * dp * pp * num_replicas` will still fail fast at config time.
|
||||||
|
|
||||||
|
!!! warning "Passing `--strategy-config` against the legacy `--stage-configs-path`"
|
||||||
|
Composable parallel only applies on the registry-based deploy path. If a model still resolves through the legacy `stage_args` YAML (loaded via `--stage-configs-path`), `--strategy-config` is silently inapplicable and emits:
|
||||||
|
|
||||||
|
```text
|
||||||
|
--strategy-config (<path>) was provided but model '<model>' resolves via the
|
||||||
|
legacy stage_configs YAML path, which does not support composable-parallel
|
||||||
|
strategies; the strategy is ignored. Use a registry-based model to apply it.
|
||||||
|
```
|
||||||
|
|
||||||
|
Migrate the model to the new deploy schema (see `configuration/stage_configs.md`) before applying a strategy.
|
||||||
|
|
||||||
|
!!! warning "Conflicting `omni_lb_policy` across stages"
|
||||||
|
`omni_lb_policy` is a single pipeline-wide knob. If two `stage_replica` axes in different stages derive different policies, `apply_strategy_specs` raises:
|
||||||
|
|
||||||
|
```text
|
||||||
|
StrategyApplyError: role 'talker' derives omni_lb_policy='least-queue-length'
|
||||||
|
but role 'thinker' already derived 'round-robin'. The load-balancer policy is
|
||||||
|
pipeline-wide; only one value is allowed.
|
||||||
|
```
|
||||||
|
|
||||||
|
Make the policies match, drop one, or supply the value explicitly via `--omni-lb-policy`.
|
||||||
@@ -60,3 +60,24 @@ If media file paths are not provided, the script will use default assets. Suppor
|
|||||||
- `mixed_modalities`: Audio + image + video
|
- `mixed_modalities`: Audio + image + video
|
||||||
- `use_audio_in_video`: Extract audio from video
|
- `use_audio_in_video`: Extract audio from video
|
||||||
- `text`: Text-only query
|
- `text`: Text-only query
|
||||||
|
|
||||||
|
### Composable parallelism (strategy configs)
|
||||||
|
|
||||||
|
You can shard or replicate a stage with a small composable-parallel
|
||||||
|
`strategy.yaml` overlaid onto the **bundled default deploy config** -- no bespoke
|
||||||
|
deploy YAML required. Supply the strategy with `--strategy-config` and any matching
|
||||||
|
device layout via `--stage-overrides` (explicit kwargs forwarded to `Omni`).
|
||||||
|
|
||||||
|
Both examples below require >= 3 GPUs (the thinker uses 2, talker + code2wav share a third):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Tensor-parallel the thinker (TP=2)
|
||||||
|
python end2end.py --query-type text --num-prompts 6 \
|
||||||
|
--strategy-config strategy_tp2.yaml \
|
||||||
|
--stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
|
||||||
|
|
||||||
|
# Replicate the thinker across 2 engines (round-robin load balancing)
|
||||||
|
python end2end.py --query-type text --num-prompts 6 \
|
||||||
|
--strategy-config strategy_stage_replica.yaml \
|
||||||
|
--stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
|
||||||
|
```
|
||||||
@@ -553,6 +553,39 @@ def parse_args():
|
|||||||
default=False,
|
default=False,
|
||||||
help="Use py_generator mode. The returned type of Omni.generate() is a Python Generator object.",
|
help="Use py_generator mode. The returned type of Omni.generate() is a Python Generator object.",
|
||||||
)
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--deploy-config",
|
||||||
|
type=str,
|
||||||
|
default=None,
|
||||||
|
help="Optional explicit deploy YAML (otherwise the bundled default is used).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--strategy-config",
|
||||||
|
type=str,
|
||||||
|
default=None,
|
||||||
|
help=(
|
||||||
|
"Optional composable-parallel strategy.yaml mapping role -> parallel axes "
|
||||||
|
"(e.g. tp / stage_replica). Overlaid onto the merged stage configs."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--stage-overrides",
|
||||||
|
type=str,
|
||||||
|
default=None,
|
||||||
|
help=(
|
||||||
|
"Optional JSON of per-stage overrides applied on top of the default "
|
||||||
|
'deploy config, e.g. \'{"0": {"devices": "0,1"}}\' to give the '
|
||||||
|
"thinker a 2-GPU pool. Lets you run a strategy on the bundled default "
|
||||||
|
"deploy config without writing a bespoke deploy YAML."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--omni-lb-policy",
|
||||||
|
type=str,
|
||||||
|
default=None,
|
||||||
|
choices=["random", "round-robin", "least-queue-length"],
|
||||||
|
help="StagePool load-balancer policy for replicated stages (orchestrator-level knob).",
|
||||||
|
)
|
||||||
return parser.parse_args()
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Composable-parallel strategy for Qwen2.5-Omni.
|
||||||
|
#
|
||||||
|
# Replicate the thinker stage across 2 independent engines and let the omni
|
||||||
|
# StagePool balance requests round-robin across them. The thinker's `devices`
|
||||||
|
# entry acts as a per-replica template, so omni expands it to one GPU per replica.
|
||||||
|
# The `routing: round_robin` below auto-derives the StagePool load-balancer policy,
|
||||||
|
# so --omni-lb-policy is not needed.
|
||||||
|
#
|
||||||
|
# Rather than hand-write a bespoke deploy YAML, run on the bundled default deploy
|
||||||
|
# config and pass the matching 2-GPU device template via --stage-overrides (explicit
|
||||||
|
# kwargs): the thinker gets a two-GPU pool while talker + code2wav share a third
|
||||||
|
# card. Requires >= 3 GPUs.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# python3 end2end.py --query-type text --num-prompts 6 \
|
||||||
|
# --strategy-config strategy_stage_replica.yaml \
|
||||||
|
# --stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
|
||||||
|
strategies:
|
||||||
|
thinker:
|
||||||
|
- axis: stage_replica
|
||||||
|
size: 2
|
||||||
|
routing: round_robin
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Composable-parallel strategy for Qwen2.5-Omni: tensor-parallel the thinker.
|
||||||
|
#
|
||||||
|
# Shard the thinker stage across 2 GPUs with tensor parallelism (TP=2). TP routing
|
||||||
|
# is fixed (broadcast), so no `routing` / load-balancer policy is involved -- this
|
||||||
|
# is the simplest axis and a good starting point for the mental model.
|
||||||
|
#
|
||||||
|
# The strategy sets tensor_parallel_size=2 on the thinker. Rather than hand-write a
|
||||||
|
# bespoke deploy YAML, run on the bundled default deploy config and pass a matching
|
||||||
|
# 2-GPU device layout via --stage-overrides (explicit kwargs), so the thinker gets a
|
||||||
|
# two-GPU pool while talker + code2wav share a third card. Requires >= 3 GPUs.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# python3 end2end.py --query-type text --num-prompts 6 \
|
||||||
|
# --strategy-config strategy_tp2.yaml \
|
||||||
|
# --stage-overrides '{"0": {"devices": "0,1"}, "1": {"devices": "2"}, "2": {"devices": "2"}}'
|
||||||
|
strategies:
|
||||||
|
thinker:
|
||||||
|
- axis: tp
|
||||||
|
size: 2
|
||||||
@@ -0,0 +1,154 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Tests for applying strategy specs onto merged stage configs."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel import (
|
||||||
|
Broadcast,
|
||||||
|
FanInByStage,
|
||||||
|
MeshAxisSpec,
|
||||||
|
RouteByStage,
|
||||||
|
StrategyApplyError,
|
||||||
|
StrategySpec,
|
||||||
|
TakeRank,
|
||||||
|
apply_strategy_specs,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.stage_config import StageConfig
|
||||||
|
|
||||||
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
|
|
||||||
|
|
||||||
|
def _tp(size: int) -> StrategySpec:
|
||||||
|
return StrategySpec("tp", MeshAxisSpec("tp", size), Broadcast(), TakeRank())
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_replica(size: int, policy: str = "round_robin") -> StrategySpec:
|
||||||
|
return StrategySpec("stage_replica", MeshAxisSpec("stage_replica", size), RouteByStage(policy), FanInByStage())
|
||||||
|
|
||||||
|
|
||||||
|
def _stage(stage_id: int, model_stage: str, engine_args=None, runtime=None) -> StageConfig:
|
||||||
|
return StageConfig(
|
||||||
|
stage_id=stage_id,
|
||||||
|
model_stage=model_stage,
|
||||||
|
yaml_engine_args=dict(engine_args or {}),
|
||||||
|
yaml_runtime=dict(runtime or {"num_replicas": 1}),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _qwen_stages() -> list[StageConfig]:
|
||||||
|
return [
|
||||||
|
_stage(0, "thinker"),
|
||||||
|
_stage(1, "talker"),
|
||||||
|
_stage(2, "code2wav"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_tp_by_role():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
assert stages[0].yaml_engine_args["tensor_parallel_size"] == 2
|
||||||
|
# untouched roles keep their config
|
||||||
|
assert "tensor_parallel_size" not in stages[1].yaml_engine_args
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_by_model_stage():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
assert stages[0].yaml_engine_args["tensor_parallel_size"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_stage_replica_sets_num_replicas_and_surfaces_lb():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
result = apply_strategy_specs(stages, {"talker": [_stage_replica(2, "round_robin")]})
|
||||||
|
assert stages[1].yaml_runtime["num_replicas"] == 2
|
||||||
|
assert result.omni_lb_policy == "round-robin"
|
||||||
|
|
||||||
|
|
||||||
|
def test_only_declared_axes_are_written():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
# strategy declares only stage_replica -> tp must not be forced.
|
||||||
|
apply_strategy_specs(stages, {"talker": [_stage_replica(2)]})
|
||||||
|
assert "tensor_parallel_size" not in stages[1].yaml_engine_args
|
||||||
|
|
||||||
|
|
||||||
|
def test_conflict_on_explicit_tp():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
stages[0].yaml_engine_args["tensor_parallel_size"] = 4
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_equal_explicit_value_is_noop():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
stages[0].yaml_engine_args["tensor_parallel_size"] = 2
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
assert stages[0].yaml_engine_args["tensor_parallel_size"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_none_conflicts_with_derived_value():
|
||||||
|
# An explicit YAML ``null`` (``tensor_parallel_size: null``) is a *present*
|
||||||
|
# value, not a missing key, so a strategy deriving a non-None size must raise
|
||||||
|
# rather than silently clobber the explicit None.
|
||||||
|
stages = _qwen_stages()
|
||||||
|
stages[0].yaml_engine_args["tensor_parallel_size"] = None
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_key_is_filled_without_conflict():
|
||||||
|
# A genuinely absent key (never set in the YAML) is filled by the strategy
|
||||||
|
# and must NOT raise — the contrast case to an explicit None.
|
||||||
|
stages = _qwen_stages()
|
||||||
|
assert "tensor_parallel_size" not in stages[0].yaml_engine_args
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
assert stages[0].yaml_engine_args["tensor_parallel_size"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_num_replicas_conflict():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
stages[1].yaml_runtime["num_replicas"] = 3
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
apply_strategy_specs(stages, {"talker": [_stage_replica(2)]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_device_count_ok_template():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
stages[0].yaml_runtime["devices"] = "0,1"
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
assert stages[0].yaml_engine_args["tensor_parallel_size"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_device_count_ok_pool():
|
||||||
|
# tp=2 -> world=2; 2 replicas -> pool of 4 device ids is valid.
|
||||||
|
stages = _qwen_stages()
|
||||||
|
stages[1].yaml_runtime["devices"] = "0,1,2,3"
|
||||||
|
apply_strategy_specs(stages, {"talker": [_tp(2), _stage_replica(2)]})
|
||||||
|
assert stages[1].yaml_runtime["num_replicas"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_device_count_mismatch():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
stages[0].yaml_runtime["devices"] = "0,1,2"
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_unknown_role_raises():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
apply_strategy_specs(stages, {"nonexistent": [_tp(2)]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_conflicting_lb_policy_across_roles():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
apply_strategy_specs(
|
||||||
|
stages,
|
||||||
|
{
|
||||||
|
"talker": [_stage_replica(2, "round_robin")],
|
||||||
|
"code2wav": [_stage_replica(2, "least_queue")],
|
||||||
|
},
|
||||||
|
)
|
||||||
@@ -0,0 +1,141 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Integration tests: strategy overlay on the real merge_pipeline_deploy seam."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel import (
|
||||||
|
Broadcast,
|
||||||
|
FanInByStage,
|
||||||
|
MeshAxisSpec,
|
||||||
|
RouteByStage,
|
||||||
|
StrategyApplyError,
|
||||||
|
StrategySpec,
|
||||||
|
TakeRank,
|
||||||
|
apply_strategy_specs,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.config_factory import StageConfigFactory
|
||||||
|
from vllm_omni.config.pipeline_registry import OMNI_PIPELINES
|
||||||
|
from vllm_omni.config.stage_config import load_deploy_config, merge_pipeline_deploy
|
||||||
|
|
||||||
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
|
|
||||||
|
_DEPLOY = Path(__file__).parents[3] / "vllm_omni" / "deploy" / "qwen2_5_omni.yaml"
|
||||||
|
|
||||||
|
|
||||||
|
def _tp(size: int) -> StrategySpec:
|
||||||
|
return StrategySpec("tp", MeshAxisSpec("tp", size), Broadcast(), TakeRank())
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_replica(size: int, policy: str = "round_robin") -> StrategySpec:
|
||||||
|
return StrategySpec("stage_replica", MeshAxisSpec("stage_replica", size), RouteByStage(policy), FanInByStage())
|
||||||
|
|
||||||
|
|
||||||
|
def _qwen_stages():
|
||||||
|
pipeline = OMNI_PIPELINES["qwen2_5_omni"]
|
||||||
|
deploy = load_deploy_config(_DEPLOY)
|
||||||
|
return merge_pipeline_deploy(pipeline, deploy)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage(stages, role):
|
||||||
|
return next(s for s in stages if s.model_stage == role)
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_tp_on_thinker():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
# The bundled deploy pins the thinker to one GPU; a TP=2 strategy needs a
|
||||||
|
# matching 2-GPU layout (mirrors what a TP2 deploy would declare).
|
||||||
|
_stage(stages, "thinker").yaml_runtime["devices"] = "0,1"
|
||||||
|
result = apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
assert _stage(result.stages, "thinker").yaml_engine_args["tensor_parallel_size"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_device_guard_rejects_tp_on_single_gpu_deploy():
|
||||||
|
# The bundled deploy pins the thinker to a single GPU, so the pre-spawn
|
||||||
|
# device check must refuse a TP=2 strategy on it.
|
||||||
|
stages = _qwen_stages()
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
apply_strategy_specs(stages, {"thinker": [_tp(2)]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_stage_replica_on_talker():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
result = apply_strategy_specs(stages, {"talker": [_stage_replica(2, "round_robin")]})
|
||||||
|
assert _stage(result.stages, "talker").yaml_runtime["num_replicas"] == 2
|
||||||
|
assert result.omni_lb_policy == "round-robin"
|
||||||
|
|
||||||
|
|
||||||
|
def test_overlay_mixed_roles():
|
||||||
|
stages = _qwen_stages()
|
||||||
|
_stage(stages, "thinker").yaml_runtime["devices"] = "0,1"
|
||||||
|
result = apply_strategy_specs(
|
||||||
|
stages,
|
||||||
|
{
|
||||||
|
"thinker": [_tp(2)],
|
||||||
|
"talker": [_stage_replica(2, "round_robin")],
|
||||||
|
"code2wav": [_stage_replica(2, "round_robin")],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
by_role = {s.model_stage: s for s in result.stages}
|
||||||
|
assert by_role["thinker"].yaml_engine_args["tensor_parallel_size"] == 2
|
||||||
|
assert by_role["talker"].yaml_runtime["num_replicas"] == 2
|
||||||
|
assert by_role["code2wav"].yaml_runtime["num_replicas"] == 2
|
||||||
|
assert result.omni_lb_policy == "round-robin"
|
||||||
|
|
||||||
|
|
||||||
|
def _resolved(stages, role):
|
||||||
|
"""Return a role's fully-resolved (post-CLI) OmegaConf stage config."""
|
||||||
|
stage = next(s for s in stages if s.model_stage == role)
|
||||||
|
return stage.to_omegaconf()
|
||||||
|
|
||||||
|
|
||||||
|
def test_device_check_survives_cli_override():
|
||||||
|
# Strategy replicates the talker (1-GPU template -> valid at apply time),
|
||||||
|
# but a CLI --stage_1_devices with 3 ids must NOT slip past the device
|
||||||
|
# guard: effective world=1, replicas=2 admits only 1 or 2 device ids.
|
||||||
|
pipeline_cfg = OMNI_PIPELINES["qwen2_5_omni"]
|
||||||
|
with pytest.raises(StrategyApplyError):
|
||||||
|
StageConfigFactory._create_from_registry(
|
||||||
|
"qwen2_5_omni",
|
||||||
|
pipeline_cfg,
|
||||||
|
cli_overrides={"stage_1_devices": "0,1,2"},
|
||||||
|
strategy_specs={"talker": [_stage_replica(2, "round_robin")]},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_overrides_strategy_with_warning():
|
||||||
|
# Strategy derives num_replicas=2 for the talker; a CLI override to 3 wins
|
||||||
|
# (it is the most explicit user action) but must be surfaced loudly rather
|
||||||
|
# than silently. The resulting layout (1 template device) stays valid.
|
||||||
|
#
|
||||||
|
# vLLM's logger sets propagate=False, so attach a handler directly to it
|
||||||
|
# rather than relying on pytest's caplog (which listens on the root logger).
|
||||||
|
messages: list[str] = []
|
||||||
|
|
||||||
|
class _Capture(logging.Handler):
|
||||||
|
def emit(self, record: logging.LogRecord) -> None:
|
||||||
|
messages.append(record.getMessage())
|
||||||
|
|
||||||
|
log = logging.getLogger("vllm_omni.config.config_factory")
|
||||||
|
handler = _Capture(level=logging.WARNING)
|
||||||
|
log.addHandler(handler)
|
||||||
|
try:
|
||||||
|
pipeline_cfg = OMNI_PIPELINES["qwen2_5_omni"]
|
||||||
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
|
"qwen2_5_omni",
|
||||||
|
pipeline_cfg,
|
||||||
|
cli_overrides={"stage_1_num_replicas": 3},
|
||||||
|
strategy_specs={"talker": [_stage_replica(2, "round_robin")]},
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
log.removeHandler(handler)
|
||||||
|
|
||||||
|
# CLI value wins in the resolved config.
|
||||||
|
assert _resolved(stages, "talker").runtime.num_replicas == 3
|
||||||
|
# ...and the override was warned about, naming the conflicting field.
|
||||||
|
assert any("num_replicas" in m and "overrides the strategy-derived" in m for m in messages)
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Tests for the vendored spec/pattern data types."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel.aggregation import AggregationPattern, TakeRank
|
||||||
|
from vllm_omni.config.composable_parallel.routing import RoutingPattern
|
||||||
|
from vllm_omni.config.composable_parallel.spec import MeshAxisSpec
|
||||||
|
|
||||||
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
|
|
||||||
|
|
||||||
|
def test_mesh_axis_rejects_bad_kind():
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
MeshAxisSpec(kind="not_a_kind", size=2) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_mesh_axis_rejects_nonpositive_size():
|
||||||
|
with pytest.raises(ValueError):
|
||||||
|
MeshAxisSpec(kind="tp", size=0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_routing_pattern_is_closed():
|
||||||
|
with pytest.raises(TypeError):
|
||||||
|
|
||||||
|
class RogueRouting(RoutingPattern): # noqa: F811
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def test_aggregation_pattern_is_closed():
|
||||||
|
with pytest.raises(TypeError):
|
||||||
|
|
||||||
|
class RogueAggregation(AggregationPattern): # noqa: F811
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def test_take_rank_type_guard():
|
||||||
|
with pytest.raises(TypeError):
|
||||||
|
TakeRank(rank=True) # bool is not a valid rank
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Tests for parsing/loading strategy files."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel.routing import Broadcast, RouteByStage
|
||||||
|
from vllm_omni.config.composable_parallel.strategy_loader import (
|
||||||
|
StrategyLoadError,
|
||||||
|
load_strategy_specs,
|
||||||
|
parse_strategy_specs,
|
||||||
|
)
|
||||||
|
|
||||||
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_basic():
|
||||||
|
data = {
|
||||||
|
"strategies": {
|
||||||
|
"thinker": [{"axis": "tp", "size": 2}],
|
||||||
|
"talker": [{"axis": "stage_replica", "size": 2, "routing": "round_robin"}],
|
||||||
|
}
|
||||||
|
}
|
||||||
|
specs = parse_strategy_specs(data)
|
||||||
|
assert set(specs) == {"thinker", "talker"}
|
||||||
|
|
||||||
|
(tp_spec,) = specs["thinker"]
|
||||||
|
assert tp_spec.mesh_axis.kind == "tp"
|
||||||
|
assert tp_spec.mesh_axis.size == 2
|
||||||
|
assert isinstance(tp_spec.routing, Broadcast)
|
||||||
|
|
||||||
|
(sr_spec,) = specs["talker"]
|
||||||
|
assert sr_spec.mesh_axis.kind == "stage_replica"
|
||||||
|
assert isinstance(sr_spec.routing, RouteByStage)
|
||||||
|
assert sr_spec.routing.routing_policy == "round_robin"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_without_strategies_key():
|
||||||
|
# A bare role mapping (no top-level "strategies") is also accepted.
|
||||||
|
specs = parse_strategy_specs({"thinker": [{"axis": "tp", "size": 4}]})
|
||||||
|
assert specs["thinker"][0].mesh_axis.size == 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_l1_owner_goes_to_shard_extension():
|
||||||
|
specs = parse_strategy_specs({"talker": [{"axis": "stage_replica", "size": 2, "l1_owner": "delegated"}]})
|
||||||
|
assert specs["talker"][0].shard_extension["l1_owner"] == "delegated"
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_axis_raises():
|
||||||
|
with pytest.raises(StrategyLoadError):
|
||||||
|
parse_strategy_specs({"thinker": [{"size": 2}]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_size_raises():
|
||||||
|
with pytest.raises(StrategyLoadError):
|
||||||
|
parse_strategy_specs({"thinker": [{"axis": "tp"}]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_routing_on_non_policy_axis_raises():
|
||||||
|
with pytest.raises(StrategyLoadError):
|
||||||
|
parse_strategy_specs({"thinker": [{"axis": "tp", "size": 2, "routing": "random"}]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_bad_size_raises():
|
||||||
|
with pytest.raises(StrategyLoadError):
|
||||||
|
parse_strategy_specs({"thinker": [{"axis": "tp", "size": "two"}]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_entries_must_be_list():
|
||||||
|
with pytest.raises(StrategyLoadError):
|
||||||
|
parse_strategy_specs({"thinker": {"axis": "tp", "size": 2}})
|
||||||
|
|
||||||
|
|
||||||
|
def test_non_mapping_entry_raises():
|
||||||
|
# A list whose elements are not mappings (e.g. a bare string) must raise a
|
||||||
|
# StrategyLoadError, not an opaque TypeError from dict(entry).
|
||||||
|
with pytest.raises(StrategyLoadError):
|
||||||
|
parse_strategy_specs({"thinker": ["tp"]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_load_from_file(tmp_path):
|
||||||
|
path = tmp_path / "strategy.yaml"
|
||||||
|
path.write_text(
|
||||||
|
"strategies:\n"
|
||||||
|
" thinker:\n"
|
||||||
|
" - axis: tp\n"
|
||||||
|
" size: 2\n"
|
||||||
|
" talker:\n"
|
||||||
|
" - axis: stage_replica\n"
|
||||||
|
" size: 2\n"
|
||||||
|
" routing: least_queue\n"
|
||||||
|
)
|
||||||
|
specs = load_strategy_specs(str(path))
|
||||||
|
assert specs["thinker"][0].mesh_axis.size == 2
|
||||||
|
assert specs["talker"][0].routing.routing_policy == "least_queue"
|
||||||
@@ -0,0 +1,195 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Tests for StrategySpec -> OmniParallelConfig translation."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import dataclasses
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel import (
|
||||||
|
AxisTranslationError,
|
||||||
|
Broadcast,
|
||||||
|
FanInByStage,
|
||||||
|
MeshAxisSpec,
|
||||||
|
PartitionByHash,
|
||||||
|
PipelineMicrobatch,
|
||||||
|
RouteByStage,
|
||||||
|
StrategySpec,
|
||||||
|
TakeRank,
|
||||||
|
Union,
|
||||||
|
translate_strategy_stack,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.aggregation import StitchPipeline
|
||||||
|
|
||||||
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
|
|
||||||
|
|
||||||
|
def _tp(size: int) -> StrategySpec:
|
||||||
|
return StrategySpec("tp", MeshAxisSpec("tp", size), Broadcast(), TakeRank())
|
||||||
|
|
||||||
|
|
||||||
|
def _dp(size: int, policy: str = "round_robin") -> StrategySpec:
|
||||||
|
return StrategySpec("dp", MeshAxisSpec("dp", size), RouteByStage(policy), Union())
|
||||||
|
|
||||||
|
|
||||||
|
def _pp(size: int) -> StrategySpec:
|
||||||
|
return StrategySpec("pp", MeshAxisSpec("pp", size), PipelineMicrobatch(), StitchPipeline())
|
||||||
|
|
||||||
|
|
||||||
|
def _ep(size: int) -> StrategySpec:
|
||||||
|
return StrategySpec("ep", MeshAxisSpec("ep", size), Broadcast(), Union())
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_replica(size: int, policy: str = "round_robin") -> StrategySpec:
|
||||||
|
return StrategySpec("stage_replica", MeshAxisSpec("stage_replica", size), RouteByStage(policy), FanInByStage())
|
||||||
|
|
||||||
|
|
||||||
|
def test_tp_only():
|
||||||
|
cfg = translate_strategy_stack([_tp(2)])
|
||||||
|
assert cfg.tensor_parallel_size == 2
|
||||||
|
assert cfg.data_parallel_size == 1
|
||||||
|
assert cfg.world_size == 2
|
||||||
|
assert cfg.stage_replica_size == 1
|
||||||
|
assert cfg.l1_owners == {"tp": "engine"}
|
||||||
|
|
||||||
|
|
||||||
|
def test_dp_is_engine_owned():
|
||||||
|
cfg = translate_strategy_stack([_dp(2)])
|
||||||
|
assert cfg.data_parallel_size == 2
|
||||||
|
assert cfg.world_size == 2
|
||||||
|
# engine DP does not emit an omni LB policy (vLLM's internal DP LB is used).
|
||||||
|
assert cfg.omni_lb_policy is None
|
||||||
|
assert cfg.as_engine_kwargs().get("omni_lb_policy") is None
|
||||||
|
assert cfg.l1_owners["dp"] == "engine"
|
||||||
|
|
||||||
|
|
||||||
|
def test_tp_times_dp_world_size():
|
||||||
|
cfg = translate_strategy_stack([_tp(2), _dp(2)])
|
||||||
|
assert cfg.world_size == 4
|
||||||
|
assert cfg.tensor_parallel_size == 2
|
||||||
|
assert cfg.data_parallel_size == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_replica_is_delegated():
|
||||||
|
cfg = translate_strategy_stack([_stage_replica(3, "least_queue")])
|
||||||
|
# stage_replica is NOT a world dimension.
|
||||||
|
assert cfg.world_size == 1
|
||||||
|
assert cfg.stage_replica_size == 3
|
||||||
|
assert cfg.omni_lb_policy == "least-queue-length"
|
||||||
|
assert cfg.l1_owners["stage_replica"] == "delegated"
|
||||||
|
# stage_replica_size is not an engine kwarg; it's a per-stage deploy knob.
|
||||||
|
assert "stage_replica_size" not in cfg.as_engine_kwargs()
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_replica_lb_policy_only_when_replicated():
|
||||||
|
cfg = translate_strategy_stack([_stage_replica(1)])
|
||||||
|
assert cfg.stage_replica_size == 1
|
||||||
|
assert cfg.as_engine_kwargs().get("omni_lb_policy") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_tp_with_stage_replica():
|
||||||
|
cfg = translate_strategy_stack([_tp(2), _stage_replica(2)])
|
||||||
|
assert cfg.world_size == 2 # tp only
|
||||||
|
assert cfg.tensor_parallel_size == 2
|
||||||
|
assert cfg.stage_replica_size == 2
|
||||||
|
assert cfg.omni_lb_policy == "round-robin"
|
||||||
|
|
||||||
|
|
||||||
|
def test_ep_must_match_tp_dp_product():
|
||||||
|
cfg = translate_strategy_stack([_tp(2), _ep(2)])
|
||||||
|
assert cfg.enable_expert_parallel is True
|
||||||
|
assert cfg.as_engine_kwargs()["enable_expert_parallel"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_ep_mismatch_raises():
|
||||||
|
with pytest.raises(Exception) as exc:
|
||||||
|
translate_strategy_stack([_tp(2), _ep(4)])
|
||||||
|
assert "must equal" in str(exc.value)
|
||||||
|
|
||||||
|
|
||||||
|
def test_duplicate_kind_rejected():
|
||||||
|
with pytest.raises(AxisTranslationError):
|
||||||
|
translate_strategy_stack([_tp(2), _tp(2)])
|
||||||
|
|
||||||
|
|
||||||
|
def test_unsupported_kind_rejected():
|
||||||
|
spec = StrategySpec("sp", MeshAxisSpec("sp_ulysses", 2), Broadcast(), TakeRank())
|
||||||
|
with pytest.raises(AxisTranslationError):
|
||||||
|
translate_strategy_stack([spec])
|
||||||
|
|
||||||
|
|
||||||
|
def test_dp_hash_routing_rejected():
|
||||||
|
spec = StrategySpec("dp", MeshAxisSpec("dp", 2), PartitionByHash(), Union())
|
||||||
|
with pytest.raises(NotImplementedError):
|
||||||
|
translate_strategy_stack([spec])
|
||||||
|
|
||||||
|
|
||||||
|
def test_dp_invalid_routing_policy_rejected():
|
||||||
|
spec = StrategySpec("dp", MeshAxisSpec("dp", 2), RouteByStage("bogus"), Union())
|
||||||
|
with pytest.raises(AxisTranslationError):
|
||||||
|
translate_strategy_stack([spec])
|
||||||
|
|
||||||
|
|
||||||
|
def test_lb_policy_never_in_engine_kwargs_even_when_replicated():
|
||||||
|
# omni_lb_policy is pipeline-wide (applied at engine construction), not a
|
||||||
|
# per-stage engine arg, so it must never leak into as_engine_kwargs().
|
||||||
|
cfg = translate_strategy_stack([_tp(2), _stage_replica(2, "least_queue")])
|
||||||
|
assert cfg.omni_lb_policy == "least-queue-length"
|
||||||
|
assert "omni_lb_policy" not in cfg.as_engine_kwargs()
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_replica_hash_routing_rejected():
|
||||||
|
spec = StrategySpec("sr", MeshAxisSpec("stage_replica", 2), RouteByStage("hash"), FanInByStage())
|
||||||
|
with pytest.raises(NotImplementedError):
|
||||||
|
translate_strategy_stack([spec])
|
||||||
|
|
||||||
|
|
||||||
|
def test_dp_wrong_owner_rejected():
|
||||||
|
spec = StrategySpec(
|
||||||
|
"dp",
|
||||||
|
MeshAxisSpec("dp", 2),
|
||||||
|
RouteByStage("round_robin"),
|
||||||
|
Union(),
|
||||||
|
shard_extension={"l1_owner": "delegated"},
|
||||||
|
)
|
||||||
|
with pytest.raises(AxisTranslationError):
|
||||||
|
translate_strategy_stack([spec])
|
||||||
|
|
||||||
|
|
||||||
|
def test_stage_replica_wrong_owner_rejected():
|
||||||
|
spec = StrategySpec(
|
||||||
|
"sr",
|
||||||
|
MeshAxisSpec("stage_replica", 2),
|
||||||
|
RouteByStage("round_robin"),
|
||||||
|
FanInByStage(),
|
||||||
|
shard_extension={"l1_owner": "engine"},
|
||||||
|
)
|
||||||
|
with pytest.raises(AxisTranslationError):
|
||||||
|
translate_strategy_stack([spec])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"specs",
|
||||||
|
[
|
||||||
|
[_tp(2)],
|
||||||
|
[_dp(2)],
|
||||||
|
[_tp(2), _dp(2)],
|
||||||
|
[_tp(2), _ep(2)],
|
||||||
|
[_tp(2), _stage_replica(2)],
|
||||||
|
[_pp(2)],
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_as_engine_kwargs_are_valid_omni_engine_args(specs):
|
||||||
|
# Guard against desync: every key emitted by as_engine_kwargs() must be a
|
||||||
|
# real OmniEngineArgs field, otherwise constructing the per-stage engine args
|
||||||
|
# would explode at runtime. Keep this CPU-only by checking field names rather
|
||||||
|
# than instantiating the engine.
|
||||||
|
from vllm_omni.engine.arg_utils import OmniEngineArgs
|
||||||
|
|
||||||
|
valid_fields = {f.name for f in dataclasses.fields(OmniEngineArgs)}
|
||||||
|
cfg = translate_strategy_stack(specs)
|
||||||
|
kwargs = cfg.as_engine_kwargs()
|
||||||
|
unknown = set(kwargs) - valid_fields
|
||||||
|
assert not unknown, f"as_engine_kwargs emitted non-OmniEngineArgs fields: {unknown}"
|
||||||
@@ -739,7 +739,7 @@ def test_resolve_stage_configs_injects_global_diffusion_attention_when_missing(m
|
|||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
engine_mod,
|
engine_mod,
|
||||||
"load_and_resolve_stage_configs",
|
"load_and_resolve_stage_configs",
|
||||||
lambda *args, **kwargs: ("dummy-config", [stage_cfg]),
|
lambda *args, **kwargs: ("dummy-config", [stage_cfg], None),
|
||||||
)
|
)
|
||||||
|
|
||||||
_config_path, stage_configs = engine._resolve_stage_configs(
|
_config_path, stage_configs = engine._resolve_stage_configs(
|
||||||
@@ -772,7 +772,7 @@ def test_resolve_stage_configs_preserves_stage_diffusion_attention(monkeypatch):
|
|||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
engine_mod,
|
engine_mod,
|
||||||
"load_and_resolve_stage_configs",
|
"load_and_resolve_stage_configs",
|
||||||
lambda *args, **kwargs: ("dummy-config", [stage_cfg]),
|
lambda *args, **kwargs: ("dummy-config", [stage_cfg], None),
|
||||||
)
|
)
|
||||||
|
|
||||||
_config_path, stage_configs = engine._resolve_stage_configs(
|
_config_path, stage_configs = engine._resolve_stage_configs(
|
||||||
@@ -798,7 +798,7 @@ def test_resolve_stage_configs_does_not_inject_diffusion_attention_into_llm_stag
|
|||||||
monkeypatch.setattr(
|
monkeypatch.setattr(
|
||||||
engine_mod,
|
engine_mod,
|
||||||
"load_and_resolve_stage_configs",
|
"load_and_resolve_stage_configs",
|
||||||
lambda *args, **kwargs: ("dummy-config", [stage_cfg]),
|
lambda *args, **kwargs: ("dummy-config", [stage_cfg], None),
|
||||||
)
|
)
|
||||||
|
|
||||||
_config_path, stage_configs = engine._resolve_stage_configs(
|
_config_path, stage_configs = engine._resolve_stage_configs(
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Tests for strategy-derived omni_lb_policy threading into AsyncOmniEngine.
|
||||||
|
|
||||||
|
``AsyncOmniEngine._apply_strategy_lb_policy`` only touches ``self._omni_lb_policy``
|
||||||
|
and module-level logging, so we exercise it with a lightweight stub ``self`` to
|
||||||
|
avoid standing up a full engine (which needs GPUs/model weights).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from vllm_omni.engine.async_omni_engine import AsyncOmniEngine
|
||||||
|
|
||||||
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
|
|
||||||
|
|
||||||
|
class _Stub:
|
||||||
|
def __init__(self, current: str = "random") -> None:
|
||||||
|
self._omni_lb_policy = current
|
||||||
|
|
||||||
|
|
||||||
|
_apply = AsyncOmniEngine._apply_strategy_lb_policy
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_derived_policy_is_noop():
|
||||||
|
stub = _Stub("random")
|
||||||
|
_apply(stub, None, {})
|
||||||
|
assert stub._omni_lb_policy == "random"
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_overrides_default_random():
|
||||||
|
# "random" is the engine default == "unset", so the strategy value wins.
|
||||||
|
stub = _Stub("random")
|
||||||
|
_apply(stub, "round-robin", {})
|
||||||
|
assert stub._omni_lb_policy == "round-robin"
|
||||||
|
|
||||||
|
|
||||||
|
def test_derived_overrides_when_flag_absent():
|
||||||
|
stub = _Stub("random")
|
||||||
|
_apply(stub, "least-queue-length", {"omni_lb_policy": None})
|
||||||
|
assert stub._omni_lb_policy == "least-queue-length"
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_matching_flag_is_fine():
|
||||||
|
stub = _Stub("round-robin")
|
||||||
|
_apply(stub, "round-robin", {"omni_lb_policy": "round-robin"})
|
||||||
|
assert stub._omni_lb_policy == "round-robin"
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_conflicting_flag_raises():
|
||||||
|
stub = _Stub("round-robin")
|
||||||
|
with pytest.raises(ValueError, match="Conflicting load-balancer policy"):
|
||||||
|
_apply(stub, "least-queue-length", {"omni_lb_policy": "round-robin"})
|
||||||
|
|
||||||
|
|
||||||
|
def test_explicit_random_is_treated_as_unset():
|
||||||
|
# An explicit --omni-lb-policy=random is indistinguishable from the default,
|
||||||
|
# so a strategy value still overrides it (documented Phase-1 semantics).
|
||||||
|
stub = _Stub("random")
|
||||||
|
_apply(stub, "round-robin", {"omni_lb_policy": "random"})
|
||||||
|
assert stub._omni_lb_policy == "round-robin"
|
||||||
@@ -343,7 +343,7 @@ def test_resolve_stage_configs_injects_additional_config_into_diffusion_stage(mo
|
|||||||
)
|
)
|
||||||
mocker.patch(
|
mocker.patch(
|
||||||
"vllm_omni.engine.async_omni_engine.load_and_resolve_stage_configs",
|
"vllm_omni.engine.async_omni_engine.load_and_resolve_stage_configs",
|
||||||
return_value=("dummy.yaml", [fake_llm_stage, fake_diffusion_stage]),
|
return_value=("dummy.yaml", [fake_llm_stage, fake_diffusion_stage], None),
|
||||||
)
|
)
|
||||||
|
|
||||||
engine = AsyncOmniEngine.__new__(AsyncOmniEngine)
|
engine = AsyncOmniEngine.__new__(AsyncOmniEngine)
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ def test_dreamzero_config_sets_model_class_and_policy_config(monkeypatch):
|
|||||||
lambda _model: True,
|
lambda _model: True,
|
||||||
)
|
)
|
||||||
|
|
||||||
stage_configs = load_stage_configs_from_model("GEAR-Dreams/DreamZero-DROID")
|
stage_configs, _ = load_stage_configs_from_model("GEAR-Dreams/DreamZero-DROID")
|
||||||
engine_args = stage_configs[0].engine_args
|
engine_args = stage_configs[0].engine_args
|
||||||
|
|
||||||
assert engine_args.model_class_name == "DreamZeroPipeline"
|
assert engine_args.model_class_name == "DreamZeroPipeline"
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ import pytest
|
|||||||
from pytest_mock import MockerFixture
|
from pytest_mock import MockerFixture
|
||||||
|
|
||||||
from vllm_omni.entrypoints.cli.serve import OmniServeCommand, run_headless
|
from vllm_omni.entrypoints.cli.serve import OmniServeCommand, run_headless
|
||||||
|
from vllm_omni.entrypoints.utils import parse_stage_overrides
|
||||||
from vllm_omni.utils.tracking_parser import TrackingArgumentParser, TrackingNamespace
|
from vllm_omni.utils.tracking_parser import TrackingArgumentParser, TrackingNamespace
|
||||||
|
|
||||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||||
@@ -31,6 +32,20 @@ def test_serve_parser_accepts_no_async_chunk_and_marks_it_explicit() -> None:
|
|||||||
assert not explicit["async_chunk"]
|
assert not explicit["async_chunk"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_serve_parser_accepts_strategy_config() -> None:
|
||||||
|
"""``--strategy-config`` must parse onto the ``strategy_config`` dest and be
|
||||||
|
forwarded as an explicit kwarg so the engine can overlay the strategy."""
|
||||||
|
parser = TrackingArgumentParser()
|
||||||
|
subparsers = parser.add_subparsers(dest="subcommand")
|
||||||
|
cmd = OmniServeCommand()
|
||||||
|
cmd.subparser_init(subparsers)
|
||||||
|
|
||||||
|
argv = ["serve", "fake-model", "--omni", "--strategy-config", "/tmp/strategy.yaml"]
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
assert args.strategy_config == "/tmp/strategy.yaml"
|
||||||
|
assert args.get_explicit_kwargs_dict()["strategy_config"] == "/tmp/strategy.yaml"
|
||||||
|
|
||||||
|
|
||||||
def _make_headless_args(**kwargs) -> TrackingNamespace:
|
def _make_headless_args(**kwargs) -> TrackingNamespace:
|
||||||
defaults = {
|
defaults = {
|
||||||
"model": "fake-model",
|
"model": "fake-model",
|
||||||
@@ -80,13 +95,97 @@ def test_run_headless_rejects_non_multiprocess_worker_backend() -> None:
|
|||||||
run_headless(args)
|
run_headless(args)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# --stage-overrides parsing parity (headless vs standard path)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_stage_overrides_valid_json() -> None:
|
||||||
|
"""A valid JSON string is parsed into the nested per-stage dict."""
|
||||||
|
parsed = parse_stage_overrides('{"0": {"devices": "0,1"}, "1": {"devices": "2"}}')
|
||||||
|
assert parsed == {"0": {"devices": "0,1"}, "1": {"devices": "2"}}
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_stage_overrides_none_and_empty_return_none() -> None:
|
||||||
|
"""No overrides (None / empty string) resolve to ``None``."""
|
||||||
|
assert parse_stage_overrides(None) is None
|
||||||
|
assert parse_stage_overrides("") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_stage_overrides_empty_dict_returns_none() -> None:
|
||||||
|
"""An empty dict is falsy and must resolve to ``None``, locking in parity
|
||||||
|
with the original standard-path ``if stage_overrides_json:`` falsy check."""
|
||||||
|
assert parse_stage_overrides({}) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_stage_overrides_passes_through_non_str() -> None:
|
||||||
|
"""An already-parsed mapping is returned unchanged (identity)."""
|
||||||
|
overrides = {"0": {"devices": "0"}}
|
||||||
|
assert parse_stage_overrides(overrides) is overrides
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_stage_overrides_invalid_json_raises() -> None:
|
||||||
|
"""Invalid JSON raises ValueError whose message matches the standard path
|
||||||
|
verbatim: the ``--stage-overrides is not valid JSON:`` prefix AND the
|
||||||
|
``Got: <repr>`` suffix echoing the raw input."""
|
||||||
|
bad = "{not valid json}"
|
||||||
|
with pytest.raises(ValueError) as excinfo:
|
||||||
|
parse_stage_overrides(bad)
|
||||||
|
message = str(excinfo.value)
|
||||||
|
assert message.startswith("--stage-overrides is not valid JSON:")
|
||||||
|
assert f"Got: {bad!r}" in message
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_headless_parses_and_forwards_stage_overrides(mocker: MockerFixture) -> None:
|
||||||
|
"""Regression: the headless path must parse ``--stage-overrides`` (a JSON
|
||||||
|
string) and forward the parsed dict to ``load_and_resolve_stage_configs``,
|
||||||
|
mirroring the standard engine path. Previously it was dropped entirely,
|
||||||
|
silently producing a different per-stage device layout."""
|
||||||
|
captured: dict = {}
|
||||||
|
|
||||||
|
def _fake_resolve(*args, **kwargs):
|
||||||
|
captured.update(kwargs)
|
||||||
|
# Return a stage that does NOT match stage_id=0 so run_headless stops
|
||||||
|
# right after the resolver call (we only care about how it was called).
|
||||||
|
return ("/fake/stages.yaml", [SimpleNamespace(stage_id=99)], None)
|
||||||
|
|
||||||
|
mocker.patch(
|
||||||
|
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
||||||
|
side_effect=_fake_resolve,
|
||||||
|
)
|
||||||
|
|
||||||
|
args = _make_headless_args(
|
||||||
|
stage_id=0,
|
||||||
|
strategy_config="/tmp/strategy.yaml",
|
||||||
|
stage_overrides='{"0": {"devices": "0,1"}, "1": {"devices": "2"}}',
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError, match="No stage config found for stage_id=0"):
|
||||||
|
run_headless(args)
|
||||||
|
|
||||||
|
assert captured["stage_overrides"] == {"0": {"devices": "0,1"}, "1": {"devices": "2"}}
|
||||||
|
assert captured["strategy_config_path"] == "/tmp/strategy.yaml"
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_headless_invalid_stage_overrides_raises(mocker: MockerFixture) -> None:
|
||||||
|
"""Invalid ``--stage-overrides`` JSON in headless mode fails fast with the
|
||||||
|
shared ValueError instead of being silently ignored."""
|
||||||
|
mocker.patch(
|
||||||
|
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
||||||
|
return_value=("/fake/stages.yaml", [SimpleNamespace(stage_id=0)], None),
|
||||||
|
)
|
||||||
|
|
||||||
|
args = _make_headless_args(stage_id=0, stage_overrides="{not valid json}")
|
||||||
|
with pytest.raises(ValueError, match="--stage-overrides is not valid JSON"):
|
||||||
|
run_headless(args)
|
||||||
|
|
||||||
|
|
||||||
def test_run_headless_raises_when_stage_id_not_in_configs(mocker: MockerFixture) -> None:
|
def test_run_headless_raises_when_stage_id_not_in_configs(mocker: MockerFixture) -> None:
|
||||||
"""Headless looks up its assigned stage_id in the loaded deploy YAML and
|
"""Headless looks up its assigned stage_id in the loaded deploy YAML and
|
||||||
fails fast when the launcher's --stage-id doesn't match any entry."""
|
fails fast when the launcher's --stage-id doesn't match any entry."""
|
||||||
other_stage = SimpleNamespace(stage_id=99)
|
other_stage = SimpleNamespace(stage_id=99)
|
||||||
mocker.patch(
|
mocker.patch(
|
||||||
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
||||||
return_value=("/fake/stages.yaml", [other_stage]),
|
return_value=("/fake/stages.yaml", [other_stage], None),
|
||||||
)
|
)
|
||||||
|
|
||||||
args = _make_headless_args(stage_id=0)
|
args = _make_headless_args(stage_id=0)
|
||||||
@@ -134,7 +233,7 @@ def test_run_headless_llm_registers_with_auto_assigned_replica_id(mocker: Mocker
|
|||||||
|
|
||||||
mocker.patch(
|
mocker.patch(
|
||||||
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
||||||
return_value=("/fake/stages.yaml", [stage_cfg]),
|
return_value=("/fake/stages.yaml", [stage_cfg], None),
|
||||||
)
|
)
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
||||||
@@ -210,7 +309,7 @@ def test_run_headless_llm_launches_one_manager_per_omni_dp_size_local(mocker: Mo
|
|||||||
|
|
||||||
mocker.patch(
|
mocker.patch(
|
||||||
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
||||||
return_value=("/fake/stages.yaml", [stage_cfg]),
|
return_value=("/fake/stages.yaml", [stage_cfg], None),
|
||||||
)
|
)
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
||||||
@@ -269,7 +368,7 @@ def test_run_headless_diffusion_registers_and_spawns_proc(mocker: MockerFixture)
|
|||||||
|
|
||||||
mocker.patch(
|
mocker.patch(
|
||||||
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
||||||
return_value=("/fake/stages.yaml", [stage_cfg]),
|
return_value=("/fake/stages.yaml", [stage_cfg], None),
|
||||||
)
|
)
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
||||||
@@ -349,7 +448,7 @@ def test_run_headless_diffusion_raises_on_nonzero_proc_exit(mocker: MockerFixtur
|
|||||||
|
|
||||||
mocker.patch(
|
mocker.patch(
|
||||||
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
"vllm_omni.entrypoints.utils.load_and_resolve_stage_configs",
|
||||||
return_value=("/fake/stages.yaml", [stage_cfg]),
|
return_value=("/fake/stages.yaml", [stage_cfg], None),
|
||||||
)
|
)
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
mocker.patch("vllm_omni.engine.stage_init_utils.prepare_engine_environment")
|
||||||
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
mocker.patch("vllm_omni.engine.stage_init_utils.load_omni_transfer_config_for_model", return_value=None)
|
||||||
|
|||||||
@@ -318,7 +318,7 @@ class TestLoadAndResolveStageConfigs:
|
|||||||
def test_load_and_resolve_with_kwargs(self):
|
def test_load_and_resolve_with_kwargs(self):
|
||||||
"""Ensure that dtype survives default stage creation."""
|
"""Ensure that dtype survives default stage creation."""
|
||||||
kwargs = {"dtype": torch.float32}
|
kwargs = {"dtype": torch.float32}
|
||||||
config_path, stage_configs = load_and_resolve_stage_configs(
|
config_path, stage_configs, _ = load_and_resolve_stage_configs(
|
||||||
model="black-forest-labs/FLUX.2-klein-4B",
|
model="black-forest-labs/FLUX.2-klein-4B",
|
||||||
stage_configs_path=None,
|
stage_configs_path=None,
|
||||||
kwargs=kwargs,
|
kwargs=kwargs,
|
||||||
@@ -349,10 +349,10 @@ class TestLoadAndResolveStageConfigs:
|
|||||||
]
|
]
|
||||||
load_stage_configs = mocker.patch(
|
load_stage_configs = mocker.patch(
|
||||||
"vllm_omni.entrypoints.utils.load_stage_configs_from_model",
|
"vllm_omni.entrypoints.utils.load_stage_configs_from_model",
|
||||||
return_value=returned_stage_configs,
|
return_value=(returned_stage_configs, None),
|
||||||
)
|
)
|
||||||
|
|
||||||
config_path, stage_configs = load_and_resolve_stage_configs(
|
config_path, stage_configs, _ = load_and_resolve_stage_configs(
|
||||||
model="dummy-model",
|
model="dummy-model",
|
||||||
stage_configs_path=str(deploy_path),
|
stage_configs_path=str(deploy_path),
|
||||||
kwargs={},
|
kwargs={},
|
||||||
@@ -363,6 +363,7 @@ class TestLoadAndResolveStageConfigs:
|
|||||||
base_engine_args={},
|
base_engine_args={},
|
||||||
deploy_config_path=str(deploy_path),
|
deploy_config_path=str(deploy_path),
|
||||||
stage_overrides=None,
|
stage_overrides=None,
|
||||||
|
strategy_config_path=None,
|
||||||
)
|
)
|
||||||
assert config_path == str(deploy_path)
|
assert config_path == str(deploy_path)
|
||||||
assert len(stage_configs) == 2
|
assert len(stage_configs) == 2
|
||||||
|
|||||||
@@ -1721,7 +1721,7 @@ class TestAuraOmniDeploy:
|
|||||||
def test_aura_omni_deploy_resolves_four_native_stages(self):
|
def test_aura_omni_deploy_resolves_four_native_stages(self):
|
||||||
pipeline_cfg = StageConfigFactory.resolve_pipeline_config("aura_omni")
|
pipeline_cfg = StageConfigFactory.resolve_pipeline_config("aura_omni")
|
||||||
|
|
||||||
stages = StageConfigFactory._create_from_registry(
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
"qwen3_tts",
|
"qwen3_tts",
|
||||||
pipeline_cfg,
|
pipeline_cfg,
|
||||||
cli_overrides={},
|
cli_overrides={},
|
||||||
@@ -1943,11 +1943,12 @@ class TestSentinelDefaultPrecedence:
|
|||||||
model_type,
|
model_type,
|
||||||
Q3_OMNI_ALL_STAGES_HF_CONFIG,
|
Q3_OMNI_ALL_STAGES_HF_CONFIG,
|
||||||
)
|
)
|
||||||
return StageConfigFactory._create_from_registry(
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
"qwen3_omni_moe",
|
"qwen3_omni_moe",
|
||||||
pipeline_cfg,
|
pipeline_cfg,
|
||||||
cli_overrides=cli_overrides,
|
cli_overrides=cli_overrides,
|
||||||
)
|
)
|
||||||
|
return stages
|
||||||
|
|
||||||
def test_typed_kwarg_overrides_yaml(self):
|
def test_typed_kwarg_overrides_yaml(self):
|
||||||
stages = self._stages({"max_num_seqs": 999})
|
stages = self._stages({"max_num_seqs": 999})
|
||||||
|
|||||||
@@ -591,7 +591,7 @@ def test_explicit_cli_arg_reaches_runtime_overrides(mock_stages):
|
|||||||
ns = p.parse_args(["--max-num-seqs", "999"])
|
ns = p.parse_args(["--max-num-seqs", "999"])
|
||||||
|
|
||||||
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
||||||
stages = StageConfigFactory._create_from_registry(
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
_TEST_MODEL,
|
_TEST_MODEL,
|
||||||
_TEST_PIPELINE,
|
_TEST_PIPELINE,
|
||||||
explicit_kwargs,
|
explicit_kwargs,
|
||||||
@@ -608,7 +608,7 @@ def test_omitted_default_not_in_runtime_overrides(mock_stages):
|
|||||||
ns = p.parse_args([])
|
ns = p.parse_args([])
|
||||||
|
|
||||||
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
||||||
stages = StageConfigFactory._create_from_registry(
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
_TEST_MODEL,
|
_TEST_MODEL,
|
||||||
_TEST_PIPELINE,
|
_TEST_PIPELINE,
|
||||||
explicit_kwargs,
|
explicit_kwargs,
|
||||||
@@ -630,7 +630,7 @@ def test_config_file_args_reach_runtime_overrides(mock_stages):
|
|||||||
ns = p.parse_args(["--config", "fake.yaml"])
|
ns = p.parse_args(["--config", "fake.yaml"])
|
||||||
|
|
||||||
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
||||||
stages = StageConfigFactory._create_from_registry(
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
_TEST_MODEL,
|
_TEST_MODEL,
|
||||||
_TEST_PIPELINE,
|
_TEST_PIPELINE,
|
||||||
explicit_kwargs,
|
explicit_kwargs,
|
||||||
@@ -648,7 +648,7 @@ def test_per_stage_override_routes_correctly(mock_stages):
|
|||||||
ns = p.parse_args(["--stage-0-gpu-memory-utilization", "0.42"])
|
ns = p.parse_args(["--stage-0-gpu-memory-utilization", "0.42"])
|
||||||
|
|
||||||
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
||||||
stages = StageConfigFactory._create_from_registry(
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
_TEST_MODEL,
|
_TEST_MODEL,
|
||||||
_TEST_PIPELINE,
|
_TEST_PIPELINE,
|
||||||
explicit_kwargs,
|
explicit_kwargs,
|
||||||
@@ -670,7 +670,7 @@ def test_explicit_args_omitted_from_yaml(mock_stages):
|
|||||||
ns = p.parse_args(["--enforce-eager"])
|
ns = p.parse_args(["--enforce-eager"])
|
||||||
|
|
||||||
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
explicit_kwargs = ns.get_explicit_kwargs_dict()
|
||||||
stages = StageConfigFactory._create_from_registry(
|
stages, _ = StageConfigFactory._create_from_registry(
|
||||||
_TEST_MODEL,
|
_TEST_MODEL,
|
||||||
_TEST_PIPELINE,
|
_TEST_PIPELINE,
|
||||||
explicit_kwargs,
|
explicit_kwargs,
|
||||||
|
|||||||
@@ -0,0 +1,104 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Composable parallel strategies for vLLM-Omni.
|
||||||
|
|
||||||
|
A small, runtime-agnostic vocabulary for declaring a stage's parallel layout
|
||||||
|
(:class:`StrategySpec` + mesh/routing/aggregation patterns), plus the logic to
|
||||||
|
translate a per-role stack of specs into concrete engine sizing and overlay it
|
||||||
|
onto merged stage configs.
|
||||||
|
|
||||||
|
The spec/pattern *types* are deliberately pure data with no engine imports; the
|
||||||
|
translation and apply logic lives alongside them and consumes vLLM-Omni's own
|
||||||
|
config objects.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel.aggregation import (
|
||||||
|
AGGREGATION_PATTERN_VARIANTS,
|
||||||
|
AggregationConflictError,
|
||||||
|
AggregationPattern,
|
||||||
|
AllGather,
|
||||||
|
Combine,
|
||||||
|
FanInByStage,
|
||||||
|
GatherDim,
|
||||||
|
StitchPipeline,
|
||||||
|
StitchSpatial,
|
||||||
|
TakeRank,
|
||||||
|
Union,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.apply import (
|
||||||
|
StrategyApplyError,
|
||||||
|
StrategyApplyResult,
|
||||||
|
apply_strategy_specs,
|
||||||
|
check_device_layout,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.routing import (
|
||||||
|
ROUTING_PATTERN_VARIANTS,
|
||||||
|
Broadcast,
|
||||||
|
DuplicateWithCondTag,
|
||||||
|
PartitionByHash,
|
||||||
|
PipelineMicrobatch,
|
||||||
|
RouteByExpert,
|
||||||
|
RouteByStage,
|
||||||
|
RoutingKeyError,
|
||||||
|
RoutingPattern,
|
||||||
|
ShardSequence,
|
||||||
|
ShardSpatial,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.spec import (
|
||||||
|
MESH_AXIS_KINDS,
|
||||||
|
KernelSpec,
|
||||||
|
LayerHookSpec,
|
||||||
|
MeshAxisKind,
|
||||||
|
MeshAxisSpec,
|
||||||
|
SpecMergeConflictError,
|
||||||
|
StrategySpec,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.translator import (
|
||||||
|
AxisTranslationError,
|
||||||
|
OmniParallelConfig,
|
||||||
|
translate_strategy_stack,
|
||||||
|
)
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
# spec types
|
||||||
|
"MeshAxisKind",
|
||||||
|
"MESH_AXIS_KINDS",
|
||||||
|
"MeshAxisSpec",
|
||||||
|
"LayerHookSpec",
|
||||||
|
"KernelSpec",
|
||||||
|
"StrategySpec",
|
||||||
|
"SpecMergeConflictError",
|
||||||
|
# routing patterns
|
||||||
|
"RoutingPattern",
|
||||||
|
"RoutingKeyError",
|
||||||
|
"Broadcast",
|
||||||
|
"PartitionByHash",
|
||||||
|
"ShardSequence",
|
||||||
|
"ShardSpatial",
|
||||||
|
"RouteByExpert",
|
||||||
|
"PipelineMicrobatch",
|
||||||
|
"DuplicateWithCondTag",
|
||||||
|
"RouteByStage",
|
||||||
|
"ROUTING_PATTERN_VARIANTS",
|
||||||
|
# aggregation patterns
|
||||||
|
"AggregationPattern",
|
||||||
|
"AggregationConflictError",
|
||||||
|
"TakeRank",
|
||||||
|
"Union",
|
||||||
|
"GatherDim",
|
||||||
|
"AllGather",
|
||||||
|
"StitchSpatial",
|
||||||
|
"StitchPipeline",
|
||||||
|
"Combine",
|
||||||
|
"FanInByStage",
|
||||||
|
"AGGREGATION_PATTERN_VARIANTS",
|
||||||
|
# translation
|
||||||
|
"OmniParallelConfig",
|
||||||
|
"translate_strategy_stack",
|
||||||
|
"AxisTranslationError",
|
||||||
|
# apply
|
||||||
|
"apply_strategy_specs",
|
||||||
|
"StrategyApplyResult",
|
||||||
|
"check_device_layout",
|
||||||
|
"StrategyApplyError",
|
||||||
|
]
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Closed aggregation-pattern hierarchy for declarative parallel strategies.
|
||||||
|
|
||||||
|
An aggregation pattern describes how the per-worker results of one mesh axis
|
||||||
|
are recombined into a single logical result (e.g. take one tensor-parallel
|
||||||
|
rank's output, union disjoint data-parallel shards). The hierarchy is *closed*:
|
||||||
|
subclasses may only be defined in this module.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class AggregationPattern:
|
||||||
|
"""Base class for closed aggregation patterns. Subclass only within this module."""
|
||||||
|
|
||||||
|
def __init_subclass__(cls, **kwargs: object) -> None:
|
||||||
|
super().__init_subclass__(**kwargs)
|
||||||
|
if cls.__module__ != __name__:
|
||||||
|
raise TypeError("AggregationPattern is closed to external subclassing")
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
|
||||||
|
class AggregationConflictError(ValueError):
|
||||||
|
"""Raised when two aggregation inputs conflict for the same request id."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class TakeRank(AggregationPattern):
|
||||||
|
"""Select one rank's result (TP — all ranks agree)."""
|
||||||
|
|
||||||
|
rank: int = 0
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
if not isinstance(self.rank, int) or isinstance(self.rank, bool):
|
||||||
|
raise TypeError(f"TakeRank.rank must be int, got {type(self.rank).__name__}")
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "take_rank"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Union(AggregationPattern):
|
||||||
|
"""Union of disjoint replica results (DP, HSDP)."""
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "union"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class GatherDim(AggregationPattern):
|
||||||
|
"""Gather along a dimension and materialize on one designated rank.
|
||||||
|
|
||||||
|
Default designated rank is axis-rank 0 unless adapter overrides.
|
||||||
|
"""
|
||||||
|
|
||||||
|
dim: int = 1
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "gather_dim"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class AllGather(AggregationPattern):
|
||||||
|
"""All ranks materialize the same aggregate result across the axis."""
|
||||||
|
|
||||||
|
dim: int | None = None
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "all_gather"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class StitchSpatial(AggregationPattern):
|
||||||
|
"""Stitch spatial patches (VAE-PP)."""
|
||||||
|
|
||||||
|
grid: tuple[int, int] = (1, 1)
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "stitch_spatial"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class StitchPipeline(AggregationPattern):
|
||||||
|
"""Stitch pipeline stage outputs (PP)."""
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "stitch_pipeline"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Combine(AggregationPattern):
|
||||||
|
"""Custom reducer combination (CFG). Adapter resolves reducer_id."""
|
||||||
|
|
||||||
|
reducer_id: str = "default"
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "combine"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FanInByStage(AggregationPattern):
|
||||||
|
"""Fan-in results routed by stage (stage replicas)."""
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "fan_in_by_stage"
|
||||||
|
|
||||||
|
|
||||||
|
AGGREGATION_PATTERN_VARIANTS: tuple[type[AggregationPattern], ...] = (
|
||||||
|
TakeRank,
|
||||||
|
Union,
|
||||||
|
GatherDim,
|
||||||
|
AllGather,
|
||||||
|
StitchSpatial,
|
||||||
|
StitchPipeline,
|
||||||
|
Combine,
|
||||||
|
FanInByStage,
|
||||||
|
)
|
||||||
|
|
||||||
|
ReducerFn = Callable[..., object]
|
||||||
@@ -0,0 +1,264 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Apply declarative parallel strategies onto merged stage configs.
|
||||||
|
|
||||||
|
This is the *override-after-merge* seam. ``merge_pipeline_deploy`` first fuses
|
||||||
|
the pipeline topology + deploy YAML into a list of ``StageConfig`` objects; this
|
||||||
|
module then overlays the engine sizing derived from a per-role
|
||||||
|
``StrategySpec`` stack onto those stages.
|
||||||
|
|
||||||
|
Design rules (one writer per axis):
|
||||||
|
|
||||||
|
* A strategy only writes the axes it actually declares. If a stack has no ``tp``
|
||||||
|
axis, this module never touches ``tensor_parallel_size`` — it does not force a
|
||||||
|
default of 1.
|
||||||
|
* **Conflict-on-explicit.** If a knob was already set explicitly by the deploy
|
||||||
|
YAML and the strategy derives a different value, we raise rather than silently
|
||||||
|
override. Equal values are a no-op; unset values are filled.
|
||||||
|
* **Pre-spawn device check.** When a stage declares ``devices``, the device
|
||||||
|
count must be consistent with the engine world size (``tp * dp * pp``) and the
|
||||||
|
replica count, so misconfigurations fail at config time instead of at spawn.
|
||||||
|
|
||||||
|
``omni_lb_policy`` is *not* a per-stage config knob: omni reads it once at
|
||||||
|
orchestrator construction (``AsyncOmniEngine``), not from stage configs. So a
|
||||||
|
stage_replica axis's derived policy is surfaced on :class:`StrategyApplyResult`
|
||||||
|
for the caller to pass to the orchestrator; it is never silently written into a
|
||||||
|
stage where it would be ignored.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any, NoReturn
|
||||||
|
|
||||||
|
from vllm.logger import init_logger
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel.spec import StrategySpec
|
||||||
|
from vllm_omni.config.composable_parallel.translator import (
|
||||||
|
OmniParallelConfig,
|
||||||
|
translate_strategy_stack,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = init_logger(__name__)
|
||||||
|
|
||||||
|
# Role key -> the StageConfig fields each declared axis writes.
|
||||||
|
_ENGINE_FIELD_BY_KIND: dict[str, str] = {
|
||||||
|
"tp": "tensor_parallel_size",
|
||||||
|
"dp": "data_parallel_size",
|
||||||
|
"pp": "pipeline_parallel_size",
|
||||||
|
}
|
||||||
|
|
||||||
|
RoleKey = str
|
||||||
|
|
||||||
|
|
||||||
|
class StrategyApplyError(ValueError):
|
||||||
|
"""Error for applying a strategy onto stage configs.
|
||||||
|
|
||||||
|
A single error type (rather than per-cause subclasses) keeps the public
|
||||||
|
surface small and consistent with the rest of the codebase; the specific
|
||||||
|
cause — an unmatched role, a conflict with an explicit deploy value, or a
|
||||||
|
device-count mismatch — is in the message and is logged before the raise so
|
||||||
|
it stays visible even when the type is unavailable to a caller (e.g. across a
|
||||||
|
server boundary).
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _fail(msg: str) -> NoReturn:
|
||||||
|
"""Log and raise :class:`StrategyApplyError`."""
|
||||||
|
logger.error("[composable_parallel] %s", msg)
|
||||||
|
raise StrategyApplyError(msg)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class StrategyApplyResult:
|
||||||
|
"""Outcome of applying strategies to a stage list.
|
||||||
|
|
||||||
|
``stages`` is the same list passed in (mutated in place and returned for
|
||||||
|
convenience). ``omni_lb_policy`` is the pipeline-wide load-balancer policy
|
||||||
|
derived from any stage_replica axes (``None`` if none declared one); the
|
||||||
|
caller wires it into the orchestrator, since omni does not read it from
|
||||||
|
stage configs.
|
||||||
|
|
||||||
|
``per_role_config`` is keyed by the role key the caller supplied — a
|
||||||
|
``model_stage`` name (str). ``per_stage_config`` is the same information
|
||||||
|
keyed by the resolved integer ``stage_id`` — handy for re-validating a
|
||||||
|
stage after later layers (e.g. CLI overrides) have run.
|
||||||
|
"""
|
||||||
|
|
||||||
|
stages: list[Any]
|
||||||
|
omni_lb_policy: str | None = None
|
||||||
|
per_role_config: dict[RoleKey, OmniParallelConfig] = field(default_factory=dict)
|
||||||
|
per_stage_config: dict[int, OmniParallelConfig] = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_stage(stages: Sequence[Any], key: RoleKey) -> Any:
|
||||||
|
"""Find the single stage matching ``key`` by ``model_stage`` name.
|
||||||
|
|
||||||
|
Role keys are ``model_stage`` names (str) — the same human-readable names
|
||||||
|
that strategy.yaml authors use to refer to a stage (e.g. ``"thinker"``).
|
||||||
|
"""
|
||||||
|
matches = [s for s in stages if getattr(s, "model_stage", None) == key]
|
||||||
|
descriptor = f"model_stage={key!r}"
|
||||||
|
|
||||||
|
if not matches:
|
||||||
|
available = ", ".join(f"{getattr(s, 'model_stage', '?')!r}(id={getattr(s, 'stage_id', '?')})" for s in stages)
|
||||||
|
_fail(f"strategy role {descriptor} did not match any stage; available stages: {available}")
|
||||||
|
if len(matches) > 1:
|
||||||
|
_fail(f"strategy role {descriptor} is ambiguous; it matched {len(matches)} stages")
|
||||||
|
return matches[0]
|
||||||
|
|
||||||
|
|
||||||
|
def _set_explicit(engine_args: dict[str, Any], field_name: str, value: Any, *, role: RoleKey) -> None:
|
||||||
|
"""Write ``field_name`` with conflict-on-explicit semantics.
|
||||||
|
|
||||||
|
``engine_args`` is the raw mapping loaded straight from the deploy YAML
|
||||||
|
(before the engine-args dataclass is constructed), so a key is *present* only
|
||||||
|
when the YAML set it explicitly. We therefore key off membership rather than
|
||||||
|
``.get``: an explicit ``None`` in the YAML is a real, present value and must
|
||||||
|
not be conflated with a key miss (which ``.get`` would report as ``None``).
|
||||||
|
"""
|
||||||
|
if field_name in engine_args and engine_args[field_name] != value:
|
||||||
|
existing = engine_args[field_name]
|
||||||
|
_fail(
|
||||||
|
f"role {role!r}: strategy derives {field_name}={value} but the deploy config already "
|
||||||
|
f"set {field_name}={existing}. Remove one of them (the strategy is the single writer "
|
||||||
|
"for the axes it declares)."
|
||||||
|
)
|
||||||
|
engine_args[field_name] = value
|
||||||
|
|
||||||
|
|
||||||
|
def _set_num_replicas(runtime: dict[str, Any], value: int, *, role: RoleKey) -> None:
|
||||||
|
"""Write ``num_replicas`` with conflict-on-explicit (treating 1 as unset)."""
|
||||||
|
existing = runtime.get("num_replicas", 1)
|
||||||
|
if existing not in (1, value):
|
||||||
|
_fail(
|
||||||
|
f"role {role!r}: strategy derives num_replicas={value} but the deploy config already "
|
||||||
|
f"set num_replicas={existing}. Remove one of them."
|
||||||
|
)
|
||||||
|
runtime["num_replicas"] = value
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_device_count(devices: Any) -> int | None:
|
||||||
|
"""Return the number of device ids in a ``devices`` value, or None."""
|
||||||
|
if devices is None:
|
||||||
|
return None
|
||||||
|
if isinstance(devices, (list, tuple)):
|
||||||
|
return len([d for d in devices])
|
||||||
|
text = str(devices).strip()
|
||||||
|
if not text:
|
||||||
|
return None
|
||||||
|
return len([d for d in text.split(",") if d.strip()])
|
||||||
|
|
||||||
|
|
||||||
|
def check_device_layout(
|
||||||
|
devices: Any,
|
||||||
|
*,
|
||||||
|
tensor_parallel_size: int,
|
||||||
|
data_parallel_size: int,
|
||||||
|
pipeline_parallel_size: int,
|
||||||
|
num_replicas: int,
|
||||||
|
role: RoleKey,
|
||||||
|
) -> None:
|
||||||
|
"""Validate a stage's device count against its world size (× replicas in pool mode).
|
||||||
|
|
||||||
|
Raises :class:`StrategyApplyError` when the declared ``devices`` count is
|
||||||
|
neither the per-replica world size nor the full ``num_replicas`` pool.
|
||||||
|
Exposed publicly so the resolved (post-CLI) layout can be re-checked after
|
||||||
|
later override layers, not just the strategy-derived snapshot.
|
||||||
|
"""
|
||||||
|
count = _parse_device_count(devices)
|
||||||
|
if count is None:
|
||||||
|
return
|
||||||
|
world = int(tensor_parallel_size) * int(data_parallel_size) * int(pipeline_parallel_size)
|
||||||
|
replicas = int(num_replicas or 1)
|
||||||
|
# Two valid shapes: a single per-replica template (== world) or the full
|
||||||
|
# pool across replicas (== replicas * world). This mirrors omni's own
|
||||||
|
# per-replica device splitter.
|
||||||
|
valid = {world, replicas * world}
|
||||||
|
if count not in valid:
|
||||||
|
_fail(
|
||||||
|
f"role {role!r}: declared {count} device id(s) but the strategy world size is {world} "
|
||||||
|
f"(tp={tensor_parallel_size} * dp={data_parallel_size} * pp={pipeline_parallel_size}). "
|
||||||
|
f"Provide either {world} (per-replica template) or {replicas * world} "
|
||||||
|
f"(num_replicas={replicas} pool)."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_devices(runtime: dict[str, Any], cfg: OmniParallelConfig, *, role: RoleKey) -> None:
|
||||||
|
"""Pre-spawn check on the strategy-derived snapshot (see :func:`check_device_layout`)."""
|
||||||
|
check_device_layout(
|
||||||
|
runtime.get("devices"),
|
||||||
|
tensor_parallel_size=cfg.tensor_parallel_size,
|
||||||
|
data_parallel_size=cfg.data_parallel_size,
|
||||||
|
pipeline_parallel_size=cfg.pipeline_parallel_size,
|
||||||
|
num_replicas=int(runtime.get("num_replicas", 1) or 1),
|
||||||
|
role=role,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_to_stage(stage: Any, cfg: OmniParallelConfig, *, role: RoleKey) -> None:
|
||||||
|
engine_args = stage.yaml_engine_args
|
||||||
|
runtime = stage.yaml_runtime
|
||||||
|
|
||||||
|
declared = set(cfg.l1_owners.keys())
|
||||||
|
for kind, field_name in _ENGINE_FIELD_BY_KIND.items():
|
||||||
|
if kind in declared:
|
||||||
|
_set_explicit(engine_args, field_name, getattr(cfg, field_name), role=role)
|
||||||
|
if "ep" in declared and cfg.enable_expert_parallel:
|
||||||
|
_set_explicit(engine_args, "enable_expert_parallel", True, role=role)
|
||||||
|
if "stage_replica" in declared:
|
||||||
|
_set_num_replicas(runtime, cfg.stage_replica_size, role=role)
|
||||||
|
|
||||||
|
_check_devices(runtime, cfg, role=role)
|
||||||
|
|
||||||
|
|
||||||
|
def apply_strategy_specs(
|
||||||
|
stages: list[Any],
|
||||||
|
strategy_specs: Mapping[RoleKey, Sequence[StrategySpec]],
|
||||||
|
) -> StrategyApplyResult:
|
||||||
|
"""Overlay per-role strategy specs onto a merged stage list.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
stages: the ``list[StageConfig]`` returned by ``merge_pipeline_deploy``.
|
||||||
|
strategy_specs: maps a role (a ``model_stage`` name, e.g. ``"thinker"``)
|
||||||
|
to that stage's stack of ``StrategySpec`` (one per declared mesh
|
||||||
|
axis). Role keys are ``model_stage`` names (str).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A :class:`StrategyApplyResult` with the mutated stages and the derived
|
||||||
|
pipeline-wide ``omni_lb_policy`` (if any stage_replica axis set one).
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
StrategyApplyError: a role matched zero or multiple stages, a derived
|
||||||
|
value conflicts with an explicit deploy value (including two roles
|
||||||
|
deriving different ``omni_lb_policy`` values), or a stage's device
|
||||||
|
count is inconsistent with its world size.
|
||||||
|
AxisTranslationError: the spec stack is invalid/unsupported.
|
||||||
|
NotImplementedError: the spec stack requests routing not built yet.
|
||||||
|
"""
|
||||||
|
result = StrategyApplyResult(stages=stages)
|
||||||
|
lb_policy: str | None = None
|
||||||
|
lb_owner: RoleKey | None = None
|
||||||
|
|
||||||
|
for key, specs in strategy_specs.items():
|
||||||
|
stage = _resolve_stage(stages, key)
|
||||||
|
cfg = translate_strategy_stack(specs)
|
||||||
|
_apply_to_stage(stage, cfg, role=key)
|
||||||
|
result.per_role_config[key] = cfg
|
||||||
|
stage_id = getattr(stage, "stage_id", None)
|
||||||
|
if stage_id is not None:
|
||||||
|
result.per_stage_config[int(stage_id)] = cfg
|
||||||
|
|
||||||
|
if cfg.stage_replica_size > 1 and cfg.omni_lb_policy is not None:
|
||||||
|
if lb_policy is not None and lb_policy != cfg.omni_lb_policy:
|
||||||
|
_fail(
|
||||||
|
f"role {key!r} derives omni_lb_policy={cfg.omni_lb_policy!r} but role "
|
||||||
|
f"{lb_owner!r} already derived {lb_policy!r}. The load-balancer policy is "
|
||||||
|
"pipeline-wide; only one value is allowed."
|
||||||
|
)
|
||||||
|
lb_policy = cfg.omni_lb_policy
|
||||||
|
lb_owner = key
|
||||||
|
|
||||||
|
result.omni_lb_policy = lb_policy
|
||||||
|
return result
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Closed routing-pattern hierarchy for declarative parallel strategies.
|
||||||
|
|
||||||
|
A routing pattern describes how a request batch is distributed across the
|
||||||
|
workers of one mesh axis (e.g. broadcast to every tensor-parallel rank, hash
|
||||||
|
partition across data-parallel ranks). The hierarchy is *closed*: subclasses
|
||||||
|
may only be defined in this module, so the set of patterns is a fixed,
|
||||||
|
exhaustively matchable contract.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RoutingPattern:
|
||||||
|
"""Base class for closed routing patterns. Subclass only within this module."""
|
||||||
|
|
||||||
|
def __init_subclass__(cls, **kwargs: object) -> None:
|
||||||
|
super().__init_subclass__(**kwargs)
|
||||||
|
if cls.__module__ != __name__:
|
||||||
|
raise TypeError("RoutingPattern is closed to external subclassing")
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
|
||||||
|
class RoutingKeyError(KeyError):
|
||||||
|
"""Raised when a routing key is missing from a request."""
|
||||||
|
|
||||||
|
def __init__(self, request_id: object, missing_key: str) -> None:
|
||||||
|
self.request_id = request_id
|
||||||
|
self.missing_key = missing_key
|
||||||
|
super().__init__(f"Missing routing key {missing_key!r} for request_id={request_id!r}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Broadcast(RoutingPattern):
|
||||||
|
"""Every worker in the group sees the whole batch (TP, dense EP)."""
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "broadcast"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class PartitionByHash(RoutingPattern):
|
||||||
|
"""Stable hash partition by request key (DP, HSDP)."""
|
||||||
|
|
||||||
|
key: str = "request_id"
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "partition_by_hash"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ShardSequence(RoutingPattern):
|
||||||
|
"""Shard along a sequence dimension (SP-Ulysses, SP-Ring, CP)."""
|
||||||
|
|
||||||
|
dim: int = 1
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "shard_sequence"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ShardSpatial(RoutingPattern):
|
||||||
|
"""Shard along a spatial grid (VAE-PP)."""
|
||||||
|
|
||||||
|
grid: tuple[int, int] = (1, 1)
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "shard_spatial"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RouteByExpert(RoutingPattern):
|
||||||
|
"""Sparse MoE expert routing."""
|
||||||
|
|
||||||
|
router_id: str = "default"
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "route_by_expert"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class PipelineMicrobatch(RoutingPattern):
|
||||||
|
"""Pipeline parallelism microbatch routing (PP)."""
|
||||||
|
|
||||||
|
microbatch_size: int = 1
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "pipeline_microbatch"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DuplicateWithCondTag(RoutingPattern):
|
||||||
|
"""Duplicate batch with a conditional tag (CFG)."""
|
||||||
|
|
||||||
|
flag_name: str = "guidance_branch"
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "duplicate_with_cond_tag"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class RouteByStage(RoutingPattern):
|
||||||
|
"""Route requests by pipeline stage policy (stage replicas).
|
||||||
|
|
||||||
|
The ``random``/``round_robin``/``least_queue`` policies describe stateless
|
||||||
|
load balancing that a runtime (e.g. an engine's built-in DP load balancer,
|
||||||
|
or the omni StagePool balancer) can realize. ``hash`` describes
|
||||||
|
deterministic, key-stable routing that a load balancer cannot honor on its
|
||||||
|
own (see ``PartitionByHash``).
|
||||||
|
"""
|
||||||
|
|
||||||
|
routing_policy: Literal["random", "round_robin", "least_queue", "hash"] = "round_robin"
|
||||||
|
|
||||||
|
def pattern_kind(self) -> str:
|
||||||
|
return "route_by_stage"
|
||||||
|
|
||||||
|
|
||||||
|
ROUTING_PATTERN_VARIANTS: tuple[type[RoutingPattern], ...] = (
|
||||||
|
Broadcast,
|
||||||
|
PartitionByHash,
|
||||||
|
ShardSequence,
|
||||||
|
ShardSpatial,
|
||||||
|
RouteByExpert,
|
||||||
|
PipelineMicrobatch,
|
||||||
|
DuplicateWithCondTag,
|
||||||
|
RouteByStage,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Optional callable slot for Combine-style custom routing reducers in adapters.
|
||||||
|
RouterFn = Callable[..., object]
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Core declarative types for parallel-strategy specification.
|
||||||
|
|
||||||
|
A :class:`StrategySpec` declares one parallelism scheme: the mesh axis it sizes
|
||||||
|
(kind + degree), how a batch is routed across that axis, and how per-worker
|
||||||
|
results are aggregated back. A *stack* of specs (one per axis) fully describes a
|
||||||
|
stage's parallel layout in a runtime-agnostic, data-only form that can be
|
||||||
|
translated into concrete engine sizing.
|
||||||
|
|
||||||
|
The mesh-axis kinds below enumerate every parallelism dimension the contract can
|
||||||
|
describe. Only a subset is wired end-to-end today (see ``translator.py``):
|
||||||
|
|
||||||
|
* **Wired** (translatable now): ``tp``, ``dp``, ``pp``, ``ep``,
|
||||||
|
``stage_replica``.
|
||||||
|
* **Reserved** (declared in the type system but not yet translatable — the
|
||||||
|
translator raises ``AxisTranslationError`` for them): ``sp_ulysses``,
|
||||||
|
``sp_ring``, ``cfg``, ``vae_pp``, ``hsdp``, ``stage_pp``, ``cp``.
|
||||||
|
|
||||||
|
Reserved kinds fail fast at translation time rather than silently doing nothing,
|
||||||
|
so declaring one is an explicit "not yet" rather than a no-op.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from enum import Enum
|
||||||
|
from typing import Any, Literal
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel.aggregation import AggregationPattern
|
||||||
|
from vllm_omni.config.composable_parallel.routing import RoutingPattern
|
||||||
|
|
||||||
|
|
||||||
|
class MeshAxisKind(str, Enum):
|
||||||
|
"""Enumerates every parallelism dimension the strategy contract can describe.
|
||||||
|
|
||||||
|
Subclassing ``str`` keeps existing string comparisons and dict keys working
|
||||||
|
(``MeshAxisKind.TP == "tp"`` and both hash the same), while making the type
|
||||||
|
refactor-safe: a typo'd kind is an ``AttributeError`` on the enum rather than
|
||||||
|
a silently-wrong bare string.
|
||||||
|
"""
|
||||||
|
|
||||||
|
TP = "tp"
|
||||||
|
DP = "dp"
|
||||||
|
PP = "pp"
|
||||||
|
EP = "ep"
|
||||||
|
SP_ULYSSES = "sp_ulysses"
|
||||||
|
SP_RING = "sp_ring"
|
||||||
|
CFG = "cfg"
|
||||||
|
VAE_PP = "vae_pp"
|
||||||
|
HSDP = "hsdp"
|
||||||
|
STAGE_PP = "stage_pp"
|
||||||
|
STAGE_REPLICA = "stage_replica"
|
||||||
|
CP = "cp"
|
||||||
|
|
||||||
|
|
||||||
|
# Bare string values, kept as a plain tuple so membership checks and error
|
||||||
|
# messages read as the kind names (``"tp"``) rather than enum reprs. ``str``-enum
|
||||||
|
# equality means both ``"tp"`` and ``MeshAxisKind.TP`` satisfy ``in`` checks.
|
||||||
|
MESH_AXIS_KINDS: tuple[str, ...] = tuple(k.value for k in MeshAxisKind)
|
||||||
|
|
||||||
|
HookCategory = Literal[
|
||||||
|
"linear",
|
||||||
|
"attention_pre",
|
||||||
|
"attention_post",
|
||||||
|
"ffn_pre",
|
||||||
|
"ffn_post",
|
||||||
|
"moe_dispatch",
|
||||||
|
"other",
|
||||||
|
]
|
||||||
|
|
||||||
|
HOOK_CATEGORY_ORDER: dict[HookCategory, int] = {
|
||||||
|
"linear": 0,
|
||||||
|
"attention_pre": 1,
|
||||||
|
"attention_post": 2,
|
||||||
|
"ffn_pre": 3,
|
||||||
|
"ffn_post": 4,
|
||||||
|
"moe_dispatch": 5,
|
||||||
|
"other": 6,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class SpecMergeConflictError(ValueError):
|
||||||
|
"""Raised when merged hook/kernel specs have conflicting declarations."""
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class MeshAxisSpec:
|
||||||
|
"""Declares one axis of a process mesh (kind + size)."""
|
||||||
|
|
||||||
|
kind: MeshAxisKind
|
||||||
|
size: int
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
if self.kind not in MESH_AXIS_KINDS:
|
||||||
|
raise ValueError(f"MeshAxisSpec.kind must be one of {MESH_AXIS_KINDS}, got {self.kind!r}")
|
||||||
|
if self.size <= 0:
|
||||||
|
raise ValueError(f"MeshAxisSpec.size must be > 0, got {self.size}")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class LayerHookSpec:
|
||||||
|
"""L2 hook slot referenced by StrategySpec; resolved by the model walker."""
|
||||||
|
|
||||||
|
hook_id: str
|
||||||
|
target: str | None = None
|
||||||
|
category: HookCategory = "other"
|
||||||
|
priority: int = 0
|
||||||
|
axis_index: int = 0
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class KernelSpec:
|
||||||
|
"""L3 kernel slot referenced by StrategySpec; resolved by the kernel registry."""
|
||||||
|
|
||||||
|
kernel_id: str
|
||||||
|
target: str | None = None
|
||||||
|
category: HookCategory = "other"
|
||||||
|
priority: int = 0
|
||||||
|
axis_index: int = 0
|
||||||
|
group_axis_kind: MeshAxisKind | None = None
|
||||||
|
requires_collective: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class StrategySpec:
|
||||||
|
"""Declarative contract for one parallelism scheme (data + hook/kernel slots)."""
|
||||||
|
|
||||||
|
name: str
|
||||||
|
mesh_axis: MeshAxisSpec
|
||||||
|
routing: RoutingPattern
|
||||||
|
aggregation: AggregationPattern
|
||||||
|
layer_hook_specs: tuple[LayerHookSpec, ...] = ()
|
||||||
|
kernel_specs: tuple[KernelSpec, ...] = ()
|
||||||
|
shard_extension: Mapping[str, Any] = field(default_factory=dict)
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Load per-role parallel strategies from a strategy file.
|
||||||
|
|
||||||
|
The strategy file is a small, optional companion to the deploy config. It maps a
|
||||||
|
logical role (``model_stage``, e.g. ``thinker``) to a list of axis declarations,
|
||||||
|
each of which becomes one :class:`StrategySpec`. YAML is the only format wired up
|
||||||
|
today, but the module/symbol names stay format-agnostic so a different source can
|
||||||
|
be added without renaming the public surface::
|
||||||
|
|
||||||
|
strategies:
|
||||||
|
thinker:
|
||||||
|
- axis: tp
|
||||||
|
size: 2
|
||||||
|
talker:
|
||||||
|
- axis: stage_replica
|
||||||
|
size: 2
|
||||||
|
routing: round_robin # random | round_robin | least_queue
|
||||||
|
code2wav:
|
||||||
|
- axis: stage_replica
|
||||||
|
size: 2
|
||||||
|
routing: round_robin
|
||||||
|
|
||||||
|
Per-axis fields:
|
||||||
|
|
||||||
|
* ``axis`` (required): mesh-axis kind (``tp``, ``dp``, ``pp``, ``ep``,
|
||||||
|
``stage_replica`` are translatable today).
|
||||||
|
* ``size`` (required): axis degree (> 0).
|
||||||
|
* ``routing`` (optional): stateless policy for ``dp`` / ``stage_replica`` axes
|
||||||
|
(``random`` / ``round_robin`` / ``least_queue``). Ignored for axes whose
|
||||||
|
routing is fixed (``tp`` broadcast, ``pp`` microbatch).
|
||||||
|
* ``l1_owner`` (optional): ``engine`` or ``delegated``; overrides the default
|
||||||
|
owner for the axis (recorded in ``shard_extension``).
|
||||||
|
* ``name`` (optional): a label for the spec; defaults to the axis kind.
|
||||||
|
|
||||||
|
Routing/aggregation patterns are filled in from sensible per-kind defaults so
|
||||||
|
the file stays compact; the translator validates the combination.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel.aggregation import (
|
||||||
|
AggregationPattern,
|
||||||
|
FanInByStage,
|
||||||
|
StitchPipeline,
|
||||||
|
TakeRank,
|
||||||
|
Union,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.routing import (
|
||||||
|
Broadcast,
|
||||||
|
PipelineMicrobatch,
|
||||||
|
RouteByStage,
|
||||||
|
RoutingPattern,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.spec import MeshAxisSpec, StrategySpec
|
||||||
|
from vllm_omni.config.yaml_util import load_yaml_config, to_dict
|
||||||
|
|
||||||
|
# Per-kind defaults for routing + aggregation, so the file only needs axis+size.
|
||||||
|
_ROUTING_POLICY_KINDS = frozenset({"dp", "stage_replica"})
|
||||||
|
|
||||||
|
|
||||||
|
class StrategyLoadError(ValueError):
|
||||||
|
"""Raised when a strategy file is malformed."""
|
||||||
|
|
||||||
|
|
||||||
|
def _default_routing(kind: str, routing_policy: str | None) -> RoutingPattern:
|
||||||
|
if kind in _ROUTING_POLICY_KINDS:
|
||||||
|
return RouteByStage(routing_policy=routing_policy or "round_robin") # type: ignore[arg-type]
|
||||||
|
if kind == "pp":
|
||||||
|
return PipelineMicrobatch()
|
||||||
|
# tp, ep, and any other kind default to broadcast (the translator rejects
|
||||||
|
# kinds it cannot handle before it ever inspects routing).
|
||||||
|
return Broadcast()
|
||||||
|
|
||||||
|
|
||||||
|
def _default_aggregation(kind: str) -> AggregationPattern:
|
||||||
|
if kind == "stage_replica":
|
||||||
|
return FanInByStage()
|
||||||
|
if kind == "pp":
|
||||||
|
return StitchPipeline()
|
||||||
|
if kind in ("dp", "ep"):
|
||||||
|
return Union()
|
||||||
|
return TakeRank()
|
||||||
|
|
||||||
|
|
||||||
|
def _build_spec(role: str, entry: Mapping[str, Any]) -> StrategySpec:
|
||||||
|
if "axis" not in entry:
|
||||||
|
raise StrategyLoadError(f"role {role!r}: each strategy entry needs an 'axis' field")
|
||||||
|
if "size" not in entry:
|
||||||
|
raise StrategyLoadError(f"role {role!r}: strategy entry for axis {entry['axis']!r} needs a 'size'")
|
||||||
|
|
||||||
|
kind = str(entry["axis"])
|
||||||
|
try:
|
||||||
|
size = int(entry["size"])
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise StrategyLoadError(f"role {role!r}: axis {kind!r} size must be an integer, got {entry['size']!r}") from exc
|
||||||
|
|
||||||
|
routing_policy = entry.get("routing")
|
||||||
|
if routing_policy is not None and kind not in _ROUTING_POLICY_KINDS:
|
||||||
|
raise StrategyLoadError(
|
||||||
|
f"role {role!r}: axis {kind!r} does not accept a 'routing' policy (only {sorted(_ROUTING_POLICY_KINDS)} do)"
|
||||||
|
)
|
||||||
|
|
||||||
|
shard_extension: dict[str, Any] = {}
|
||||||
|
l1_owner = entry.get("l1_owner")
|
||||||
|
if l1_owner is not None:
|
||||||
|
shard_extension["l1_owner"] = str(l1_owner)
|
||||||
|
|
||||||
|
return StrategySpec(
|
||||||
|
name=str(entry.get("name", kind)),
|
||||||
|
mesh_axis=MeshAxisSpec(kind=kind, size=size), # type: ignore[arg-type]
|
||||||
|
routing=_default_routing(kind, routing_policy),
|
||||||
|
aggregation=_default_aggregation(kind),
|
||||||
|
shard_extension=shard_extension,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_strategy_specs(data: Mapping[str, Any]) -> dict[str, list[StrategySpec]]:
|
||||||
|
"""Parse an already-loaded strategy mapping into per-role spec stacks."""
|
||||||
|
strategies = data.get("strategies", data)
|
||||||
|
if not isinstance(strategies, Mapping):
|
||||||
|
raise StrategyLoadError("strategy file must contain a 'strategies' mapping of role -> list of axes")
|
||||||
|
|
||||||
|
result: dict[str, list[StrategySpec]] = {}
|
||||||
|
for role, entries in strategies.items():
|
||||||
|
if not isinstance(entries, (list, tuple)):
|
||||||
|
raise StrategyLoadError(
|
||||||
|
f"role {role!r}: expected a list of axis declarations, got {type(entries).__name__}"
|
||||||
|
)
|
||||||
|
specs: list[StrategySpec] = []
|
||||||
|
for entry in entries:
|
||||||
|
if not isinstance(entry, Mapping):
|
||||||
|
raise StrategyLoadError(
|
||||||
|
f"role {role!r}: each axis declaration must be a mapping, got {type(entry).__name__}"
|
||||||
|
)
|
||||||
|
specs.append(_build_spec(str(role), dict(entry)))
|
||||||
|
result[str(role)] = specs
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def load_strategy_specs(path: str) -> dict[str, list[StrategySpec]]:
|
||||||
|
"""Load and parse a strategy file into per-role spec stacks."""
|
||||||
|
data = to_dict(load_yaml_config(path))
|
||||||
|
if not isinstance(data, Mapping):
|
||||||
|
raise StrategyLoadError(f"strategy file {path!r} did not parse to a mapping")
|
||||||
|
return parse_strategy_specs(data)
|
||||||
@@ -0,0 +1,378 @@
|
|||||||
|
# SPDX-License-Identifier: Apache-2.0
|
||||||
|
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||||
|
"""Translate a ``StrategySpec`` stack into vLLM-Omni parallel sizing.
|
||||||
|
|
||||||
|
Read a stack of strategy specs (one per mesh axis) and work out the
|
||||||
|
tensor/data/pipeline parallel sizes, the dense-EP flag, and the number of stage
|
||||||
|
replicas, plus who owns each axis's request routing. The result is a plain,
|
||||||
|
CPU-computable sizing struct (:class:`OmniParallelConfig`) keyed by the real
|
||||||
|
``OmniEngineArgs`` / ``EngineArgs`` field names, so the deploy layer can splat it
|
||||||
|
onto a stage's engine args.
|
||||||
|
|
||||||
|
The distinction this module enforces — engine data parallelism (a true vLLM
|
||||||
|
intra-engine world dimension) vs. omni stage replicas (independent engines fanned
|
||||||
|
out by omni's coordinator) — is documented on the axis validators that depend on
|
||||||
|
it (see :func:`_validate_dp` and :func:`_stage_replica_lb_policy`). Routing we do
|
||||||
|
not support yet (key-stable / affinity routing) raises ``NotImplementedError``;
|
||||||
|
any other invalid spec raises :class:`AxisTranslationError`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Literal, NoReturn, cast, get_args
|
||||||
|
|
||||||
|
from vllm.logger import init_logger
|
||||||
|
|
||||||
|
from vllm_omni.config.composable_parallel.routing import (
|
||||||
|
Broadcast,
|
||||||
|
PartitionByHash,
|
||||||
|
PipelineMicrobatch,
|
||||||
|
RouteByStage,
|
||||||
|
RoutingPattern,
|
||||||
|
)
|
||||||
|
from vllm_omni.config.composable_parallel.spec import MeshAxisKind, StrategySpec
|
||||||
|
|
||||||
|
logger = init_logger(__name__)
|
||||||
|
|
||||||
|
# Who owns an axis's request routing. A closed string type so an unexpected raw
|
||||||
|
# value is caught statically and at runtime (``_VALID_L1_OWNERS`` is derived from
|
||||||
|
# it, keeping one source of truth) rather than silently flowing through.
|
||||||
|
L1Owner = Literal["delegated", "engine"]
|
||||||
|
|
||||||
|
# DP, TP, PP, (dense) EP and stage_replica translate today. DP/TP/PP are true
|
||||||
|
# world dimensions vLLM realizes intra-engine; dense EP is a flag over the
|
||||||
|
# existing TP*DP ranks; stage_replica is an omni-coordinator-level fan-out of
|
||||||
|
# independent engine replicas (NOT a vLLM world dimension). The remaining kinds
|
||||||
|
# (sequence/context parallel, CFG, VAE pipelines, ...) need per-layer hooks or
|
||||||
|
# custom collectives and arrive in later stages.
|
||||||
|
_SUPPORTED_KINDS: tuple[MeshAxisKind, ...] = ("dp", "tp", "pp", "ep", "stage_replica")
|
||||||
|
|
||||||
|
# Which EngineArgs field each axis kind *sizes*. EP and stage_replica are
|
||||||
|
# deliberately absent: EP is not an independent world dimension but an
|
||||||
|
# ``enable_expert_parallel`` flag whose degree equals tp*dp; stage_replica is a
|
||||||
|
# per-stage deploy ``num_replicas`` count, not an intra-engine world dimension.
|
||||||
|
_AXIS_TO_ENGINE_FIELD: dict[MeshAxisKind, str] = {
|
||||||
|
"tp": "tensor_parallel_size",
|
||||||
|
"dp": "data_parallel_size",
|
||||||
|
"pp": "pipeline_parallel_size",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Default L1 owner per axis kind. ``stage_replica`` is the only delegated axis:
|
||||||
|
# omni's coordinator/StagePool owns routing across replicas. ``dp`` is engine
|
||||||
|
# data parallelism, realized by vLLM's own intra-engine DP load balancer.
|
||||||
|
_DEFAULT_L1_OWNER: dict[MeshAxisKind, L1Owner] = {
|
||||||
|
"dp": "engine",
|
||||||
|
"tp": "engine",
|
||||||
|
"pp": "engine",
|
||||||
|
"ep": "engine",
|
||||||
|
"stage_replica": "delegated",
|
||||||
|
}
|
||||||
|
|
||||||
|
# How a RouteByStage policy maps onto omni's load balancer policy string. These
|
||||||
|
# are the stateless options omni can actually do; note there's no "hash" here,
|
||||||
|
# because omni has no key-stable balancer.
|
||||||
|
_STAGE_POLICY_TO_OMNI_LB: dict[str, str] = {
|
||||||
|
"random": "random",
|
||||||
|
"round_robin": "round-robin",
|
||||||
|
"least_queue": "least-queue-length",
|
||||||
|
}
|
||||||
|
|
||||||
|
_VALID_L1_OWNERS: frozenset[str] = frozenset(get_args(L1Owner))
|
||||||
|
|
||||||
|
|
||||||
|
class AxisTranslationError(ValueError):
|
||||||
|
"""Error for an invalid or unsupported strategy spec.
|
||||||
|
|
||||||
|
A single error type (rather than a tree of subclasses) keeps the public
|
||||||
|
surface small and consistent with the rest of the codebase; the specific
|
||||||
|
cause is in the message and is logged before the raise so it is visible even
|
||||||
|
when the type is unavailable to a caller (e.g. across a server boundary).
|
||||||
|
|
||||||
|
Strategies that are *valid but not built yet* — key-stable / affinity routing
|
||||||
|
today — raise ``NotImplementedError`` instead, to distinguish "we haven't
|
||||||
|
implemented this" from "your config is wrong".
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _fail(msg: str) -> NoReturn:
|
||||||
|
"""Log and raise :class:`AxisTranslationError` for an invalid/unsupported spec."""
|
||||||
|
logger.error("[composable_parallel] %s", msg)
|
||||||
|
raise AxisTranslationError(msg)
|
||||||
|
|
||||||
|
|
||||||
|
def _not_implemented(msg: str) -> NoReturn:
|
||||||
|
"""Log and raise ``NotImplementedError`` for a valid-but-unbuilt strategy."""
|
||||||
|
logger.error("[composable_parallel] %s", msg)
|
||||||
|
raise NotImplementedError(msg)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class OmniParallelConfig:
|
||||||
|
"""Result of translating a spec stack into omni parallel sizing."""
|
||||||
|
|
||||||
|
tensor_parallel_size: int = 1
|
||||||
|
data_parallel_size: int = 1
|
||||||
|
pipeline_parallel_size: int = 1
|
||||||
|
# Dense expert parallelism: a flag over the TP*DP ranks, not a world axis.
|
||||||
|
enable_expert_parallel: bool = False
|
||||||
|
# Number of independent omni stage replicas (the ``stage_replica`` axis).
|
||||||
|
# This is NOT a vLLM world dimension; it is the per-stage deploy
|
||||||
|
# ``num_replicas`` count. 1 means a single (un-replicated) engine.
|
||||||
|
stage_replica_size: int = 1
|
||||||
|
# The omni StagePool LB policy string for the (delegated) stage_replica
|
||||||
|
# axis, if any. Only meaningful when ``stage_replica_size > 1``.
|
||||||
|
omni_lb_policy: str | None = None
|
||||||
|
# axis kind -> resolved L1 owner ("delegated" | "engine").
|
||||||
|
l1_owners: Mapping[MeshAxisKind, L1Owner] = field(default_factory=dict)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def world_size(self) -> int:
|
||||||
|
# EP and stage_replica are intentionally excluded. EP reuses the TP*DP
|
||||||
|
# ranks rather than adding a dimension; stage_replica spins up separate
|
||||||
|
# engines (each its own world), not extra ranks in this engine's group.
|
||||||
|
# This product is what vLLM calls ``world_size_across_dp`` (tp * pp * dp).
|
||||||
|
return self.tensor_parallel_size * self.data_parallel_size * self.pipeline_parallel_size
|
||||||
|
|
||||||
|
@property
|
||||||
|
def delegated_axes(self) -> tuple[MeshAxisKind, ...]:
|
||||||
|
return tuple(kind for kind, owner in self.l1_owners.items() if owner == "delegated")
|
||||||
|
|
||||||
|
def as_engine_kwargs(self) -> dict[str, object]:
|
||||||
|
"""Return per-stage kwargs keyed by real OmniEngineArgs/EngineArgs field names.
|
||||||
|
|
||||||
|
Two derived values are intentionally *not* emitted here because they are
|
||||||
|
not per-stage engine args:
|
||||||
|
|
||||||
|
* ``stage_replica_size`` — a per-stage deploy ``num_replicas`` knob
|
||||||
|
(``StageDeployConfig``); the deploy layer consumes it separately.
|
||||||
|
* ``omni_lb_policy`` — a pipeline-wide load-balancer policy the engine
|
||||||
|
reads once at construction (see ``StrategyApplyResult.omni_lb_policy``);
|
||||||
|
it is applied at the orchestrator level, not folded into per-stage args.
|
||||||
|
"""
|
||||||
|
kwargs: dict[str, object] = {
|
||||||
|
"tensor_parallel_size": self.tensor_parallel_size,
|
||||||
|
"data_parallel_size": self.data_parallel_size,
|
||||||
|
"pipeline_parallel_size": self.pipeline_parallel_size,
|
||||||
|
}
|
||||||
|
if self.enable_expert_parallel:
|
||||||
|
kwargs["enable_expert_parallel"] = True
|
||||||
|
return kwargs
|
||||||
|
|
||||||
|
|
||||||
|
def _is_affinity_dp_routing(routing: RoutingPattern) -> bool:
|
||||||
|
"""True when DP routing demands key-stable (hash) placement."""
|
||||||
|
if isinstance(routing, PartitionByHash):
|
||||||
|
return True
|
||||||
|
if isinstance(routing, RouteByStage) and routing.routing_policy == "hash":
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_l1_owner(spec: StrategySpec) -> L1Owner:
|
||||||
|
kind = spec.mesh_axis.kind
|
||||||
|
declared_owner = spec.shard_extension.get("l1_owner")
|
||||||
|
owner = declared_owner
|
||||||
|
if owner is None:
|
||||||
|
owner = _DEFAULT_L1_OWNER.get(kind, "engine")
|
||||||
|
if kind not in _DEFAULT_L1_OWNER:
|
||||||
|
logger.debug(
|
||||||
|
"[composable_parallel] axis %r has unknown kind %r with no default l1_owner; falling back to %r",
|
||||||
|
spec.name,
|
||||||
|
kind,
|
||||||
|
owner,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.debug(
|
||||||
|
"[composable_parallel] axis %r (kind %r) declared no l1_owner; using default %r",
|
||||||
|
spec.name,
|
||||||
|
kind,
|
||||||
|
owner,
|
||||||
|
)
|
||||||
|
if owner not in _VALID_L1_OWNERS:
|
||||||
|
# Distinguish a bad value supplied via shard_extension from a bad default
|
||||||
|
# indexed out of _DEFAULT_L1_OWNER, so the source of the invalid owner is
|
||||||
|
# debuggable rather than swallowed.
|
||||||
|
source = "shard_extension" if declared_owner is not None else "_DEFAULT_L1_OWNER"
|
||||||
|
logger.debug(
|
||||||
|
"[composable_parallel] axis %r (kind %r) resolved invalid l1_owner %r from %s; valid owners are %s",
|
||||||
|
spec.name,
|
||||||
|
kind,
|
||||||
|
owner,
|
||||||
|
source,
|
||||||
|
sorted(_VALID_L1_OWNERS),
|
||||||
|
)
|
||||||
|
_fail(f"axis {kind!r} has invalid l1_owner {owner!r}; expected one of {sorted(_VALID_L1_OWNERS)}")
|
||||||
|
return cast(L1Owner, owner)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_dp(spec: StrategySpec, owner: L1Owner) -> None:
|
||||||
|
"""Validate an engine data-parallel axis.
|
||||||
|
|
||||||
|
DP is realized intra-engine by vLLM's own DP load balancer, so it is
|
||||||
|
engine-owned and emits no ``omni_lb_policy`` (that string configures omni's
|
||||||
|
*replica* balancer, a different layer). Key-stable (hash) request affinity is
|
||||||
|
not something vLLM's DP LB guarantees, so it is rejected rather than silently
|
||||||
|
dropped.
|
||||||
|
"""
|
||||||
|
if owner != "engine":
|
||||||
|
_fail(
|
||||||
|
f"dp axis {spec.name!r} is engine data parallelism realized intra-engine by "
|
||||||
|
f"vLLM's DP load balancer; l1_owner must be 'engine', got {owner!r}. For "
|
||||||
|
"omni-coordinator-level request fan-out across independent replicas, use a "
|
||||||
|
"'stage_replica' axis."
|
||||||
|
)
|
||||||
|
if _is_affinity_dp_routing(spec.routing):
|
||||||
|
_not_implemented(
|
||||||
|
f"dp axis {spec.name!r} requests key-stable (hash) routing, which vLLM's "
|
||||||
|
"intra-engine DP load balancer does not guarantee — not supported yet. Use "
|
||||||
|
"RouteByStage(random|round_robin|least_queue) for stateless DP balancing."
|
||||||
|
)
|
||||||
|
if not isinstance(spec.routing, RouteByStage):
|
||||||
|
_fail(
|
||||||
|
f"dp axis {spec.name!r} expects RouteByStage(random|round_robin|least_queue) routing, "
|
||||||
|
f"got {type(spec.routing).__name__}"
|
||||||
|
)
|
||||||
|
if spec.routing.routing_policy not in _STAGE_POLICY_TO_OMNI_LB:
|
||||||
|
# Recognized-but-unimplemented routing (key-stable/hash) is already
|
||||||
|
# handled above via _is_affinity_dp_routing -> _not_implemented. Anything
|
||||||
|
# left here is an unknown/invalid policy value (e.g. a typo from YAML),
|
||||||
|
# i.e. invalid input -> AxisTranslationError, not NotImplementedError.
|
||||||
|
_fail(
|
||||||
|
f"dp axis {spec.name!r} has invalid routing_policy "
|
||||||
|
f"{spec.routing.routing_policy!r}; expected one of "
|
||||||
|
f"{sorted(_STAGE_POLICY_TO_OMNI_LB)}."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _stage_replica_lb_policy(spec: StrategySpec, owner: L1Owner) -> str:
|
||||||
|
"""Return the omni StagePool LB policy for a delegated stage_replica axis.
|
||||||
|
|
||||||
|
A ``stage_replica`` axis is *not* a vLLM world dimension: it stands up N
|
||||||
|
independent engine replicas of one pipeline stage, coordinated by the omni
|
||||||
|
coordinator and balanced over a StagePool with stateless policies only
|
||||||
|
(random / round-robin / least-queue-length). It maps to the per-stage
|
||||||
|
``num_replicas`` count plus the pipeline-level ``omni_lb_policy`` string, so
|
||||||
|
its ``l1_owner`` must be ``"delegated"`` (omni owns the routing). Key-stable
|
||||||
|
(hash) routing is rejected because omni has no key-stable balancer yet.
|
||||||
|
"""
|
||||||
|
if owner != "delegated":
|
||||||
|
_fail(
|
||||||
|
f"stage_replica axis {spec.name!r} must be 'delegated' to omni's StagePool load "
|
||||||
|
f"balancer; got owner {owner!r}. Replica routing is owned by omni's coordinator."
|
||||||
|
)
|
||||||
|
|
||||||
|
routing = spec.routing
|
||||||
|
if _is_affinity_dp_routing(routing):
|
||||||
|
_not_implemented(
|
||||||
|
f"stage_replica axis {spec.name!r} requests key-stable (hash) routing, which needs a "
|
||||||
|
"dedicated load balancer — not implemented yet. Use "
|
||||||
|
"RouteByStage(random|round_robin|least_queue) to delegate to omni's load balancer."
|
||||||
|
)
|
||||||
|
if not isinstance(routing, RouteByStage):
|
||||||
|
_fail(
|
||||||
|
f"stage_replica axis {spec.name!r} expects RouteByStage(random|round_robin|least_queue) "
|
||||||
|
f"routing, got {type(routing).__name__}"
|
||||||
|
)
|
||||||
|
policy = _STAGE_POLICY_TO_OMNI_LB.get(routing.routing_policy)
|
||||||
|
if policy is None:
|
||||||
|
_not_implemented(
|
||||||
|
f"stage_replica axis {spec.name!r} routing_policy {routing.routing_policy!r} has no omni LB policy"
|
||||||
|
)
|
||||||
|
return policy
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_tp(spec: StrategySpec, owner: L1Owner) -> None:
|
||||||
|
if not isinstance(spec.routing, Broadcast):
|
||||||
|
_fail(f"tp axis {spec.name!r} expects Broadcast routing, got {type(spec.routing).__name__}")
|
||||||
|
if owner != "engine":
|
||||||
|
_fail(f"tp axis {spec.name!r} is realized intra-engine; l1_owner must be 'engine', got {owner!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_pp(spec: StrategySpec, owner: L1Owner) -> None:
|
||||||
|
if not isinstance(spec.routing, PipelineMicrobatch):
|
||||||
|
_fail(f"pp axis {spec.name!r} expects PipelineMicrobatch routing, got {type(spec.routing).__name__}")
|
||||||
|
if owner != "engine":
|
||||||
|
_fail(f"pp axis {spec.name!r} is realized intra-engine; l1_owner must be 'engine', got {owner!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_ep(spec: StrategySpec, owner: L1Owner) -> None:
|
||||||
|
# Dense EP: every rank still sees the whole batch, experts are sharded
|
||||||
|
# across ranks inside the engine (sparse MoE all-to-all is a later stage).
|
||||||
|
if not isinstance(spec.routing, Broadcast):
|
||||||
|
_fail(
|
||||||
|
f"ep axis {spec.name!r} expects Broadcast routing (dense expert parallel), "
|
||||||
|
f"got {type(spec.routing).__name__}"
|
||||||
|
)
|
||||||
|
if owner != "engine":
|
||||||
|
_fail(f"ep axis {spec.name!r} is realized intra-engine; l1_owner must be 'engine', got {owner!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def translate_strategy_stack(specs: Sequence[StrategySpec]) -> OmniParallelConfig:
|
||||||
|
"""Translate a spec stack into an ``OmniParallelConfig``.
|
||||||
|
|
||||||
|
Supported kinds: dp (engine data parallel), tp, pp, (dense) ep, and
|
||||||
|
stage_replica (omni replicas). Raises ``NotImplementedError`` for deferred
|
||||||
|
(affinity / key-stable) routing, and :class:`AxisTranslationError` for any
|
||||||
|
other invalid spec (a kind not yet translatable, a repeated kind, an owner
|
||||||
|
incompatible with the axis kind, or unsupported routing). The EP degree must
|
||||||
|
equal tensor_parallel_size * data_parallel_size.
|
||||||
|
"""
|
||||||
|
sizes: dict[str, int] = {"tensor_parallel_size": 1, "data_parallel_size": 1, "pipeline_parallel_size": 1}
|
||||||
|
owners: dict[MeshAxisKind, L1Owner] = {}
|
||||||
|
omni_lb_policy: str | None = None
|
||||||
|
stage_replica_size = 1
|
||||||
|
enable_expert_parallel = False
|
||||||
|
ep_size: int | None = None
|
||||||
|
|
||||||
|
for spec in specs:
|
||||||
|
kind = spec.mesh_axis.kind
|
||||||
|
if kind not in _SUPPORTED_KINDS:
|
||||||
|
_fail(
|
||||||
|
f"axis kind {kind!r} is not translatable yet (supported: {list(_SUPPORTED_KINDS)}); "
|
||||||
|
"it is designed-for and lands in a later stage"
|
||||||
|
)
|
||||||
|
if kind in owners:
|
||||||
|
_fail(f"axis kind {kind!r} appears more than once in the spec stack")
|
||||||
|
|
||||||
|
owner = _resolve_l1_owner(spec)
|
||||||
|
if kind == "dp":
|
||||||
|
_validate_dp(spec, owner)
|
||||||
|
elif kind == "tp":
|
||||||
|
_validate_tp(spec, owner)
|
||||||
|
elif kind == "pp":
|
||||||
|
_validate_pp(spec, owner)
|
||||||
|
elif kind == "ep":
|
||||||
|
_validate_ep(spec, owner)
|
||||||
|
enable_expert_parallel = True
|
||||||
|
ep_size = spec.mesh_axis.size
|
||||||
|
elif kind == "stage_replica":
|
||||||
|
omni_lb_policy = _stage_replica_lb_policy(spec, owner)
|
||||||
|
stage_replica_size = spec.mesh_axis.size
|
||||||
|
|
||||||
|
if kind in _AXIS_TO_ENGINE_FIELD:
|
||||||
|
sizes[_AXIS_TO_ENGINE_FIELD[kind]] = spec.mesh_axis.size
|
||||||
|
owners[kind] = owner
|
||||||
|
|
||||||
|
if enable_expert_parallel:
|
||||||
|
# Dense EP shards experts across exactly the TP*DP ranks, so the declared
|
||||||
|
# EP degree must match that product (it is not its own world dimension,
|
||||||
|
# and PP is excluded). An EP axis of size 1 is a degenerate no-op that
|
||||||
|
# still sets the flag; it is only valid when TP*DP == 1.
|
||||||
|
ep_ranks = sizes["tensor_parallel_size"] * sizes["data_parallel_size"]
|
||||||
|
if ep_size != ep_ranks:
|
||||||
|
_fail(
|
||||||
|
f"ep axis size {ep_size} must equal tensor_parallel_size*data_parallel_size "
|
||||||
|
f"(={ep_ranks}); dense expert parallelism shards experts across exactly those ranks"
|
||||||
|
)
|
||||||
|
|
||||||
|
return OmniParallelConfig(
|
||||||
|
tensor_parallel_size=sizes["tensor_parallel_size"],
|
||||||
|
data_parallel_size=sizes["data_parallel_size"],
|
||||||
|
pipeline_parallel_size=sizes["pipeline_parallel_size"],
|
||||||
|
enable_expert_parallel=enable_expert_parallel,
|
||||||
|
stage_replica_size=stage_replica_size,
|
||||||
|
omni_lb_policy=omni_lb_policy,
|
||||||
|
l1_owners=owners,
|
||||||
|
)
|
||||||
@@ -6,6 +6,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import dataclasses
|
import dataclasses
|
||||||
import functools
|
import functools
|
||||||
|
from collections.abc import Mapping
|
||||||
from dataclasses import asdict
|
from dataclasses import asdict
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
@@ -33,6 +34,16 @@ from vllm_omni.diffusion.utils.hf_utils import _looks_like_dreamzero
|
|||||||
logger = init_logger(__name__)
|
logger = init_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
# Default degree for any parallel axis / replica count that isn't set anywhere
|
||||||
|
# (CLI, deploy YAML, or pipeline default): a single, un-parallelized rank.
|
||||||
|
# TODO(composable_parallel): this "1" default and the per-axis device-layout
|
||||||
|
# fallbacks are currently re-derived in the merge, apply, and reconcile layers.
|
||||||
|
# Centralize them in one schema so the pre-spawn device guard and the engine-args
|
||||||
|
# defaults can't drift apart. This is the light slice; the full device-layout
|
||||||
|
# centralization is tracked as a follow-up.
|
||||||
|
_DEFAULT_PARALLEL_DEGREE = 1
|
||||||
|
|
||||||
|
|
||||||
class StageConfigFactory:
|
class StageConfigFactory:
|
||||||
"""Factory that loads pipeline YAML and merges CLI overrides.
|
"""Factory that loads pipeline YAML and merges CLI overrides.
|
||||||
|
|
||||||
@@ -267,14 +278,27 @@ class StageConfigFactory:
|
|||||||
trust_remote_code: bool = False,
|
trust_remote_code: bool = False,
|
||||||
cli_overrides: dict[str, Any] | None = None,
|
cli_overrides: dict[str, Any] | None = None,
|
||||||
deploy_config_path: str | None = None,
|
deploy_config_path: str | None = None,
|
||||||
|
strategy_specs: Mapping[Any, Any] | None = None,
|
||||||
**deprecated_kwargs: Any,
|
**deprecated_kwargs: Any,
|
||||||
) -> list[StageConfig] | None:
|
) -> tuple[list[StageConfig] | None, str | None]:
|
||||||
"""Load pipeline + deploy config, merge with CLI overrides.
|
"""Load pipeline + deploy config, merge with CLI overrides.
|
||||||
|
|
||||||
Checks OMNI_PIPELINES first, since supported models should be explicitly
|
Checks OMNI_PIPELINES first, since supported models should be explicitly
|
||||||
registered. If a model is not registered in OMNI_PIPELINES, tries to fall
|
registered. If a model is not registered in OMNI_PIPELINES, tries to fall
|
||||||
back to using the Transformers config & finding pipelines that have overlapping
|
back to using the Transformers config & finding pipelines that have overlapping
|
||||||
supported architectures.
|
supported architectures.
|
||||||
|
|
||||||
|
When ``strategy_specs`` is provided (a mapping of role -> list of
|
||||||
|
``StrategySpec``), the derived parallel sizing is overlaid onto the
|
||||||
|
merged stages (see ``vllm_omni.config.composable_parallel``). This is
|
||||||
|
opt-in: omitting it leaves the existing merge path untouched.
|
||||||
|
|
||||||
|
Returns ``(stages, omni_lb_policy)``: the merged stages (``None`` when the
|
||||||
|
model is not in the pipeline registry and the caller should fall back to
|
||||||
|
the legacy YAML path) and the strategy-derived, pipeline-wide
|
||||||
|
``omni_lb_policy`` (``None`` when no stage_replica axis set one). The
|
||||||
|
policy is returned rather than threaded through a mutable out-param so the
|
||||||
|
engine can apply it without every intermediate call carrying the dict.
|
||||||
"""
|
"""
|
||||||
if cli_overrides is None:
|
if cli_overrides is None:
|
||||||
cli_overrides = {}
|
cli_overrides = {}
|
||||||
@@ -290,8 +314,9 @@ class StageConfigFactory:
|
|||||||
pipeline_cfg,
|
pipeline_cfg,
|
||||||
cli_overrides,
|
cli_overrides,
|
||||||
deploy_config_path,
|
deploy_config_path,
|
||||||
|
strategy_specs=strategy_specs,
|
||||||
)
|
)
|
||||||
return None
|
return None, None
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def _create_from_registry(
|
def _create_from_registry(
|
||||||
@@ -300,12 +325,17 @@ class StageConfigFactory:
|
|||||||
pipeline_cfg: PipelineConfig,
|
pipeline_cfg: PipelineConfig,
|
||||||
cli_overrides: dict[str, Any],
|
cli_overrides: dict[str, Any],
|
||||||
deploy_config_path: str | None = None,
|
deploy_config_path: str | None = None,
|
||||||
|
strategy_specs: Mapping[Any, Any] | None = None,
|
||||||
**deprecated_kwargs: Any,
|
**deprecated_kwargs: Any,
|
||||||
) -> list[StageConfig]:
|
) -> tuple[list[StageConfig], str | None]:
|
||||||
"""Create StageConfigs from pipeline registry + deploy YAML.
|
"""Create StageConfigs from pipeline registry + deploy YAML.
|
||||||
|
|
||||||
Precedence: caller-typed (non-None) value > deploy YAML >
|
Precedence: caller-typed (non-None) value > deploy YAML >
|
||||||
StageDeployConfig dataclass default.
|
StageDeployConfig dataclass default.
|
||||||
|
|
||||||
|
Returns ``(stages, omni_lb_policy)`` — the strategy-derived pipeline-wide
|
||||||
|
load-balancer policy (``None`` when no strategy set one) travels with the
|
||||||
|
stages instead of through a mutable out-param.
|
||||||
"""
|
"""
|
||||||
# Resolve deploy config path
|
# Resolve deploy config path
|
||||||
if deploy_config_path is None:
|
if deploy_config_path is None:
|
||||||
@@ -338,12 +368,127 @@ class StageConfigFactory:
|
|||||||
|
|
||||||
stages = merge_pipeline_deploy(pipeline_cfg, deploy_cfg, cli_overrides)
|
stages = merge_pipeline_deploy(pipeline_cfg, deploy_cfg, cli_overrides)
|
||||||
|
|
||||||
|
# Overlay declarative parallel strategies (opt-in) before CLI overrides.
|
||||||
|
applied = cls._apply_strategy_specs(stages, strategy_specs)
|
||||||
|
|
||||||
explicit_overrides = {k: v for k, v in cli_overrides.items() if v is not None}
|
explicit_overrides = {k: v for k, v in cli_overrides.items() if v is not None}
|
||||||
|
|
||||||
for stage in stages:
|
for stage in stages:
|
||||||
stage.runtime_overrides = cls._merge_cli_overrides(stage, explicit_overrides)
|
stage.runtime_overrides = cls._merge_cli_overrides(stage, explicit_overrides)
|
||||||
|
|
||||||
return stages
|
# Re-validate the resolved layout now that CLI overrides are on top.
|
||||||
|
cls._reconcile_strategy_with_cli(stages, applied)
|
||||||
|
|
||||||
|
omni_lb_policy = applied.omni_lb_policy if applied is not None else None
|
||||||
|
return stages, omni_lb_policy
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _apply_strategy_specs(
|
||||||
|
stages: list[StageConfig],
|
||||||
|
strategy_specs: Mapping[Any, Any] | None,
|
||||||
|
) -> Any:
|
||||||
|
"""Overlay derived parallel sizing onto merged stages (opt-in).
|
||||||
|
|
||||||
|
``omni_lb_policy`` cannot be set from stage configs (omni reads it once
|
||||||
|
at orchestrator construction), so a derived non-default policy is logged
|
||||||
|
here and carried on the returned ``StrategyApplyResult`` for the caller
|
||||||
|
to hand to the orchestrator.
|
||||||
|
|
||||||
|
Returns the ``StrategyApplyResult`` (or ``None`` when no strategy was
|
||||||
|
supplied) so the caller can re-validate the resolved layout once CLI
|
||||||
|
overrides have been merged on top, and read its ``omni_lb_policy``.
|
||||||
|
"""
|
||||||
|
if not strategy_specs:
|
||||||
|
return None
|
||||||
|
from vllm_omni.config.composable_parallel import apply_strategy_specs
|
||||||
|
|
||||||
|
applied = apply_strategy_specs(stages, strategy_specs)
|
||||||
|
if applied.omni_lb_policy is not None:
|
||||||
|
logger.info(
|
||||||
|
"[composable_parallel] strategy derived omni_lb_policy=%r; it will be applied "
|
||||||
|
"to the orchestrator unless an explicit --omni-lb-policy was given.",
|
||||||
|
applied.omni_lb_policy,
|
||||||
|
)
|
||||||
|
return applied
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _reconcile_strategy_with_cli(
|
||||||
|
stages: list[StageConfig],
|
||||||
|
applied: Any,
|
||||||
|
) -> None:
|
||||||
|
"""Reconcile CLI overrides applied *after* a strategy overlay.
|
||||||
|
|
||||||
|
CLI overrides are applied last (in ``to_omegaconf``), so for any stage a
|
||||||
|
strategy touched we must (a) warn loudly when a CLI arg overrides a
|
||||||
|
strategy-declared axis — the strategy is meant to be the single writer
|
||||||
|
for the axes it declares, so a silent CLI win is surprising — and
|
||||||
|
(b) re-run the device-count check against the *effective* (post-CLI)
|
||||||
|
``tp``/``dp``/``pp``/``num_replicas``/``devices`` so a CLI
|
||||||
|
``--devices``/``--tensor-parallel-size``/``--num-replicas`` cannot slip
|
||||||
|
past the pre-spawn guard that exists to prevent silent OOMs.
|
||||||
|
"""
|
||||||
|
if applied is None:
|
||||||
|
return
|
||||||
|
from vllm_omni.config.composable_parallel import check_device_layout
|
||||||
|
|
||||||
|
# Axis kind -> (engine-arg field, strategy-derived attribute on the cfg).
|
||||||
|
axis_fields = {
|
||||||
|
"tp": ("tensor_parallel_size", "tensor_parallel_size"),
|
||||||
|
"dp": ("data_parallel_size", "data_parallel_size"),
|
||||||
|
"pp": ("pipeline_parallel_size", "pipeline_parallel_size"),
|
||||||
|
}
|
||||||
|
|
||||||
|
for stage in stages:
|
||||||
|
cfg = applied.per_stage_config.get(stage.stage_id)
|
||||||
|
if cfg is None:
|
||||||
|
continue
|
||||||
|
overrides = stage.runtime_overrides or {}
|
||||||
|
declared = set(cfg.l1_owners.keys())
|
||||||
|
|
||||||
|
for kind, (field_name, attr) in axis_fields.items():
|
||||||
|
if kind in declared and overrides.get(field_name) is not None:
|
||||||
|
cli_val = overrides[field_name]
|
||||||
|
derived = getattr(cfg, attr)
|
||||||
|
if cli_val != derived:
|
||||||
|
logger.warning(
|
||||||
|
"[composable_parallel] stage %s: CLI %s=%s overrides the "
|
||||||
|
"strategy-derived %s=%s. The CLI value wins; remove one to avoid ambiguity.",
|
||||||
|
stage.stage_id,
|
||||||
|
field_name,
|
||||||
|
cli_val,
|
||||||
|
field_name,
|
||||||
|
derived,
|
||||||
|
)
|
||||||
|
if "stage_replica" in declared and overrides.get("num_replicas") is not None:
|
||||||
|
cli_val = overrides["num_replicas"]
|
||||||
|
if cli_val != cfg.stage_replica_size:
|
||||||
|
logger.warning(
|
||||||
|
"[composable_parallel] stage %s: CLI num_replicas=%s overrides the "
|
||||||
|
"strategy-derived num_replicas=%s. The CLI value wins; remove one to avoid ambiguity.",
|
||||||
|
stage.stage_id,
|
||||||
|
cli_val,
|
||||||
|
cfg.stage_replica_size,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _eff(field_name: str, fallback: Any) -> Any:
|
||||||
|
val = overrides.get(field_name)
|
||||||
|
return val if val is not None else fallback
|
||||||
|
|
||||||
|
def _eff_degree(field_name: str, source: dict[str, Any]) -> int:
|
||||||
|
# Single place the per-axis "default to 1" fallback is applied,
|
||||||
|
# for both the override and the YAML-default sides (see the
|
||||||
|
# _DEFAULT_PARALLEL_DEGREE TODO).
|
||||||
|
value = _eff(field_name, source.get(field_name, _DEFAULT_PARALLEL_DEGREE))
|
||||||
|
return int(value or _DEFAULT_PARALLEL_DEGREE)
|
||||||
|
|
||||||
|
check_device_layout(
|
||||||
|
_eff("devices", stage.yaml_runtime.get("devices")),
|
||||||
|
tensor_parallel_size=_eff_degree("tensor_parallel_size", stage.yaml_engine_args),
|
||||||
|
data_parallel_size=_eff_degree("data_parallel_size", stage.yaml_engine_args),
|
||||||
|
pipeline_parallel_size=_eff_degree("pipeline_parallel_size", stage.yaml_engine_args),
|
||||||
|
num_replicas=_eff_degree("num_replicas", stage.yaml_runtime),
|
||||||
|
role=stage.model_stage,
|
||||||
|
)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def create_default_diffusion(cls, kwargs: dict[str, Any]) -> list[dict[str, Any]]:
|
def create_default_diffusion(cls, kwargs: dict[str, Any]) -> list[dict[str, Any]]:
|
||||||
|
|||||||
@@ -434,6 +434,9 @@ class OrchestratorArgs:
|
|||||||
stage_configs_path: str | None = None
|
stage_configs_path: str | None = None
|
||||||
deploy_config: str | None = None
|
deploy_config: str | None = None
|
||||||
stage_overrides: str | None = None # raw JSON string; parsed downstream
|
stage_overrides: str | None = None # raw JSON string; parsed downstream
|
||||||
|
# Optional composable-parallel strategy.yaml; orchestrator reads it, overlays
|
||||||
|
# derived sizing onto merged stages, then drops it before per-stage engine args.
|
||||||
|
strategy_config: str | None = None
|
||||||
|
|
||||||
# === Mode Switches (orchestrator reads, DeployConfig redistributes) ===
|
# === Mode Switches (orchestrator reads, DeployConfig redistributes) ===
|
||||||
async_chunk: bool | None = None
|
async_chunk: bool | None = None
|
||||||
|
|||||||
@@ -58,7 +58,10 @@ from vllm_omni.engine.stage_runtime import (
|
|||||||
create_stage_runtime,
|
create_stage_runtime,
|
||||||
)
|
)
|
||||||
from vllm_omni.entrypoints.pd_utils import PDDisaggregationMixin
|
from vllm_omni.entrypoints.pd_utils import PDDisaggregationMixin
|
||||||
from vllm_omni.entrypoints.utils import load_and_resolve_stage_configs
|
from vllm_omni.entrypoints.utils import (
|
||||||
|
load_and_resolve_stage_configs,
|
||||||
|
parse_stage_overrides,
|
||||||
|
)
|
||||||
from vllm_omni.inputs.data import OmniSamplingParams
|
from vllm_omni.inputs.data import OmniSamplingParams
|
||||||
from vllm_omni.metrics.prometheus import OmniRequestCounter
|
from vllm_omni.metrics.prometheus import OmniRequestCounter
|
||||||
|
|
||||||
@@ -1123,11 +1126,41 @@ class AsyncOmniEngine:
|
|||||||
|
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
def _apply_strategy_lb_policy(self, derived: str | None, kwargs: dict[str, Any]) -> None:
|
||||||
|
"""Apply a strategy-derived ``omni_lb_policy`` to the engine.
|
||||||
|
|
||||||
|
Precedence: an explicit ``--omni-lb-policy`` always wins. ``"random"`` is
|
||||||
|
the engine default and is treated as "unset" (indistinguishable from no
|
||||||
|
flag), so a strategy value overrides it. If the user explicitly passed a
|
||||||
|
non-default policy that conflicts with the strategy-derived one, raise so
|
||||||
|
the mismatch is not silently ignored.
|
||||||
|
"""
|
||||||
|
if not derived:
|
||||||
|
return
|
||||||
|
explicit = kwargs.get("omni_lb_policy")
|
||||||
|
user_set = explicit is not None and str(explicit) != "random"
|
||||||
|
if user_set:
|
||||||
|
if str(explicit) != str(derived):
|
||||||
|
raise ValueError(
|
||||||
|
f"Conflicting load-balancer policy: --omni-lb-policy={explicit!r} was given "
|
||||||
|
f"but the composable-parallel strategy derived omni_lb_policy={derived!r}. "
|
||||||
|
"Drop --omni-lb-policy to use the strategy value, or make them match."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if self._omni_lb_policy != str(derived):
|
||||||
|
logger.info(
|
||||||
|
"[composable_parallel] applying strategy-derived omni_lb_policy=%r (was %r).",
|
||||||
|
derived,
|
||||||
|
self._omni_lb_policy,
|
||||||
|
)
|
||||||
|
self._omni_lb_policy = str(derived)
|
||||||
|
|
||||||
def _resolve_stage_configs(self, model: str, kwargs: dict[str, Any]) -> tuple[str, list[Any]]:
|
def _resolve_stage_configs(self, model: str, kwargs: dict[str, Any]) -> tuple[str, list[Any]]:
|
||||||
"""Resolve stage configs and inject defaults shared by orchestrator/headless."""
|
"""Resolve stage configs and inject defaults shared by orchestrator/headless."""
|
||||||
|
|
||||||
stage_configs_path = kwargs.get("stage_configs_path", None)
|
stage_configs_path = kwargs.get("stage_configs_path", None)
|
||||||
deploy_config_path = kwargs.pop("deploy_config", None)
|
deploy_config_path = kwargs.pop("deploy_config", None)
|
||||||
|
strategy_config_path = kwargs.pop("strategy_config", None)
|
||||||
stage_overrides_json = kwargs.pop("stage_overrides", None)
|
stage_overrides_json = kwargs.pop("stage_overrides", None)
|
||||||
explicit_stage_configs = kwargs.pop("stage_configs", None)
|
explicit_stage_configs = kwargs.pop("stage_configs", None)
|
||||||
if explicit_stage_configs is not None:
|
if explicit_stage_configs is not None:
|
||||||
@@ -1142,27 +1175,23 @@ class AsyncOmniEngine:
|
|||||||
base_kwargs = kwargs
|
base_kwargs = kwargs
|
||||||
|
|
||||||
# Parse --stage-overrides JSON string if provided
|
# Parse --stage-overrides JSON string if provided
|
||||||
stage_overrides = None
|
stage_overrides = parse_stage_overrides(stage_overrides_json)
|
||||||
if stage_overrides_json:
|
|
||||||
if isinstance(stage_overrides_json, str):
|
|
||||||
try:
|
|
||||||
stage_overrides = json.loads(stage_overrides_json)
|
|
||||||
except json.JSONDecodeError as exc:
|
|
||||||
raise ValueError(
|
|
||||||
f"--stage-overrides is not valid JSON: {exc}. Got: {stage_overrides_json!r}"
|
|
||||||
) from exc
|
|
||||||
else:
|
|
||||||
stage_overrides = stage_overrides_json
|
|
||||||
|
|
||||||
config_path, stage_configs = load_and_resolve_stage_configs(
|
config_path, stage_configs, strategy_lb_policy = load_and_resolve_stage_configs(
|
||||||
model,
|
model,
|
||||||
stage_configs_path,
|
stage_configs_path,
|
||||||
base_kwargs,
|
base_kwargs,
|
||||||
default_stage_cfg_factory=lambda: self._create_default_diffusion_stage_cfg(kwargs),
|
default_stage_cfg_factory=lambda: self._create_default_diffusion_stage_cfg(kwargs),
|
||||||
deploy_config_path=deploy_config_path,
|
deploy_config_path=deploy_config_path,
|
||||||
stage_overrides=stage_overrides,
|
stage_overrides=stage_overrides,
|
||||||
|
strategy_config_path=strategy_config_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# A strategy.yaml may derive a pipeline-wide load-balancer policy. It is
|
||||||
|
# an orchestrator-level knob (read once at construction), so apply it here
|
||||||
|
# rather than as a per-stage config field.
|
||||||
|
self._apply_strategy_lb_policy(strategy_lb_policy, kwargs)
|
||||||
|
|
||||||
# Inject diffusion LoRA-related knobs from kwargs if not present in the stage config.
|
# Inject diffusion LoRA-related knobs from kwargs if not present in the stage config.
|
||||||
for cfg in stage_configs:
|
for cfg in stage_configs:
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -254,6 +254,15 @@ class OmniServeCommand(CLISubcommand):
|
|||||||
help="Path to a deploy config YAML (new format with stages/engine_args). "
|
help="Path to a deploy config YAML (new format with stages/engine_args). "
|
||||||
"Mutually exclusive with --stage-configs-path.",
|
"Mutually exclusive with --stage-configs-path.",
|
||||||
)
|
)
|
||||||
|
omni_config_group.add_argument(
|
||||||
|
"--strategy-config",
|
||||||
|
type=str,
|
||||||
|
default=None,
|
||||||
|
help="Path to a composable-parallel strategy.yaml. Only applies to "
|
||||||
|
"registry-based models (e.g. qwen2_5_omni): its derived parallel sizing "
|
||||||
|
"is overlaid onto the registry-merged stages before per-stage engine "
|
||||||
|
"args are built.",
|
||||||
|
)
|
||||||
omni_config_group.add_argument(
|
omni_config_group.add_argument(
|
||||||
"--stage-overrides",
|
"--stage-overrides",
|
||||||
type=str,
|
type=str,
|
||||||
@@ -766,7 +775,10 @@ def run_headless(args: TrackingNamespace) -> None:
|
|||||||
load_omni_transfer_config_for_model,
|
load_omni_transfer_config_for_model,
|
||||||
prepare_engine_environment,
|
prepare_engine_environment,
|
||||||
)
|
)
|
||||||
from vllm_omni.entrypoints.utils import load_and_resolve_stage_configs
|
from vllm_omni.entrypoints.utils import (
|
||||||
|
load_and_resolve_stage_configs,
|
||||||
|
parse_stage_overrides,
|
||||||
|
)
|
||||||
|
|
||||||
model = args.model
|
model = args.model
|
||||||
stage_id: int | None = args.stage_id
|
stage_id: int | None = args.stage_id
|
||||||
@@ -802,11 +814,18 @@ def run_headless(args: TrackingNamespace) -> None:
|
|||||||
args.replica_id,
|
args.replica_id,
|
||||||
)
|
)
|
||||||
|
|
||||||
config_path, stage_configs = load_and_resolve_stage_configs(
|
# Parse --stage-overrides (raw JSON string) exactly like the standard
|
||||||
|
# engine path (AsyncOmniEngine._resolve_stage_configs) so headless and
|
||||||
|
# standard launches resolve to the same per-stage device layout.
|
||||||
|
stage_overrides = parse_stage_overrides(args_dict.get("stage_overrides"))
|
||||||
|
|
||||||
|
config_path, stage_configs, _ = load_and_resolve_stage_configs(
|
||||||
model,
|
model,
|
||||||
stage_configs_path,
|
stage_configs_path,
|
||||||
args_dict,
|
args_dict,
|
||||||
deploy_config_path=args_dict.get("deploy_config"),
|
deploy_config_path=args_dict.get("deploy_config"),
|
||||||
|
stage_overrides=stage_overrides,
|
||||||
|
strategy_config_path=args_dict.get("strategy_config"),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Locate the stage config that matches stage_id.
|
# Locate the stage config that matches stage_id.
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import json
|
||||||
import os
|
import os
|
||||||
import types
|
import types
|
||||||
from collections import Counter
|
from collections import Counter
|
||||||
@@ -316,7 +317,8 @@ def load_stage_configs_from_model(
|
|||||||
base_engine_args: dict | None = None,
|
base_engine_args: dict | None = None,
|
||||||
deploy_config_path: str | None = None,
|
deploy_config_path: str | None = None,
|
||||||
stage_overrides: dict[str, dict[str, Any]] | None = None,
|
stage_overrides: dict[str, dict[str, Any]] | None = None,
|
||||||
) -> list:
|
strategy_config_path: str | None = None,
|
||||||
|
) -> tuple[list, str | None]:
|
||||||
"""Load stage configurations from model's default config file.
|
"""Load stage configurations from model's default config file.
|
||||||
|
|
||||||
For models registered in the pipeline registry (new path), uses
|
For models registered in the pipeline registry (new path), uses
|
||||||
@@ -330,9 +332,15 @@ def load_stage_configs_from_model(
|
|||||||
base_engine_args: Base engine args to merge as CLI overrides.
|
base_engine_args: Base engine args to merge as CLI overrides.
|
||||||
deploy_config_path: Optional explicit deploy config path.
|
deploy_config_path: Optional explicit deploy config path.
|
||||||
stage_overrides: Per-stage overrides from --stage-overrides.
|
stage_overrides: Per-stage overrides from --stage-overrides.
|
||||||
|
strategy_config_path: Optional path to a composable-parallel
|
||||||
|
``strategy.yaml`` whose derived sizing is overlaid onto the
|
||||||
|
registry-merged stages (opt-in; ignored on the legacy YAML path).
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of stage configuration dictionaries
|
``(stage_configs, omni_lb_policy)``: the list of stage configuration
|
||||||
|
dictionaries plus the strategy-derived pipeline-wide ``omni_lb_policy``
|
||||||
|
(``None`` when no strategy set one). The policy is returned rather than
|
||||||
|
written into a caller-provided mutable dict.
|
||||||
"""
|
"""
|
||||||
if base_engine_args is None:
|
if base_engine_args is None:
|
||||||
base_engine_args = {}
|
base_engine_args = {}
|
||||||
@@ -343,26 +351,44 @@ def load_stage_configs_from_model(
|
|||||||
for key, val in overrides.items():
|
for key, val in overrides.items():
|
||||||
cli_overrides[f"stage_{stage_id_str}_{key}"] = val
|
cli_overrides[f"stage_{stage_id_str}_{key}"] = val
|
||||||
|
|
||||||
stages = StageConfigFactory.create_from_model(
|
strategy_specs = None
|
||||||
|
if strategy_config_path is not None:
|
||||||
|
from vllm_omni.config.composable_parallel.strategy_loader import load_strategy_specs
|
||||||
|
|
||||||
|
strategy_specs = load_strategy_specs(strategy_config_path)
|
||||||
|
|
||||||
|
stages, omni_lb_policy = StageConfigFactory.create_from_model(
|
||||||
model,
|
model,
|
||||||
trust_remote_code=cli_overrides.get("trust_remote_code", False),
|
trust_remote_code=cli_overrides.get("trust_remote_code", False),
|
||||||
cli_overrides=cli_overrides,
|
cli_overrides=cli_overrides,
|
||||||
deploy_config_path=deploy_config_path,
|
deploy_config_path=deploy_config_path,
|
||||||
|
strategy_specs=strategy_specs,
|
||||||
)
|
)
|
||||||
if stages is not None:
|
if stages is not None:
|
||||||
# Convert StageConfig objects to OmegaConf for backward compat
|
# Convert StageConfig objects to OmegaConf for backward compat
|
||||||
return [stage.to_omegaconf() for stage in stages]
|
return [stage.to_omegaconf() for stage in stages], omni_lb_policy
|
||||||
|
|
||||||
# Legacy fallback: load from YAML
|
# Legacy fallback: load from YAML. A composable-parallel strategy cannot be
|
||||||
|
# applied here (it overlays onto registry-merged stages), so warn rather than
|
||||||
|
# silently dropping the operator's --strategy-config.
|
||||||
|
if strategy_config_path is not None:
|
||||||
|
logger.warning(
|
||||||
|
"--strategy-config (%s) was provided but model %r resolves via the "
|
||||||
|
"legacy stage_configs YAML path, which does not support "
|
||||||
|
"composable-parallel strategies; the strategy is ignored. Use a "
|
||||||
|
"registry-based model to apply it.",
|
||||||
|
strategy_config_path,
|
||||||
|
model,
|
||||||
|
)
|
||||||
stage_config_path = resolve_model_config_path(model)
|
stage_config_path = resolve_model_config_path(model)
|
||||||
if stage_config_path is None:
|
if stage_config_path is None:
|
||||||
return []
|
return [], None
|
||||||
stage_configs = load_stage_configs_from_yaml(
|
stage_configs = load_stage_configs_from_yaml(
|
||||||
config_path=stage_config_path,
|
config_path=stage_config_path,
|
||||||
base_engine_args=base_engine_args,
|
base_engine_args=base_engine_args,
|
||||||
prefer_stage_engine_args=True,
|
prefer_stage_engine_args=True,
|
||||||
)
|
)
|
||||||
return stage_configs
|
return stage_configs, None
|
||||||
|
|
||||||
|
|
||||||
def load_stage_configs_from_yaml(
|
def load_stage_configs_from_yaml(
|
||||||
@@ -501,6 +527,25 @@ def filter_stages(
|
|||||||
return stage_configs
|
return stage_configs
|
||||||
|
|
||||||
|
|
||||||
|
def parse_stage_overrides(value: Any) -> dict[str, dict[str, Any]] | None:
|
||||||
|
"""Parse the ``--stage-overrides`` value into a per-stage override dict.
|
||||||
|
|
||||||
|
``value`` may be a raw JSON string (as supplied on the CLI) or an
|
||||||
|
already-parsed mapping. Returns ``None`` when no overrides are given.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
ValueError: when ``value`` is a string that is not valid JSON.
|
||||||
|
"""
|
||||||
|
if not value:
|
||||||
|
return None
|
||||||
|
if isinstance(value, str):
|
||||||
|
try:
|
||||||
|
return json.loads(value)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise ValueError(f"--stage-overrides is not valid JSON: {exc}. Got: {value!r}") from exc
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
def load_and_resolve_stage_configs(
|
def load_and_resolve_stage_configs(
|
||||||
model: str,
|
model: str,
|
||||||
stage_configs_path: str | None,
|
stage_configs_path: str | None,
|
||||||
@@ -508,7 +553,8 @@ def load_and_resolve_stage_configs(
|
|||||||
default_stage_cfg_factory: Any = None,
|
default_stage_cfg_factory: Any = None,
|
||||||
deploy_config_path: str | None = None,
|
deploy_config_path: str | None = None,
|
||||||
stage_overrides: dict[str, dict[str, Any]] | None = None,
|
stage_overrides: dict[str, dict[str, Any]] | None = None,
|
||||||
) -> tuple[str, list]:
|
strategy_config_path: str | None = None,
|
||||||
|
) -> tuple[str, list, str | None]:
|
||||||
"""Load stage configurations from model or YAML file with fallback to defaults.
|
"""Load stage configurations from model or YAML file with fallback to defaults.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
@@ -521,9 +567,13 @@ def load_and_resolve_stage_configs(
|
|||||||
Mutually exclusive with ``stage_configs_path``.
|
Mutually exclusive with ``stage_configs_path``.
|
||||||
stage_overrides: Per-stage overrides from ``--stage-overrides`` JSON.
|
stage_overrides: Per-stage overrides from ``--stage-overrides`` JSON.
|
||||||
Keys are stage_id strings, values are dicts of overrides.
|
Keys are stage_id strings, values are dicts of overrides.
|
||||||
|
strategy_config_path: Optional path to a composable-parallel
|
||||||
|
``strategy.yaml`` overlaid onto the registry-merged stages.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Tuple of (config_path, stage_configs)
|
Tuple of ``(config_path, stage_configs, omni_lb_policy)`` — the last is
|
||||||
|
the strategy-derived pipeline-wide load-balancer policy (``None`` when no
|
||||||
|
strategy set one), returned for the engine to apply.
|
||||||
"""
|
"""
|
||||||
if stage_configs_path is not None and deploy_config_path is not None:
|
if stage_configs_path is not None and deploy_config_path is not None:
|
||||||
raise ValueError(
|
raise ValueError(
|
||||||
@@ -551,13 +601,15 @@ def load_and_resolve_stage_configs(
|
|||||||
stage_configs_path,
|
stage_configs_path,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
omni_lb_policy: str | None = None
|
||||||
if deploy_config_path is not None:
|
if deploy_config_path is not None:
|
||||||
config_path = deploy_config_path
|
config_path = deploy_config_path
|
||||||
stage_configs = load_stage_configs_from_model(
|
stage_configs, omni_lb_policy = load_stage_configs_from_model(
|
||||||
model,
|
model,
|
||||||
base_engine_args=kwargs,
|
base_engine_args=kwargs,
|
||||||
deploy_config_path=deploy_config_path,
|
deploy_config_path=deploy_config_path,
|
||||||
stage_overrides=stage_overrides,
|
stage_overrides=stage_overrides,
|
||||||
|
strategy_config_path=strategy_config_path,
|
||||||
)
|
)
|
||||||
if not stage_configs:
|
if not stage_configs:
|
||||||
if default_stage_cfg_factory is not None:
|
if default_stage_cfg_factory is not None:
|
||||||
@@ -567,10 +619,11 @@ def load_and_resolve_stage_configs(
|
|||||||
stage_configs = []
|
stage_configs = []
|
||||||
elif stage_configs_path is None:
|
elif stage_configs_path is None:
|
||||||
config_path = resolve_model_config_path(model)
|
config_path = resolve_model_config_path(model)
|
||||||
stage_configs = load_stage_configs_from_model(
|
stage_configs, omni_lb_policy = load_stage_configs_from_model(
|
||||||
model,
|
model,
|
||||||
base_engine_args=kwargs,
|
base_engine_args=kwargs,
|
||||||
stage_overrides=stage_overrides,
|
stage_overrides=stage_overrides,
|
||||||
|
strategy_config_path=strategy_config_path,
|
||||||
)
|
)
|
||||||
if not stage_configs:
|
if not stage_configs:
|
||||||
if default_stage_cfg_factory is not None:
|
if default_stage_cfg_factory is not None:
|
||||||
@@ -585,7 +638,7 @@ def load_and_resolve_stage_configs(
|
|||||||
stage_configs = filter_stages(config_path, stage_configs, kwargs)
|
stage_configs = filter_stages(config_path, stage_configs, kwargs)
|
||||||
logger.debug(f"stage_configs: {stage_configs}")
|
logger.debug(f"stage_configs: {stage_configs}")
|
||||||
|
|
||||||
return config_path, stage_configs
|
return config_path, stage_configs, omni_lb_policy
|
||||||
|
|
||||||
|
|
||||||
def get_final_stage_id_for_e2e(
|
def get_final_stage_id_for_e2e(
|
||||||
|
|||||||
Reference in new issue
Block a user