[Config] Reuse vLLM configs in VllmOmniConfig (#6050)
Signed-off-by: Acerak01-fy <wfy2003324@163.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
This commit is contained in:
1 parent
02069a5bbf
commit
e1c5f59aab
5 files changed
+734
-88
No files matched your search
@@ -18,8 +18,8 @@ validation_paths:
|
||||
- tests/config/**
|
||||
upstream_refs:
|
||||
- vllm.config
|
||||
last_reviewed: 2026-08-07
|
||||
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
|
||||
last_reviewed: 2026-08-11
|
||||
last_verified_commit: e356708da4cd39992405686dacfc32b18fcccc7f
|
||||
---
|
||||
|
||||
# vLLM-Omni configuration
|
||||
@@ -39,11 +39,66 @@ The PR review proposal to capture environment-derived values in configuration
|
||||
objects at initialization is an open design question, not a current
|
||||
invariant.
|
||||
|
||||
## Current structured-stage reuse behavior
|
||||
|
||||
The current RFC #4021 implementation uses upstream vLLM config classes as the
|
||||
schema source for AR and generation stages:
|
||||
|
||||
- `OmniStageLoadConfig` inherits `vllm.config.LoadConfig`.
|
||||
- `OmniStageCacheConfig` inherits `vllm.config.CacheConfig`.
|
||||
- `OmniStageSchedulerConfig` inherits `vllm.config.SchedulerConfig`.
|
||||
- `OmniStageParallelConfig` inherits `vllm.config.ParallelConfig`.
|
||||
|
||||
This inheritance preserves the existing load, cache, scheduler, and parallel
|
||||
concern boundaries while reducing duplicate downstream declarations. It does
|
||||
not, by itself, make every inherited field an effective vLLM-Omni runtime
|
||||
option. Construction and runtime consumption are separate surfaces:
|
||||
|
||||
| Surface | Current behavior |
|
||||
| --- | --- |
|
||||
| Structured schema | Upstream dataclass fields are inherited. Their defaults, default factories, and applicable Pydantic validation can participate when the structured object is constructed. |
|
||||
| Omni input ownership | Pipeline/deploy/CLI construction continues to accept only fields with an existing structured owner. Direct construction of an inherited sub-config can expose a wider upstream schema. |
|
||||
| Engine projection | For AR and generation stages, reusable fields are discovered by intersecting each upstream config dataclass with upstream `EngineArgs`. Only constructor-explicit values are emitted; inherited defaults remain deferred to terminal materialization. |
|
||||
| Terminal materialization | The engine-owning process constructs the final upstream `VllmConfig` and performs model-, platform-, rank-, port-, and backend-dependent initialization. |
|
||||
|
||||
The generated projection maps include known upstream naming differences such
|
||||
as `cache_dtype` to `kv_cache_dtype`, `policy` to `scheduling_policy`, and
|
||||
`data_parallel_master_ip` to `data_parallel_address`. A field added to both an
|
||||
upstream concern config and `EngineArgs` therefore enters the AR/generation
|
||||
projection without requiring a second downstream allowlist. Explicit inherited
|
||||
fields that have no projection raise an error rather than being silently
|
||||
dropped.
|
||||
|
||||
Ownership exclusions remain deliberate. Stage topology owns `scheduler_cls`,
|
||||
cache owns `disable_hybrid_kv_cache_manager`, and Omni runtime owns
|
||||
`distributed_executor_backend` and `worker_cls`; vLLM's private API-process
|
||||
fields are terminal internals. These exclusions prevent one input from being
|
||||
constructed or projected through two config concerns. Effective-engine-argument
|
||||
tests cover both the dynamic mapping and this exclusion boundary.
|
||||
|
||||
`CompilationConfig` and `ProfilerConfig` are direct structured-stage fields:
|
||||
`stage.compilation_config` and `stage.profiler_config`. They accept mapping/YAML
|
||||
inputs and complete upstream value objects. Mapping inputs are validated and
|
||||
materialized as their concrete upstream types when the structured config is
|
||||
constructed, so downstream consumers see one resolved representation.
|
||||
Prebuilt upstream objects retain their type across the structured projection
|
||||
boundary.
|
||||
|
||||
Quantization remains on its existing Omni-owned transport contract. This
|
||||
reuse step does not adopt upstream `QuantizationConfigArgs`, because the
|
||||
current engine-specific quantization split needs to be resolved separately.
|
||||
|
||||
Diffusion keeps its existing engine-field ownership and projection surface.
|
||||
Shared Python inheritance must not be treated as evidence that an LLM-only
|
||||
parallel input is effective for a diffusion stage.
|
||||
|
||||
## Safe-change guide while deferred
|
||||
|
||||
Changes should trace every affected producer to its runtime consumer and test
|
||||
the structured, legacy, CLI, and deployment paths that are actually supported.
|
||||
Do not infer a stable contract from this placeholder page.
|
||||
For inherited vLLM fields, tests should separately cover constructor behavior
|
||||
and effective `EngineArgs` projection. Do not infer a stable contract from
|
||||
this placeholder page.
|
||||
|
||||
## Promotion gate
|
||||
|
||||
|
||||
@@ -5,11 +5,20 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import fields
|
||||
from inspect import Parameter, signature
|
||||
from pathlib import Path
|
||||
|
||||
import msgspec
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
from pydantic.fields import FieldInfo
|
||||
from transformers import Qwen3OmniMoeConfig
|
||||
from vllm.config import CacheConfig as VllmCacheConfig
|
||||
from vllm.config import CompilationConfig as VllmCompilationConfig
|
||||
from vllm.config import LoadConfig as VllmLoadConfig
|
||||
from vllm.config import ParallelConfig as VllmParallelConfig
|
||||
from vllm.config import ProfilerConfig as VllmProfilerConfig
|
||||
from vllm.config import SchedulerConfig as VllmSchedulerConfig
|
||||
|
||||
from tests.helpers.stage_config import get_deploy_config_path
|
||||
from vllm_omni.config import omni_config as omni_config_module
|
||||
@@ -40,6 +49,7 @@ from vllm_omni.config.stage_config import (
|
||||
merge_pipeline_deploy,
|
||||
)
|
||||
from vllm_omni.diffusion.diffusion_kv.config import DiffusionKVCacheMode
|
||||
from vllm_omni.engine.stage_engine_startup import _serialize_stage_config
|
||||
from vllm_omni.engine.stage_init_utils import build_legacy_engine_args_dict
|
||||
|
||||
pytestmark = [pytest.mark.core_model, pytest.mark.cpu]
|
||||
@@ -243,9 +253,29 @@ def test_from_pipeline_config_rejects_unowned_deploy_engine_extras(engine_extras
|
||||
],
|
||||
ids=["global", "ar-stage", "generation-stage"],
|
||||
)
|
||||
def test_from_pipeline_config_rejects_diffusion_only_cli_fields_for_llm_stages(cli_overrides, stage_id):
|
||||
with pytest.raises(ValueError, match=rf"Stage {stage_id} .*no structured config owner: kv_cache_dtype"):
|
||||
_from_pipeline_key("qwen3_tts", cli_overrides=cli_overrides)
|
||||
def test_from_pipeline_config_accepts_upstream_cache_dtype_for_llm_stages(cli_overrides, stage_id):
|
||||
stage = _from_pipeline_key("qwen3_tts", cli_overrides=cli_overrides).stage_by_id(stage_id)
|
||||
|
||||
assert stage.cache_config.cache_dtype == "fp8"
|
||||
assert "cache_dtype" in stage.cache_config._omni_explicit_fields
|
||||
|
||||
|
||||
@pytest.mark.parametrize("field_name", ["data_parallel_master_ip", "data_parallel_address"])
|
||||
def test_from_pipeline_config_normalizes_nested_parallel_config_aliases(field_name):
|
||||
pipeline = _resolve_pipeline_or_skip("qwen3_tts")
|
||||
deploy = DeployConfig(
|
||||
stages=[
|
||||
StageDeployConfig(
|
||||
stage_id=0,
|
||||
engine_extras={"parallel_config": {field_name: "10.0.0.1"}},
|
||||
)
|
||||
]
|
||||
)
|
||||
|
||||
stage = VllmOmniConfig.from_pipeline_config(pipeline, user_deploy_config=deploy).stage_by_id(0)
|
||||
|
||||
assert stage.parallel_config.data_parallel_master_ip == "10.0.0.1"
|
||||
assert "data_parallel_master_ip" in stage.parallel_config._omni_explicit_fields
|
||||
|
||||
|
||||
def test_from_pipeline_config_accepts_diffusion_only_cli_fields_for_diffusion_stage():
|
||||
@@ -437,6 +467,8 @@ def test_vllm_omni_stage_config_public_fields_use_typed_stage_realizations():
|
||||
"connector_config",
|
||||
"runtime_config",
|
||||
"parallel_config",
|
||||
"compilation_config",
|
||||
"profiler_config",
|
||||
"quantization_config",
|
||||
}
|
||||
assert "diffusion_config" not in public_fields
|
||||
@@ -455,10 +487,47 @@ def test_runtime_config_fields_match_structured_runtime_scope():
|
||||
"num_gpus",
|
||||
"log_level",
|
||||
"log_stats",
|
||||
"profiler_config",
|
||||
}
|
||||
|
||||
|
||||
def test_upstream_config_compatible_fields_materialize_mapping_inputs():
|
||||
compilation_config = {"backend": "eager"}
|
||||
profiler_config = {"profiler": "cuda"}
|
||||
|
||||
stage_config = VllmOmniARStageConfig(
|
||||
stage_pipeline_config=_resolve_pipeline_or_skip("qwen3_tts").stages[0],
|
||||
compilation_config=compilation_config,
|
||||
profiler_config=profiler_config,
|
||||
)
|
||||
|
||||
assert isinstance(stage_config.compilation_config, VllmCompilationConfig)
|
||||
assert stage_config.compilation_config.backend == "eager"
|
||||
assert isinstance(stage_config.profiler_config, VllmProfilerConfig)
|
||||
assert stage_config.profiler_config.profiler == "cuda"
|
||||
|
||||
|
||||
def test_upstream_config_compatible_fields_validate_mapping_inputs():
|
||||
with pytest.raises(ValidationError):
|
||||
VllmOmniARStageConfig(
|
||||
stage_pipeline_config=_resolve_pipeline_or_skip("qwen3_tts").stages[0],
|
||||
profiler_config={"delay_iterations": -1},
|
||||
)
|
||||
|
||||
|
||||
def test_upstream_config_compatible_fields_keep_prebuilt_config_types():
|
||||
compilation_config = VllmCompilationConfig(backend="eager")
|
||||
profiler_config = VllmProfilerConfig(profiler="cuda")
|
||||
|
||||
stage_config = VllmOmniARStageConfig(
|
||||
stage_pipeline_config=_resolve_pipeline_or_skip("qwen3_tts").stages[0],
|
||||
compilation_config=compilation_config,
|
||||
profiler_config=profiler_config,
|
||||
)
|
||||
|
||||
assert stage_config.compilation_config is compilation_config
|
||||
assert stage_config.profiler_config is profiler_config
|
||||
|
||||
|
||||
def test_sub_config_fields_match_structured_scopes():
|
||||
assert {f.name for f in fields(OmniStageModelConfig)} == {
|
||||
"model",
|
||||
@@ -489,7 +558,6 @@ def test_sub_config_fields_match_structured_scopes():
|
||||
"enforce_eager",
|
||||
"max_cudagraph_capture_size",
|
||||
"enable_flashinfer_autotune",
|
||||
"compilation_config",
|
||||
"enable_multithread_weight_load",
|
||||
"num_weight_load_threads",
|
||||
"disable_autocast",
|
||||
@@ -499,29 +567,25 @@ def test_sub_config_fields_match_structured_scopes():
|
||||
"model_subdir",
|
||||
"tokenizer_subdir",
|
||||
}
|
||||
assert {f.name for f in fields(OmniStageLoadConfig)} == {
|
||||
vllm_load_fields = {f.name for f in fields(VllmLoadConfig)}
|
||||
assert issubclass(OmniStageLoadConfig, VllmLoadConfig)
|
||||
assert {f.name for f in fields(OmniStageLoadConfig)} == vllm_load_fields | {
|
||||
"tokenizer",
|
||||
"download_dir",
|
||||
"skip_tokenizer_init",
|
||||
"load_format",
|
||||
"tokenizer_mode",
|
||||
"config_format",
|
||||
"skip_mm_profiling",
|
||||
}
|
||||
assert {f.name for f in fields(OmniStageCacheConfig)} == {
|
||||
"kv_cache_memory_bytes",
|
||||
"gpu_memory_utilization",
|
||||
"enable_prefix_caching",
|
||||
assert OmniStageLoadConfig(load_format="PT").load_format == "pt"
|
||||
assert issubclass(OmniStageCacheConfig, VllmCacheConfig)
|
||||
assert {f.name for f in fields(OmniStageCacheConfig)} == {f.name for f in fields(VllmCacheConfig)} | {
|
||||
"disable_hybrid_kv_cache_manager",
|
||||
"mm_processor_cache_gb",
|
||||
"mamba_ssm_cache_dtype",
|
||||
}
|
||||
assert {f.name for f in fields(OmniStageSchedulerConfig)} == {
|
||||
"max_num_seqs",
|
||||
"max_num_batched_tokens",
|
||||
assert issubclass(OmniStageSchedulerConfig, VllmSchedulerConfig)
|
||||
assert {f.name for f in fields(OmniStageSchedulerConfig)} == {f.name for f in fields(VllmSchedulerConfig)} | {
|
||||
"max_model_len",
|
||||
"enable_chunked_prefill",
|
||||
"async_scheduling",
|
||||
}
|
||||
assert {f.name for f in fields(OmniStageConnectorConfig)} == {
|
||||
"async_chunk",
|
||||
@@ -530,17 +594,10 @@ def test_sub_config_fields_match_structured_scopes():
|
||||
"output_connectors",
|
||||
"input_connectors",
|
||||
}
|
||||
assert {f.name for f in fields(OmniStageParallelConfig)} == {
|
||||
"pipeline_parallel_size",
|
||||
"data_parallel_size",
|
||||
"tensor_parallel_size",
|
||||
"enable_expert_parallel",
|
||||
"world_size",
|
||||
}
|
||||
assert {f.name for f in fields(OmniStageDiffusionParallelConfig)} == {
|
||||
"pipeline_parallel_size",
|
||||
"data_parallel_size",
|
||||
"tensor_parallel_size",
|
||||
vllm_parallel_fields = {f.name for f in fields(VllmParallelConfig)}
|
||||
assert issubclass(OmniStageParallelConfig, VllmParallelConfig)
|
||||
assert {f.name for f in fields(OmniStageParallelConfig)} == vllm_parallel_fields
|
||||
assert {f.name for f in fields(OmniStageDiffusionParallelConfig)} == vllm_parallel_fields | {
|
||||
"sequence_parallel_size",
|
||||
"ulysses_degree",
|
||||
"ring_degree",
|
||||
@@ -554,20 +611,86 @@ def test_sub_config_fields_match_structured_scopes():
|
||||
"mask_sp_padding",
|
||||
"hsdp_shard_size",
|
||||
"hsdp_replicate_size",
|
||||
"enable_expert_parallel",
|
||||
"world_size",
|
||||
}
|
||||
|
||||
|
||||
def test_inherited_sub_configs_initialize_transport_safe_derived_fields():
|
||||
scheduler_config = OmniStageSchedulerConfig(max_num_batched_tokens=4096)
|
||||
assert scheduler_config.max_num_encoder_input_tokens == 4096
|
||||
assert scheduler_config.encoder_cache_size == 4096
|
||||
|
||||
unresolved_scheduler_config = OmniStageSchedulerConfig()
|
||||
assert unresolved_scheduler_config.max_num_encoder_input_tokens is None
|
||||
assert unresolved_scheduler_config.encoder_cache_size is None
|
||||
|
||||
parallel_config = OmniStageParallelConfig(data_parallel_size=3, data_parallel_rank=2)
|
||||
assert parallel_config.data_parallel_index == 2
|
||||
|
||||
diffusion_parallel_config = OmniStageDiffusionParallelConfig(data_parallel_size=3, data_parallel_rank=2)
|
||||
assert diffusion_parallel_config.data_parallel_index == 2
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"config_cls",
|
||||
[
|
||||
OmniStageLoadConfig,
|
||||
OmniStageCacheConfig,
|
||||
OmniStageSchedulerConfig,
|
||||
OmniStageParallelConfig,
|
||||
OmniStageDiffusionParallelConfig,
|
||||
],
|
||||
)
|
||||
def test_inherited_sub_configs_are_keyword_only(config_cls):
|
||||
assert all(parameter.kind is Parameter.KEYWORD_ONLY for parameter in signature(config_cls).parameters.values())
|
||||
|
||||
with pytest.raises(TypeError):
|
||||
config_cls(1)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"config_cls",
|
||||
[
|
||||
OmniStageLoadConfig,
|
||||
OmniStageCacheConfig,
|
||||
OmniStageSchedulerConfig,
|
||||
OmniStageParallelConfig,
|
||||
OmniStageDiffusionParallelConfig,
|
||||
],
|
||||
)
|
||||
def test_inherited_sub_configs_remain_msgpack_transport_safe(config_cls):
|
||||
config = config_cls()
|
||||
|
||||
uninitialized_fields = [
|
||||
config_field.name
|
||||
for config_field in fields(config)
|
||||
if isinstance(getattr(config, config_field.name), FieldInfo)
|
||||
]
|
||||
assert uninitialized_fields == []
|
||||
|
||||
msgspec.msgpack.encode(_serialize_stage_config(config))
|
||||
|
||||
|
||||
def test_structured_llm_stage_registration_payloads_remain_msgpack_transport_safe():
|
||||
omni_config = _from_pipeline_key("qwen3_tts")
|
||||
|
||||
for stage_config in omni_config.stage_configs:
|
||||
payload = {
|
||||
"stage_id": stage_config.stage_id,
|
||||
"stage_config": _serialize_stage_config(stage_config),
|
||||
}
|
||||
msgspec.msgpack.encode(payload)
|
||||
|
||||
|
||||
def test_diffusion_parallel_config_fields_cover_legacy_surface():
|
||||
from vllm_omni.diffusion.data import DiffusionParallelConfig
|
||||
|
||||
legacy_fields = {f.name for f in fields(DiffusionParallelConfig)}
|
||||
structured_fields = {f.name for f in fields(OmniStageDiffusionParallelConfig)}
|
||||
expected_upstream_fields = {"mask_sp_padding"}
|
||||
vllm_parallel_fields = {f.name for f in fields(VllmParallelConfig)}
|
||||
|
||||
assert legacy_fields | expected_upstream_fields <= structured_fields
|
||||
assert structured_fields - legacy_fields - expected_upstream_fields == {"world_size"}
|
||||
assert structured_fields - legacy_fields - expected_upstream_fields - vllm_parallel_fields == set()
|
||||
|
||||
|
||||
def test_diffusion_parallel_config_keeps_current_diffusion_parallel_surface():
|
||||
@@ -1078,3 +1201,21 @@ def test_diffusion_config_projection_keeps_mapping_quantization_config_serializa
|
||||
cfg = omni_config_module._DiffusionConfigProjection.from_kwargs(quantization_config=quantization_config)
|
||||
|
||||
assert cfg.quantization_config == quantization_config
|
||||
|
||||
|
||||
def test_diffusion_quantization_mapping_reaches_terminal_config(monkeypatch):
|
||||
from vllm_omni.diffusion.data import OmniDiffusionConfig
|
||||
|
||||
quantization_config = {"method": "int8", "activation_scheme": "dynamic"}
|
||||
cfg = omni_config_module._DiffusionConfigProjection.from_kwargs(
|
||||
quantization_config=quantization_config,
|
||||
)
|
||||
|
||||
# Exercise the terminal construction performed by the future typed startup
|
||||
# path without probing ports or loading remote model metadata.
|
||||
monkeypatch.setattr(OmniDiffusionConfig, "_resolve_master_port", lambda _self: 29500)
|
||||
monkeypatch.setattr(OmniDiffusionConfig, "enrich_config", lambda _self: None)
|
||||
cfg.enrich_config()
|
||||
|
||||
assert cfg.quantization_config is not None
|
||||
assert cfg.quantization_config.get_name() == "int8"
|
||||
@@ -8,8 +8,23 @@ from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from pydantic.fields import FieldInfo
|
||||
from vllm.config import CacheConfig as VllmCacheConfig
|
||||
from vllm.config import CompilationConfig as VllmCompilationConfig
|
||||
from vllm.config import LoadConfig as VllmLoadConfig
|
||||
from vllm.config import ParallelConfig as VllmParallelConfig
|
||||
from vllm.config import ProfilerConfig as VllmProfilerConfig
|
||||
from vllm.config import SchedulerConfig as VllmSchedulerConfig
|
||||
from vllm.engine.arg_utils import EngineArgs
|
||||
|
||||
from vllm_omni.config.omni_config import VllmOmniConfig
|
||||
from vllm_omni.config.omni_config import (
|
||||
_LLM_STAGE_ENGINE_FIELDS,
|
||||
OmniStageCacheConfig,
|
||||
OmniStageLoadConfig,
|
||||
OmniStageParallelConfig,
|
||||
OmniStageSchedulerConfig,
|
||||
VllmOmniARStageConfig,
|
||||
VllmOmniConfig,
|
||||
)
|
||||
from vllm_omni.config.pipeline_registry import OMNI_PIPELINES, resolve_pipeline_config
|
||||
from vllm_omni.config.stage_config import (
|
||||
DeployConfig,
|
||||
@@ -53,6 +68,30 @@ def _effective_backend_values(config_cls: type, engine_args: dict) -> dict[str,
|
||||
|
||||
_LLM_BACKEND_FIELDS = frozenset(field.name for field in fields(OmniEngineArgs))
|
||||
_DIFFUSION_BACKEND_FIELDS = frozenset(field.name for field in fields(OmniDiffusionConfig))
|
||||
_OMNI_ONLY_LLM_STAGE_ENGINE_FIELDS = frozenset(
|
||||
{
|
||||
"active_stream_window",
|
||||
"codec_frame_rate_hz",
|
||||
"custom_voice_dir",
|
||||
"devices",
|
||||
"disable_autocast",
|
||||
"enable_multithread_weight_load",
|
||||
"env",
|
||||
"has_sampling_extra_args",
|
||||
"log_level",
|
||||
"log_stats",
|
||||
"model_arch",
|
||||
"model_subdir",
|
||||
"num_gpus",
|
||||
"num_replicas",
|
||||
"num_weight_load_threads",
|
||||
"omni_kv_config",
|
||||
"parallel_config",
|
||||
"subtalker_sampling_params",
|
||||
"task_type",
|
||||
"tokenizer_subdir",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -197,6 +236,100 @@ def _legacy_and_typed_stages(
|
||||
return legacy_stages, omni_config
|
||||
|
||||
|
||||
def test_llm_stage_engine_field_schema_tracks_upstream_engine_args():
|
||||
upstream_engine_fields = frozenset(field.name for field in fields(EngineArgs))
|
||||
|
||||
assert _LLM_STAGE_ENGINE_FIELDS - upstream_engine_fields == _OMNI_ONLY_LLM_STAGE_ENGINE_FIELDS
|
||||
|
||||
|
||||
def test_typed_llm_projection_discovers_explicit_upstream_config_fields():
|
||||
stage_config = VllmOmniARStageConfig(
|
||||
stage_pipeline_config=StagePipelineConfig(stage_id=0, model_stage="test"),
|
||||
load_config=OmniStageLoadConfig(ignore_patterns=["*.bin"]),
|
||||
cache_config=OmniStageCacheConfig(block_size=32, cache_dtype="fp8"),
|
||||
scheduler_config=OmniStageSchedulerConfig(
|
||||
prefill_schedule_interval=2,
|
||||
policy="priority",
|
||||
),
|
||||
parallel_config=OmniStageParallelConfig(
|
||||
data_parallel_master_ip="10.0.0.1",
|
||||
enable_dbo=True,
|
||||
),
|
||||
)
|
||||
|
||||
engine_args = stage_init_utils._project_omni_stage_engine_args(stage_config)
|
||||
|
||||
assert engine_args["ignore_patterns"] == ["*.bin"]
|
||||
assert engine_args["block_size"] == 32
|
||||
assert engine_args["kv_cache_dtype"] == "fp8"
|
||||
assert engine_args["prefill_schedule_interval"] == 2
|
||||
assert engine_args["scheduling_policy"] == "priority"
|
||||
assert engine_args["data_parallel_address"] == "10.0.0.1"
|
||||
assert engine_args["enable_dbo"] is True
|
||||
|
||||
|
||||
def test_typed_llm_projection_does_not_emit_inherited_upstream_defaults():
|
||||
stage_config = VllmOmniARStageConfig(
|
||||
stage_pipeline_config=StagePipelineConfig(stage_id=0, model_stage="test"),
|
||||
)
|
||||
|
||||
engine_args = stage_init_utils._project_omni_stage_engine_args(stage_config)
|
||||
|
||||
inherited_defaults = {
|
||||
"ignore_patterns",
|
||||
"block_size",
|
||||
"kv_cache_dtype",
|
||||
"prefill_schedule_interval",
|
||||
"scheduling_policy",
|
||||
"data_parallel_address",
|
||||
"data_parallel_rank",
|
||||
"data_parallel_size_local",
|
||||
"data_parallel_rpc_port",
|
||||
"enable_dbo",
|
||||
}
|
||||
assert inherited_defaults.isdisjoint(engine_args)
|
||||
|
||||
|
||||
def test_typed_llm_projection_rejects_explicit_fields_owned_by_another_boundary():
|
||||
stage_config = VllmOmniARStageConfig(
|
||||
stage_pipeline_config=StagePipelineConfig(stage_id=0, model_stage="test"),
|
||||
scheduler_config=OmniStageSchedulerConfig(scheduler_cls="test.CustomScheduler"),
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match=r"no EngineArgs projection: scheduler_cls"):
|
||||
stage_init_utils._project_omni_stage_engine_args(stage_config)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stage_id", [0, 1], ids=["ar", "generation"])
|
||||
def test_typed_llm_engine_args_preserve_upstream_config_objects(tmp_path, stage_id):
|
||||
pipeline, deploy, model = _engine_arg_inputs(tmp_path)
|
||||
compilation_config = VllmCompilationConfig(backend="eager")
|
||||
profiler_config = VllmProfilerConfig(profiler="cuda")
|
||||
stage_prefix = f"stage_{stage_id}_"
|
||||
omni_config = VllmOmniConfig.from_pipeline_config(
|
||||
pipeline,
|
||||
user_deploy_config=copy.deepcopy(deploy),
|
||||
cli_overrides={
|
||||
"model": model,
|
||||
f"{stage_prefix}compilation_config": compilation_config,
|
||||
f"{stage_prefix}profiler_config": profiler_config,
|
||||
},
|
||||
)
|
||||
|
||||
stage_config = omni_config.stage_by_id(stage_id)
|
||||
typed_args = build_engine_args_dict_from_omni_stage_config(stage_config, model)
|
||||
|
||||
assert isinstance(stage_config.compilation_config, VllmCompilationConfig)
|
||||
assert isinstance(typed_args["compilation_config"], VllmCompilationConfig)
|
||||
assert typed_args["compilation_config"] is not stage_config.compilation_config
|
||||
assert typed_args["compilation_config"].backend == "eager"
|
||||
|
||||
assert isinstance(stage_config.profiler_config, VllmProfilerConfig)
|
||||
assert isinstance(typed_args["profiler_config"], VllmProfilerConfig)
|
||||
assert typed_args["profiler_config"] is not stage_config.profiler_config
|
||||
assert typed_args["profiler_config"].profiler == "cuda"
|
||||
|
||||
|
||||
def test_typed_llm_engine_args_preserve_legacy_adapter_behavior(tmp_path):
|
||||
pipeline, deploy, model = _engine_arg_inputs(tmp_path)
|
||||
legacy_stages, omni_config = _legacy_and_typed_stages(pipeline, deploy, model)
|
||||
@@ -222,6 +355,13 @@ def test_typed_llm_engine_args_preserve_legacy_adapter_behavior(tmp_path):
|
||||
assert {name: typed_args[name] for name in legacy_args} == legacy_args
|
||||
|
||||
thinker_args = typed_args_by_stage[0]
|
||||
inherited_vllm_fields = {
|
||||
field.name
|
||||
for config_cls in (VllmLoadConfig, VllmCacheConfig, VllmSchedulerConfig, VllmParallelConfig)
|
||||
for field in fields(config_cls)
|
||||
}
|
||||
topology_projected_fields = {"scheduler_cls"}
|
||||
assert (inherited_vllm_fields - _LLM_STAGE_ENGINE_FIELDS - topology_projected_fields).isdisjoint(thinker_args)
|
||||
assert thinker_args["model"] == str(tmp_path / "stage-model" / "ar-model")
|
||||
assert thinker_args["tokenizer"] == str(tmp_path / "stage-model" / "ar-tokenizer")
|
||||
assert thinker_args["stage_id"] == 0
|
||||
|
||||
+311
-45
@@ -12,12 +12,22 @@ from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass, field, fields
|
||||
from dataclasses import InitVar, dataclass, field, fields
|
||||
from functools import wraps
|
||||
from inspect import Parameter, signature
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal, TypeAlias, TypedDict, cast
|
||||
|
||||
from pydantic import ConfigDict, Field
|
||||
from pydantic import ConfigDict, Field, model_validator
|
||||
from typing_extensions import Self
|
||||
from vllm.config import CacheConfig as VllmCacheConfig
|
||||
from vllm.config import CompilationConfig as VllmCompilationConfig
|
||||
from vllm.config import LoadConfig as VllmLoadConfig
|
||||
from vllm.config import ParallelConfig as VllmParallelConfig
|
||||
from vllm.config import ProfilerConfig as VllmProfilerConfig
|
||||
from vllm.config import SchedulerConfig as VllmSchedulerConfig
|
||||
from vllm.config.utils import config
|
||||
from vllm.engine.arg_utils import EngineArgs as VllmEngineArgs
|
||||
from vllm.model_executor.layers.quantization.base_config import QuantizationConfig
|
||||
|
||||
from vllm_omni.config.stage_config import (
|
||||
@@ -77,6 +87,49 @@ class _QuantizationEngineOverrides(TypedDict, total=False):
|
||||
quantization: str
|
||||
|
||||
|
||||
class _TrackExplicitConfigFields:
|
||||
"""Record constructor inputs without adding a serialized config field."""
|
||||
|
||||
@model_validator(mode="wrap")
|
||||
@classmethod
|
||||
def _record_explicit_config_fields(cls, value: Any, handler: Any) -> Any:
|
||||
result = handler(value)
|
||||
kwargs = getattr(value, "kwargs", None)
|
||||
if kwargs is not None:
|
||||
explicit_fields = frozenset(kwargs)
|
||||
elif isinstance(value, cls):
|
||||
explicit_fields = getattr(value, "_omni_explicit_fields", frozenset())
|
||||
else:
|
||||
explicit_fields = frozenset()
|
||||
object.__setattr__(result, "_omni_explicit_fields", explicit_fields)
|
||||
return result
|
||||
|
||||
|
||||
def _enforce_keyword_only_init(cls: type[Any]) -> type[Any]:
|
||||
"""Make inherited Pydantic dataclass fields keyword-only as well."""
|
||||
generated_init = cls.__init__
|
||||
generated_signature = signature(cls)
|
||||
keyword_only_signature = generated_signature.replace(
|
||||
parameters=[
|
||||
parameter.replace(kind=Parameter.KEYWORD_ONLY)
|
||||
if parameter.kind in {Parameter.POSITIONAL_ONLY, Parameter.POSITIONAL_OR_KEYWORD}
|
||||
else parameter
|
||||
for parameter in generated_signature.parameters.values()
|
||||
]
|
||||
)
|
||||
|
||||
@wraps(generated_init)
|
||||
def keyword_only_init(self: Any, *args: Any, **kwargs: Any) -> None:
|
||||
if args:
|
||||
raise TypeError(f"{cls.__name__}() accepts keyword arguments only; got {len(args)} positional argument(s)")
|
||||
generated_init(self, **kwargs)
|
||||
|
||||
cls.__init__ = keyword_only_init
|
||||
cls.__signature__ = keyword_only_signature
|
||||
cls.__match_args__ = ()
|
||||
return cls
|
||||
|
||||
|
||||
class _ModelEngineOverrides(TypedDict, total=False):
|
||||
model: str
|
||||
model_arch: str
|
||||
@@ -106,7 +159,6 @@ class _ModelEngineOverrides(TypedDict, total=False):
|
||||
enforce_eager: bool
|
||||
max_cudagraph_capture_size: int
|
||||
enable_flashinfer_autotune: bool
|
||||
compilation_config: dict[str, Any]
|
||||
enable_multithread_weight_load: bool
|
||||
num_weight_load_threads: int
|
||||
disable_autocast: bool
|
||||
@@ -148,7 +200,6 @@ class _RuntimeEngineOverrides(TypedDict, total=False):
|
||||
num_gpus: int
|
||||
log_level: str
|
||||
log_stats: bool
|
||||
profiler_config: dict[str, Any]
|
||||
|
||||
|
||||
class _ParallelConfigEngineOverrides(TypedDict, total=False):
|
||||
@@ -192,6 +243,8 @@ class _StageEngineValues:
|
||||
runtime: _RuntimeEngineOverrides
|
||||
parallel: _ParallelEngineOverrides
|
||||
diffusion: _DiffusionEngineOverrides
|
||||
compilation_config: Mapping[str, Any] | VllmCompilationConfig | None
|
||||
profiler_config: Mapping[str, Any] | VllmProfilerConfig | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -341,7 +394,6 @@ class OmniStageModelConfig:
|
||||
enforce_eager: bool = False
|
||||
max_cudagraph_capture_size: int | None = Field(default=None, ge=0)
|
||||
enable_flashinfer_autotune: bool | None = None
|
||||
compilation_config: dict[str, Any] | None = None
|
||||
enable_multithread_weight_load: bool = True
|
||||
num_weight_load_threads: int = Field(default=4, ge=1)
|
||||
disable_autocast: bool = False
|
||||
@@ -352,21 +404,21 @@ class OmniStageModelConfig:
|
||||
tokenizer_subdir: str | None = None
|
||||
|
||||
|
||||
@config
|
||||
class OmniStageLoadConfig:
|
||||
"""Per-stage loading behavior and resolved tokenizer input."""
|
||||
@_enforce_keyword_only_init
|
||||
@config(kw_only=True)
|
||||
class OmniStageLoadConfig(_TrackExplicitConfigFields, VllmLoadConfig):
|
||||
"""vLLM loading behavior plus Omni stage-specific tokenizer inputs."""
|
||||
|
||||
tokenizer: str | None = None
|
||||
download_dir: str | None = None
|
||||
skip_tokenizer_init: bool = False
|
||||
load_format: str = "auto"
|
||||
tokenizer_mode: str = "auto"
|
||||
config_format: str | None = None
|
||||
skip_mm_profiling: bool | None = None
|
||||
|
||||
|
||||
@config
|
||||
class OmniStageCacheConfig:
|
||||
@_enforce_keyword_only_init
|
||||
@config(kw_only=True)
|
||||
class OmniStageCacheConfig(_TrackExplicitConfigFields, VllmCacheConfig):
|
||||
"""Per-stage engine cache and memory behavior.
|
||||
|
||||
This is separate from ``_DiffusionConfigProjection.cache_config``, which configures
|
||||
@@ -383,17 +435,26 @@ class OmniStageCacheConfig:
|
||||
mamba_ssm_cache_dtype: str | None = None
|
||||
|
||||
|
||||
@config
|
||||
class OmniStageSchedulerConfig:
|
||||
@_enforce_keyword_only_init
|
||||
@config(kw_only=True)
|
||||
class OmniStageSchedulerConfig(_TrackExplicitConfigFields, VllmSchedulerConfig):
|
||||
"""Per-stage request scheduling behavior."""
|
||||
|
||||
# Upstream receives max_model_len only while materializing SchedulerConfig.
|
||||
# Omni retains it as unresolved stage input until the owning engine process.
|
||||
max_num_seqs: int | None = Field(default=None, ge=1)
|
||||
max_num_batched_tokens: int | None = Field(default=None, ge=1)
|
||||
max_model_len: int | None = Field(default=None, ge=-1)
|
||||
is_encoder_decoder: InitVar[bool] = False
|
||||
enable_chunked_prefill: bool | None = None
|
||||
async_scheduling: bool | None = None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
def __post_init__(self, is_encoder_decoder: bool = False) -> None:
|
||||
# Upstream initializes these derived fields in its terminal post-init.
|
||||
# Keep them serializable here without running model-dependent checks.
|
||||
self.max_num_encoder_input_tokens = self.max_num_batched_tokens
|
||||
self.encoder_cache_size = self.max_num_batched_tokens
|
||||
|
||||
if (
|
||||
self.max_num_batched_tokens is not None
|
||||
and self.max_num_seqs is not None
|
||||
@@ -432,24 +493,73 @@ class OmniStageRuntimeConfig:
|
||||
num_gpus: int = Field(default=1, ge=1)
|
||||
log_level: str = "info"
|
||||
log_stats: bool = False
|
||||
profiler_config: dict[str, Any] | None = None
|
||||
|
||||
|
||||
@config
|
||||
class OmniStageParallelConfig:
|
||||
@_enforce_keyword_only_init
|
||||
@config(kw_only=True)
|
||||
class OmniStageParallelConfig(_TrackExplicitConfigFields, VllmParallelConfig):
|
||||
"""Common per-stage distributed parallelism behavior."""
|
||||
|
||||
pipeline_parallel_size: int = Field(default=1, ge=1)
|
||||
data_parallel_size: int = Field(default=1, ge=1)
|
||||
tensor_parallel_size: int = Field(default=1, ge=1)
|
||||
enable_expert_parallel: bool = False
|
||||
world_size: int = Field(default=1, ge=1, init=False)
|
||||
# EngineArgs intentionally leaves these unresolved. The upstream terminal
|
||||
# ParallelConfig defaults (rank 0, local size 1, port 29550, worker
|
||||
# ``"auto"``) must not turn into explicit engine inputs in the head
|
||||
# process merely because this transport class inherits ParallelConfig.
|
||||
data_parallel_size_local: int | None = Field(default=None, ge=0)
|
||||
data_parallel_rank: int | None = Field(default=None, ge=0)
|
||||
data_parallel_rpc_port: int | None = None
|
||||
worker_cls: str | None = None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def _validate_parallel_config(self) -> Self:
|
||||
"""Run upstream validation without resolving deferred engine inputs."""
|
||||
deferred_values = (
|
||||
self.data_parallel_size_local,
|
||||
self.data_parallel_rank,
|
||||
self.data_parallel_rpc_port,
|
||||
self.worker_cls,
|
||||
self.all2all_backend,
|
||||
self.disable_custom_all_reduce,
|
||||
)
|
||||
if self.data_parallel_size_local is not None and self.data_parallel_size_local > self.data_parallel_size:
|
||||
raise ValueError(
|
||||
f"data_parallel_size_local ({self.data_parallel_size_local}) "
|
||||
f"must be <= data_parallel_size ({self.data_parallel_size})"
|
||||
)
|
||||
if self.data_parallel_rank is not None and not 0 <= self.data_parallel_rank < self.data_parallel_size:
|
||||
raise ValueError(
|
||||
f"data_parallel_rank ({self.data_parallel_rank}) must be in the range [0, {self.data_parallel_size})"
|
||||
)
|
||||
self.data_parallel_size_local = self.data_parallel_size
|
||||
self.data_parallel_rank = 0
|
||||
self.data_parallel_rpc_port = VllmParallelConfig.data_parallel_rpc_port
|
||||
self.worker_cls = VllmParallelConfig.worker_cls
|
||||
try:
|
||||
VllmParallelConfig._validate_parallel_config(self)
|
||||
finally:
|
||||
(
|
||||
self.data_parallel_size_local,
|
||||
self.data_parallel_rank,
|
||||
self.data_parallel_rpc_port,
|
||||
self.worker_cls,
|
||||
self.all2all_backend,
|
||||
self.disable_custom_all_reduce,
|
||||
) = deferred_values
|
||||
return self
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
# Keep config construction transport-safe. Upstream runtime backend,
|
||||
# rank, port, and platform resolution happens in the owning process.
|
||||
self.data_parallel_index = self.data_parallel_rank
|
||||
self.world_size = self.pipeline_parallel_size * self.data_parallel_size * self.tensor_parallel_size
|
||||
|
||||
@property
|
||||
def world_size_across_dp(self) -> int:
|
||||
# Omni's public world_size has historically included DP.
|
||||
return self.world_size
|
||||
|
||||
@config
|
||||
|
||||
@_enforce_keyword_only_init
|
||||
@config(kw_only=True)
|
||||
class OmniStageDiffusionParallelConfig(OmniStageParallelConfig):
|
||||
"""Diffusion-stage distributed parallelism behavior."""
|
||||
|
||||
@@ -468,6 +578,7 @@ class OmniStageDiffusionParallelConfig(OmniStageParallelConfig):
|
||||
hsdp_replicate_size: int = Field(default=1, ge=1)
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.data_parallel_index = self.data_parallel_rank
|
||||
self.sequence_parallel_size = (
|
||||
self.allgather_degree if self.allgather_degree > 1 else self.ulysses_degree * self.ring_degree
|
||||
)
|
||||
@@ -843,6 +954,58 @@ _DIFFUSION_STAGE_ENGINE_FIELDS = (_DIFFUSION_CONFIG_FIELDS | _DIFFUSION_BACKCOMP
|
||||
"stage_id",
|
||||
}
|
||||
|
||||
|
||||
def _upstream_engine_field_map(
|
||||
config_cls: type[Any],
|
||||
*,
|
||||
aliases: Mapping[str, str] = {},
|
||||
exclude: frozenset[str] = frozenset(),
|
||||
) -> dict[str, str]:
|
||||
"""Map reusable upstream config fields to their EngineArgs inputs."""
|
||||
engine_fields = frozenset(config_field.name for config_field in fields(VllmEngineArgs))
|
||||
return {
|
||||
config_field.name: engine_name
|
||||
for config_field in fields(config_cls)
|
||||
if config_field.init
|
||||
and config_field.name not in exclude
|
||||
and (engine_name := aliases.get(config_field.name, config_field.name)) in engine_fields
|
||||
}
|
||||
|
||||
|
||||
_LOAD_CONFIG_ENGINE_FIELD_MAP = _upstream_engine_field_map(VllmLoadConfig)
|
||||
_CACHE_CONFIG_ENGINE_FIELD_MAP = _upstream_engine_field_map(
|
||||
VllmCacheConfig,
|
||||
aliases={"cache_dtype": "kv_cache_dtype"},
|
||||
)
|
||||
_SCHEDULER_CONFIG_ENGINE_FIELD_MAP = _upstream_engine_field_map(
|
||||
VllmSchedulerConfig,
|
||||
aliases={"policy": "scheduling_policy"},
|
||||
# ``scheduler_cls`` is selected by the immutable stage topology, while
|
||||
# ``disable_hybrid_kv_cache_manager`` belongs to the cache concern in the
|
||||
# Omni schema. Keep both out of the dynamic SchedulerConfig projection so
|
||||
# one input cannot acquire two owners.
|
||||
exclude=frozenset(
|
||||
{
|
||||
"scheduler_cls",
|
||||
"disable_hybrid_kv_cache_manager",
|
||||
}
|
||||
),
|
||||
)
|
||||
_PARALLEL_CONFIG_ENGINE_FIELD_MAP = _upstream_engine_field_map(
|
||||
VllmParallelConfig,
|
||||
aliases={"data_parallel_master_ip": "data_parallel_address"},
|
||||
# Runtime owns worker selection. The private API-process fields are
|
||||
# terminal vLLM internals rather than per-stage user inputs.
|
||||
exclude=frozenset(
|
||||
{
|
||||
"distributed_executor_backend",
|
||||
"worker_cls",
|
||||
"_api_process_count",
|
||||
"_api_process_rank",
|
||||
}
|
||||
),
|
||||
)
|
||||
|
||||
_QUANTIZATION_ENGINE_FIELDS = frozenset(_QuantizationEngineOverrides.__annotations__)
|
||||
_MODEL_ENGINE_FIELDS = frozenset(_ModelEngineOverrides.__annotations__)
|
||||
_LOAD_ENGINE_FIELDS = frozenset(_LoadEngineOverrides.__annotations__)
|
||||
@@ -850,7 +1013,17 @@ _CACHE_ENGINE_FIELDS = frozenset(_CacheEngineOverrides.__annotations__)
|
||||
_SCHEDULER_ENGINE_FIELDS = frozenset(_SchedulerEngineOverrides.__annotations__)
|
||||
_CONNECTOR_ENGINE_FIELDS = frozenset(_ConnectorEngineOverrides.__annotations__)
|
||||
_RUNTIME_ENGINE_FIELDS = frozenset(_RuntimeEngineOverrides.__annotations__)
|
||||
_PARALLEL_CONFIG_ENGINE_FIELDS = frozenset(_ParallelConfigEngineOverrides.__annotations__)
|
||||
_DIRECT_VLLM_CONFIG_ENGINE_FIELDS = frozenset({"compilation_config", "profiler_config"})
|
||||
_LLM_LOAD_ENGINE_FIELDS = _LOAD_ENGINE_FIELDS | frozenset(_LOAD_CONFIG_ENGINE_FIELD_MAP.values())
|
||||
_LLM_CACHE_ENGINE_FIELDS = _CACHE_ENGINE_FIELDS | frozenset(_CACHE_CONFIG_ENGINE_FIELD_MAP.values())
|
||||
_LLM_SCHEDULER_ENGINE_FIELDS = _SCHEDULER_ENGINE_FIELDS | frozenset(_SCHEDULER_CONFIG_ENGINE_FIELD_MAP.values())
|
||||
_LLM_PARALLEL_CONFIG_ENGINE_FIELDS = frozenset(_PARALLEL_CONFIG_ENGINE_FIELD_MAP.values())
|
||||
_DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS = frozenset(
|
||||
f.name for f in fields(OmniStageDiffusionParallelConfig)
|
||||
) & frozenset(_ParallelConfigEngineOverrides.__annotations__)
|
||||
_DIFFUSION_PARALLEL_CONFIG_FIELD_MAP = {name: name for name in _DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS}
|
||||
_LLM_PARALLEL_CONFIG_FIELDS = frozenset(_PARALLEL_CONFIG_ENGINE_FIELD_MAP)
|
||||
_PARALLEL_CONFIG_ENGINE_FIELDS = _LLM_PARALLEL_CONFIG_ENGINE_FIELDS | _DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS
|
||||
_PARALLEL_ENGINE_FIELDS = _PARALLEL_CONFIG_ENGINE_FIELDS | {"parallel_config"}
|
||||
_COMMON_STAGE_ENGINE_FIELDS = (
|
||||
_QUANTIZATION_ENGINE_FIELDS
|
||||
@@ -860,12 +1033,16 @@ _COMMON_STAGE_ENGINE_FIELDS = (
|
||||
| _SCHEDULER_ENGINE_FIELDS
|
||||
| _CONNECTOR_ENGINE_FIELDS
|
||||
| _RUNTIME_ENGINE_FIELDS
|
||||
| _DIRECT_VLLM_CONFIG_ENGINE_FIELDS
|
||||
)
|
||||
_LLM_STAGE_ENGINE_FIELDS = (
|
||||
_COMMON_STAGE_ENGINE_FIELDS
|
||||
| _LLM_LOAD_ENGINE_FIELDS
|
||||
| _LLM_CACHE_ENGINE_FIELDS
|
||||
| _LLM_SCHEDULER_ENGINE_FIELDS
|
||||
| _LLM_PARALLEL_CONFIG_ENGINE_FIELDS
|
||||
| {"parallel_config"}
|
||||
)
|
||||
_LLM_PARALLEL_CONFIG_ENGINE_FIELDS = frozenset(f.name for f in fields(OmniStageParallelConfig)) - {"world_size"}
|
||||
_DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS = frozenset(f.name for f in fields(OmniStageDiffusionParallelConfig)) - {
|
||||
"world_size"
|
||||
}
|
||||
_LLM_STAGE_ENGINE_FIELDS = _COMMON_STAGE_ENGINE_FIELDS | _LLM_PARALLEL_CONFIG_ENGINE_FIELDS | {"parallel_config"}
|
||||
_DIFFUSION_OWNED_STAGE_ENGINE_FIELDS = (
|
||||
_COMMON_STAGE_ENGINE_FIELDS
|
||||
| _DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS
|
||||
@@ -886,6 +1063,27 @@ _PARALLEL_CONFIG_ENGINE_FIELDS_BY_EXECUTION_TYPE = {
|
||||
StageExecutionType.LLM_GENERATION: _LLM_PARALLEL_CONFIG_ENGINE_FIELDS,
|
||||
StageExecutionType.DIFFUSION: _DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS,
|
||||
}
|
||||
_PARALLEL_CONFIG_FIELDS_BY_EXECUTION_TYPE = {
|
||||
StageExecutionType.LLM_AR: _LLM_PARALLEL_CONFIG_FIELDS,
|
||||
StageExecutionType.LLM_GENERATION: _LLM_PARALLEL_CONFIG_FIELDS,
|
||||
StageExecutionType.DIFFUSION: frozenset(_DIFFUSION_PARALLEL_CONFIG_FIELD_MAP),
|
||||
}
|
||||
|
||||
_LOAD_STAGE_ENGINE_FIELD_MAP = {
|
||||
**{name: name for name in _LOAD_ENGINE_FIELDS},
|
||||
**_LOAD_CONFIG_ENGINE_FIELD_MAP,
|
||||
}
|
||||
_CACHE_STAGE_ENGINE_FIELD_MAP = {
|
||||
**{name: name for name in _CACHE_ENGINE_FIELDS},
|
||||
**_CACHE_CONFIG_ENGINE_FIELD_MAP,
|
||||
}
|
||||
_SCHEDULER_STAGE_ENGINE_FIELD_MAP = {
|
||||
**{name: name for name in _SCHEDULER_ENGINE_FIELDS},
|
||||
**_SCHEDULER_CONFIG_ENGINE_FIELD_MAP,
|
||||
}
|
||||
_DIFFUSION_LOAD_STAGE_ENGINE_FIELD_MAP = {name: name for name in _LOAD_ENGINE_FIELDS}
|
||||
_DIFFUSION_CACHE_STAGE_ENGINE_FIELD_MAP = {name: name for name in _CACHE_ENGINE_FIELDS}
|
||||
_DIFFUSION_SCHEDULER_STAGE_ENGINE_FIELD_MAP = {name: name for name in _SCHEDULER_ENGINE_FIELDS}
|
||||
|
||||
|
||||
def _validate_stage_engine_override_ownership(
|
||||
@@ -896,13 +1094,19 @@ def _validate_stage_engine_override_ownership(
|
||||
try:
|
||||
owner_fields = _STAGE_ENGINE_FIELDS_BY_EXECUTION_TYPE[execution_type]
|
||||
parallel_owner_fields = _PARALLEL_CONFIG_ENGINE_FIELDS_BY_EXECUTION_TYPE[execution_type]
|
||||
parallel_config_fields = _PARALLEL_CONFIG_FIELDS_BY_EXECUTION_TYPE[execution_type]
|
||||
except KeyError as exc:
|
||||
raise ValueError(f"Unsupported stage execution type: {execution_type!r}") from exc
|
||||
|
||||
unowned_fields = set(overrides) - owner_fields
|
||||
parallel_config = overrides.get("parallel_config")
|
||||
if isinstance(parallel_config, Mapping):
|
||||
unowned_fields.update(f"parallel_config.{name}" for name in set(parallel_config) - parallel_owner_fields)
|
||||
# Nested values traditionally use upstream config names, while flat
|
||||
# EngineArgs uses aliases such as ``data_parallel_address``. Accept
|
||||
# either spelling at the boundary; the builder canonicalizes aliases
|
||||
# before instantiating the inherited config.
|
||||
accepted_parallel_names = parallel_config_fields | parallel_owner_fields
|
||||
unowned_fields.update(f"parallel_config.{name}" for name in set(parallel_config) - accepted_parallel_names)
|
||||
if unowned_fields:
|
||||
names = ", ".join(sorted(unowned_fields))
|
||||
raise ValueError(
|
||||
@@ -973,17 +1177,28 @@ def _stage_engine_values(
|
||||
topology.execution_type,
|
||||
engine,
|
||||
)
|
||||
if topology.execution_type in {
|
||||
StageExecutionType.LLM_AR,
|
||||
StageExecutionType.LLM_GENERATION,
|
||||
}:
|
||||
load_engine_fields = _LLM_LOAD_ENGINE_FIELDS
|
||||
cache_engine_fields = _LLM_CACHE_ENGINE_FIELDS
|
||||
scheduler_engine_fields = _LLM_SCHEDULER_ENGINE_FIELDS
|
||||
else:
|
||||
load_engine_fields = _LOAD_ENGINE_FIELDS
|
||||
cache_engine_fields = _CACHE_ENGINE_FIELDS
|
||||
scheduler_engine_fields = _SCHEDULER_ENGINE_FIELDS
|
||||
return _StageEngineValues(
|
||||
quantization=cast(
|
||||
_QuantizationEngineOverrides,
|
||||
_select_engine_overrides(engine, _QUANTIZATION_ENGINE_FIELDS),
|
||||
),
|
||||
model=cast(_ModelEngineOverrides, _select_engine_overrides(engine, _MODEL_ENGINE_FIELDS)),
|
||||
load=cast(_LoadEngineOverrides, _select_engine_overrides(engine, _LOAD_ENGINE_FIELDS)),
|
||||
cache=cast(_CacheEngineOverrides, _select_engine_overrides(engine, _CACHE_ENGINE_FIELDS)),
|
||||
load=cast(_LoadEngineOverrides, _select_engine_overrides(engine, load_engine_fields)),
|
||||
cache=cast(_CacheEngineOverrides, _select_engine_overrides(engine, cache_engine_fields)),
|
||||
scheduler=cast(
|
||||
_SchedulerEngineOverrides,
|
||||
_select_engine_overrides(engine, _SCHEDULER_ENGINE_FIELDS),
|
||||
_select_engine_overrides(engine, scheduler_engine_fields),
|
||||
),
|
||||
connector=cast(
|
||||
_ConnectorEngineOverrides,
|
||||
@@ -992,6 +1207,8 @@ def _stage_engine_values(
|
||||
runtime=cast(_RuntimeEngineOverrides, _select_engine_overrides(engine, _RUNTIME_ENGINE_FIELDS)),
|
||||
parallel=cast(_ParallelEngineOverrides, _select_engine_overrides(engine, _PARALLEL_ENGINE_FIELDS)),
|
||||
diffusion=_DiffusionEngineOverrides.from_engine(engine),
|
||||
compilation_config=_copy_value(engine.get("compilation_config")),
|
||||
profiler_config=_copy_value(engine.get("profiler_config")),
|
||||
)
|
||||
|
||||
|
||||
@@ -1046,6 +1263,8 @@ class BaseVllmOmniStageConfig:
|
||||
connector_config: OmniStageConnectorConfig = field(default_factory=OmniStageConnectorConfig)
|
||||
runtime_config: OmniStageRuntimeConfig = field(default_factory=OmniStageRuntimeConfig)
|
||||
parallel_config: OmniStageParallelConfig = field(default_factory=OmniStageParallelConfig)
|
||||
compilation_config: VllmCompilationConfig | None = None
|
||||
profiler_config: VllmProfilerConfig | None = None
|
||||
quantization_config: _QuantizationConfigType = None
|
||||
|
||||
@property
|
||||
@@ -1197,6 +1416,8 @@ def _build_common_stage_config_kwargs(
|
||||
parallel_config,
|
||||
),
|
||||
"parallel_config": parallel_config,
|
||||
"compilation_config": _copy_value(engine.compilation_config),
|
||||
"profiler_config": _copy_value(engine.profiler_config),
|
||||
"quantization_config": _copy_value(quantization_config),
|
||||
},
|
||||
input_proc,
|
||||
@@ -1391,12 +1612,51 @@ def _build_load_config(
|
||||
return OmniStageLoadConfig(**kwargs)
|
||||
|
||||
|
||||
def _config_kwargs_from_engine_args(
|
||||
engine: Mapping[str, Any],
|
||||
field_map: Mapping[str, str],
|
||||
) -> dict[str, Any]:
|
||||
"""Convert EngineArgs names back to their upstream config field names."""
|
||||
return {
|
||||
config_name: _copy_value(engine[engine_name])
|
||||
for config_name, engine_name in field_map.items()
|
||||
if engine_name in engine and engine[engine_name] is not None
|
||||
}
|
||||
|
||||
|
||||
def _normalize_config_mapping(
|
||||
values: Mapping[str, Any],
|
||||
field_map: Mapping[str, str],
|
||||
) -> dict[str, Any]:
|
||||
"""Canonicalize config and EngineArgs spellings to config field names."""
|
||||
engine_to_config = {engine_name: config_name for config_name, engine_name in field_map.items()}
|
||||
normalized: dict[str, Any] = {}
|
||||
source_names: dict[str, str] = {}
|
||||
for name, value in values.items():
|
||||
if value is None:
|
||||
continue
|
||||
config_name = name if name in field_map else engine_to_config.get(name, name)
|
||||
if config_name in normalized:
|
||||
raise ValueError(
|
||||
f"Config mapping specifies both {source_names[config_name]!r} and {name!r} "
|
||||
f"for upstream field {config_name!r}"
|
||||
)
|
||||
normalized[config_name] = _copy_value(value)
|
||||
source_names[config_name] = name
|
||||
return normalized
|
||||
|
||||
|
||||
def _omni_config_kwargs(engine: Mapping[str, Any], fields_: frozenset[str]) -> dict[str, Any]:
|
||||
return {name: _copy_value(engine[name]) for name in fields_ if name in engine and engine[name] is not None}
|
||||
|
||||
|
||||
def _build_cache_config(
|
||||
deploy: DeployConfig,
|
||||
engine: _CacheEngineOverrides,
|
||||
execution_type: StageExecutionType,
|
||||
) -> OmniStageCacheConfig:
|
||||
kwargs = _config_kwargs(engine)
|
||||
kwargs = _config_kwargs_from_engine_args(engine, _CACHE_CONFIG_ENGINE_FIELD_MAP)
|
||||
kwargs.update(_omni_config_kwargs(engine, _CACHE_ENGINE_FIELDS))
|
||||
if "enable_prefix_caching" not in kwargs and deploy.enable_prefix_caching is not None:
|
||||
kwargs["enable_prefix_caching"] = _copy_value(deploy.enable_prefix_caching)
|
||||
if "disable_hybrid_kv_cache_manager" not in kwargs and execution_type == StageExecutionType.LLM_GENERATION:
|
||||
@@ -1410,7 +1670,8 @@ def _build_scheduler_config(
|
||||
engine: _SchedulerEngineOverrides,
|
||||
execution_type: StageExecutionType,
|
||||
) -> OmniStageSchedulerConfig:
|
||||
kwargs = _config_kwargs(engine)
|
||||
kwargs = _config_kwargs_from_engine_args(engine, _SCHEDULER_CONFIG_ENGINE_FIELD_MAP)
|
||||
kwargs.update(_omni_config_kwargs(engine, _SCHEDULER_ENGINE_FIELDS))
|
||||
if "enable_chunked_prefill" not in kwargs and deploy.enable_chunked_prefill is not None:
|
||||
kwargs["enable_chunked_prefill"] = _copy_value(deploy.enable_chunked_prefill)
|
||||
if "async_scheduling" not in kwargs and execution_type == StageExecutionType.LLM_AR:
|
||||
@@ -1458,16 +1719,21 @@ def _build_parallel_config(
|
||||
config_cls: type[OmniStageParallelConfig] = OmniStageParallelConfig,
|
||||
) -> OmniStageParallelConfig:
|
||||
parallel_config = _mapping_or_empty(engine.get("parallel_config"))
|
||||
config_fields = {
|
||||
config_field.name
|
||||
for config_field in fields(config_cls)
|
||||
if getattr(config_field.default, "init", None) is not False
|
||||
}
|
||||
kwargs = {
|
||||
name: _copy_value(value) for name in _PARALLEL_CONFIG_ENGINE_FIELDS if (value := engine.get(name)) is not None
|
||||
}
|
||||
config_fields = frozenset(signature(config_cls).parameters)
|
||||
is_diffusion = issubclass(config_cls, OmniStageDiffusionParallelConfig)
|
||||
field_map = _DIFFUSION_PARALLEL_CONFIG_FIELD_MAP if is_diffusion else _PARALLEL_CONFIG_ENGINE_FIELD_MAP
|
||||
owned_parallel_fields = frozenset(field_map)
|
||||
kwargs = _config_kwargs(engine) if is_diffusion else _config_kwargs_from_engine_args(engine, field_map)
|
||||
kwargs = {name: value for name, value in kwargs.items() if name in owned_parallel_fields}
|
||||
kwargs = {name: value for name, value in kwargs.items() if name in config_fields}
|
||||
kwargs.update({name: value for name, value in _config_kwargs(parallel_config).items() if name in config_fields})
|
||||
nested_kwargs = _normalize_config_mapping(parallel_config, field_map)
|
||||
kwargs.update(
|
||||
{
|
||||
name: value
|
||||
for name, value in nested_kwargs.items()
|
||||
if name in owned_parallel_fields and name in config_fields
|
||||
}
|
||||
)
|
||||
if "pipeline_parallel_size" not in kwargs and deploy.pipeline_parallel_size is not None:
|
||||
kwargs["pipeline_parallel_size"] = _copy_value(deploy.pipeline_parallel_size)
|
||||
if "data_parallel_size" not in kwargs and deploy.data_parallel_size is not None:
|
||||
|
||||
@@ -14,7 +14,7 @@ import importlib
|
||||
import multiprocessing as mp
|
||||
import os
|
||||
import time
|
||||
from collections.abc import Callable, Generator, Sequence
|
||||
from collections.abc import Callable, Generator, Mapping, Sequence
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass, fields, replace
|
||||
from typing import Any, Literal, cast
|
||||
@@ -29,6 +29,14 @@ from vllm.v1.engine.input_processor import InputProcessor
|
||||
from vllm.v1.executor import Executor
|
||||
|
||||
from vllm_omni.config.omni_config import (
|
||||
_CACHE_STAGE_ENGINE_FIELD_MAP,
|
||||
_DIFFUSION_CACHE_STAGE_ENGINE_FIELD_MAP,
|
||||
_DIFFUSION_LOAD_STAGE_ENGINE_FIELD_MAP,
|
||||
_DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS,
|
||||
_DIFFUSION_SCHEDULER_STAGE_ENGINE_FIELD_MAP,
|
||||
_LOAD_STAGE_ENGINE_FIELD_MAP,
|
||||
_PARALLEL_CONFIG_ENGINE_FIELD_MAP,
|
||||
_SCHEDULER_STAGE_ENGINE_FIELD_MAP,
|
||||
BaseVllmOmniStageConfig,
|
||||
VllmOmniDiffusionStageConfig,
|
||||
)
|
||||
@@ -781,20 +789,38 @@ def stage_runtime_env(stage_id: int, runtime_cfg: Any) -> Generator[None, None,
|
||||
def _project_omni_config_fields(
|
||||
config: Any,
|
||||
*,
|
||||
field_map: Mapping[str, str] | None = None,
|
||||
exclude: frozenset[str] = frozenset(),
|
||||
) -> dict[str, Any]:
|
||||
"""Copy defined typed config fields into backend adapter kwargs."""
|
||||
projected: dict[str, Any] = {}
|
||||
for config_field in fields(config):
|
||||
name = config_field.name
|
||||
if name in exclude:
|
||||
if name in exclude or (field_map is not None and name not in field_map):
|
||||
continue
|
||||
value = getattr(config, name)
|
||||
if value is not None:
|
||||
projected[name] = copy.deepcopy(value)
|
||||
projected[field_map.get(name, name) if field_map is not None else name] = copy.deepcopy(value)
|
||||
return projected
|
||||
|
||||
|
||||
def _project_upstream_config_fields(
|
||||
config: Any,
|
||||
field_map: Mapping[str, str],
|
||||
) -> dict[str, Any]:
|
||||
"""Project every explicit upstream input, including newly added fields."""
|
||||
explicit_fields = getattr(config, "_omni_explicit_fields", frozenset())
|
||||
unprojected_fields = explicit_fields - frozenset(field_map)
|
||||
if unprojected_fields:
|
||||
names = ", ".join(sorted(unprojected_fields))
|
||||
raise ValueError(f"{type(config).__name__} has explicit field(s) with no EngineArgs projection: {names}")
|
||||
return _project_omni_config_fields(
|
||||
config,
|
||||
field_map=field_map,
|
||||
exclude=frozenset(field_map) - explicit_fields,
|
||||
)
|
||||
|
||||
|
||||
def _project_omni_stage_engine_args(
|
||||
stage_config: BaseVllmOmniStageConfig,
|
||||
) -> dict[str, Any]:
|
||||
@@ -810,9 +836,6 @@ def _project_omni_stage_engine_args(
|
||||
stage_config.model_config,
|
||||
frozenset({"default_sampling_params", "has_sampling_extra_args"}),
|
||||
),
|
||||
(stage_config.load_config, frozenset()),
|
||||
(stage_config.cache_config, frozenset()),
|
||||
(stage_config.scheduler_config, frozenset()),
|
||||
(
|
||||
stage_config.runtime_config,
|
||||
frozenset({"devices", "num_replicas", "env", "num_gpus"}),
|
||||
@@ -825,6 +848,26 @@ def _project_omni_stage_engine_args(
|
||||
)
|
||||
)
|
||||
|
||||
if is_diffusion:
|
||||
for config, field_map in (
|
||||
(stage_config.load_config, _DIFFUSION_LOAD_STAGE_ENGINE_FIELD_MAP),
|
||||
(stage_config.cache_config, _DIFFUSION_CACHE_STAGE_ENGINE_FIELD_MAP),
|
||||
(stage_config.scheduler_config, _DIFFUSION_SCHEDULER_STAGE_ENGINE_FIELD_MAP),
|
||||
):
|
||||
engine_args.update(_project_omni_config_fields(config, field_map=field_map))
|
||||
else:
|
||||
for config, field_map in (
|
||||
(stage_config.load_config, _LOAD_STAGE_ENGINE_FIELD_MAP),
|
||||
(stage_config.cache_config, _CACHE_STAGE_ENGINE_FIELD_MAP),
|
||||
(stage_config.scheduler_config, _SCHEDULER_STAGE_ENGINE_FIELD_MAP),
|
||||
):
|
||||
engine_args.update(_project_upstream_config_fields(config, field_map))
|
||||
|
||||
for name in ("compilation_config", "profiler_config"):
|
||||
value = getattr(stage_config, name)
|
||||
if value is not None:
|
||||
engine_args[name] = copy.deepcopy(value)
|
||||
|
||||
# The legacy builder always emits this key, including for pipelines such
|
||||
# as Audex that intentionally defer architecture discovery to HF config.
|
||||
engine_args["model_arch"] = copy.deepcopy(stage_config.model_config.model_arch)
|
||||
@@ -851,13 +894,14 @@ def _project_omni_stage_engine_args(
|
||||
if is_diffusion:
|
||||
engine_args["parallel_config"] = _project_omni_config_fields(
|
||||
stage_config.parallel_config,
|
||||
field_map={name: name for name in _DIFFUSION_PARALLEL_CONFIG_ENGINE_FIELDS},
|
||||
exclude=frozenset({"world_size"}),
|
||||
)
|
||||
else:
|
||||
engine_args.update(
|
||||
_project_omni_config_fields(
|
||||
_project_upstream_config_fields(
|
||||
stage_config.parallel_config,
|
||||
exclude=frozenset({"world_size"}),
|
||||
_PARALLEL_CONFIG_ENGINE_FIELD_MAP,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
Reference in new issue
Block a user