Compare commits

..
31 changed files with 147 additions and 1514 deletions

No files matched your search

+6 -22
View File
@@ -138,28 +138,12 @@ nav:
- Prometheus Metrics: design/metrics.md - Prometheus Metrics: design/metrics.md
- Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md - Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md
- Module Design: - Module Design:
- Entrypoints: design/module/entrypoints.md - Runtime Modules:
- vLLM-Omni Configuration: design/module/vllm_omni_config.md - AR Module: design/module/ar_module.md
- Input, Output, and Modality Contracts: design/module/input_output_modality_contracts.md - DIT Module: design/module/dit_module.md
- Error Contracts: design/module/error_contracts.md - Orchestration:
- Engine Orchestration: design/module/engine_orchestration.md - Entrypoint Module: design/module/entrypoint_module.md
- Stage Runtime: design/module/stage_runtime.md - AsyncOmni Architecture: design/module/async_omni_architecture.md
- OmniConnector: design/module/omni_connector.md
- Model Integration: design/module/model_integration.md
- Autoregressive Runtime: design/module/ar_runtime.md
- Diffusion:
- Overview: design/module/diffusion/index.md
- Runtime: design/module/diffusion/diffusion_runtime.md
- Model Integration: design/module/diffusion/diffusion_model_integration.md
- Continuous Batching: design/module/diffusion/continuous_batching.md
- Parallelism: design/module/diffusion/parallelism.md
- Offloader: design/module/diffusion/offloader.md
- Execution Platforms: design/module/execution_platforms.md
- Cache Management: design/module/cache_management.md
- Quantization: design/module/quantization.md
- Observability: design/module/observability.md
- Profiling: design/module/profiling.md
- Benchmarking: design/module/benchmarking.md
- Docs Guide: contributing/DOCS_GUIDE.md - Docs Guide: contributing/DOCS_GUIDE.md
- API Reference: - API Reference:
- api/README.md - api/README.md
+10 -28
View File
@@ -1,9 +1,8 @@
# Design Documents # Design Documents
This section contains design documents and architecture specifications for This section contains design documents and architecture specifications for
vLLM-Omni. The sidebar groups documents by the system concern they describe; vLLM-Omni. The sidebar groups documents by the system concern they describe,
retired module pages are preserved in the legacy archive instead of remaining while the source paths remain stable so existing links continue to work.
in the active navigation.
## Architecture Documents ## Architecture Documents
@@ -62,29 +61,12 @@ in the active navigation.
## Module Design Documents ## Module Design Documents
- [Entrypoints and Serving Boundaries](module/entrypoints.md) ### Runtime Modules
- [vLLM-Omni Configuration](module/vllm_omni_config.md)
- [Input, Output, and Modality Contracts](module/input_output_modality_contracts.md)
- [Error Classification, Propagation, and Rendering](module/error_contracts.md)
- [Engine Orchestration](module/engine_orchestration.md)
- [Stage Runtime and Replica Lifecycle](module/stage_runtime.md)
- [OmniConnector](module/omni_connector.md)
- [Model Integration](module/model_integration.md)
- [Autoregressive Runtime](module/ar_runtime.md)
- Diffusion
- [Overview](module/diffusion/index.md)
- [Runtime](module/diffusion/diffusion_runtime.md)
- [Model Integration](module/diffusion/diffusion_model_integration.md)
- [Continuous Batching](module/diffusion/continuous_batching.md)
- [Parallelism](module/diffusion/parallelism.md)
- [Offloader](module/diffusion/offloader.md)
- [Execution Platforms](module/execution_platforms.md)
- [Cache Management](module/cache_management.md)
- [Quantization](module/quantization.md)
- [Observability](module/observability.md)
- [Profiling](module/profiling.md)
- [Benchmarking](module/benchmarking.md)
The pre-#5137 pages are preserved in the - [AR Module](module/ar_module.md)
[legacy module archive](module/archive/README.md) for historical reference and - [DIT Module](module/dit_module.md)
are not active design contracts.
### Orchestration
- [Entrypoint Module](module/entrypoint_module.md)
- [AsyncOmni Architecture (Qwen3-Omni Example)](module/async_omni_architecture.md)
@@ -12,7 +12,7 @@ The AR module of vLLM-Omni extends vLLM's core components to support:
- **Hidden state exposure**: Exposing per-request hidden representations for downstream stages - **Hidden state exposure**: Exposing per-request hidden representations for downstream stages
- **Basic generator support**: Support some basic heterogeneous architecture such as Convolution, LSTM, etc. - **Basic generator support**: Support some basic heterogeneous architecture such as Convolution, LSTM, etc.
As shown in the [end2end example](../../../user_guide/examples/offline_inference/qwen3_omni.md), AR module can be widely applied across multiple stages, generating text tokens in thinker(AR), audio latent tokens in talker(AR) and audio wave in code2wav(Convolution). As shown in the [end2end example](../../user_guide/examples/offline_inference/qwen3_omni.md), AR module can be widely applied across multiple stages, generating text tokens in thinker(AR), audio latent tokens in talker(AR) and audio wave in code2wav(Convolution).
## 2. Relationship with vLLM ## 2. Relationship with vLLM
-52
View File
@@ -1,52 +0,0 @@
---
title: Autoregressive Runtime
kind: module
status: draft
owners:
- "@tzhouam"
- "@yinpeiqi"
- "@fake0fan"
- "@Sy0307"
primary_code_paths:
- vllm_omni/core/**
- vllm_omni/worker/**
related_code_paths:
- vllm_omni/model_executor/**
depends_on:
- engine_orchestration.md
- model_integration.md
- input_output_modality_contracts.md
validation_paths:
- tests/core/**
- tests/worker/**
upstream_refs:
- vllm.v1.core
- vllm.v1.worker
last_reviewed: 2026-07-16
---
# Autoregressive runtime
The AR runtime extends vLLM scheduling and worker execution for omni-stage
inputs and outputs while preserving vLLM scheduling and cache semantics.
## Candidate invariants
### AR-INV-001: vLLM owns base scheduling semantics
**Rule:** Omni schedulers MUST preserve upstream request-state and cache
transitions unless an Omni-specific difference is documented and tested.
### AR-INV-002: Omni data crosses explicit adapters
**Rule:** Modality-specific stage data MUST be converted at an input or output
adapter, not injected through unrelated scheduler state.
### AR-INV-003: Workers execute assigned work
**Rule:** Workers and model runners MUST NOT implement cross-stage routing.
## Safe-change guide
Test request lifecycle, abort, cache state, and every affected worker execution
mode against the supported upstream vLLM contract.
-18
View File
@@ -1,18 +0,0 @@
# Archived module design pages
This directory preserves module-design pages that predate the structure
proposed in [issue #5137](https://github.com/vllm-project/vllm-omni/issues/5137).
They are historical snapshots, not active architectural contracts, and are
intentionally excluded from the documentation navigation.
| Archived page | Active destination |
| --- | --- |
| [AR Module](ar_module.md) | [Autoregressive Runtime](../ar_runtime.md) |
| [AsyncOmni Architecture](async_omni_architecture.md) | [Engine Orchestration](../engine_orchestration.md) and [Stage Runtime](../stage_runtime.md) |
| [DiT Module](dit_module.md) | [Diffusion module overview](../diffusion/index.md) |
| [Entrypoint Module](entrypoint_module.md) | [Entrypoints and Serving Boundaries](../entrypoints.md) |
Do not update these pages with current design guidance. Move still-useful
material into the active destination and review it against current code and
tests instead. No compatibility redirect pages are provided at the former
paths.
-47
View File
@@ -1,47 +0,0 @@
---
title: Benchmarking
kind: module
status: draft
owners:
- "@alex-jw-brooks"
primary_code_paths:
- vllm_omni/benchmarks/**
- vllm_omni/entrypoints/cli/benchmark/**
related_code_paths:
- benchmarks/**
depends_on:
- entrypoints.md
- profiling.md
validation_paths:
- tests/benchmarks/**
- tests/dfx/perf/**
upstream_refs:
- vllm.benchmarks
last_reviewed: 2026-07-16
---
# Benchmarking
Benchmarking defines reproducible workloads, metrics, warm-up, measurement,
and result metadata for comparing vLLM-Omni performance.
## Candidate invariants
### BENCH-INV-001: Results are reproducible
**Rule:** Results MUST identify the model, revision, commit, hardware,
configuration, workload, concurrency, warm-up, and measurement window.
### BENCH-INV-002: Correctness precedes performance
**Rule:** A benchmark MUST validate successful and semantically acceptable
outputs before using timings as performance evidence.
### BENCH-INV-003: Metric definitions are explicit
**Rule:** Metrics MUST state their unit, population, and aggregation.
## Safe-change guide
Test workload parsing and metric calculations with deterministic fixtures. Do
not compare materially different workloads or runtime settings.
-47
View File
@@ -1,47 +0,0 @@
---
title: Cache Management
kind: module
status: draft
owners:
- "@Isotr0py"
- "@princepride"
primary_code_paths:
- vllm_omni/diffusion/cache/**
related_code_paths:
- vllm_omni/experimental/ar_diffusion/kv_cache/**
- vllm_omni/core/**
depends_on:
- ar_runtime.md
- diffusion/diffusion_runtime.md
validation_paths:
- tests/diffusion/cache/**
upstream_refs:
- vllm.v1.core.kv_cache_manager
last_reviewed: 2026-07-16
---
# Cache management
Cache management defines reusable state, cache identity, validity, lifecycle,
and eviction across AR and diffusion execution.
## Candidate invariants
### CACHE-INV-001: Cache identity is complete
**Rule:** A cache key MUST include every input and execution property that can
change the reused value.
### CACHE-INV-002: Reuse preserves correctness
**Rule:** A cache optimization MUST provide a disabled path and MUST NOT reuse
state after its validity conditions change.
### CACHE-INV-003: Ownership has an end
**Rule:** Request-scoped and model-scoped cache entries MUST have explicit
reset, eviction, or teardown behavior.
## Safe-change guide
Test hit, miss, invalidation, disabled, concurrent-request, and teardown paths.
@@ -1,51 +0,0 @@
---
title: Diffusion Continuous Batching
kind: module
status: draft
owners:
- "@Isotr0py"
- "@princepride"
primary_code_paths:
- vllm_omni/diffusion/sched/**
related_code_paths:
- vllm_omni/diffusion/executor/**
depends_on:
- diffusion_runtime.md
validation_paths:
- tests/diffusion/batching/**
upstream_refs: []
last_reviewed: 2026-07-16
---
# Diffusion continuous batching
Continuous batching defines when diffusion requests are compatible, how they
share an execution step, and how each request advances independently.
## Candidate invariants
### BATCH-INV-001: Compatibility is explicit
**Rule:** Requests MAY share a batch only when all properties required by the
selected pipeline, scheduler step, shape, precision, and execution features are
compatible.
### BATCH-INV-002: Per-request state remains isolated
**Rule:** Batch assembly MUST preserve each request's random generator, progress,
conditioning, outputs, cancellation, and errors.
### BATCH-INV-003: Admission is bounded
**Rule:** Admission MUST respect configured capacity and MUST NOT rely on an
unbounded waiting or active-request collection.
### BATCH-INV-004: Batch order is not completion order
**Rule:** Output association MUST use stable request identity rather than batch
position across execution steps.
## Safe-change guide
Test heterogeneous requests, partial completion, cancellation, deterministic
seeds, capacity limits, and batch sizes of one and many.
@@ -1,58 +0,0 @@
---
title: Diffusion Model Integration
kind: module
status: draft
owners:
- "@Isotr0py"
- "@wtomin"
- "@ZJY0516"
primary_code_paths:
- vllm_omni/diffusion/models/**
- vllm_omni/diffusion/model_loader/**
related_code_paths:
- vllm_omni/diffusion/layers/**
- vllm_omni/diffusion/lora/**
- vllm_omni/diffusion/utils/**
depends_on:
- diffusion_runtime.md
- ../input_output_modality_contracts.md
validation_paths:
- tests/diffusion/models/**
- tests/diffusion/model_loader/**
- tests/diffusion/layers/**
- tests/diffusion/lora/**
upstream_refs:
- diffusers.DiffusionPipeline
last_reviewed: 2026-07-16
---
# Diffusion model integration
Diffusion model integration owns pipeline contracts, registration, checkpoint
loading, adapters, shared layers, and model-specific processing.
## Candidate invariants
### DIFF-MODEL-INV-001: Pipelines implement one runtime contract
**Rule:** A pipeline MUST declare its supported modalities, configuration,
inputs, outputs, loading path, and runtime capabilities.
### DIFF-MODEL-INV-002: Registration is the selection boundary
**Rule:** Runtime code MUST select model implementations through the registry or
loader contract rather than scattered model-name conditionals.
### DIFF-MODEL-INV-003: Model code does not schedule requests
**Rule:** Pipeline code MUST NOT own admission, batching, cancellation, or
cross-stage routing.
### DIFF-MODEL-INV-004: Shared behavior stays shared
**Rule:** Model directories SHOULD contain only genuine model differences.
## Safe-change guide
Test registry selection, checkpoint loading, minimal inference, input and output
contracts, and every declared optional capability.
@@ -1,58 +0,0 @@
---
title: Diffusion Runtime
kind: module
status: draft
owners:
- "@Isotr0py"
- "@princepride"
- "@SamitHuang"
primary_code_paths:
- vllm_omni/diffusion/sched/**
- vllm_omni/diffusion/executor/**
- vllm_omni/diffusion/worker/**
related_code_paths:
- vllm_omni/diffusion/hooks/**
- vllm_omni/diffusion/postprocess/**
depends_on:
- ../engine_orchestration.md
- ../input_output_modality_contracts.md
- diffusion_model_integration.md
validation_paths:
- tests/diffusion/diffusion_backend/**
- tests/diffusion/hooks/**
upstream_refs:
- diffusers.DiffusionPipeline
last_reviewed: 2026-07-16
---
# Diffusion runtime
The diffusion runtime owns request admission, scheduling, execution, progress,
output, cancellation, and cleanup inside a diffusion stage.
## Candidate invariants
### DIFF-RUNTIME-INV-001: One lifecycle owner
**Rule:** Every admitted request MUST have exactly one scheduler-owned lifecycle
until completion, cancellation, or failure.
### DIFF-RUNTIME-INV-002: Execution follows scheduler output
**Rule:** Executors and workers MUST execute scheduler decisions without
admitting, reordering, or forwarding requests independently.
### DIFF-RUNTIME-INV-003: Terminal cleanup is complete
**Rule:** Every terminal path MUST release request state, temporary tensors,
hooks, and runtime-owned resources.
### DIFF-RUNTIME-INV-004: Optional features use runtime hooks
**Rule:** Cache, profiling, offload, and parallel features SHOULD integrate at
defined hooks instead of creating another request lifecycle.
## Safe-change guide
Test success, cancellation, failure, shutdown, multiple requests, and resource
cleanup.
-42
View File
@@ -1,42 +0,0 @@
---
title: Diffusion Modules
kind: module-index
status: draft
owners:
- "@Isotr0py"
- "@princepride"
- "@SamitHuang"
- "@wtomin"
- "@ZJY0516"
- "@RuixiangMa"
- "@david6666666"
- "@xuechendi"
primary_code_paths:
- vllm_omni/diffusion/**
related_code_paths:
- vllm_omni/platforms/**
depends_on:
- ../engine_orchestration.md
- ../input_output_modality_contracts.md
validation_paths:
- tests/diffusion/**
upstream_refs:
- diffusers
last_reviewed: 2026-07-16
---
# Diffusion modules
Diffusion is a module family containing runtime, model integration, batching,
distributed execution, and memory-management subsystems.
## Module documents
- [Diffusion runtime](diffusion_runtime.md)
- [Diffusion model integration](diffusion_model_integration.md)
- [Continuous batching](continuous_batching.md)
- [Parallelism](parallelism.md)
- [Offloader](offloader.md)
Cache, quantization, profiling, and benchmarking follow their top-level module
documents.
-53
View File
@@ -1,53 +0,0 @@
---
title: Diffusion Offloader
kind: module
status: draft
owners:
- "@yuanheng-zhao"
- "@Isotr0py"
primary_code_paths:
- vllm_omni/diffusion/offloader/**
related_code_paths:
- vllm_omni/diffusion/models/**
- vllm_omni/diffusion/worker/**
depends_on:
- diffusion_runtime.md
- ../execution_platforms.md
validation_paths:
- tests/diffusion/offloader/**
upstream_refs:
- torch.nn.Module.to
last_reviewed: 2026-07-16
---
# Diffusion offloader
The offloader owns component residency, transfer scheduling, memory accounting,
prefetch, and teardown for diffusion execution.
## Candidate invariants
### OFFLOAD-INV-001: Residency is explicit
**Rule:** Each managed component MUST have one known residency state and one
owner responsible for transitions.
### OFFLOAD-INV-002: Use follows readiness
**Rule:** Runtime execution MUST NOT consume a component until its transfer to
the required device is complete.
### OFFLOAD-INV-003: Transfers preserve model state
**Rule:** Offloading MUST preserve parameter and buffer identity, dtype, device,
and correctness required by the selected execution mode.
### OFFLOAD-INV-004: Memory is bounded and released
**Rule:** Retained host and device copies MUST respect configured limits and
provide deterministic teardown.
## Safe-change guide
Test enabled and disabled paths, repeated execution, asynchronous transfer,
memory limits, failure, teardown, and numerical equivalence.
@@ -1,56 +0,0 @@
---
title: Diffusion Parallelism
kind: module
status: draft
owners:
- "@princepride"
- "@yuanheng-zhao"
- "@xuechendi"
primary_code_paths:
- vllm_omni/diffusion/distributed/**
- vllm_omni/diffusion/attention/parallel/**
related_code_paths:
- vllm_omni/distributed/**
- vllm_omni/config/composable_parallel/**
depends_on:
- diffusion_runtime.md
- ../vllm_omni_config.md
validation_paths:
- tests/diffusion/distributed/**
- tests/diffusion/attention/**
upstream_refs:
- torch.distributed
last_reviewed: 2026-07-16
---
# Diffusion parallelism
Parallelism owns diffusion rank topology, process groups, tensor and sequence
sharding, collectives, and distributed execution.
## Candidate invariants
### PARALLEL-INV-001: Topology has one source of truth
**Rule:** Rank coordinates, group membership, and parallel dimensions MUST be
derived from validated configuration.
### PARALLEL-INV-002: Sharding contracts are explicit
**Rule:** Every distributed boundary MUST define tensor shape, sharded
dimension, placement, dtype, producer, and consumer.
### PARALLEL-INV-003: Collectives are symmetric
**Rule:** Process-group members MUST invoke compatible collectives in the same
logical order.
### PARALLEL-INV-004: Single-rank behavior remains valid
**Rule:** Distributed implementations SHOULD preserve a supported single-rank
path without distributed initialization.
## Safe-change guide
Test topology validation, single-rank execution, affected parallel modes,
combined modes, invalid shapes, and orderly teardown.
@@ -252,7 +252,7 @@ while True:
batch of compatible independent requests for one full pipeline forward when batch of compatible independent requests for one full pipeline forward when
the pipeline supports request-level batching; the step-wise path can also the pipeline supports request-level batching; the step-wise path can also
admit or remove compatible requests between denoise steps. See admit or remove compatible requests between denoise steps. See
[Diffusion Continuous Batching](../../feature/diffusion_continuous_batching.md). [Diffusion Continuous Batching](../feature/diffusion_continuous_batching.md).
--- ---
@@ -541,7 +541,7 @@ def get_attn_backend_for_role(role, head_size, attention_config=None, role_categ
A backend that needs configuration exposes it as a typed field on `AttentionSpec`; `AttentionSpec.backend_kwargs()` serializes it into the `backend_kwargs` the impl consumes. Today those fields are `skip_softmax` (TRTLLM_ATTN) and `block_sparse` (any block-sparse backend, currently RAINFUSION_ATTN), so e.g. `target_sparsity` or `sparsity` can be set without changing model code. A backend that needs configuration exposes it as a typed field on `AttentionSpec`; `AttentionSpec.backend_kwargs()` serializes it into the `backend_kwargs` the impl consumes. Today those fields are `skip_softmax` (TRTLLM_ATTN) and `block_sparse` (any block-sparse backend, currently RAINFUSION_ATTN), so e.g. `target_sparsity` or `sparsity` can be set without changing model code.
For the user-facing CLI surface, see [Diffusion Attention Backends](../../../user_guide/diffusion/attention_backends.md). For the role declaration contract on the model side, see [Adding a Diffusion Model](../../../contributing/model/adding_diffusion_model.md). For the user-facing CLI surface, see [Diffusion Attention Backends](../../user_guide/diffusion/attention_backends.md). For the role declaration contract on the model side, see [Adding a Diffusion Model](../../contributing/model/adding_diffusion_model.md).
**Backend Availability**: **Backend Availability**:
-122
View File
@@ -1,122 +0,0 @@
---
title: Engine Orchestration
kind: module
status: draft
architecture_state: current-plus-in-flight
owners:
- "@tzhouam"
- "@fake0fan"
document_stewards:
- "@hsliuustc0106"
- "@Gaohan123"
- "@david6666666"
required_reviewers:
- "@yinpeiqi"
- "@Sy0307"
- "@yenuo26"
- "@NickCao"
primary_code_paths:
- vllm_omni/engine/async_omni_engine.py
- vllm_omni/engine/async_engine_utils.py
- vllm_omni/engine/orchestrator.py
- vllm_omni/engine/cfg_companion_tracker.py
- vllm_omni/engine/rpc_result_router.py
related_code_paths:
- vllm_omni/engine/messages.py
- vllm_omni/engine/orchestrator_monitor.py
- vllm_omni/engine/stage_client.py
- vllm_omni/engine/stage_pool.py
- vllm_omni/outputs/output_processor.py
- vllm_omni/distributed/omni_coordinator/**
depends_on:
- input_output_modality_contracts.md
- error_contracts.md
- stage_runtime.md
- omni_connector.md
validation_paths:
- tests/engine/test_async_omni_engine_input.py
- tests/engine/test_async_omni_engine_outputs.py
- tests/engine/test_async_omni_engine_abort.py
- tests/engine/test_async_omni_engine_stage_init.py
- tests/engine/test_orchestrator.py
- tests/engine/test_orchestrator_error_handling.py
- tests/engine/test_orchestrator_stage_input_bridge.py
- tests/engine/test_cfg_companion_lifecycle.py
- tests/engine/test_rpc_result_router.py
- tests/e2e/features/fullduplex/engine/**
upstream_refs:
- vllm.v1.engine.EngineCoreRequest
- vllm.v1.engine.EngineCoreOutputs
- vllm.v1.engine.exceptions.EngineDeadError
- vllm.outputs.RequestOutput
invariant_namespace: ORCH-INV
last_reviewed: 2026-08-07
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
---
# Engine orchestration
Engine orchestration coordinates configured AR and diffusion stages while
keeping public clients independent from stage processes and transports.
## Contract status
This document is a draft description of current behavior. It also identifies
the boundary affected by the in-flight stage client/process refactor in
[#5441](https://github.com/vllm-project/vllm-omni/pull/5441). Names and
responsibilities proposed only by that PR are not current contracts.
## Ownership boundary
This document owns `AsyncOmniEngine`, `Orchestrator`, request-state creation,
cross-stage routing, output ordering, companion tracking, control/RPC
correlation, cancellation propagation, and terminal-state convergence.
It does not own stage placement, replica selection, stage-process startup,
payload schemas, public protocol rendering, connector transport, or semantic
error classification. Those responsibilities belong to the stage runtime,
I/O, entrypoint, connector, and error contracts.
## Candidate invariants
These identifiers are proposals while the document is `draft`.
### ORCH-INV-001: The orchestrator owns cross-stage routing
**Rule:** Entrypoints and stage clients MUST NOT independently forward a
request to a downstream logical stage.
### ORCH-INV-002: Stage clients do not own routing policy
**Rule:** Stage clients MUST implement communication and lifecycle operations
without selecting the next logical stage.
### ORCH-INV-100: Terminal state is monotonic
**Rule:** Once a request reaches a terminal state, orchestration MUST NOT
forward new work for that request.
## Invariant namespace
`ORCH-INV` reserves `001-099` for dependency direction, `100-199` for
request state and ordering, `200-299` for failure/cancellation/cleanup, and
`300-399` for extension points, shutdown ordering, and upstream alignment.
Numbers become append-only after normative promotion.
## Safe-change guide
Test routing, output ordering, queue correlation, cancellation, failure
propagation, shutdown ordering, and representative multi-stage execution.
Changes that cross into stage runtime or public error behavior require review
from that contract's owners.
## Promotion gate
- Reconcile current names and boundaries after #5441 merges, closes, or is
replaced.
- Demonstrate one terminal outcome per request across success, abort,
request-scoped failure, and fatal engine failure.
- Verify queue correlation and orchestrator-before-runtime shutdown ordering
in the cited validation paths.
- Obtain approval from a technical owner and an independent validation
reviewer.
-134
View File
@@ -1,134 +0,0 @@
---
title: Entrypoints and Serving Boundaries
kind: module
status: draft
architecture_state: current-plus-in-flight
owners:
- "@alex-jw-brooks"
- "@linyueqian"
document_stewards:
- "@hsliuustc0106"
- "@Gaohan123"
- "@david6666666"
required_reviewers:
- "@tzhouam"
- "@herotai214"
- "@yenuo26"
- "@NickCao"
primary_code_paths:
- vllm_omni/entrypoints/omni.py
- vllm_omni/entrypoints/async_omni.py
- vllm_omni/entrypoints/omni_base.py
- vllm_omni/entrypoints/cli/**
- vllm_omni/entrypoints/openai/**
- vllm_omni/entrypoints/openpi/**
- vllm_omni/entrypoints/client_request_state.py
- vllm_omni/entrypoints/stage_utils.py
- vllm_omni/entrypoints/utils.py
primary_path_exceptions:
- path: vllm_omni/entrypoints/openai/errors.py
owner: error_contracts.md
related_code_paths:
- vllm_omni/errors.py
- vllm_omni/inputs/**
- vllm_omni/outputs/**
- vllm_omni/engine/async_omni_engine.py
- vllm_omni/config/**
- vllm_omni/deploy/**
depends_on:
- input_output_modality_contracts.md
- error_contracts.md
- engine_orchestration.md
validation_paths:
- tests/entrypoints/test_omni_entrypoints.py
- tests/entrypoints/test_async_omni.py
- tests/entrypoints/test_async_omni_duplex.py
- tests/entrypoints/test_serve.py
- tests/entrypoints/test_stream_finish_reason.py
- tests/entrypoints/openai/**
- tests/entrypoints/openai_api/**
- tests/e2e/online_serving/**
upstream_refs:
- vllm.engine.protocol.EngineClient
- vllm.entrypoints.openai/**
- vllm.entrypoints.serve/**
- vllm.renderers/**
- vllm.v1.engine.exceptions/**
invariant_namespace: ENTRY-INV
last_reviewed: 2026-08-07
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
---
# Entrypoints and serving boundaries
Entrypoints translate offline, CLI, and serving requests into stable engine
operations and translate engine outputs into public responses.
## Contract status
This document describes current entrypoint responsibilities plus the boundary
under review in the open roadmap
[#5227](https://github.com/vllm-project/vllm-omni/issues/5227) and helper-move
PR [#5453](https://github.com/vllm-project/vllm-omni/pull/5453). In-flight
helper locations are not treated as current paths.
## Ownership boundary
This document owns offline API semantics, CLI and serve composition,
supported OpenAI-compatible routes, request validation and normalization,
response conversion, streaming/session behavior, and engine handoff.
It does not own configuration precedence, cross-stage routing, stage
placement, payload implementation, or semantic error classification.
`entrypoints/openai/errors.py` is an explicit primary-path exception owned by
`error_contracts.md`.
## Candidate invariants
These identifiers are proposals while the document is `draft`.
### ENTRY-INV-001: Entrypoints adapt but do not orchestrate
**Rule:** Entrypoints MUST NOT implement cross-stage routing or stage lifecycle
policy.
### ENTRY-INV-100: Public requests are normalized once
**Rule:** Public protocol values MUST be validated and converted to internal
request contracts before engine submission.
### ENTRY-INV-101: Streaming preserves request identity
**Rule:** Every streamed response MUST remain associated with the request and
output modality that produced it.
The reviewer-proposed rule that model-specific behavior should stay behind a
common adapter or processor abstraction remains an unnumbered candidate until
the entrypoint refactor makes that boundary enforceable.
## Invariant namespace
`ENTRY-INV` reserves `001-099` for boundary and dependency direction,
`100-199` for normalization, sessions, and streaming, `200-299` for
disconnect, cancellation, rendering, and cleanup, and `300-399` for upstream
route and renderer compatibility. Numbers become append-only after normative
promotion.
## Safe-change guide
Test request validation, protocol conversion, model-adapter routing,
streaming, session identity, disconnect/cancellation, and error mapping for
each affected offline or serving entrypoint.
## Promotion gate
- Reconcile the page after #5227's P0 ownership work and #5453's final
disposition.
- Publish a supported route/transport matrix rather than claiming blanket
OpenAI parity.
- Verify one normalization and handoff path per route family and one terminal
outcome per streaming transport.
- Promote the `api_server.py` composition-root rule only after helper moves and
import boundaries enforce it.
- Obtain approval from a technical owner and an independent validation
reviewer.
-109
View File
@@ -1,109 +0,0 @@
---
title: Error Classification, Propagation, and Rendering
kind: module
status: draft
architecture_state: current-plus-rfc
owners:
- "@alex-jw-brooks"
- "@NickCao"
document_stewards:
- "@hsliuustc0106"
- "@Gaohan123"
- "@david6666666"
required_reviewers:
- "@yinpeiqi"
- "@tzhouam"
- "@yenuo26"
primary_code_paths:
- vllm_omni/errors.py
- vllm_omni/entrypoints/openai/errors.py
related_code_paths:
- vllm_omni/engine/messages.py
- vllm_omni/engine/orchestrator.py
- vllm_omni/entrypoints/omni_base.py
- vllm_omni/entrypoints/async_omni.py
- vllm_omni/entrypoints/openai/api_server.py
- vllm_omni/entrypoints/openai/serving_chat.py
- vllm_omni/entrypoints/openai/serving_speech.py
- vllm_omni/entrypoints/openai/serving_audio_generate.py
- vllm_omni/entrypoints/openai/protocol/images.py
- vllm_omni/entrypoints/openai/protocol/videos.py
depends_on: []
validation_paths:
- tests/engine/test_orchestrator_error_handling.py
- tests/engine/test_async_omni_engine_outputs.py
- tests/entrypoints/test_omni_entrypoints.py
- tests/entrypoints/test_stream_finish_reason.py
- tests/entrypoints/openai_api/test_image_server.py
- tests/entrypoints/openai_api/test_video_server.py
- tests/entrypoints/openai_api/test_serving_audio_generate.py
- tests/entrypoints/openai_api/test_serving_speech.py
- tests/metrics/test_prometheus.py
upstream_refs:
- vllm.v1.engine.exceptions.EngineDeadError
- vllm.v1.engine.exceptions.EngineGenerateError
- vllm.entrypoints.serve.utils.error_response
- vLLM RFC #48227
- vLLM PR #49665
invariant_namespace: ERR-INV
last_reviewed: 2026-08-07
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
---
# Error classification, propagation, and rendering
This contract separates the meaning of a failure from the transport used to
carry it and the public protocol used to render it.
## Contract status
Current behavior is distributed across engine, message, and entrypoint code.
The semantic hierarchy proposed by
[#5570](https://github.com/vllm-project/vllm-omni/issues/5570) remains an open
RFC. Proposed types and wire records from that RFC are not current contracts.
## Ownership boundary
This document owns semantic classification, fatal versus request-scoped
failure, cross-process error context, compatibility policy, sanitization, and
offline, HTTP, SSE, WebSocket, and asynchronous-job rendering rules.
`engine/messages.py` remains primarily owned by the I/O contract; this page
owns the semantics of its error fields. Route and serving files remain
primarily owned by `entrypoints.md`; this page owns their rendering
constraints.
## Candidate invariants
The following topics require owner review before receiving stable IDs:
- A request-scoped failure terminates only the affected request unless the
underlying engine is no longer usable.
- Every public rendering path preserves a stable classification while
removing sensitive internal context.
- Cancellation remains distinguishable from internal failure across process
and protocol boundaries.
## Invariant namespace
`ERR-INV` reserves `001-099` for classification ownership, `100-199` for wire
context and compatibility, `200-299` for fatality, termination, sanitization,
and rendering, and `300-399` for upstream alignment and transport extension
points. Numbers become append-only after normative promotion.
## Safe-change guide
Exercise exception classification, stage wire round trips, fatal and
request-scoped propagation, cancellation, sanitization, offline exceptions,
HTTP status/body consistency, streaming terminal events, asynchronous-job
persistence, and metrics classification.
## Promotion gate
- Resolve #5570's hierarchy, wire record, sanitization, and mixed-version
questions.
- Add table-driven exception/status/type tests and stage wire round trips.
- Verify all public transports terminate once and do not leak sensitive
payload data.
- Obtain approval from both technical owners and an independent validation
reviewer.
-48
View File
@@ -1,48 +0,0 @@
---
title: Execution Platforms
kind: module
status: draft
owners:
- "@gcanlin"
- "@tjtanaa"
- "@xuechendi"
primary_code_paths:
- vllm_omni/platforms/**
related_code_paths:
- vllm_omni/attention/**
depends_on:
- ar_runtime.md
- diffusion/index.md
validation_paths:
- tests/platforms/**
upstream_refs:
- vllm.platforms
last_reviewed: 2026-07-16
---
# Execution platforms
Execution platforms isolate hardware-specific capability detection, worker
selection, patches, kernels, and configuration adjustments.
## Candidate invariants
### PLATFORM-INV-001: Capabilities are explicit
**Rule:** Hardware-dependent behavior MUST be guarded by platform selection or
capability detection and MUST fail clearly when unsupported.
### PLATFORM-INV-002: Portable code remains portable
**Rule:** Platform-neutral modules MUST NOT import a vendor implementation
directly when a common selection interface exists.
### PLATFORM-INV-003: Overrides are minimal
**Rule:** A platform implementation SHOULD override only behavior that differs
from the common or upstream implementation.
## Safe-change guide
Validate common import paths without the target accelerator and run focused
tests in a fresh `uv` environment on every affected device platform.
@@ -1,133 +0,0 @@
---
title: Input, Output, and Modality Contracts
kind: module
status: draft
architecture_state: current
owners:
- "@Sy0307"
- "@amy-why-3459"
document_stewards:
- "@hsliuustc0106"
- "@Gaohan123"
- "@david6666666"
required_reviewers:
- "@tzhouam"
- "@linyueqian"
- "@david6666666"
- "@yenuo26"
- "@NickCao"
primary_code_paths:
- vllm_omni/inputs/**
- vllm_omni/outputs/**
- vllm_omni/request.py
- vllm_omni/data_entry_keys.py
- vllm_omni/engine/messages.py
- vllm_omni/engine/serialization.py
- vllm_omni/engine/mm_outputs.py
- vllm_omni/engine/output_modality.py
- vllm_omni/engine/output_processor.py
related_code_paths:
- vllm_omni/model_executor/stage_input_processors/**
- vllm_omni/core/sched/**
- vllm_omni/distributed/omni_connectors/**
- vllm_omni/diffusion/request.py
- vllm_omni/diffusion/worker/request_batch.py
- vllm_omni/errors.py
depends_on:
- error_contracts.md
validation_paths:
- tests/inputs/**
- tests/engine/test_data_entry_keys.py
- tests/engine/test_omni_request.py
- tests/engine/test_omni_request_output.py
- tests/engine/test_multimodal_accumulation.py
- tests/engine/test_output_modality.py
- tests/engine/test_output_processor.py
- tests/engine/test_wire_multimodal_output.py
- tests/utils/test_mm_outputs.py
- tests/utils/test_mm_outputs_partition.py
- tests/diffusion/test_diffusion_output_metadata.py
upstream_refs:
- vllm.inputs/**
- vllm.outputs/**
- vllm.v1.request.Request
- vllm.v1.engine.EngineCoreRequest
- vllm.v1.engine.EngineCoreOutput
- vllm.v1.engine.output_processor.OutputProcessor
invariant_namespace: IO-INV
last_reviewed: 2026-08-07
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
---
# Input, output, and modality contracts
These contracts define the data that may cross entrypoint, orchestration,
stage, connector, and model boundaries.
## Contract status
This document is a draft description of the current request, message,
serialization, modality, accumulation, and output types. It does not assign
semantic error ownership to the wire schema.
## Ownership boundary
This document owns request identity, prompt/input types, modality keys and
metadata, queue and wire message schemas, serialization, output types,
accumulation, completion semantics, and compatibility shims.
It does not own route-specific validation or rendering, model-specific
interpretation, connector transport mechanics, scheduling policy, or semantic
error classification. The `ErrorMessage` schema belongs here; the meaning and
public rendering of its error fields belong to `error_contracts.md`.
## Candidate invariants
These identifiers are proposals while the document is `draft`.
### IO-INV-001: Boundary data has an explicit modality
**Rule:** Data crossing a module or stage boundary MUST identify its modality
and use the corresponding validated contract.
### IO-INV-100: Request identity is stable
**Rule:** Request identity MUST be preserved across conversions, stages,
streaming updates, cancellation, and errors.
### IO-INV-101: Completion is explicit
**Rule:** Producers MUST distinguish partial updates from the terminal update
for every output modality.
### IO-INV-300: Internal objects do not leak into public protocols
**Rule:** Entrypoints MUST explicitly translate internal output objects into
public response types.
## Invariant namespace
`IO-INV` reserves `001-099` for schema ownership and dependency direction,
`100-199` for identity, modality, accumulation, ordering, and completion,
`200-299` for serialization failure and payload cleanup, and `300-399` for
upstream extension, deprecation, and compatibility. Numbers become
append-only after normative promotion.
## Safe-change guide
Test construction, validation, serialization, round trips, streaming,
accumulation, completion, optional fields, unknown fields, and compatibility
at every affected producer-consumer boundary.
## Promotion gate
- Add explicit CODEOWNERS for the primary I/O paths or confirm that fallback
ownership is intentional.
- Document the compatibility window for deprecated `vllm_omni.engine`
I/O-related imports.
- Verify wire round trips and rejection of unknown or incompatible schema
fields.
- Define payload-versus-metadata rules and completion semantics for streaming
and batched outputs.
- Obtain approval from a technical owner and independent AR, diffusion,
connector, entrypoint, and validation reviewers as applicable.
-51
View File
@@ -1,51 +0,0 @@
---
title: Model Integration
kind: module
status: draft
owners:
- "@tzhouam"
- "@gcanlin"
primary_code_paths:
- vllm_omni/model_executor/**
- vllm_omni/model_extras/**
- vllm_omni/plugins/**
related_code_paths:
- vllm_omni/transformers_utils/**
- vllm_omni/tokenizers/**
depends_on:
- input_output_modality_contracts.md
- ar_runtime.md
validation_paths:
- tests/model_executor/**
- tests/model_extras/**
- tests/plugin/**
upstream_refs:
- vllm.model_executor
last_reviewed: 2026-07-16
---
# Model integration
Model integration adapts model-specific configuration, preprocessing, loading,
and execution behavior to stable runtime contracts.
## Candidate invariants
### MODEL-INV-001: Registration is explicit
**Rule:** A model integration MUST declare how its model class, loader, input
processor, and stage configuration are selected.
### MODEL-INV-002: Model code does not route stages
**Rule:** Model-specific code MUST NOT select or invoke a downstream omni stage.
### MODEL-INV-003: Upstream behavior is reused deliberately
**Rule:** An upstream vLLM implementation SHOULD be reused when its contract is
sufficient; an override MUST document the behavioral difference.
## Safe-change guide
Test registration, checkpoint loading, input conversion, and representative
model execution. Test shared utilities against more than one integration.
-46
View File
@@ -1,46 +0,0 @@
---
title: Observability
kind: module
status: draft
owners:
- "@lishunyang12"
primary_code_paths:
- vllm_omni/metrics/**
related_code_paths:
- vllm_omni/engine/**
- vllm_omni/entrypoints/**
depends_on:
- engine_orchestration.md
validation_paths:
- tests/metrics/**
upstream_refs:
- vllm.v1.metrics
last_reviewed: 2026-07-16
---
# Observability
Observability defines stable metrics, logs, and correlation fields for tracing a
request across entrypoints, orchestration, stages, and workers.
## Candidate invariants
### OBS-INV-001: Request correlation survives stage boundaries
**Rule:** Logs and metrics for the same request SHOULD carry a stable request
identifier and stage identity.
### OBS-INV-002: Metric meaning is stable
**Rule:** A metric's unit, labels, aggregation, and lifecycle MUST be documented;
changing any of them is a contract change.
### OBS-INV-003: Labels are bounded
**Rule:** Metrics MUST NOT use request IDs, prompts, outputs, or other unbounded
values as labels.
## Safe-change guide
Test registration, label sets, units, lifecycle resets, and representative
multi-stage emission.
-50
View File
@@ -1,50 +0,0 @@
---
title: OmniConnector
kind: module
status: draft
owners:
- "@princepride"
- "@yuanheng-zhao"
- "@xuechendi"
- "@natureofnature"
- "@fake0fan"
primary_code_paths:
- vllm_omni/distributed/omni_connectors/**
related_code_paths:
- vllm_omni/platforms/*/omni_connectors/**
depends_on:
- input_output_modality_contracts.md
- vllm_omni_config.md
validation_paths:
- tests/distributed/omni_connectors/**
upstream_refs:
- vllm.distributed
last_reviewed: 2026-07-16
---
# OmniConnector
OmniConnector defines model-agnostic transport and synchronization contracts
for data exchanged across stages, processes, devices, and nodes.
## Candidate invariants
### CONNECTOR-INV-001: Connectors transport but do not route
**Rule:** A connector MUST NOT select the next stage or implement model-specific
execution policy.
### CONNECTOR-INV-002: Producer and consumer contracts agree
**Rule:** Both ends MUST agree on data identity, shape, dtype, placement,
ownership, and completion semantics.
### CONNECTOR-INV-003: Resources have deterministic cleanup
**Rule:** Connections, buffers, handles, and background work MUST be released on
normal completion, cancellation, failure, and shutdown.
## Safe-change guide
Test setup, transfer, synchronization, timeout, cancellation, failure, and
cleanup across every affected backend.
-46
View File
@@ -1,46 +0,0 @@
---
title: Profiling
kind: module
status: draft
owners:
- "@gcanlin"
primary_code_paths:
- vllm_omni/profiler/**
- vllm_omni/diffusion/profiler/**
related_code_paths:
- vllm_omni/entrypoints/**
depends_on:
- observability.md
validation_paths:
- tests/profile/**
upstream_refs:
- vllm.profiler
last_reviewed: 2026-07-16
---
# Profiling
Profiling owns opt-in instrumentation and trace collection used to locate
runtime bottlenecks without changing model results or execution policy.
## Candidate invariants
### PROFILE-INV-001: Profiling is opt-in
**Rule:** Expensive trace collection MUST be disabled by default and bounded by
an explicit start and stop lifecycle.
### PROFILE-INV-002: Instrumentation preserves semantics
**Rule:** Profiling MUST NOT change scheduling decisions, generated outputs, or
resource ownership beyond documented synchronization overhead.
### PROFILE-INV-003: Trace context is sufficient
**Rule:** Trace events SHOULD identify stage, rank, worker, operation, and
request where available without embedding user payloads.
## Safe-change guide
Test disabled overhead, repeated start and stop, cleanup, multi-rank trace
naming, and each supported profiler backend.
-50
View File
@@ -1,50 +0,0 @@
---
title: Quantization
kind: module
status: draft
owners:
- "@david6666666"
- "@Isotr0py"
- "@lishunyang12"
primary_code_paths:
- vllm_omni/quantization/**
- vllm_omni/diffusion/quantization/**
related_code_paths:
- vllm_omni/platforms/*/quant/**
depends_on:
- model_integration.md
- diffusion/diffusion_model_integration.md
validation_paths:
- tests/quantization/**
- tests/diffusion/quantization/**
upstream_refs:
- vllm.model_executor.layers.quantization
last_reviewed: 2026-07-16
---
# Quantization
Quantization owns method registration, checkpoint interpretation, layer
replacement, and runtime constraints for reduced-precision execution.
## Candidate invariants
### QUANT-INV-001: Checkpoint and runtime methods agree
**Rule:** The selected method MUST validate checkpoint metadata, supported
layers, device capabilities, and required dependencies before use.
### QUANT-INV-002: Unsupported combinations fail clearly
**Rule:** Quantization MUST NOT silently fall back to a different precision or
method when a requested combination is unsupported.
### QUANT-INV-003: Numerical validation is required
**Rule:** A new method or layer mapping MUST include correctness comparison
against a defined reference result.
## Safe-change guide
Test loading, representative layers, metadata, platform guards, and numerical
tolerances separately from performance.
-126
View File
@@ -1,126 +0,0 @@
---
title: Stage Runtime and Replica Lifecycle
kind: module
status: draft
architecture_state: current-plus-in-flight
owners:
- "@tzhouam"
- "@fake0fan"
document_stewards:
- "@hsliuustc0106"
- "@Gaohan123"
- "@david6666666"
required_reviewers:
- "@Sy0307"
- "@princepride"
- "@chickeyton"
- "@yenuo26"
- "@NickCao"
primary_code_paths:
- vllm_omni/engine/stage_runtime.py
- vllm_omni/engine/stage_pool.py
- vllm_omni/engine/stage_client.py
- vllm_omni/engine/membership_controller.py
- vllm_omni/engine/stage_engine_startup.py
- vllm_omni/engine/stage_engine_core_client.py
- vllm_omni/engine/stage_engine_core_proc.py
- vllm_omni/engine/stage_engine_core_proc_manager.py
- vllm_omni/engine/stage_init_utils.py
related_code_paths:
- vllm_omni/distributed/omni_coordinator/**
- vllm_omni/distributed/ray_utils/**
- vllm_omni/diffusion/inline_stage_diffusion_client.py
- vllm_omni/diffusion/stage_diffusion_client.py
- vllm_omni/diffusion/stage_diffusion_proc.py
- vllm_omni/config/**
depends_on:
- input_output_modality_contracts.md
- error_contracts.md
- omni_connector.md
validation_paths:
- tests/engine/test_async_omni_engine_stage_init.py
- tests/engine/test_membership_controller.py
- tests/engine/test_single_stage_mode.py
- tests/engine/test_stage_engine_core_client.py
- tests/engine/test_stage_engine_core_proc.py
- tests/diffusion/test_inline_stage_diffusion_client.py
- tests/diffusion/test_stage_diffusion_proc.py
- tests/distributed/omni_coordinator/**
upstream_refs:
- vllm.v1.engine.core_client.AsyncMPClient
- vllm.v1.engine.core_client.DPLBAsyncMPClient
- vllm.v1.engine.core.EngineCoreProc
- vllm.v1.engine.utils.CoreEngineProcManager
- vllm.v1.engine.coordinator.DPCoordinator
invariant_namespace: STAGE-INV
last_reviewed: 2026-08-07
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
---
# Stage runtime and replica lifecycle
The stage runtime turns a logical stage definition into ready local or
distributed replicas and provides the lifecycle boundary used by the
orchestrator.
## Contract status
This document describes the current `StageRuntime`, `StagePool`, stage-client,
and stage-process layout. The unified LLM/diffusion direction in
[#5441](https://github.com/vllm-project/vllm-omni/pull/5441) remains in flight
and is not presented as current behavior.
## Ownership boundary
This document owns local and distributed stage placement, startup and
readiness, one pool per logical stage, replica identity and membership,
selection and affinity, stage-client/process lifecycle, liveness, draining,
and shutdown.
It does not own cross-stage request policy, model scheduler policy, connector
transfer semantics, configuration precedence, or public error rendering.
## Candidate invariants
These identifiers are proposals while the document is `draft`.
### STAGE-INV-001: The runtime owns replica lifecycle
**Rule:** Orchestration MUST acquire stage capacity through the stage runtime
and MUST NOT construct or retire backend replicas directly.
### STAGE-INV-100: Affinity uses stable replica identity
**Rule:** Follow-up operations for a request with replica affinity MUST resolve
the same logical replica until the request terminates or the replica is
declared lost.
### STAGE-INV-200: Shutdown is idempotent
**Rule:** Repeated shutdown or cleanup signals MUST NOT leak stage clients,
process managers, or membership registrations.
## Invariant namespace
`STAGE-INV` reserves `001-099` for placement and ownership, `100-199` for
replica identity/readiness/affinity, `200-299` for liveness loss, draining and
cleanup, and `300-399` for local/distributed and upstream compatibility.
Numbers become append-only after normative promotion.
## Safe-change guide
Exercise local and distributed startup, readiness, replica selection,
affinity, membership changes, abort routing, process failure, draining, and
repeated shutdown. Cross-stage policy changes belong in
`engine_orchestration.md`.
## Promotion gate
- Reconcile terminology and ownership after #5441 reaches a final state.
- Add or identify a dedicated StageRuntime/StagePool lifecycle suite; current
evidence is distributed across the cited tests.
- Demonstrate affinity preservation for update, interaction, and abort paths.
- Demonstrate membership loss and repeated shutdown without leaked clients or
processes.
- Obtain approval from a technical owner and an independent validation
reviewer.
-56
View File
@@ -1,56 +0,0 @@
---
title: vLLM-Omni Configuration
kind: module
status: draft
architecture_state: deferred-pending-refactor
ownership_status: provisional
owners:
- "@lishunyang12"
- "@alex-jw-brooks"
primary_code_paths:
- vllm_omni/config/**
- vllm_omni/deploy/**
- vllm_omni/model_executor/stage_configs/**
related_code_paths:
- vllm_omni/platforms/*/stage_configs/**
depends_on: []
validation_paths:
- tests/config/**
upstream_refs:
- vllm.config
last_reviewed: 2026-08-07
last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc
---
# vLLM-Omni configuration
This page intentionally records discovery paths only. Substantive
configuration ownership and contract work is deferred until the active
configuration refactoring has settled.
## Deferred contract scope
The current code and tests remain authoritative for parsing, defaults,
overrides, deployment topology, stage construction, and runtime projection.
This draft does not define a stable precedence model, environment-variable
capture rule, canonical configuration object, or invariant namespace.
The PR review proposal to capture environment-derived values in configuration
objects at initialization is an open design question, not a current
invariant.
## Safe-change guide while deferred
Changes should trace every affected producer to its runtime consumer and test
the structured, legacy, CLI, and deployment paths that are actually supported.
Do not infer a stable contract from this placeholder page.
## Promotion gate
- Finish or stabilize the configuration refactor and identify the canonical
runtime configuration representation.
- Confirm technical owners, primary path exceptions, and validation owners.
- Document parsing and override precedence, including when environment values
are captured.
- Allocate an invariant namespace only after those behaviors are enforced and
owner-approved.
@@ -0,0 +1,76 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import pytest
import torch
pytestmark = [pytest.mark.core_model, pytest.mark.cpu, pytest.mark.diffusion]
def test_scheduler_default_skips_large_finite_scans_and_preserves_results(monkeypatch):
from vllm_omni.diffusion.models.minimax_h3 import (
scheduling_minimax_h3_euler_ancestral as scheduler,
)
monkeypatch.delenv("VLLM_OMNI_H3_VALIDATE_FINITE_TENSORS", raising=False)
original_isfinite = torch.isfinite
scanned_numels = []
def record_isfinite(tensor):
scanned_numels.append(tensor.numel())
return original_isfinite(tensor)
monkeypatch.setattr(torch, "isfinite", record_isfinite)
state = torch.linspace(-1.0, 1.0, 8)
velocity = torch.linspace(0.25, -0.25, 8)
timestep = torch.tensor(0.5)
default_x0 = scheduler.minimax_h3_rf_v_to_x0(state, velocity, timestep)
default = scheduler.minimax_h3_euler_eta0_step(
state,
default_x0,
sigma_curr=0.5,
sigma_next=0.25,
)
assert scanned_numels == [1]
checked_x0 = scheduler.minimax_h3_rf_v_to_x0(
state,
velocity,
timestep,
validate_finite_tensors=True,
)
checked = scheduler.minimax_h3_euler_eta0_step(
state,
checked_x0,
sigma_curr=0.5,
sigma_next=0.25,
validate_finite_tensors=True,
)
torch.testing.assert_close(default_x0, checked_x0, rtol=0, atol=0)
torch.testing.assert_close(default, checked, rtol=0, atol=0)
@pytest.mark.parametrize("invalid_value", [float("nan"), float("inf")])
def test_scheduler_finite_gate_rejects_nonfinite_inputs(monkeypatch, invalid_value):
from vllm_omni.diffusion.models.minimax_h3 import (
scheduling_minimax_h3_euler_ancestral as scheduler,
)
monkeypatch.setenv("VLLM_OMNI_H3_VALIDATE_FINITE_TENSORS", "1")
invalid = torch.ones(8)
invalid[0] = invalid_value
with pytest.raises(ValueError, match="must be finite"):
scheduler.minimax_h3_rf_v_to_x0(
invalid,
torch.ones_like(invalid),
torch.tensor(0.5),
)
with pytest.raises(ValueError, match="must be finite"):
scheduler.minimax_h3_euler_eta0_step(
invalid,
torch.ones_like(invalid),
sigma_curr=0.5,
sigma_next=0.25,
)
@@ -20,6 +20,7 @@ from vllm_omni.diffusion.forward_context import set_forward_context_denoise_step
from .scheduling_minimax_h3_euler_ancestral import ( from .scheduling_minimax_h3_euler_ancestral import (
minimax_h3_euler_eta0_step, minimax_h3_euler_eta0_step,
minimax_h3_rf_v_to_x0, minimax_h3_rf_v_to_x0,
minimax_h3_validate_finite_tensors_enabled,
) )
MINIMAX_H3_IMGVID_COND_TIMESTEP = 0.999 MINIMAX_H3_IMGVID_COND_TIMESTEP = 0.999
@@ -198,6 +199,7 @@ def minimax_h3_denoise_loop(
if audio_anchor is not None: if audio_anchor is not None:
audio_rows[~audio_update] = audio_anchor audio_rows[~audio_update] = audio_anchor
validate_finite_tensors = minimax_h3_validate_finite_tensors_enabled()
num_steps = len(sigmas_video) - 1 num_steps = len(sigmas_video) - 1
for step in range(num_steps): for step in range(num_steps):
step_cm = step_profiler(step) if step_profiler is not None else nullcontext() step_cm = step_profiler(step) if step_profiler is not None else nullcontext()
@@ -231,8 +233,15 @@ def minimax_h3_denoise_loop(
video_rows[update], video_rows[update],
mv_video_t, mv_video_t,
torch.tensor(t_v, dtype=torch.float32, device=device), torch.tensor(t_v, dtype=torch.float32, device=device),
validate_finite_tensors=validate_finite_tensors,
)
new_target = minimax_h3_euler_eta0_step(
video_rows[update],
x0_video,
sigma_curr=s_v,
sigma_next=s_v_next,
validate_finite_tensors=validate_finite_tensors,
) )
new_target = minimax_h3_euler_eta0_step(video_rows[update], x0_video, sigma_curr=s_v, sigma_next=s_v_next)
video_rows = video_rows.clone() video_rows = video_rows.clone()
video_rows[update] = new_target video_rows[update] = new_target
if cond_anchor is not None: if cond_anchor is not None:
@@ -242,9 +251,14 @@ def minimax_h3_denoise_loop(
audio_rows[audio_update], audio_rows[audio_update],
mv_audio_t, mv_audio_t,
torch.tensor(t_a, dtype=torch.float32, device=device), torch.tensor(t_a, dtype=torch.float32, device=device),
validate_finite_tensors=validate_finite_tensors,
) )
new_audio = minimax_h3_euler_eta0_step( new_audio = minimax_h3_euler_eta0_step(
audio_rows[audio_update], x0_audio, sigma_curr=s_a, sigma_next=s_a_next audio_rows[audio_update],
x0_audio,
sigma_curr=s_a,
sigma_next=s_a_next,
validate_finite_tensors=validate_finite_tensors,
) )
audio_rows = audio_rows.clone() audio_rows = audio_rows.clone()
audio_rows[audio_update] = new_audio audio_rows[audio_update] = new_audio
@@ -2,10 +2,24 @@
from __future__ import annotations from __future__ import annotations
import math import math
import os
from typing import Any from typing import Any
import torch import torch
MINIMAX_H3_VALIDATE_FINITE_TENSORS_ENV = "VLLM_OMNI_H3_VALIDATE_FINITE_TENSORS"
def minimax_h3_validate_finite_tensors_enabled() -> bool:
value = os.getenv(MINIMAX_H3_VALIDATE_FINITE_TENSORS_ENV, "")
return value.strip().lower() in {"1", "true", "yes", "on"}
def _resolve_validate_finite_tensors(value: bool | None) -> bool:
if value is None:
return minimax_h3_validate_finite_tensors_enabled()
return bool(value)
def _require_finite_tensor(tensor: torch.Tensor, name: str) -> None: def _require_finite_tensor(tensor: torch.Tensor, name: str) -> None:
if not bool(torch.isfinite(tensor).all().item()): if not bool(torch.isfinite(tensor).all().item()):
@@ -50,6 +64,8 @@ def minimax_h3_rf_v_to_x0(
xt: torch.Tensor, xt: torch.Tensor,
v: torch.Tensor, v: torch.Tensor,
timestep: torch.Tensor, timestep: torch.Tensor,
*,
validate_finite_tensors: bool | None = None,
) -> torch.Tensor: ) -> torch.Tensor:
if xt.shape != v.shape: if xt.shape != v.shape:
raise ValueError(f"xt and v shapes must match, got {xt.shape} vs {v.shape}") raise ValueError(f"xt and v shapes must match, got {xt.shape} vs {v.shape}")
@@ -57,15 +73,18 @@ def minimax_h3_rf_v_to_x0(
raise ValueError("xt must be a floating point tensor") raise ValueError("xt must be a floating point tensor")
if not torch.is_floating_point(v): if not torch.is_floating_point(v):
raise ValueError("v must be a floating point tensor") raise ValueError("v must be a floating point tensor")
_require_finite_tensor(xt, "xt") validate_finite_tensors = _resolve_validate_finite_tensors(validate_finite_tensors)
_require_finite_tensor(v, "v") if validate_finite_tensors:
_require_finite_tensor(xt, "xt")
_require_finite_tensor(v, "v")
_validate_unit_timestep(timestep, "timestep") _validate_unit_timestep(timestep, "timestep")
cond_t = timestep.to(device=xt.device, dtype=xt.dtype) cond_t = timestep.to(device=xt.device, dtype=xt.dtype)
while cond_t.ndim < xt.ndim: while cond_t.ndim < xt.ndim:
cond_t = cond_t.unsqueeze(-1) cond_t = cond_t.unsqueeze(-1)
sigma_t = 1 - cond_t sigma_t = 1 - cond_t
x0 = xt + sigma_t * v x0 = xt + sigma_t * v
_require_finite_tensor(x0, "x0") if validate_finite_tensors:
_require_finite_tensor(x0, "x0")
return x0 return x0
@@ -75,6 +94,7 @@ def minimax_h3_euler_eta0_step(
*, *,
sigma_curr: float, sigma_curr: float,
sigma_next: float, sigma_next: float,
validate_finite_tensors: bool | None = None,
) -> torch.Tensor: ) -> torch.Tensor:
if state.shape != denoised.shape: if state.shape != denoised.shape:
raise ValueError(f"state and denoised shapes must match, got {state.shape} vs {denoised.shape}") raise ValueError(f"state and denoised shapes must match, got {state.shape} vs {denoised.shape}")
@@ -82,8 +102,10 @@ def minimax_h3_euler_eta0_step(
raise ValueError("state must be a floating point tensor") raise ValueError("state must be a floating point tensor")
if not torch.is_floating_point(denoised): if not torch.is_floating_point(denoised):
raise ValueError("denoised must be a floating point tensor") raise ValueError("denoised must be a floating point tensor")
_require_finite_tensor(state, "state") validate_finite_tensors = _resolve_validate_finite_tensors(validate_finite_tensors)
_require_finite_tensor(denoised, "denoised") if validate_finite_tensors:
_require_finite_tensor(state, "state")
_require_finite_tensor(denoised, "denoised")
sigma_curr = _validate_sigma(sigma_curr, "sigma_curr") sigma_curr = _validate_sigma(sigma_curr, "sigma_curr")
sigma_next = _validate_sigma(sigma_next, "sigma_next") sigma_next = _validate_sigma(sigma_next, "sigma_next")
if sigma_curr == 0.0: if sigma_curr == 0.0:
@@ -98,7 +120,8 @@ def minimax_h3_euler_eta0_step(
sigma_ratio = sigma_next_t / sigma_curr_t sigma_ratio = sigma_next_t / sigma_curr_t
out = sigma_ratio * state.to(dtype=compute_dtype) + (1.0 - sigma_ratio) * denoised.to(dtype=compute_dtype) out = sigma_ratio * state.to(dtype=compute_dtype) + (1.0 - sigma_ratio) * denoised.to(dtype=compute_dtype)
out = out.to(dtype=state.dtype) out = out.to(dtype=state.dtype)
_require_finite_tensor(out, "euler_eta0_step output") if validate_finite_tensors:
_require_finite_tensor(out, "euler_eta0_step output")
return out return out
@@ -127,6 +150,7 @@ class MiniMaxH3EulerAncestralEta0SchedulerAdapter:
audio_sigma_curr: float | None = None, audio_sigma_curr: float | None = None,
audio_sigma_next: float | None = None, audio_sigma_next: float | None = None,
) -> dict[str, torch.Tensor]: ) -> dict[str, torch.Tensor]:
validate_finite_tensors = minimax_h3_validate_finite_tensors_enabled()
visual_timestep = timestep if video_timestep is None else video_timestep visual_timestep = timestep if video_timestep is None else video_timestep
audio_timestep = timestep if audio_timestep is None else audio_timestep audio_timestep = timestep if audio_timestep is None else audio_timestep
visual_sigma_curr = sigma_curr if video_sigma_curr is None else video_sigma_curr visual_sigma_curr = sigma_curr if video_sigma_curr is None else video_sigma_curr
@@ -148,11 +172,13 @@ class MiniMaxH3EulerAncestralEta0SchedulerAdapter:
input_visual_latent, input_visual_latent,
noise_pred_visual, noise_pred_visual,
visual_timestep, visual_timestep,
validate_finite_tensors=validate_finite_tensors,
) )
denoised_audio = minimax_h3_rf_v_to_x0( denoised_audio = minimax_h3_rf_v_to_x0(
input_audio_latent, input_audio_latent,
noise_pred_audio, noise_pred_audio,
audio_timestep, audio_timestep,
validate_finite_tensors=validate_finite_tensors,
) )
return { return {
"output_visual_latent": minimax_h3_euler_eta0_step( "output_visual_latent": minimax_h3_euler_eta0_step(
@@ -160,12 +186,14 @@ class MiniMaxH3EulerAncestralEta0SchedulerAdapter:
denoised_visual, denoised_visual,
sigma_curr=visual_sigma_curr, sigma_curr=visual_sigma_curr,
sigma_next=visual_sigma_next, sigma_next=visual_sigma_next,
validate_finite_tensors=validate_finite_tensors,
), ),
"output_audio_latent": minimax_h3_euler_eta0_step( "output_audio_latent": minimax_h3_euler_eta0_step(
input_audio_latent, input_audio_latent,
denoised_audio, denoised_audio,
sigma_curr=audio_sigma_curr, sigma_curr=audio_sigma_curr,
sigma_next=audio_sigma_next, sigma_next=audio_sigma_next,
validate_finite_tensors=validate_finite_tensors,
), ),
} }
@@ -173,7 +201,9 @@ class MiniMaxH3EulerAncestralEta0SchedulerAdapter:
EntryClass = MiniMaxH3EulerAncestralEta0SchedulerAdapter EntryClass = MiniMaxH3EulerAncestralEta0SchedulerAdapter
__all__ = [ __all__ = [
"MINIMAX_H3_VALIDATE_FINITE_TENSORS_ENV",
"MiniMaxH3EulerAncestralEta0SchedulerAdapter", "MiniMaxH3EulerAncestralEta0SchedulerAdapter",
"minimax_h3_euler_eta0_step", "minimax_h3_euler_eta0_step",
"minimax_h3_rf_v_to_x0", "minimax_h3_rf_v_to_x0",
"minimax_h3_validate_finite_tensors_enabled",
] ]