diff --git a/docs/.nav.yml b/docs/.nav.yml index ca17d550d..fc11bb838 100644 --- a/docs/.nav.yml +++ b/docs/.nav.yml @@ -138,12 +138,28 @@ nav: - Prometheus Metrics: design/metrics.md - Speech Generation Performance Optimizations: design/qwen3_omni_tts_performance_optimization.md - Module Design: - - Runtime Modules: - - AR Module: design/module/ar_module.md - - DIT Module: design/module/dit_module.md - - Orchestration: - - Entrypoint Module: design/module/entrypoint_module.md - - AsyncOmni Architecture: design/module/async_omni_architecture.md + - Entrypoints: design/module/entrypoints.md + - vLLM-Omni Configuration: design/module/vllm_omni_config.md + - Input, Output, and Modality Contracts: design/module/input_output_modality_contracts.md + - Error Contracts: design/module/error_contracts.md + - Engine Orchestration: design/module/engine_orchestration.md + - Stage Runtime: design/module/stage_runtime.md + - OmniConnector: design/module/omni_connector.md + - Model Integration: design/module/model_integration.md + - Autoregressive Runtime: design/module/ar_runtime.md + - Diffusion: + - Overview: design/module/diffusion/index.md + - Runtime: design/module/diffusion/diffusion_runtime.md + - Model Integration: design/module/diffusion/diffusion_model_integration.md + - Continuous Batching: design/module/diffusion/continuous_batching.md + - Parallelism: design/module/diffusion/parallelism.md + - Offloader: design/module/diffusion/offloader.md + - Execution Platforms: design/module/execution_platforms.md + - Cache Management: design/module/cache_management.md + - Quantization: design/module/quantization.md + - Observability: design/module/observability.md + - Profiling: design/module/profiling.md + - Benchmarking: design/module/benchmarking.md - Docs Guide: contributing/DOCS_GUIDE.md - API Reference: - api/README.md diff --git a/docs/design/index.md b/docs/design/index.md index 5434b27fb..c98af378d 100644 --- a/docs/design/index.md +++ b/docs/design/index.md @@ -1,8 +1,9 @@ # Design Documents This section contains design documents and architecture specifications for -vLLM-Omni. The sidebar groups documents by the system concern they describe, -while the source paths remain stable so existing links continue to work. +vLLM-Omni. The sidebar groups documents by the system concern they describe; +retired module pages are preserved in the legacy archive instead of remaining +in the active navigation. ## Architecture Documents @@ -61,12 +62,29 @@ while the source paths remain stable so existing links continue to work. ## Module Design Documents -### Runtime Modules +- [Entrypoints and Serving Boundaries](module/entrypoints.md) +- [vLLM-Omni Configuration](module/vllm_omni_config.md) +- [Input, Output, and Modality Contracts](module/input_output_modality_contracts.md) +- [Error Classification, Propagation, and Rendering](module/error_contracts.md) +- [Engine Orchestration](module/engine_orchestration.md) +- [Stage Runtime and Replica Lifecycle](module/stage_runtime.md) +- [OmniConnector](module/omni_connector.md) +- [Model Integration](module/model_integration.md) +- [Autoregressive Runtime](module/ar_runtime.md) +- Diffusion + - [Overview](module/diffusion/index.md) + - [Runtime](module/diffusion/diffusion_runtime.md) + - [Model Integration](module/diffusion/diffusion_model_integration.md) + - [Continuous Batching](module/diffusion/continuous_batching.md) + - [Parallelism](module/diffusion/parallelism.md) + - [Offloader](module/diffusion/offloader.md) +- [Execution Platforms](module/execution_platforms.md) +- [Cache Management](module/cache_management.md) +- [Quantization](module/quantization.md) +- [Observability](module/observability.md) +- [Profiling](module/profiling.md) +- [Benchmarking](module/benchmarking.md) -- [AR Module](module/ar_module.md) -- [DIT Module](module/dit_module.md) - -### Orchestration - -- [Entrypoint Module](module/entrypoint_module.md) -- [AsyncOmni Architecture (Qwen3-Omni Example)](module/async_omni_architecture.md) +The pre-#5137 pages are preserved in the +[legacy module archive](module/archive/README.md) for historical reference and +are not active design contracts. diff --git a/docs/design/module/ar_runtime.md b/docs/design/module/ar_runtime.md new file mode 100644 index 000000000..194aade5f --- /dev/null +++ b/docs/design/module/ar_runtime.md @@ -0,0 +1,52 @@ +--- +title: Autoregressive Runtime +kind: module +status: draft +owners: + - "@tzhouam" + - "@yinpeiqi" + - "@fake0fan" + - "@Sy0307" +primary_code_paths: + - vllm_omni/core/** + - vllm_omni/worker/** +related_code_paths: + - vllm_omni/model_executor/** +depends_on: + - engine_orchestration.md + - model_integration.md + - input_output_modality_contracts.md +validation_paths: + - tests/core/** + - tests/worker/** +upstream_refs: + - vllm.v1.core + - vllm.v1.worker +last_reviewed: 2026-07-16 +--- + +# Autoregressive runtime + +The AR runtime extends vLLM scheduling and worker execution for omni-stage +inputs and outputs while preserving vLLM scheduling and cache semantics. + +## Candidate invariants + +### AR-INV-001: vLLM owns base scheduling semantics + +**Rule:** Omni schedulers MUST preserve upstream request-state and cache +transitions unless an Omni-specific difference is documented and tested. + +### AR-INV-002: Omni data crosses explicit adapters + +**Rule:** Modality-specific stage data MUST be converted at an input or output +adapter, not injected through unrelated scheduler state. + +### AR-INV-003: Workers execute assigned work + +**Rule:** Workers and model runners MUST NOT implement cross-stage routing. + +## Safe-change guide + +Test request lifecycle, abort, cache state, and every affected worker execution +mode against the supported upstream vLLM contract. diff --git a/docs/design/module/archive/README.md b/docs/design/module/archive/README.md new file mode 100644 index 000000000..6afc76947 --- /dev/null +++ b/docs/design/module/archive/README.md @@ -0,0 +1,18 @@ +# Archived module design pages + +This directory preserves module-design pages that predate the structure +proposed in [issue #5137](https://github.com/vllm-project/vllm-omni/issues/5137). +They are historical snapshots, not active architectural contracts, and are +intentionally excluded from the documentation navigation. + +| Archived page | Active destination | +| --- | --- | +| [AR Module](ar_module.md) | [Autoregressive Runtime](../ar_runtime.md) | +| [AsyncOmni Architecture](async_omni_architecture.md) | [Engine Orchestration](../engine_orchestration.md) and [Stage Runtime](../stage_runtime.md) | +| [DiT Module](dit_module.md) | [Diffusion module overview](../diffusion/index.md) | +| [Entrypoint Module](entrypoint_module.md) | [Entrypoints and Serving Boundaries](../entrypoints.md) | + +Do not update these pages with current design guidance. Move still-useful +material into the active destination and review it against current code and +tests instead. No compatibility redirect pages are provided at the former +paths. diff --git a/docs/design/module/ar_module.md b/docs/design/module/archive/ar_module.md similarity index 98% rename from docs/design/module/ar_module.md rename to docs/design/module/archive/ar_module.md index 5e0aa5b07..f0085fa42 100644 --- a/docs/design/module/ar_module.md +++ b/docs/design/module/archive/ar_module.md @@ -12,7 +12,7 @@ The AR module of vLLM-Omni extends vLLM's core components to support: - **Hidden state exposure**: Exposing per-request hidden representations for downstream stages - **Basic generator support**: Support some basic heterogeneous architecture such as Convolution, LSTM, etc. -As shown in the [end2end example](../../user_guide/examples/offline_inference/qwen3_omni.md), AR module can be widely applied across multiple stages, generating text tokens in thinker(AR), audio latent tokens in talker(AR) and audio wave in code2wav(Convolution). +As shown in the [end2end example](../../../user_guide/examples/offline_inference/qwen3_omni.md), AR module can be widely applied across multiple stages, generating text tokens in thinker(AR), audio latent tokens in talker(AR) and audio wave in code2wav(Convolution). ## 2. Relationship with vLLM diff --git a/docs/design/module/async_omni_architecture.md b/docs/design/module/archive/async_omni_architecture.md similarity index 100% rename from docs/design/module/async_omni_architecture.md rename to docs/design/module/archive/async_omni_architecture.md diff --git a/docs/design/module/dit_module.md b/docs/design/module/archive/dit_module.md similarity index 99% rename from docs/design/module/dit_module.md rename to docs/design/module/archive/dit_module.md index 36fd16877..f505d1e67 100644 --- a/docs/design/module/dit_module.md +++ b/docs/design/module/archive/dit_module.md @@ -252,7 +252,7 @@ while True: batch of compatible independent requests for one full pipeline forward when the pipeline supports request-level batching; the step-wise path can also admit or remove compatible requests between denoise steps. See - [Diffusion Continuous Batching](../feature/diffusion_continuous_batching.md). + [Diffusion Continuous Batching](../../feature/diffusion_continuous_batching.md). --- @@ -541,7 +541,7 @@ def get_attn_backend_for_role(role, head_size, attention_config=None, role_categ A backend that needs configuration exposes it as a typed field on `AttentionSpec`; `AttentionSpec.backend_kwargs()` serializes it into the `backend_kwargs` the impl consumes. Today those fields are `skip_softmax` (TRTLLM_ATTN) and `block_sparse` (any block-sparse backend, currently RAINFUSION_ATTN), so e.g. `target_sparsity` or `sparsity` can be set without changing model code. -For the user-facing CLI surface, see [Diffusion Attention Backends](../../user_guide/diffusion/attention_backends.md). For the role declaration contract on the model side, see [Adding a Diffusion Model](../../contributing/model/adding_diffusion_model.md). +For the user-facing CLI surface, see [Diffusion Attention Backends](../../../user_guide/diffusion/attention_backends.md). For the role declaration contract on the model side, see [Adding a Diffusion Model](../../../contributing/model/adding_diffusion_model.md). **Backend Availability**: diff --git a/docs/design/module/entrypoint_module.md b/docs/design/module/archive/entrypoint_module.md similarity index 100% rename from docs/design/module/entrypoint_module.md rename to docs/design/module/archive/entrypoint_module.md diff --git a/docs/design/module/benchmarking.md b/docs/design/module/benchmarking.md new file mode 100644 index 000000000..93eba2c72 --- /dev/null +++ b/docs/design/module/benchmarking.md @@ -0,0 +1,47 @@ +--- +title: Benchmarking +kind: module +status: draft +owners: + - "@alex-jw-brooks" +primary_code_paths: + - vllm_omni/benchmarks/** + - vllm_omni/entrypoints/cli/benchmark/** +related_code_paths: + - benchmarks/** +depends_on: + - entrypoints.md + - profiling.md +validation_paths: + - tests/benchmarks/** + - tests/dfx/perf/** +upstream_refs: + - vllm.benchmarks +last_reviewed: 2026-07-16 +--- + +# Benchmarking + +Benchmarking defines reproducible workloads, metrics, warm-up, measurement, +and result metadata for comparing vLLM-Omni performance. + +## Candidate invariants + +### BENCH-INV-001: Results are reproducible + +**Rule:** Results MUST identify the model, revision, commit, hardware, +configuration, workload, concurrency, warm-up, and measurement window. + +### BENCH-INV-002: Correctness precedes performance + +**Rule:** A benchmark MUST validate successful and semantically acceptable +outputs before using timings as performance evidence. + +### BENCH-INV-003: Metric definitions are explicit + +**Rule:** Metrics MUST state their unit, population, and aggregation. + +## Safe-change guide + +Test workload parsing and metric calculations with deterministic fixtures. Do +not compare materially different workloads or runtime settings. diff --git a/docs/design/module/cache_management.md b/docs/design/module/cache_management.md new file mode 100644 index 000000000..6f22cf919 --- /dev/null +++ b/docs/design/module/cache_management.md @@ -0,0 +1,47 @@ +--- +title: Cache Management +kind: module +status: draft +owners: + - "@Isotr0py" + - "@princepride" +primary_code_paths: + - vllm_omni/diffusion/cache/** +related_code_paths: + - vllm_omni/experimental/ar_diffusion/kv_cache/** + - vllm_omni/core/** +depends_on: + - ar_runtime.md + - diffusion/diffusion_runtime.md +validation_paths: + - tests/diffusion/cache/** +upstream_refs: + - vllm.v1.core.kv_cache_manager +last_reviewed: 2026-07-16 +--- + +# Cache management + +Cache management defines reusable state, cache identity, validity, lifecycle, +and eviction across AR and diffusion execution. + +## Candidate invariants + +### CACHE-INV-001: Cache identity is complete + +**Rule:** A cache key MUST include every input and execution property that can +change the reused value. + +### CACHE-INV-002: Reuse preserves correctness + +**Rule:** A cache optimization MUST provide a disabled path and MUST NOT reuse +state after its validity conditions change. + +### CACHE-INV-003: Ownership has an end + +**Rule:** Request-scoped and model-scoped cache entries MUST have explicit +reset, eviction, or teardown behavior. + +## Safe-change guide + +Test hit, miss, invalidation, disabled, concurrent-request, and teardown paths. diff --git a/docs/design/module/diffusion/continuous_batching.md b/docs/design/module/diffusion/continuous_batching.md new file mode 100644 index 000000000..bffe24309 --- /dev/null +++ b/docs/design/module/diffusion/continuous_batching.md @@ -0,0 +1,51 @@ +--- +title: Diffusion Continuous Batching +kind: module +status: draft +owners: + - "@Isotr0py" + - "@princepride" +primary_code_paths: + - vllm_omni/diffusion/sched/** +related_code_paths: + - vllm_omni/diffusion/executor/** +depends_on: + - diffusion_runtime.md +validation_paths: + - tests/diffusion/batching/** +upstream_refs: [] +last_reviewed: 2026-07-16 +--- + +# Diffusion continuous batching + +Continuous batching defines when diffusion requests are compatible, how they +share an execution step, and how each request advances independently. + +## Candidate invariants + +### BATCH-INV-001: Compatibility is explicit + +**Rule:** Requests MAY share a batch only when all properties required by the +selected pipeline, scheduler step, shape, precision, and execution features are +compatible. + +### BATCH-INV-002: Per-request state remains isolated + +**Rule:** Batch assembly MUST preserve each request's random generator, progress, +conditioning, outputs, cancellation, and errors. + +### BATCH-INV-003: Admission is bounded + +**Rule:** Admission MUST respect configured capacity and MUST NOT rely on an +unbounded waiting or active-request collection. + +### BATCH-INV-004: Batch order is not completion order + +**Rule:** Output association MUST use stable request identity rather than batch +position across execution steps. + +## Safe-change guide + +Test heterogeneous requests, partial completion, cancellation, deterministic +seeds, capacity limits, and batch sizes of one and many. diff --git a/docs/design/module/diffusion/diffusion_model_integration.md b/docs/design/module/diffusion/diffusion_model_integration.md new file mode 100644 index 000000000..3673db9c1 --- /dev/null +++ b/docs/design/module/diffusion/diffusion_model_integration.md @@ -0,0 +1,58 @@ +--- +title: Diffusion Model Integration +kind: module +status: draft +owners: + - "@Isotr0py" + - "@wtomin" + - "@ZJY0516" +primary_code_paths: + - vllm_omni/diffusion/models/** + - vllm_omni/diffusion/model_loader/** +related_code_paths: + - vllm_omni/diffusion/layers/** + - vllm_omni/diffusion/lora/** + - vllm_omni/diffusion/utils/** +depends_on: + - diffusion_runtime.md + - ../input_output_modality_contracts.md +validation_paths: + - tests/diffusion/models/** + - tests/diffusion/model_loader/** + - tests/diffusion/layers/** + - tests/diffusion/lora/** +upstream_refs: + - diffusers.DiffusionPipeline +last_reviewed: 2026-07-16 +--- + +# Diffusion model integration + +Diffusion model integration owns pipeline contracts, registration, checkpoint +loading, adapters, shared layers, and model-specific processing. + +## Candidate invariants + +### DIFF-MODEL-INV-001: Pipelines implement one runtime contract + +**Rule:** A pipeline MUST declare its supported modalities, configuration, +inputs, outputs, loading path, and runtime capabilities. + +### DIFF-MODEL-INV-002: Registration is the selection boundary + +**Rule:** Runtime code MUST select model implementations through the registry or +loader contract rather than scattered model-name conditionals. + +### DIFF-MODEL-INV-003: Model code does not schedule requests + +**Rule:** Pipeline code MUST NOT own admission, batching, cancellation, or +cross-stage routing. + +### DIFF-MODEL-INV-004: Shared behavior stays shared + +**Rule:** Model directories SHOULD contain only genuine model differences. + +## Safe-change guide + +Test registry selection, checkpoint loading, minimal inference, input and output +contracts, and every declared optional capability. diff --git a/docs/design/module/diffusion/diffusion_runtime.md b/docs/design/module/diffusion/diffusion_runtime.md new file mode 100644 index 000000000..9925ed487 --- /dev/null +++ b/docs/design/module/diffusion/diffusion_runtime.md @@ -0,0 +1,58 @@ +--- +title: Diffusion Runtime +kind: module +status: draft +owners: + - "@Isotr0py" + - "@princepride" + - "@SamitHuang" +primary_code_paths: + - vllm_omni/diffusion/sched/** + - vllm_omni/diffusion/executor/** + - vllm_omni/diffusion/worker/** +related_code_paths: + - vllm_omni/diffusion/hooks/** + - vllm_omni/diffusion/postprocess/** +depends_on: + - ../engine_orchestration.md + - ../input_output_modality_contracts.md + - diffusion_model_integration.md +validation_paths: + - tests/diffusion/diffusion_backend/** + - tests/diffusion/hooks/** +upstream_refs: + - diffusers.DiffusionPipeline +last_reviewed: 2026-07-16 +--- + +# Diffusion runtime + +The diffusion runtime owns request admission, scheduling, execution, progress, +output, cancellation, and cleanup inside a diffusion stage. + +## Candidate invariants + +### DIFF-RUNTIME-INV-001: One lifecycle owner + +**Rule:** Every admitted request MUST have exactly one scheduler-owned lifecycle +until completion, cancellation, or failure. + +### DIFF-RUNTIME-INV-002: Execution follows scheduler output + +**Rule:** Executors and workers MUST execute scheduler decisions without +admitting, reordering, or forwarding requests independently. + +### DIFF-RUNTIME-INV-003: Terminal cleanup is complete + +**Rule:** Every terminal path MUST release request state, temporary tensors, +hooks, and runtime-owned resources. + +### DIFF-RUNTIME-INV-004: Optional features use runtime hooks + +**Rule:** Cache, profiling, offload, and parallel features SHOULD integrate at +defined hooks instead of creating another request lifecycle. + +## Safe-change guide + +Test success, cancellation, failure, shutdown, multiple requests, and resource +cleanup. diff --git a/docs/design/module/diffusion/index.md b/docs/design/module/diffusion/index.md new file mode 100644 index 000000000..51d66eda9 --- /dev/null +++ b/docs/design/module/diffusion/index.md @@ -0,0 +1,42 @@ +--- +title: Diffusion Modules +kind: module-index +status: draft +owners: + - "@Isotr0py" + - "@princepride" + - "@SamitHuang" + - "@wtomin" + - "@ZJY0516" + - "@RuixiangMa" + - "@david6666666" + - "@xuechendi" +primary_code_paths: + - vllm_omni/diffusion/** +related_code_paths: + - vllm_omni/platforms/** +depends_on: + - ../engine_orchestration.md + - ../input_output_modality_contracts.md +validation_paths: + - tests/diffusion/** +upstream_refs: + - diffusers +last_reviewed: 2026-07-16 +--- + +# Diffusion modules + +Diffusion is a module family containing runtime, model integration, batching, +distributed execution, and memory-management subsystems. + +## Module documents + +- [Diffusion runtime](diffusion_runtime.md) +- [Diffusion model integration](diffusion_model_integration.md) +- [Continuous batching](continuous_batching.md) +- [Parallelism](parallelism.md) +- [Offloader](offloader.md) + +Cache, quantization, profiling, and benchmarking follow their top-level module +documents. diff --git a/docs/design/module/diffusion/offloader.md b/docs/design/module/diffusion/offloader.md new file mode 100644 index 000000000..52e634a2f --- /dev/null +++ b/docs/design/module/diffusion/offloader.md @@ -0,0 +1,53 @@ +--- +title: Diffusion Offloader +kind: module +status: draft +owners: + - "@yuanheng-zhao" + - "@Isotr0py" +primary_code_paths: + - vllm_omni/diffusion/offloader/** +related_code_paths: + - vllm_omni/diffusion/models/** + - vllm_omni/diffusion/worker/** +depends_on: + - diffusion_runtime.md + - ../execution_platforms.md +validation_paths: + - tests/diffusion/offloader/** +upstream_refs: + - torch.nn.Module.to +last_reviewed: 2026-07-16 +--- + +# Diffusion offloader + +The offloader owns component residency, transfer scheduling, memory accounting, +prefetch, and teardown for diffusion execution. + +## Candidate invariants + +### OFFLOAD-INV-001: Residency is explicit + +**Rule:** Each managed component MUST have one known residency state and one +owner responsible for transitions. + +### OFFLOAD-INV-002: Use follows readiness + +**Rule:** Runtime execution MUST NOT consume a component until its transfer to +the required device is complete. + +### OFFLOAD-INV-003: Transfers preserve model state + +**Rule:** Offloading MUST preserve parameter and buffer identity, dtype, device, +and correctness required by the selected execution mode. + +### OFFLOAD-INV-004: Memory is bounded and released + +**Rule:** Retained host and device copies MUST respect configured limits and +provide deterministic teardown. + +## Safe-change guide + +Test enabled and disabled paths, repeated execution, asynchronous transfer, +memory limits, failure, teardown, and numerical equivalence. diff --git a/docs/design/module/diffusion/parallelism.md b/docs/design/module/diffusion/parallelism.md new file mode 100644 index 000000000..39034d5d3 --- /dev/null +++ b/docs/design/module/diffusion/parallelism.md @@ -0,0 +1,56 @@ +--- +title: Diffusion Parallelism +kind: module +status: draft +owners: + - "@princepride" + - "@yuanheng-zhao" + - "@xuechendi" +primary_code_paths: + - vllm_omni/diffusion/distributed/** + - vllm_omni/diffusion/attention/parallel/** +related_code_paths: + - vllm_omni/distributed/** + - vllm_omni/config/composable_parallel/** +depends_on: + - diffusion_runtime.md + - ../vllm_omni_config.md +validation_paths: + - tests/diffusion/distributed/** + - tests/diffusion/attention/** +upstream_refs: + - torch.distributed +last_reviewed: 2026-07-16 +--- + +# Diffusion parallelism + +Parallelism owns diffusion rank topology, process groups, tensor and sequence +sharding, collectives, and distributed execution. + +## Candidate invariants + +### PARALLEL-INV-001: Topology has one source of truth + +**Rule:** Rank coordinates, group membership, and parallel dimensions MUST be +derived from validated configuration. + +### PARALLEL-INV-002: Sharding contracts are explicit + +**Rule:** Every distributed boundary MUST define tensor shape, sharded +dimension, placement, dtype, producer, and consumer. + +### PARALLEL-INV-003: Collectives are symmetric + +**Rule:** Process-group members MUST invoke compatible collectives in the same +logical order. + +### PARALLEL-INV-004: Single-rank behavior remains valid + +**Rule:** Distributed implementations SHOULD preserve a supported single-rank +path without distributed initialization. + +## Safe-change guide + +Test topology validation, single-rank execution, affected parallel modes, +combined modes, invalid shapes, and orderly teardown. diff --git a/docs/design/module/engine_orchestration.md b/docs/design/module/engine_orchestration.md new file mode 100644 index 000000000..5bb0ea0df --- /dev/null +++ b/docs/design/module/engine_orchestration.md @@ -0,0 +1,122 @@ +--- +title: Engine Orchestration +kind: module +status: draft +architecture_state: current-plus-in-flight +owners: + - "@tzhouam" + - "@fake0fan" +document_stewards: + - "@hsliuustc0106" + - "@Gaohan123" + - "@david6666666" +required_reviewers: + - "@yinpeiqi" + - "@Sy0307" + - "@yenuo26" + - "@NickCao" +primary_code_paths: + - vllm_omni/engine/async_omni_engine.py + - vllm_omni/engine/async_engine_utils.py + - vllm_omni/engine/orchestrator.py + - vllm_omni/engine/cfg_companion_tracker.py + - vllm_omni/engine/rpc_result_router.py +related_code_paths: + - vllm_omni/engine/messages.py + - vllm_omni/engine/orchestrator_monitor.py + - vllm_omni/engine/stage_client.py + - vllm_omni/engine/stage_pool.py + - vllm_omni/outputs/output_processor.py + - vllm_omni/distributed/omni_coordinator/** +depends_on: + - input_output_modality_contracts.md + - error_contracts.md + - stage_runtime.md + - omni_connector.md +validation_paths: + - tests/engine/test_async_omni_engine_input.py + - tests/engine/test_async_omni_engine_outputs.py + - tests/engine/test_async_omni_engine_abort.py + - tests/engine/test_async_omni_engine_stage_init.py + - tests/engine/test_orchestrator.py + - tests/engine/test_orchestrator_error_handling.py + - tests/engine/test_orchestrator_stage_input_bridge.py + - tests/engine/test_cfg_companion_lifecycle.py + - tests/engine/test_rpc_result_router.py + - tests/e2e/features/fullduplex/engine/** +upstream_refs: + - vllm.v1.engine.EngineCoreRequest + - vllm.v1.engine.EngineCoreOutputs + - vllm.v1.engine.exceptions.EngineDeadError + - vllm.outputs.RequestOutput +invariant_namespace: ORCH-INV +last_reviewed: 2026-08-07 +last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc +--- + +# Engine orchestration + +Engine orchestration coordinates configured AR and diffusion stages while +keeping public clients independent from stage processes and transports. + +## Contract status + +This document is a draft description of current behavior. It also identifies +the boundary affected by the in-flight stage client/process refactor in +[#5441](https://github.com/vllm-project/vllm-omni/pull/5441). Names and +responsibilities proposed only by that PR are not current contracts. + +## Ownership boundary + +This document owns `AsyncOmniEngine`, `Orchestrator`, request-state creation, +cross-stage routing, output ordering, companion tracking, control/RPC +correlation, cancellation propagation, and terminal-state convergence. + +It does not own stage placement, replica selection, stage-process startup, +payload schemas, public protocol rendering, connector transport, or semantic +error classification. Those responsibilities belong to the stage runtime, +I/O, entrypoint, connector, and error contracts. + +## Candidate invariants + +These identifiers are proposals while the document is `draft`. + +### ORCH-INV-001: The orchestrator owns cross-stage routing + +**Rule:** Entrypoints and stage clients MUST NOT independently forward a +request to a downstream logical stage. + +### ORCH-INV-002: Stage clients do not own routing policy + +**Rule:** Stage clients MUST implement communication and lifecycle operations +without selecting the next logical stage. + +### ORCH-INV-100: Terminal state is monotonic + +**Rule:** Once a request reaches a terminal state, orchestration MUST NOT +forward new work for that request. + +## Invariant namespace + +`ORCH-INV` reserves `001-099` for dependency direction, `100-199` for +request state and ordering, `200-299` for failure/cancellation/cleanup, and +`300-399` for extension points, shutdown ordering, and upstream alignment. +Numbers become append-only after normative promotion. + +## Safe-change guide + +Test routing, output ordering, queue correlation, cancellation, failure +propagation, shutdown ordering, and representative multi-stage execution. +Changes that cross into stage runtime or public error behavior require review +from that contract's owners. + +## Promotion gate + +- Reconcile current names and boundaries after #5441 merges, closes, or is + replaced. +- Demonstrate one terminal outcome per request across success, abort, + request-scoped failure, and fatal engine failure. +- Verify queue correlation and orchestrator-before-runtime shutdown ordering + in the cited validation paths. +- Obtain approval from a technical owner and an independent validation + reviewer. diff --git a/docs/design/module/entrypoints.md b/docs/design/module/entrypoints.md new file mode 100644 index 000000000..80b576f96 --- /dev/null +++ b/docs/design/module/entrypoints.md @@ -0,0 +1,134 @@ +--- +title: Entrypoints and Serving Boundaries +kind: module +status: draft +architecture_state: current-plus-in-flight +owners: + - "@alex-jw-brooks" + - "@linyueqian" +document_stewards: + - "@hsliuustc0106" + - "@Gaohan123" + - "@david6666666" +required_reviewers: + - "@tzhouam" + - "@herotai214" + - "@yenuo26" + - "@NickCao" +primary_code_paths: + - vllm_omni/entrypoints/omni.py + - vllm_omni/entrypoints/async_omni.py + - vllm_omni/entrypoints/omni_base.py + - vllm_omni/entrypoints/cli/** + - vllm_omni/entrypoints/openai/** + - vllm_omni/entrypoints/openpi/** + - vllm_omni/entrypoints/client_request_state.py + - vllm_omni/entrypoints/stage_utils.py + - vllm_omni/entrypoints/utils.py +primary_path_exceptions: + - path: vllm_omni/entrypoints/openai/errors.py + owner: error_contracts.md +related_code_paths: + - vllm_omni/errors.py + - vllm_omni/inputs/** + - vllm_omni/outputs/** + - vllm_omni/engine/async_omni_engine.py + - vllm_omni/config/** + - vllm_omni/deploy/** +depends_on: + - input_output_modality_contracts.md + - error_contracts.md + - engine_orchestration.md +validation_paths: + - tests/entrypoints/test_omni_entrypoints.py + - tests/entrypoints/test_async_omni.py + - tests/entrypoints/test_async_omni_duplex.py + - tests/entrypoints/test_serve.py + - tests/entrypoints/test_stream_finish_reason.py + - tests/entrypoints/openai/** + - tests/entrypoints/openai_api/** + - tests/e2e/online_serving/** +upstream_refs: + - vllm.engine.protocol.EngineClient + - vllm.entrypoints.openai/** + - vllm.entrypoints.serve/** + - vllm.renderers/** + - vllm.v1.engine.exceptions/** +invariant_namespace: ENTRY-INV +last_reviewed: 2026-08-07 +last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc +--- + +# Entrypoints and serving boundaries + +Entrypoints translate offline, CLI, and serving requests into stable engine +operations and translate engine outputs into public responses. + +## Contract status + +This document describes current entrypoint responsibilities plus the boundary +under review in the open roadmap +[#5227](https://github.com/vllm-project/vllm-omni/issues/5227) and helper-move +PR [#5453](https://github.com/vllm-project/vllm-omni/pull/5453). In-flight +helper locations are not treated as current paths. + +## Ownership boundary + +This document owns offline API semantics, CLI and serve composition, +supported OpenAI-compatible routes, request validation and normalization, +response conversion, streaming/session behavior, and engine handoff. + +It does not own configuration precedence, cross-stage routing, stage +placement, payload implementation, or semantic error classification. +`entrypoints/openai/errors.py` is an explicit primary-path exception owned by +`error_contracts.md`. + +## Candidate invariants + +These identifiers are proposals while the document is `draft`. + +### ENTRY-INV-001: Entrypoints adapt but do not orchestrate + +**Rule:** Entrypoints MUST NOT implement cross-stage routing or stage lifecycle +policy. + +### ENTRY-INV-100: Public requests are normalized once + +**Rule:** Public protocol values MUST be validated and converted to internal +request contracts before engine submission. + +### ENTRY-INV-101: Streaming preserves request identity + +**Rule:** Every streamed response MUST remain associated with the request and +output modality that produced it. + +The reviewer-proposed rule that model-specific behavior should stay behind a +common adapter or processor abstraction remains an unnumbered candidate until +the entrypoint refactor makes that boundary enforceable. + +## Invariant namespace + +`ENTRY-INV` reserves `001-099` for boundary and dependency direction, +`100-199` for normalization, sessions, and streaming, `200-299` for +disconnect, cancellation, rendering, and cleanup, and `300-399` for upstream +route and renderer compatibility. Numbers become append-only after normative +promotion. + +## Safe-change guide + +Test request validation, protocol conversion, model-adapter routing, +streaming, session identity, disconnect/cancellation, and error mapping for +each affected offline or serving entrypoint. + +## Promotion gate + +- Reconcile the page after #5227's P0 ownership work and #5453's final + disposition. +- Publish a supported route/transport matrix rather than claiming blanket + OpenAI parity. +- Verify one normalization and handoff path per route family and one terminal + outcome per streaming transport. +- Promote the `api_server.py` composition-root rule only after helper moves and + import boundaries enforce it. +- Obtain approval from a technical owner and an independent validation + reviewer. diff --git a/docs/design/module/error_contracts.md b/docs/design/module/error_contracts.md new file mode 100644 index 000000000..02469d075 --- /dev/null +++ b/docs/design/module/error_contracts.md @@ -0,0 +1,109 @@ +--- +title: Error Classification, Propagation, and Rendering +kind: module +status: draft +architecture_state: current-plus-rfc +owners: + - "@alex-jw-brooks" + - "@NickCao" +document_stewards: + - "@hsliuustc0106" + - "@Gaohan123" + - "@david6666666" +required_reviewers: + - "@yinpeiqi" + - "@tzhouam" + - "@yenuo26" +primary_code_paths: + - vllm_omni/errors.py + - vllm_omni/entrypoints/openai/errors.py +related_code_paths: + - vllm_omni/engine/messages.py + - vllm_omni/engine/orchestrator.py + - vllm_omni/entrypoints/omni_base.py + - vllm_omni/entrypoints/async_omni.py + - vllm_omni/entrypoints/openai/api_server.py + - vllm_omni/entrypoints/openai/serving_chat.py + - vllm_omni/entrypoints/openai/serving_speech.py + - vllm_omni/entrypoints/openai/serving_audio_generate.py + - vllm_omni/entrypoints/openai/protocol/images.py + - vllm_omni/entrypoints/openai/protocol/videos.py +depends_on: [] +validation_paths: + - tests/engine/test_orchestrator_error_handling.py + - tests/engine/test_async_omni_engine_outputs.py + - tests/entrypoints/test_omni_entrypoints.py + - tests/entrypoints/test_stream_finish_reason.py + - tests/entrypoints/openai_api/test_image_server.py + - tests/entrypoints/openai_api/test_video_server.py + - tests/entrypoints/openai_api/test_serving_audio_generate.py + - tests/entrypoints/openai_api/test_serving_speech.py + - tests/metrics/test_prometheus.py +upstream_refs: + - vllm.v1.engine.exceptions.EngineDeadError + - vllm.v1.engine.exceptions.EngineGenerateError + - vllm.entrypoints.serve.utils.error_response + - vLLM RFC #48227 + - vLLM PR #49665 +invariant_namespace: ERR-INV +last_reviewed: 2026-08-07 +last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc +--- + +# Error classification, propagation, and rendering + +This contract separates the meaning of a failure from the transport used to +carry it and the public protocol used to render it. + +## Contract status + +Current behavior is distributed across engine, message, and entrypoint code. +The semantic hierarchy proposed by +[#5570](https://github.com/vllm-project/vllm-omni/issues/5570) remains an open +RFC. Proposed types and wire records from that RFC are not current contracts. + +## Ownership boundary + +This document owns semantic classification, fatal versus request-scoped +failure, cross-process error context, compatibility policy, sanitization, and +offline, HTTP, SSE, WebSocket, and asynchronous-job rendering rules. + +`engine/messages.py` remains primarily owned by the I/O contract; this page +owns the semantics of its error fields. Route and serving files remain +primarily owned by `entrypoints.md`; this page owns their rendering +constraints. + +## Candidate invariants + +The following topics require owner review before receiving stable IDs: + +- A request-scoped failure terminates only the affected request unless the + underlying engine is no longer usable. +- Every public rendering path preserves a stable classification while + removing sensitive internal context. +- Cancellation remains distinguishable from internal failure across process + and protocol boundaries. + +## Invariant namespace + +`ERR-INV` reserves `001-099` for classification ownership, `100-199` for wire +context and compatibility, `200-299` for fatality, termination, sanitization, +and rendering, and `300-399` for upstream alignment and transport extension +points. Numbers become append-only after normative promotion. + +## Safe-change guide + +Exercise exception classification, stage wire round trips, fatal and +request-scoped propagation, cancellation, sanitization, offline exceptions, +HTTP status/body consistency, streaming terminal events, asynchronous-job +persistence, and metrics classification. + +## Promotion gate + +- Resolve #5570's hierarchy, wire record, sanitization, and mixed-version + questions. +- Add table-driven exception/status/type tests and stage wire round trips. +- Verify all public transports terminate once and do not leak sensitive + payload data. +- Obtain approval from both technical owners and an independent validation + reviewer. diff --git a/docs/design/module/execution_platforms.md b/docs/design/module/execution_platforms.md new file mode 100644 index 000000000..beeeef922 --- /dev/null +++ b/docs/design/module/execution_platforms.md @@ -0,0 +1,48 @@ +--- +title: Execution Platforms +kind: module +status: draft +owners: + - "@gcanlin" + - "@tjtanaa" + - "@xuechendi" +primary_code_paths: + - vllm_omni/platforms/** +related_code_paths: + - vllm_omni/attention/** +depends_on: + - ar_runtime.md + - diffusion/index.md +validation_paths: + - tests/platforms/** +upstream_refs: + - vllm.platforms +last_reviewed: 2026-07-16 +--- + +# Execution platforms + +Execution platforms isolate hardware-specific capability detection, worker +selection, patches, kernels, and configuration adjustments. + +## Candidate invariants + +### PLATFORM-INV-001: Capabilities are explicit + +**Rule:** Hardware-dependent behavior MUST be guarded by platform selection or +capability detection and MUST fail clearly when unsupported. + +### PLATFORM-INV-002: Portable code remains portable + +**Rule:** Platform-neutral modules MUST NOT import a vendor implementation +directly when a common selection interface exists. + +### PLATFORM-INV-003: Overrides are minimal + +**Rule:** A platform implementation SHOULD override only behavior that differs +from the common or upstream implementation. + +## Safe-change guide + +Validate common import paths without the target accelerator and run focused +tests in a fresh `uv` environment on every affected device platform. diff --git a/docs/design/module/input_output_modality_contracts.md b/docs/design/module/input_output_modality_contracts.md new file mode 100644 index 000000000..1083b09b0 --- /dev/null +++ b/docs/design/module/input_output_modality_contracts.md @@ -0,0 +1,133 @@ +--- +title: Input, Output, and Modality Contracts +kind: module +status: draft +architecture_state: current +owners: + - "@Sy0307" + - "@amy-why-3459" +document_stewards: + - "@hsliuustc0106" + - "@Gaohan123" + - "@david6666666" +required_reviewers: + - "@tzhouam" + - "@linyueqian" + - "@david6666666" + - "@yenuo26" + - "@NickCao" +primary_code_paths: + - vllm_omni/inputs/** + - vllm_omni/outputs/** + - vllm_omni/request.py + - vllm_omni/data_entry_keys.py + - vllm_omni/engine/messages.py + - vllm_omni/engine/serialization.py + - vllm_omni/engine/mm_outputs.py + - vllm_omni/engine/output_modality.py + - vllm_omni/engine/output_processor.py +related_code_paths: + - vllm_omni/model_executor/stage_input_processors/** + - vllm_omni/core/sched/** + - vllm_omni/distributed/omni_connectors/** + - vllm_omni/diffusion/request.py + - vllm_omni/diffusion/worker/request_batch.py + - vllm_omni/errors.py +depends_on: + - error_contracts.md +validation_paths: + - tests/inputs/** + - tests/engine/test_data_entry_keys.py + - tests/engine/test_omni_request.py + - tests/engine/test_omni_request_output.py + - tests/engine/test_multimodal_accumulation.py + - tests/engine/test_output_modality.py + - tests/engine/test_output_processor.py + - tests/engine/test_wire_multimodal_output.py + - tests/utils/test_mm_outputs.py + - tests/utils/test_mm_outputs_partition.py + - tests/diffusion/test_diffusion_output_metadata.py +upstream_refs: + - vllm.inputs/** + - vllm.outputs/** + - vllm.v1.request.Request + - vllm.v1.engine.EngineCoreRequest + - vllm.v1.engine.EngineCoreOutput + - vllm.v1.engine.output_processor.OutputProcessor +invariant_namespace: IO-INV +last_reviewed: 2026-08-07 +last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc +--- + +# Input, output, and modality contracts + +These contracts define the data that may cross entrypoint, orchestration, +stage, connector, and model boundaries. + +## Contract status + +This document is a draft description of the current request, message, +serialization, modality, accumulation, and output types. It does not assign +semantic error ownership to the wire schema. + +## Ownership boundary + +This document owns request identity, prompt/input types, modality keys and +metadata, queue and wire message schemas, serialization, output types, +accumulation, completion semantics, and compatibility shims. + +It does not own route-specific validation or rendering, model-specific +interpretation, connector transport mechanics, scheduling policy, or semantic +error classification. The `ErrorMessage` schema belongs here; the meaning and +public rendering of its error fields belong to `error_contracts.md`. + +## Candidate invariants + +These identifiers are proposals while the document is `draft`. + +### IO-INV-001: Boundary data has an explicit modality + +**Rule:** Data crossing a module or stage boundary MUST identify its modality +and use the corresponding validated contract. + +### IO-INV-100: Request identity is stable + +**Rule:** Request identity MUST be preserved across conversions, stages, +streaming updates, cancellation, and errors. + +### IO-INV-101: Completion is explicit + +**Rule:** Producers MUST distinguish partial updates from the terminal update +for every output modality. + +### IO-INV-300: Internal objects do not leak into public protocols + +**Rule:** Entrypoints MUST explicitly translate internal output objects into +public response types. + +## Invariant namespace + +`IO-INV` reserves `001-099` for schema ownership and dependency direction, +`100-199` for identity, modality, accumulation, ordering, and completion, +`200-299` for serialization failure and payload cleanup, and `300-399` for +upstream extension, deprecation, and compatibility. Numbers become +append-only after normative promotion. + +## Safe-change guide + +Test construction, validation, serialization, round trips, streaming, +accumulation, completion, optional fields, unknown fields, and compatibility +at every affected producer-consumer boundary. + +## Promotion gate + +- Add explicit CODEOWNERS for the primary I/O paths or confirm that fallback + ownership is intentional. +- Document the compatibility window for deprecated `vllm_omni.engine` + I/O-related imports. +- Verify wire round trips and rejection of unknown or incompatible schema + fields. +- Define payload-versus-metadata rules and completion semantics for streaming + and batched outputs. +- Obtain approval from a technical owner and independent AR, diffusion, + connector, entrypoint, and validation reviewers as applicable. diff --git a/docs/design/module/model_integration.md b/docs/design/module/model_integration.md new file mode 100644 index 000000000..8b2afe0a0 --- /dev/null +++ b/docs/design/module/model_integration.md @@ -0,0 +1,51 @@ +--- +title: Model Integration +kind: module +status: draft +owners: + - "@tzhouam" + - "@gcanlin" +primary_code_paths: + - vllm_omni/model_executor/** + - vllm_omni/model_extras/** + - vllm_omni/plugins/** +related_code_paths: + - vllm_omni/transformers_utils/** + - vllm_omni/tokenizers/** +depends_on: + - input_output_modality_contracts.md + - ar_runtime.md +validation_paths: + - tests/model_executor/** + - tests/model_extras/** + - tests/plugin/** +upstream_refs: + - vllm.model_executor +last_reviewed: 2026-07-16 +--- + +# Model integration + +Model integration adapts model-specific configuration, preprocessing, loading, +and execution behavior to stable runtime contracts. + +## Candidate invariants + +### MODEL-INV-001: Registration is explicit + +**Rule:** A model integration MUST declare how its model class, loader, input +processor, and stage configuration are selected. + +### MODEL-INV-002: Model code does not route stages + +**Rule:** Model-specific code MUST NOT select or invoke a downstream omni stage. + +### MODEL-INV-003: Upstream behavior is reused deliberately + +**Rule:** An upstream vLLM implementation SHOULD be reused when its contract is +sufficient; an override MUST document the behavioral difference. + +## Safe-change guide + +Test registration, checkpoint loading, input conversion, and representative +model execution. Test shared utilities against more than one integration. diff --git a/docs/design/module/observability.md b/docs/design/module/observability.md new file mode 100644 index 000000000..83971e2d3 --- /dev/null +++ b/docs/design/module/observability.md @@ -0,0 +1,46 @@ +--- +title: Observability +kind: module +status: draft +owners: + - "@lishunyang12" +primary_code_paths: + - vllm_omni/metrics/** +related_code_paths: + - vllm_omni/engine/** + - vllm_omni/entrypoints/** +depends_on: + - engine_orchestration.md +validation_paths: + - tests/metrics/** +upstream_refs: + - vllm.v1.metrics +last_reviewed: 2026-07-16 +--- + +# Observability + +Observability defines stable metrics, logs, and correlation fields for tracing a +request across entrypoints, orchestration, stages, and workers. + +## Candidate invariants + +### OBS-INV-001: Request correlation survives stage boundaries + +**Rule:** Logs and metrics for the same request SHOULD carry a stable request +identifier and stage identity. + +### OBS-INV-002: Metric meaning is stable + +**Rule:** A metric's unit, labels, aggregation, and lifecycle MUST be documented; +changing any of them is a contract change. + +### OBS-INV-003: Labels are bounded + +**Rule:** Metrics MUST NOT use request IDs, prompts, outputs, or other unbounded +values as labels. + +## Safe-change guide + +Test registration, label sets, units, lifecycle resets, and representative +multi-stage emission. diff --git a/docs/design/module/omni_connector.md b/docs/design/module/omni_connector.md new file mode 100644 index 000000000..875f79e3d --- /dev/null +++ b/docs/design/module/omni_connector.md @@ -0,0 +1,50 @@ +--- +title: OmniConnector +kind: module +status: draft +owners: + - "@princepride" + - "@yuanheng-zhao" + - "@xuechendi" + - "@natureofnature" + - "@fake0fan" +primary_code_paths: + - vllm_omni/distributed/omni_connectors/** +related_code_paths: + - vllm_omni/platforms/*/omni_connectors/** +depends_on: + - input_output_modality_contracts.md + - vllm_omni_config.md +validation_paths: + - tests/distributed/omni_connectors/** +upstream_refs: + - vllm.distributed +last_reviewed: 2026-07-16 +--- + +# OmniConnector + +OmniConnector defines model-agnostic transport and synchronization contracts +for data exchanged across stages, processes, devices, and nodes. + +## Candidate invariants + +### CONNECTOR-INV-001: Connectors transport but do not route + +**Rule:** A connector MUST NOT select the next stage or implement model-specific +execution policy. + +### CONNECTOR-INV-002: Producer and consumer contracts agree + +**Rule:** Both ends MUST agree on data identity, shape, dtype, placement, +ownership, and completion semantics. + +### CONNECTOR-INV-003: Resources have deterministic cleanup + +**Rule:** Connections, buffers, handles, and background work MUST be released on +normal completion, cancellation, failure, and shutdown. + +## Safe-change guide + +Test setup, transfer, synchronization, timeout, cancellation, failure, and +cleanup across every affected backend. diff --git a/docs/design/module/profiling.md b/docs/design/module/profiling.md new file mode 100644 index 000000000..c2f1a9f88 --- /dev/null +++ b/docs/design/module/profiling.md @@ -0,0 +1,46 @@ +--- +title: Profiling +kind: module +status: draft +owners: + - "@gcanlin" +primary_code_paths: + - vllm_omni/profiler/** + - vllm_omni/diffusion/profiler/** +related_code_paths: + - vllm_omni/entrypoints/** +depends_on: + - observability.md +validation_paths: + - tests/profile/** +upstream_refs: + - vllm.profiler +last_reviewed: 2026-07-16 +--- + +# Profiling + +Profiling owns opt-in instrumentation and trace collection used to locate +runtime bottlenecks without changing model results or execution policy. + +## Candidate invariants + +### PROFILE-INV-001: Profiling is opt-in + +**Rule:** Expensive trace collection MUST be disabled by default and bounded by +an explicit start and stop lifecycle. + +### PROFILE-INV-002: Instrumentation preserves semantics + +**Rule:** Profiling MUST NOT change scheduling decisions, generated outputs, or +resource ownership beyond documented synchronization overhead. + +### PROFILE-INV-003: Trace context is sufficient + +**Rule:** Trace events SHOULD identify stage, rank, worker, operation, and +request where available without embedding user payloads. + +## Safe-change guide + +Test disabled overhead, repeated start and stop, cleanup, multi-rank trace +naming, and each supported profiler backend. diff --git a/docs/design/module/quantization.md b/docs/design/module/quantization.md new file mode 100644 index 000000000..3e9001aab --- /dev/null +++ b/docs/design/module/quantization.md @@ -0,0 +1,50 @@ +--- +title: Quantization +kind: module +status: draft +owners: + - "@david6666666" + - "@Isotr0py" + - "@lishunyang12" +primary_code_paths: + - vllm_omni/quantization/** + - vllm_omni/diffusion/quantization/** +related_code_paths: + - vllm_omni/platforms/*/quant/** +depends_on: + - model_integration.md + - diffusion/diffusion_model_integration.md +validation_paths: + - tests/quantization/** + - tests/diffusion/quantization/** +upstream_refs: + - vllm.model_executor.layers.quantization +last_reviewed: 2026-07-16 +--- + +# Quantization + +Quantization owns method registration, checkpoint interpretation, layer +replacement, and runtime constraints for reduced-precision execution. + +## Candidate invariants + +### QUANT-INV-001: Checkpoint and runtime methods agree + +**Rule:** The selected method MUST validate checkpoint metadata, supported +layers, device capabilities, and required dependencies before use. + +### QUANT-INV-002: Unsupported combinations fail clearly + +**Rule:** Quantization MUST NOT silently fall back to a different precision or +method when a requested combination is unsupported. + +### QUANT-INV-003: Numerical validation is required + +**Rule:** A new method or layer mapping MUST include correctness comparison +against a defined reference result. + +## Safe-change guide + +Test loading, representative layers, metadata, platform guards, and numerical +tolerances separately from performance. diff --git a/docs/design/module/stage_runtime.md b/docs/design/module/stage_runtime.md new file mode 100644 index 000000000..aefba5a40 --- /dev/null +++ b/docs/design/module/stage_runtime.md @@ -0,0 +1,126 @@ +--- +title: Stage Runtime and Replica Lifecycle +kind: module +status: draft +architecture_state: current-plus-in-flight +owners: + - "@tzhouam" + - "@fake0fan" +document_stewards: + - "@hsliuustc0106" + - "@Gaohan123" + - "@david6666666" +required_reviewers: + - "@Sy0307" + - "@princepride" + - "@chickeyton" + - "@yenuo26" + - "@NickCao" +primary_code_paths: + - vllm_omni/engine/stage_runtime.py + - vllm_omni/engine/stage_pool.py + - vllm_omni/engine/stage_client.py + - vllm_omni/engine/membership_controller.py + - vllm_omni/engine/stage_engine_startup.py + - vllm_omni/engine/stage_engine_core_client.py + - vllm_omni/engine/stage_engine_core_proc.py + - vllm_omni/engine/stage_engine_core_proc_manager.py + - vllm_omni/engine/stage_init_utils.py +related_code_paths: + - vllm_omni/distributed/omni_coordinator/** + - vllm_omni/distributed/ray_utils/** + - vllm_omni/diffusion/inline_stage_diffusion_client.py + - vllm_omni/diffusion/stage_diffusion_client.py + - vllm_omni/diffusion/stage_diffusion_proc.py + - vllm_omni/config/** +depends_on: + - input_output_modality_contracts.md + - error_contracts.md + - omni_connector.md +validation_paths: + - tests/engine/test_async_omni_engine_stage_init.py + - tests/engine/test_membership_controller.py + - tests/engine/test_single_stage_mode.py + - tests/engine/test_stage_engine_core_client.py + - tests/engine/test_stage_engine_core_proc.py + - tests/diffusion/test_inline_stage_diffusion_client.py + - tests/diffusion/test_stage_diffusion_proc.py + - tests/distributed/omni_coordinator/** +upstream_refs: + - vllm.v1.engine.core_client.AsyncMPClient + - vllm.v1.engine.core_client.DPLBAsyncMPClient + - vllm.v1.engine.core.EngineCoreProc + - vllm.v1.engine.utils.CoreEngineProcManager + - vllm.v1.engine.coordinator.DPCoordinator +invariant_namespace: STAGE-INV +last_reviewed: 2026-08-07 +last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc +--- + +# Stage runtime and replica lifecycle + +The stage runtime turns a logical stage definition into ready local or +distributed replicas and provides the lifecycle boundary used by the +orchestrator. + +## Contract status + +This document describes the current `StageRuntime`, `StagePool`, stage-client, +and stage-process layout. The unified LLM/diffusion direction in +[#5441](https://github.com/vllm-project/vllm-omni/pull/5441) remains in flight +and is not presented as current behavior. + +## Ownership boundary + +This document owns local and distributed stage placement, startup and +readiness, one pool per logical stage, replica identity and membership, +selection and affinity, stage-client/process lifecycle, liveness, draining, +and shutdown. + +It does not own cross-stage request policy, model scheduler policy, connector +transfer semantics, configuration precedence, or public error rendering. + +## Candidate invariants + +These identifiers are proposals while the document is `draft`. + +### STAGE-INV-001: The runtime owns replica lifecycle + +**Rule:** Orchestration MUST acquire stage capacity through the stage runtime +and MUST NOT construct or retire backend replicas directly. + +### STAGE-INV-100: Affinity uses stable replica identity + +**Rule:** Follow-up operations for a request with replica affinity MUST resolve +the same logical replica until the request terminates or the replica is +declared lost. + +### STAGE-INV-200: Shutdown is idempotent + +**Rule:** Repeated shutdown or cleanup signals MUST NOT leak stage clients, +process managers, or membership registrations. + +## Invariant namespace + +`STAGE-INV` reserves `001-099` for placement and ownership, `100-199` for +replica identity/readiness/affinity, `200-299` for liveness loss, draining and +cleanup, and `300-399` for local/distributed and upstream compatibility. +Numbers become append-only after normative promotion. + +## Safe-change guide + +Exercise local and distributed startup, readiness, replica selection, +affinity, membership changes, abort routing, process failure, draining, and +repeated shutdown. Cross-stage policy changes belong in +`engine_orchestration.md`. + +## Promotion gate + +- Reconcile terminology and ownership after #5441 reaches a final state. +- Add or identify a dedicated StageRuntime/StagePool lifecycle suite; current + evidence is distributed across the cited tests. +- Demonstrate affinity preservation for update, interaction, and abort paths. +- Demonstrate membership loss and repeated shutdown without leaked clients or + processes. +- Obtain approval from a technical owner and an independent validation + reviewer. diff --git a/docs/design/module/vllm_omni_config.md b/docs/design/module/vllm_omni_config.md new file mode 100644 index 000000000..085e4cee6 --- /dev/null +++ b/docs/design/module/vllm_omni_config.md @@ -0,0 +1,56 @@ +--- +title: vLLM-Omni Configuration +kind: module +status: draft +architecture_state: deferred-pending-refactor +ownership_status: provisional +owners: + - "@lishunyang12" + - "@alex-jw-brooks" +primary_code_paths: + - vllm_omni/config/** + - vllm_omni/deploy/** + - vllm_omni/model_executor/stage_configs/** +related_code_paths: + - vllm_omni/platforms/*/stage_configs/** +depends_on: [] +validation_paths: + - tests/config/** +upstream_refs: + - vllm.config +last_reviewed: 2026-08-07 +last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc +--- + +# vLLM-Omni configuration + +This page intentionally records discovery paths only. Substantive +configuration ownership and contract work is deferred until the active +configuration refactoring has settled. + +## Deferred contract scope + +The current code and tests remain authoritative for parsing, defaults, +overrides, deployment topology, stage construction, and runtime projection. +This draft does not define a stable precedence model, environment-variable +capture rule, canonical configuration object, or invariant namespace. + +The PR review proposal to capture environment-derived values in configuration +objects at initialization is an open design question, not a current +invariant. + +## Safe-change guide while deferred + +Changes should trace every affected producer to its runtime consumer and test +the structured, legacy, CLI, and deployment paths that are actually supported. +Do not infer a stable contract from this placeholder page. + +## Promotion gate + +- Finish or stabilize the configuration refactor and identify the canonical + runtime configuration representation. +- Confirm technical owners, primary path exceptions, and validation owners. +- Document parsing and override precedence, including when environment values + are captured. +- Allocate an invariant namespace only after those behaviors are enforced and + owner-approved.