0 Fork 0

Files

T

bb5bf8fd0a [Bugfix] Fix colocate-async sleep admission race and deliver AR abort tokens (#6367 )

Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Samit <285365963@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Samit <285365963@qq.com>

2026-08-29 09:11:39 +00:00

5.2 KiB

Raw Blame History

title, kind, status, architecture_state, owners, document_stewards, required_reviewers, primary_code_paths, related_code_paths, depends_on, validation_paths, upstream_refs, invariant_namespace, last_reviewed, last_verified_commit

title kind status architecture_state owners document_stewards required_reviewers primary_code_paths related_code_paths depends_on validation_paths upstream_refs invariant_namespace last_reviewed last_verified_commit

Engine Orchestration

module

draft

current-plus-in-flight

@tzhouam

@fake0fan

@hsliuustc0106

@Gaohan123

@david6666666

@yinpeiqi

@Sy0307

@yenuo26

@NickCao

vllm_omni/engine/async_omni_engine.py

vllm_omni/engine/async_engine_utils.py

vllm_omni/engine/orchestrator.py

vllm_omni/engine/cfg_companion_tracker.py

vllm_omni/engine/rpc_result_router.py

vllm_omni/engine/messages.py

vllm_omni/engine/orchestrator_monitor.py

vllm_omni/engine/stage_client.py

vllm_omni/engine/stage_pool.py

vllm_omni/outputs/output_processor.py

vllm_omni/distributed/omni_coordinator/**

input_output_modality_contracts.md

error_contracts.md

stage_runtime.md

omni_connector.md

tests/engine/test_async_omni_engine_input.py

tests/engine/test_async_omni_engine_outputs.py

tests/engine/test_async_omni_engine_abort.py

tests/e2e/offline_inference/test_qwen3_omni.py

tests/engine/test_async_omni_engine_stage_init.py

tests/engine/test_orchestrator.py

tests/engine/test_orchestrator_error_handling.py

tests/engine/test_orchestrator_stage_input_bridge.py

tests/engine/test_cfg_companion_lifecycle.py

tests/engine/test_rpc_result_router.py

tests/e2e/features/fullduplex/engine/**

vllm.v1.engine.EngineCoreRequest

vllm.v1.engine.EngineCoreOutputs

vllm.v1.engine.exceptions.EngineDeadError

vllm.outputs.RequestOutput

ORCH-INV

2026-08-07

3d7fc3b9ba

Engine orchestration

Engine orchestration coordinates configured AR and diffusion stages while keeping public clients independent from stage processes and transports.

Contract status

This document is a draft description of current behavior. It also identifies the boundary affected by the in-flight stage client/process refactor in #5441. Names and responsibilities proposed only by that PR are not current contracts.

The orchestration loop also has an opt-in event-driven mode (VLLM_OMNI_EVENT_DRIVEN_ORCH=1, default off) proposed in #5221. It changes poll cadence only: the routing, ordering, and terminal-state contracts below hold identically on both loops.

Ownership boundary

This document owns AsyncOmniEngine, Orchestrator, request-state creation, cross-stage routing, output ordering, companion tracking, control/RPC correlation, cancellation propagation, and terminal-state convergence.

It does not own stage placement, replica selection, stage-process startup, payload schemas, public protocol rendering, connector transport, or semantic error classification. Those responsibilities belong to the stage runtime, I/O, entrypoint, connector, and error contracts.

Candidate invariants

These identifiers are proposals while the document is draft.

ORCH-INV-001: The orchestrator owns cross-stage routing

Rule: Entrypoints and stage clients MUST NOT independently forward a request to a downstream logical stage.

ORCH-INV-002: Stage clients do not own routing policy

Rule: Stage clients MUST implement communication and lifecycle operations without selecting the next logical stage.

ORCH-INV-100: Terminal state is monotonic

Rule: Once a request reaches a terminal state, orchestration MUST NOT forward new work for that request.

Invariant namespace

ORCH-INV reserves 001-099 for dependency direction, 100-199 for request state and ordering, 200-299 for failure/cancellation/cleanup, and 300-399 for extension points, shutdown ordering, and upstream alignment. Numbers become append-only after normative promotion.

Safe-change guide

Test routing, output ordering, queue correlation, cancellation, failure propagation, shutdown ordering, and representative multi-stage execution. AR abort of a final-stage LLM request must deliver a terminal output with finish_reason="abort" and the generated prefix so collocated training can resume; diffusion abort remains whole-sample retry. Aborting a parallel-sampling child must drop parent_requests once no children remain. EngineCore control RPCs (sleep / wake_up / pause_scheduler / resume_scheduler) must propagate worker exceptions rather than returning {"supported": False}. When a request has multiple final output stages, only the last abort message is request-terminal. Output-processor abort state is committed only after the physical EngineCore abort succeeds. Changes that cross into stage runtime or public error behavior require review from that contract's owners.

Promotion gate

Reconcile current names and boundaries after #5441 merges, closes, or is replaced.
Demonstrate one terminal outcome per request across success, abort, request-scoped failure, and fatal engine failure.
Verify queue correlation and orchestrator-before-runtime shutdown ordering in the cited validation paths.
Obtain approval from a technical owner and an independent validation reviewer.

5.2 KiB Raw Blame History