vllm-omni/docs/design/module/ar_runtime.md

title, kind, status, owners, primary_code_paths, related_code_paths, depends_on, validation_paths, upstream_refs, last_reviewed

title	kind	status	owners	primary_code_paths	related_code_paths	depends_on	validation_paths	upstream_refs	last_reviewed
Autoregressive Runtime	module	draft	@tzhouam @yinpeiqi @fake0fan @Sy0307 @Gaohan123	vllm_omni/core/** vllm_omni/worker/**	vllm_omni/model_executor/**	engine_orchestration.md model_integration.md input_output_modality_contracts.md	tests/core/** tests/worker/**	vllm.v1.core vllm.v1.worker	2026-07-16

Autoregressive runtime

The AR runtime extends vLLM scheduling and worker execution for omni-stage inputs and outputs while preserving vLLM scheduling and cache semantics.

Candidate invariants

AR-INV-001: vLLM owns base scheduling semantics

Rule: Omni schedulers MUST preserve upstream request-state and cache transitions unless an Omni-specific difference is documented and tested.

AR-INV-002: Omni data crosses explicit adapters

Rule: Modality-specific stage data MUST be converted at an input or output adapter, not injected through unrelated scheduler state.

AR-INV-003: Workers execute assigned work

Rule: Workers and model runners MUST NOT implement cross-stage routing.

Token-only output payloads

An AR stage whose downstream decoder consumes sampled tokens rather than hidden states can set omni_pooler_payload_include_hidden = False. Hidden states remain available for logits and sampling; prefill conditioning and other required multimodal outputs still cross the stage boundary.

A chunk processor that needs IDs on steps without a tensor payload declares requires_token_updates = True. The scheduler then enqueues nonempty sampled IDs while preserving final-stage exclusions and the processor's EOF handling. Processors without this declaration retain their tensor-payload trigger.

When hidden and multimodal payloads are absent, the shared runner skips empty per-request payload construction. Prefix-cache merges and pending model postprocess retain the normal path. This policy is independent of whether output materialization runs inline or in the background.

Inline construction consumes query offsets, scheduled-token counts and scheduler metadata before the next execution can reuse them. Background construction snapshots those inputs. Both paths reuse the request IDs already copied by upstream bookkeeping; connector collection stays on the main thread because it can run a tensor-parallel collective.

Safe-change guide

Test request lifecycle, abort, cache state, and every affected worker execution mode against the supported upstream vLLM contract.