77 lines
2.6 KiB
Markdown
77 lines
2.6 KiB
Markdown
---
|
|
title: Autoregressive Runtime
|
|
kind: module
|
|
status: draft
|
|
owners:
|
|
- "@tzhouam"
|
|
- "@yinpeiqi"
|
|
- "@fake0fan"
|
|
- "@Sy0307"
|
|
- "@Gaohan123"
|
|
primary_code_paths:
|
|
- vllm_omni/core/**
|
|
- vllm_omni/worker/**
|
|
related_code_paths:
|
|
- vllm_omni/model_executor/**
|
|
depends_on:
|
|
- engine_orchestration.md
|
|
- model_integration.md
|
|
- input_output_modality_contracts.md
|
|
validation_paths:
|
|
- tests/core/**
|
|
- tests/worker/**
|
|
upstream_refs:
|
|
- vllm.v1.core
|
|
- vllm.v1.worker
|
|
last_reviewed: 2026-07-16
|
|
---
|
|
|
|
# Autoregressive runtime
|
|
|
|
The AR runtime extends vLLM scheduling and worker execution for omni-stage
|
|
inputs and outputs while preserving vLLM scheduling and cache semantics.
|
|
|
|
## Candidate invariants
|
|
|
|
### AR-INV-001: vLLM owns base scheduling semantics
|
|
|
|
**Rule:** Omni schedulers MUST preserve upstream request-state and cache
|
|
transitions unless an Omni-specific difference is documented and tested.
|
|
|
|
### AR-INV-002: Omni data crosses explicit adapters
|
|
|
|
**Rule:** Modality-specific stage data MUST be converted at an input or output
|
|
adapter, not injected through unrelated scheduler state.
|
|
|
|
### AR-INV-003: Workers execute assigned work
|
|
|
|
**Rule:** Workers and model runners MUST NOT implement cross-stage routing.
|
|
|
|
## Token-only output payloads
|
|
|
|
An AR stage whose downstream decoder consumes sampled tokens rather than
|
|
hidden states can set `omni_pooler_payload_include_hidden = False`. Hidden
|
|
states remain available for logits and sampling; prefill conditioning and
|
|
other required multimodal outputs still cross the stage boundary.
|
|
|
|
A chunk processor that needs IDs on steps without a tensor payload declares
|
|
`requires_token_updates = True`. The scheduler then enqueues nonempty sampled
|
|
IDs while preserving final-stage exclusions and the processor's EOF handling.
|
|
Processors without this declaration retain their tensor-payload trigger.
|
|
|
|
When hidden and multimodal payloads are absent, the shared runner skips empty
|
|
per-request payload construction. Prefix-cache merges and pending model
|
|
postprocess retain the normal path. This policy is independent of whether
|
|
output materialization runs inline or in the background.
|
|
|
|
Inline construction consumes query offsets, scheduled-token counts and
|
|
scheduler metadata before the next execution can reuse them. Background
|
|
construction snapshots those inputs. Both paths reuse the request IDs already
|
|
copied by upstream bookkeeping; connector collection stays on the main thread
|
|
because it can run a tensor-parallel collective.
|
|
|
|
## Safe-change guide
|
|
|
|
Test request lifecycle, abort, cache state, and every affected worker execution
|
|
mode against the supported upstream vLLM contract.
|