--- title: Entrypoints and Serving Boundaries kind: module status: draft architecture_state: current-plus-in-flight owners: - "@alex-jw-brooks" - "@linyueqian" - "@NickCao" document_stewards: - "@hsliuustc0106" - "@Gaohan123" - "@david6666666" required_reviewers: - "@tzhouam" - "@herotai214" - "@yenuo26" - "@NickCao" primary_code_paths: - vllm_omni/entrypoints/omni.py - vllm_omni/entrypoints/async_omni.py - vllm_omni/entrypoints/async_omni_base.py - vllm_omni/entrypoints/omni_base.py - vllm_omni/entrypoints/duplex_omni.py - vllm_omni/entrypoints/cli/** - vllm_omni/entrypoints/openai/** - vllm_omni/entrypoints/openpi/** - vllm_omni/entrypoints/client_request_state.py - vllm_omni/entrypoints/stage_utils.py - vllm_omni/entrypoints/utils.py - vllm_omni/entrypoints/duplex/** - vllm_omni/clients/** - vllm_omni/protocol/** primary_path_exceptions: - path: vllm_omni/entrypoints/openai/errors.py owner: error_contracts.md - path: vllm_omni/entrypoints/duplex/** owner: ../fullduplex.md - path: vllm_omni/entrypoints/duplex_omni.py owner: ../fullduplex.md - path: vllm_omni/clients/** owner: ../fullduplex.md related_code_paths: - vllm_omni/errors.py - vllm_omni/inputs/** - vllm_omni/outputs/** - vllm_omni/engine/omni_engine_base.py - vllm_omni/engine/async_omni_engine.py - vllm_omni/engine/duplex_omni_engine.py - vllm_omni/config/** - vllm_omni/deploy/** depends_on: - input_output_modality_contracts.md - error_contracts.md - engine_orchestration.md validation_paths: - tests/protocol/** - tests/entrypoints/test_omni_entrypoints.py - tests/entrypoints/test_async_omni.py - tests/entrypoints/test_async_omni_pause_sleep_routing.py - tests/entrypoints/test_duplex_omni.py - tests/entrypoints/duplex/** - tests/entrypoints/openai_api/test_duplex_api_server.py - tests/entrypoints/test_serve.py - tests/entrypoints/test_stream_finish_reason.py - tests/entrypoints/openai/** - tests/entrypoints/openai_api/** - tests/e2e/online_serving/** upstream_refs: - vllm.engine.protocol.EngineClient - vllm.entrypoints.openai/** - vllm.entrypoints.serve/** - vllm.renderers/** - vllm.v1.engine.exceptions/** invariant_namespace: ENTRY-INV last_reviewed: 2026-08-07 last_verified_commit: 3d7fc3b9ba3cac88d579d4dc35b78b0b641675fc --- # Entrypoints and serving boundaries Entrypoints translate offline, CLI, and serving requests into stable engine operations and translate engine outputs into public responses. ## Contract status This document describes current entrypoint responsibilities plus the boundary under review in the open roadmap [#5227](https://github.com/vllm-project/vllm-omni/issues/5227) and helper-move PR [#5453](https://github.com/vllm-project/vllm-omni/pull/5453). In-flight helper locations are not treated as current paths. ## Ownership boundary This document owns offline API semantics, CLI and serve composition, supported OpenAI-compatible routes, request validation and normalization, response conversion, streaming/session behavior, and engine handoff. When `--api-server-count` is greater than one, the serve composition root starts the frontend process manager while a parent-owned `StageRuntime` launches the shared local stage engines. Each frontend receives only its own stage-client channel configuration and does not launch or retire backend processes. CLI validation rejects unsupported distributed, diffusion, fault-tolerant, elastic-EP, Ray, and runtime-LoRA combinations before startup. It does not own configuration precedence, cross-stage routing, stage placement, payload implementation, or semantic error classification. `entrypoints/openai/errors.py` is an explicit primary-path exception owned by `error_contracts.md`. It also owns `vllm_omni/protocol/**`, the shared wire codec promoted by RFC [#6592](https://github.com/vllm-project/vllm-omni/issues/6592) P0a. It follows the three tiers `docs/serving/realtime_duplex_api.md` already defines. `protocol/realtime/**` is Tier 1, identical to OpenAI: 30 server events, 10 client commands, their typed fields and pure wire rendering, audio-format negotiation, conversation-item rules and the error-envelope shape. `protocol/duplex/**` is Tier 2 (OpenAI names carrying vLLM-Omni extensions, each a subclass of its Tier 1 twin declaring only what it adds) and Tier 3 (ours alone, including our error-code vocabulary), kept separate so a GA-only consumer is not handed a vocabulary its clients never send. That package sits outside `entrypoints/` on purpose --- the engine depends on it, so it belongs to neither layer --- but deciding what a Realtime client may put on the wire is this document's subject, and the same reasoning already places `vllm_omni/errors.py`, `inputs/` and `outputs/` under contract documents rather than under a consumer. It is deliberately **not** owned by `../fullduplex.md`, even though duplex is the codec's only consumer today. RFC #6592 asks for a named owner precisely so a second Realtime consumer does not need duplex review to change the codec (#6592 open question 8). Ownership here is review scope, not a dependency: the codec may not import the engine, the entrypoints or the model code, which `tests/protocol/realtime/test_protocol_import_boundary.py` enforces. When a second consumer lands (#6592 P0b) the package is expected to graduate to its own module document with owners drawn from both consumers. ## Candidate invariants These identifiers are proposals while the document is `draft`. ### ENTRY-INV-001: Entrypoints adapt but do not orchestrate **Rule:** Entrypoints MUST NOT implement cross-stage routing or stage lifecycle policy. ### ENTRY-INV-002: The shared Realtime codec does not depend on its consumers **Rule:** `vllm_omni/protocol/**` MUST NOT import `vllm_omni.engine`, `vllm_omni.entrypoints`, `vllm_omni.model_executor`, `vllm_omni.worker` or `vllm_omni.clients`; `protocol/duplex/**` MAY depend on `protocol/realtime/**` but never the reverse, and a Tier 1 class MUST NOT carry a vLLM-Omni extension field; and each protocol/codec behavior MUST have exactly one implementation that every consumer uses. A wire type MUST NOT carry a consumer's internal representation: the duplex mailbox rendering (`DuplexCommand.payload()`, whose channel differs from the client event for `session.update` and the `conversation.item.*` commands) stays engine-side. A duplex consumer (`engine/duplex/**`, `entrypoints/duplex/**`, the duplex clients) MUST import `vllm_omni.protocol.duplex` and MUST NOT import `vllm_omni.protocol.realtime` directly, so the tier boundary has exactly one extension point. ### ENTRY-INV-100: Public requests are normalized once **Rule:** Public protocol values MUST be validated and converted to internal request contracts before engine submission. ### ENTRY-INV-101: Streaming preserves request identity **Rule:** Every streamed response MUST remain associated with the request and output modality that produced it. The reviewer-proposed rule that model-specific behavior should stay behind a common adapter or processor abstraction remains an unnumbered candidate until the entrypoint refactor makes that boundary enforceable. ## Invariant namespace `ENTRY-INV` reserves `001-099` for boundary and dependency direction, `100-199` for normalization, sessions, and streaming, `200-299` for disconnect, cancellation, rendering, and cleanup, and `300-399` for upstream route and renderer compatibility. Numbers become append-only after normative promotion. ## Safe-change guide Test request validation, protocol conversion, model-adapter routing, streaming, session identity, disconnect/cancellation, and error mapping for each affected offline or serving entrypoint. Sleep must wait for in-flight `generate()` admission before EngineCore offload; `wake_up` does not resume admission — callers must `resume_generation()`. Sleeping tags are tracked per stage so `wake_up(stage_ids=[0])` does not skip a later `wake_up(stage_ids=[1])`. Streaming input pumps take an admission slot immediately before each EngineCore ADD or update, not while waiting for the next client chunk. Frontend abort keeps `request_states` until `generate()` consumes the terminal output. Multi-API serving must preserve one client rank and one channel set per frontend, include every frontend in readiness/failure observation, and keep stage lifecycle ownership in the parent composition root. ## Promotion gate - Reconcile the page after #5227's P0 ownership work and #5453's final disposition. - Publish a supported route/transport matrix rather than claiming blanket OpenAI parity. - Verify one normalization and handoff path per route family and one terminal outcome per streaming transport. - Promote the `api_server.py` composition-root rule only after helper moves and import boundaries enforce it. - Obtain approval from a technical owner and an independent validation reviewer.