Rebase multi-API serving on latest main

Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
Sy03 committed 2026-09-16 12:13:04 +08:00
1 parent 78934753ff
commit dc3c7dbe16
25 files changed
+1832 -202

No files matched your search

+10
View File
@@ -89,6 +89,13 @@ This document owns offline API semantics, CLI and serve composition,
supported OpenAI-compatible routes, request validation and normalization,
response conversion, streaming/session behavior, and engine handoff.
When `--api-server-count` is greater than one, the serve composition root
starts the frontend process manager while a parent-owned `StageRuntime`
launches the shared local stage engines. Each frontend receives only its own
stage-client channel configuration and does not launch or retire backend
processes. CLI validation rejects unsupported distributed, diffusion,
fault-tolerant, elastic-EP, Ray, and runtime-LoRA combinations before startup.
It does not own configuration precedence, cross-stage routing, stage
placement, payload implementation, or semantic error classification.
`entrypoints/openai/errors.py` is an explicit primary-path exception owned by
@@ -136,6 +143,9 @@ stage so `wake_up(stage_ids=[0])` does not skip a later `wake_up(stage_ids=[1])`
Streaming input pumps take an admission slot immediately before each EngineCore
ADD or update, not while waiting for the next client chunk. Frontend abort
keeps `request_states` until `generate()` consumes the terminal output.
Multi-API serving must preserve one client rank and one channel set per
frontend, include every frontend in readiness/failure observation, and keep
stage lifecycle ownership in the parent composition root.
## Promotion gate
+20
View File
@@ -77,6 +77,22 @@ readiness, one pool per logical stage, replica identity and membership,
selection and affinity, stage-client/process lifecycle, liveness, draining,
and shutdown.
Local EngineCore initialization and multi-API serving use
`StageRuntime.launch_stage_engines(num_api_servers=1)` as their common launch
entry. It resolves plans when not supplied by the runtime's device-group
scheduler, coordinates device locks and environment overlays, starts engines,
and rolls back resources on startup or attachment failure. Single-API clients
attach locally after readiness and take ownership of the engine resources;
there is no additional API subprocess. Diffusion and remote attachment retain
their backend-specific initialization.
For local multi-API serving, the parent runtime owns and launches each stage
engine once. It supplies a distinct input/output channel for every API
frontend and stage replica; frontend workers attach clients to those engines
without acquiring stage-process ownership. Readiness observation and shutdown
remain parent-owned. This topology currently excludes diffusion and
remote/headless stages, intra-stage data parallelism, and Ray backends.
It does not own cross-stage request policy, model scheduler policy, connector
transfer semantics, configuration precedence, or public error rendering.
@@ -114,6 +130,10 @@ affinity, membership changes, abort routing, process failure, draining, and
repeated shutdown. Cross-stage policy changes belong in
`engine_orchestration.md`.
For multi-API changes, additionally exercise per-frontend channel assignment,
shared-engine startup handshakes, frontend failure observation, and parent
cleanup without duplicate stage shutdown.
## Promotion gate
- Reconcile terminology and ownership after #5441 reaches a final state.