Rebase multi-API serving on latest main
Signed-off-by: Sy03 <1370724210@qq.com>
This commit is contained in:
1 parent
78934753ff
commit
dc3c7dbe16
25 files changed
+1832
-202
No files matched your search
@@ -89,6 +89,13 @@ This document owns offline API semantics, CLI and serve composition,
|
||||
supported OpenAI-compatible routes, request validation and normalization,
|
||||
response conversion, streaming/session behavior, and engine handoff.
|
||||
|
||||
When `--api-server-count` is greater than one, the serve composition root
|
||||
starts the frontend process manager while a parent-owned `StageRuntime`
|
||||
launches the shared local stage engines. Each frontend receives only its own
|
||||
stage-client channel configuration and does not launch or retire backend
|
||||
processes. CLI validation rejects unsupported distributed, diffusion,
|
||||
fault-tolerant, elastic-EP, Ray, and runtime-LoRA combinations before startup.
|
||||
|
||||
It does not own configuration precedence, cross-stage routing, stage
|
||||
placement, payload implementation, or semantic error classification.
|
||||
`entrypoints/openai/errors.py` is an explicit primary-path exception owned by
|
||||
@@ -136,6 +143,9 @@ stage so `wake_up(stage_ids=[0])` does not skip a later `wake_up(stage_ids=[1])`
|
||||
Streaming input pumps take an admission slot immediately before each EngineCore
|
||||
ADD or update, not while waiting for the next client chunk. Frontend abort
|
||||
keeps `request_states` until `generate()` consumes the terminal output.
|
||||
Multi-API serving must preserve one client rank and one channel set per
|
||||
frontend, include every frontend in readiness/failure observation, and keep
|
||||
stage lifecycle ownership in the parent composition root.
|
||||
|
||||
## Promotion gate
|
||||
|
||||
|
||||
@@ -77,6 +77,22 @@ readiness, one pool per logical stage, replica identity and membership,
|
||||
selection and affinity, stage-client/process lifecycle, liveness, draining,
|
||||
and shutdown.
|
||||
|
||||
Local EngineCore initialization and multi-API serving use
|
||||
`StageRuntime.launch_stage_engines(num_api_servers=1)` as their common launch
|
||||
entry. It resolves plans when not supplied by the runtime's device-group
|
||||
scheduler, coordinates device locks and environment overlays, starts engines,
|
||||
and rolls back resources on startup or attachment failure. Single-API clients
|
||||
attach locally after readiness and take ownership of the engine resources;
|
||||
there is no additional API subprocess. Diffusion and remote attachment retain
|
||||
their backend-specific initialization.
|
||||
|
||||
For local multi-API serving, the parent runtime owns and launches each stage
|
||||
engine once. It supplies a distinct input/output channel for every API
|
||||
frontend and stage replica; frontend workers attach clients to those engines
|
||||
without acquiring stage-process ownership. Readiness observation and shutdown
|
||||
remain parent-owned. This topology currently excludes diffusion and
|
||||
remote/headless stages, intra-stage data parallelism, and Ray backends.
|
||||
|
||||
It does not own cross-stage request policy, model scheduler policy, connector
|
||||
transfer semantics, configuration precedence, or public error rendering.
|
||||
|
||||
@@ -114,6 +130,10 @@ affinity, membership changes, abort routing, process failure, draining, and
|
||||
repeated shutdown. Cross-stage policy changes belong in
|
||||
`engine_orchestration.md`.
|
||||
|
||||
For multi-API changes, additionally exercise per-frontend channel assignment,
|
||||
shared-engine startup handshakes, frontend failure observation, and parent
|
||||
cleanup without duplicate stage shutdown.
|
||||
|
||||
## Promotion gate
|
||||
|
||||
- Reconcile terminology and ownership after #5441 reaches a final state.
|
||||
|
||||
Reference in new issue
Block a user