Nightly Bagel failures (builds 2892/2918) traced to a companion-output
race: _handle_kv_ready_raw_outputs marked a companion done on kv_ready
alone, before its processed output was stashed, so the parent could pass
the all_companions_done gate and _forward_to_next_stage dispatched with
0/N companion outputs - silently degraded CFG conditioning, and after an
abort a TP rank desync (mismatched all_reduce sizes, NCCL watchdog kill)
in the Bagel text-embedding path.
Lifecycle now enforces one invariant - diffusion dispatch never happens
with an incomplete companion bundle:
- a companion counts as done only after set_companion_output stashes its
processed output (_route_output is the sole done-marking site);
- the forward gate checks completeness non-destructively and re-defers
the parent instead of popping partial state; a bundle that is done but
gone without ever being consumed fails the request instead of
dispatching degraded CFG (was_bundled distinguishes streaming
re-submissions);
- a companion aborted directly (parent not in the batch) fails its
deferred parent immediately via the existing cleanup plumbing.
Deferred to a follow-up (needs a liveness owner in the orchestrator
loop): a deadline on deferred parents and detection of a companion
replica dying silently.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: tzhouam <tzhouam@gmail.com>
