anodyine/vllm-omni
1
0 Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Investigate and fix the TP chroma-speckle corruption upstream (vllm-omni#8135) #3

New Issue
Open
opened 2026-10-08 00:08:21 -04:00 by anodyine · 0 comments
anodyine commented 2026-10-08 00:08:21 -04:00
Owner
Copy link
Copy source

Context

Upstream vllm-omni#8135 reports that Qwen-Image-2.1 with --tensor-parallel-size 2 produces dense multicolored chroma speckles in smooth regions (~4-8% of flat-area pixels), deterministic across seeds and prompts, in both text-to-image and editing. SGLang's independent TP implementation reproduces the identical corruption, pointing at a shared flaw in the sharding recipe rather than either runtime. The issue is open, unassigned, and untriaged; it currently blocks all TP use, which consumers need to fit 2K generation.

This is contribution work, not product work: it does not gate the novella application (our serving path does not depend on TP today). It is the highest-value upstream contribution we have found — a deterministic, seed-invariant, cross-runtime reproduction with a readable suspect list.

Why we are well positioned

  • Four identical RTX 3090s (the 4x3090 host) for rank bisection — TP=2 with two clean single-GPU references, then TP=4.
  • The fixed-seed, PSNR/pixel-diff methodology validated in #1, plus digest-pinned Diffusers baselines.
  • The reproducible environment recipe from #1 (Python 3.13 venv, vllm 0.31.0, local snapshot, offline serving).

Suspects (from the upstream reporter; all sharding-recipe level)

  1. AdaLN modulation MLPs sharded and all-reduced — summing partial scale/shift parameters would corrupt every layer's normalization while preserving structure. Matches the signature; check first.
  2. Per-rank mask/RoPE handling in the joint text+image attention.
  3. Per-head QK-norm under head sharding.

Plan

  1. Power and preflight the 4x3090 host (nvidia-smi topo -m; record P2P/NVLink topology).
  2. Stand up the #1 environment recipe on it; single-GPU reference render (fixed seed) must be clean.
  3. Reproduce at TP=2: same seed; capture PNG, pixel-diff map vs the single-GPU output; confirm determinism.
  4. Bisect: instrument or patch each suspect in turn (start with AdaLN modulation all-reduce placement); a one-line correct placement may clear the speckle.
  5. If root-caused: comment on upstream #8135 with the evidence and, if the fix is clean, prepare it as a PR (GitHub-fork flow per #1's contribution mechanics). Cross-check SGLang's recipe against the finding.
  6. Record the result here either way — a negative result with good instrumentation is also upstream-postable.

Acceptance Criteria

  • TP=2 corruption reproduced (or definitively not) on identical Ampere cards, with artifacts and diffs retained on the host outside Git.
  • Each suspect confirmed or eliminated with evidence.
  • An upstream comment (and PR, if a fix exists) drafted for owner review before posting; nothing posted without owner approval.

Out of scope

  • Making TP part of the helper system's serving path (separate decision, blocked on this bug anyway).
  • SGLang code changes (verification only).
  • Any novella-app dependency on the outcome.
## Context Upstream [vllm-omni#8135](https://github.com/vllm-project/vllm-omni/issues/8135) reports that Qwen-Image-2.1 with `--tensor-parallel-size 2` produces dense multicolored chroma speckles in smooth regions (~4-8% of flat-area pixels), deterministic across seeds and prompts, in both text-to-image and editing. SGLang's independent TP implementation reproduces the identical corruption, pointing at a shared flaw in the sharding recipe rather than either runtime. The issue is open, unassigned, and untriaged; it currently blocks all TP use, which consumers need to fit 2K generation. **This is contribution work, not product work**: it does not gate the novella application (our serving path does not depend on TP today). It is the highest-value upstream contribution we have found — a deterministic, seed-invariant, cross-runtime reproduction with a readable suspect list. ## Why we are well positioned - Four identical RTX 3090s (the 4x3090 host) for rank bisection — TP=2 with two clean single-GPU references, then TP=4. - The fixed-seed, PSNR/pixel-diff methodology validated in #1, plus digest-pinned Diffusers baselines. - The reproducible environment recipe from #1 (Python 3.13 venv, vllm 0.31.0, local snapshot, offline serving). ## Suspects (from the upstream reporter; all sharding-recipe level) 1. AdaLN modulation MLPs sharded and all-reduced — summing partial scale/shift parameters would corrupt every layer's normalization while preserving structure. Matches the signature; check first. 2. Per-rank mask/RoPE handling in the joint text+image attention. 3. Per-head QK-norm under head sharding. ## Plan 1. Power and preflight the 4x3090 host (`nvidia-smi topo -m`; record P2P/NVLink topology). 2. Stand up the #1 environment recipe on it; single-GPU reference render (fixed seed) must be clean. 3. Reproduce at TP=2: same seed; capture PNG, pixel-diff map vs the single-GPU output; confirm determinism. 4. Bisect: instrument or patch each suspect in turn (start with AdaLN modulation all-reduce placement); a one-line correct placement may clear the speckle. 5. If root-caused: comment on upstream #8135 with the evidence and, if the fix is clean, prepare it as a PR (GitHub-fork flow per #1's contribution mechanics). Cross-check SGLang's recipe against the finding. 6. Record the result here either way — a negative result with good instrumentation is also upstream-postable. ## Acceptance Criteria - TP=2 corruption reproduced (or definitively not) on identical Ampere cards, with artifacts and diffs retained on the host outside Git. - Each suspect confirmed or eliminated with evidence. - An upstream comment (and PR, if a fix exists) drafted for owner review before posting; nothing posted without owner approval. ## Out of scope - Making TP part of the helper system's serving path (separate decision, blocked on this bug anyway). - SGLang code changes (verification only). - Any novella-app dependency on the outcome.
anodyine added the type/bugarea/backendstatus/ready-for-agent labels 2026-10-08 00:08:21 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
area/agent-workflow
Agent workflow area
 area/backend
Backend area
 area/docs
Documentation area
 area/frontend
Frontend area
 area/tests
Testing area
 area/workflow
General workflow area
 priority/P0
Exceptional interrupt; not a complete priority taxonomy
 status/done
Work is complete
 status/in-progress
Implementation is in progress
 status/needs-info
Blocked pending more information
 status/needs-triage
Needs initial review and prioritization
 status/ready-for-agent
Ready for an agent to claim
 status/ready-for-human
Ready for human review or action
 status/review
Implementation is ready for review
 status/wontfix
Work will not be pursued
 type/architecture
Architecture or system-design work
 type/bug
Defect or regression
 type/docs
Documentation work
 type/epic
Epic grouping related work
 type/feature
New behavior or capability
 type/refactor
Internal restructuring without behavior change
 type/roadmap
Canonical repository work queue
 type/test
Test coverage or test infrastructure
No labels area/backend status/ready-for-agent type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
anodyine
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anodyine/vllm-omni#3