Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Hongsheng Liu
|
e384cae198 |
No files matched your search
@@ -1,64 +0,0 @@
|
||||
---
|
||||
name: 3d-animation-short-generator
|
||||
description: Develop stylized 3D narrative shorts through story, character and environment references, timed shot tables, storyboards, generation, and assembly. Use for coherent animated storytelling or its planning stages, not isolated image edits or photorealistic live action.
|
||||
metadata:
|
||||
compatibility: File-based agent workflow with sibling h3-prompt-writing. Media generation and editing require available tools or a configured backend.
|
||||
---
|
||||
|
||||
# 3D Animation Short Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
For a complete film, progress from story to references, shot table, text storyboard,
|
||||
clips, and assembly. Keep these as local artifacts, using existing user decisions
|
||||
and authorization. If only a planning stage is requested, deliver that stage.
|
||||
|
||||
## Story and visual contract
|
||||
|
||||
Capture premise, desired emotional outcome, duration, ratio, model, and audio mode.
|
||||
Use warm stylized 3D as a default only when style is unspecified: readable geometric
|
||||
silhouettes, designed hair clumps, tactile materials, soft skin shading where
|
||||
appropriate, expressive brows/eyes, and elastic posing. User-selected 2D or other
|
||||
styles override this preset rather than receiving accidental 3D texture.
|
||||
|
||||
Create an active protagonist with a want, obstacle, meaningful choice, and payoff.
|
||||
Connect the ending to an earlier visual/emotional anchor. Use as many beats as the
|
||||
duration needs; do not pad a tiny story into an eight-beat formula. Dialogue should
|
||||
change a relationship or move the action, and use the user's requested language.
|
||||
|
||||
## Reference design
|
||||
|
||||
- Character references lock name, silhouette, proportions, face, hair, costume,
|
||||
signature props, and color. Label identity in the production document; clean
|
||||
rendering anchors should not carry labels that could leak into the video.
|
||||
- Scene references contain environments only unless the user requested a composite.
|
||||
Record landmarks, lighting direction/time of day, entrances, exits, and prop state.
|
||||
- A reference board with multiple views is for design review. Prefer appropriate
|
||||
clean views for model input so grids and pose sheets do not become final frames.
|
||||
- When an asset changes, update all affected prompts and clips, with a manifest
|
||||
identifying the current revision.
|
||||
|
||||
## Shot planning
|
||||
|
||||
Read [shot-table-spec.md](references/shot-table-spec.md) for the six-column table
|
||||
and continuity checks. Then read
|
||||
[storyboard-guidelines.md](references/storyboard-guidelines.md) for the matching
|
||||
text storyboard and optional pencil previews. The text storyboard is the source
|
||||
of shot timing; preview art does not silently replace it.
|
||||
|
||||
Vary shot size and elastic performance to serve the story. Camera tilts, close-ups,
|
||||
or exaggeration are choices for specific beats, not quotas. Track off-screen
|
||||
characters and spatial landmarks so they do not teleport back into later shots.
|
||||
|
||||
## Generation and assembly
|
||||
|
||||
Before rendering, read [model-selection.md](references/model-selection.md) and
|
||||
[fallback-policy.md](references/fallback-policy.md). Use the chosen model through
|
||||
available tools. Compile prompts using the H3 base or reference guide as appropriate.
|
||||
Remove storyboard-only labels, arrows, panel borders, and sketch styling from
|
||||
rendering inputs. Tie each clip to the current scene and character references.
|
||||
|
||||
Read [qc-checklist.md](references/qc-checklist.md) for assembly and final review.
|
||||
Keep the planned shot order and audio continuity. Use a continuous score when
|
||||
requested, duck it under dialogue/SFX, and do not add subtitles unasked. Deliver
|
||||
the actual film or the requested planning package, with reproducible prompts and
|
||||
any unresolved identity, scene, or timing defects.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "3D Animation Shorts"
|
||||
short_description: "Plan stories, character anchors, shots, and edits"
|
||||
default_prompt: "Use $3d-animation-short-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,21 +0,0 @@
|
||||
# Targeted generation recovery
|
||||
|
||||
Compare the failed result against the shot's actual identity/spatial anchors.
|
||||
Classify the failure: wrong input or API mapping, reference drift, overloaded
|
||||
action, timing, audio, or a render/export failure. Fix transport/configuration
|
||||
errors before spending another generation attempt.
|
||||
|
||||
For drift, quote the relevant anchors and remove unrelated reference roles. For
|
||||
overloaded action, simplify or split the shot while preserving the requested total
|
||||
duration and handoffs. For a scene change, use the new scene reference and avoid
|
||||
conditioning on a previous scene when that would contradict the new setting.
|
||||
|
||||
Use the retry/resource limit from the shared portable workflow. A persistent
|
||||
failure should lead to a concrete choice (simpler action, a supplied reference,
|
||||
an authorized alternate model, or an explicitly incomplete shot), not automatic
|
||||
provider switching or an endless loop. Never present a placeholder as a finished
|
||||
clip or hide an incorrect scene in final assembly.
|
||||
|
||||
For optional pencil previews, reduce annotation or panel count. Use the text
|
||||
storyboard if visualization is not essential; do not rerender correct video
|
||||
solely because a human-review sketch changed.
|
||||
@@ -1,21 +0,0 @@
|
||||
# Model and format selection
|
||||
|
||||
Honor an existing model choice. Otherwise prefer MiniMax H3 when available, and
|
||||
inspect the running backend for partitions, reference modes, audio, resolutions,
|
||||
duration limits, and long-video options. Do not assume hosted and local services
|
||||
have identical limits, quality, prices, or parameter names.
|
||||
|
||||
If another model is requested, use its real documented input contract. A provider
|
||||
switch needs the user's authorization; never silently use a different service
|
||||
because a preset names it as a fallback. For an authorized mixed-model workflow,
|
||||
record model and settings per shot and check cross-shot style differences.
|
||||
|
||||
Keep delivery resolution separate from native generation resolution. Record actual
|
||||
width/height, fps, frame count, and any crop, retiming, or upscaling. Do not claim
|
||||
native 2K when the video was generated at 768p and restored later.
|
||||
|
||||
Compile shot action, camera, identity, scene, and audio into the chosen model's
|
||||
prompt format. For H3 use the bundled writing guides. Strip production labels and
|
||||
preview grids. Prompt for designed silhouettes, elastic action, material and
|
||||
lighting details when these belong to the user's 3D style; do not add text/UI just
|
||||
because a model can generate them.
|
||||
@@ -1,19 +0,0 @@
|
||||
# Assembly and review
|
||||
|
||||
Use current clip revisions in shot-table order. Preserve useful dialogue and SFX;
|
||||
mix a continuous requested score underneath them. Inspect edits at sound bridges
|
||||
and dialogue boundaries, avoiding cut vowels and accidental audio gaps. Only add
|
||||
subtitles, music, or retiming consistent with the brief.
|
||||
|
||||
Review identity, landmarks, screen direction, light/time of day, prop state,
|
||||
emotional payoff, intelligibility, SFX timing, and music balance. Look for leaked
|
||||
storyboard panels, sketch lines, arrows, labels, timecodes, or watermarks. Verify
|
||||
the complete file's duration, dimensions, fps, frame count, audio, and decode.
|
||||
|
||||
Keep planning documents, references, raw clips, and final edits in distinguishable
|
||||
folders. Record the exact assets used in the final edit. After a character, scene,
|
||||
prompt, clip, or score is revised, update its downstream dependents and archive the
|
||||
previous result; do not silently combine stale and current versions.
|
||||
|
||||
Report observed defects and what was repaired, including external editing or
|
||||
upscaling. A successful API response proves completion, not visual correctness.
|
||||
@@ -1,30 +0,0 @@
|
||||
# Six-column shot table
|
||||
|
||||
| Shot ID & Duration | Continuity Handoff | Reference Anchors (Spatial + Identity) | Hook Type | Shot Description (Per-Second Directives) | Audio & Dialogue Track |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
|
||||
Use stable IDs such as S01. Record incoming and outgoing image state, prop
|
||||
position, eyeline, motion, and emotional state in Continuity Handoff.
|
||||
|
||||
Reference Anchors contains named landmarks and screen-relative positions; each
|
||||
character's screen position, facing and pose; recently exited characters' locations;
|
||||
lighting baseline and modifiers; and exact character/scene asset paths or IDs.
|
||||
An explicit scene/time change can replace these anchors; an accidental flip cannot.
|
||||
|
||||
Hook Type names the shot's purpose: setup, reveal, reversal, joke, suspense,
|
||||
tenderness, chase, or callback. Do not force a joke or cut into a quiet scene.
|
||||
|
||||
For each second or meaningful sub-second beat, cover action/pose/expression,
|
||||
camera, spatial/prop state, sound, and the handoff to the next interval. Use compact
|
||||
phrases rather than inventing movement in intentionally still moments. Include
|
||||
anticipation, squash/stretch, overshoot, and follow-through where appropriate.
|
||||
|
||||
Keep the complete dialogue/narration script in Audio & Dialogue Track, with
|
||||
speaker, timing, delivery, and SFX. Distinguish off-screen narration from speaking
|
||||
characters; a visible listener should not lip-sync the narrator's words.
|
||||
|
||||
Before rendering, check full time coverage, supported generation durations, a
|
||||
readable number of active characters, matching scene/identity references, and
|
||||
coherent row-to-row handoffs. Mark intentional cuts/time jumps. Fix contradictory
|
||||
states rather than using a generic continuity assurance. A visual shot may span
|
||||
multiple generation windows when the verified backend supports continuation.
|
||||
@@ -1,29 +0,0 @@
|
||||
# Text storyboard and optional visualization
|
||||
|
||||
Maintain one `storyboard.md` with a section per shot. Each section contains:
|
||||
|
||||
1. Shot ID, title, duration, and narrative purpose.
|
||||
2. Exact scene and character reference IDs.
|
||||
3. Spatial anchors: landmarks, character positions/facing, off-screen characters,
|
||||
key/fill/rim lighting, and important props.
|
||||
4. Incoming and outgoing continuity state.
|
||||
5. Time-ordered beats: pose/expression, camera, audio, and anchor changes.
|
||||
|
||||
Use per-second or finer intervals only where meaningful; cover the complete shot.
|
||||
For narrated moments, specify whether a visible character speaks or only reacts.
|
||||
Optional ASCII staging diagrams can clarify screen direction but are not images
|
||||
to supply as rendering references.
|
||||
|
||||
For heavy revision of a shot, extract its section to a named file and link it from
|
||||
the main document. Choose one authoritative version; reintegrate or update the
|
||||
link after revision so later generation cannot read stale content.
|
||||
|
||||
Only generate pencil storyboards when requested or already authorized as previews.
|
||||
Use equal-sized time-ordered panels, with timecode, pose/expression, camera arrow,
|
||||
and audio/anchor notes. Suggested layouts: three panels in a strip, four in a 2×2
|
||||
grid, six in 2×3. More complex scenes may need fewer representative key poses.
|
||||
Keep shot/character/scene labels readable for human review, but never feed these
|
||||
annotated grids as clean final frames. The text timeline remains authoritative.
|
||||
|
||||
If preview panels fail, simplify annotation or rely on text within the agreed
|
||||
scope. An optional visualization failure need not block a usable text storyboard.
|
||||
@@ -1,53 +0,0 @@
|
||||
# MiniMax H3 skill provenance
|
||||
|
||||
Upstream: https://github.com/MiniMax-AI/MiniMax-H3
|
||||
|
||||
Source snapshot: `d21241f0a4b3acbb34c97dae47fa417b7065e438`, retrieved 2026-09-19.
|
||||
|
||||
Source directory:
|
||||
https://github.com/MiniMax-AI/MiniMax-H3/tree/d21241f0a4b3acbb34c97dae47fa417b7065e438/skills
|
||||
|
||||
The nine upstream skill names are retained. Each local skill corresponds to the
|
||||
same-named upstream `skills/<name>/` directory. Credit for the original creative
|
||||
workflows and templates belongs to MiniMax-AI and the upstream contributors.
|
||||
|
||||
## Files and adaptation
|
||||
|
||||
- `h3-prompt-writing/references/base-format.md` and `ref-format.md` are new,
|
||||
condensed descriptions of the public prompt interface. They preserve necessary
|
||||
field names and notation but do not copy the upstream guides or their examples.
|
||||
The entrypoint adds a portable execution reference and clarifies clip/window
|
||||
duration for verified long-video backends.
|
||||
- The eight style-specific `SKILL.md` files are rewritten portable adaptations of
|
||||
the upstream workflows. Their reference documents retain the relevant creative
|
||||
methods while removing host-specific execution assumptions.
|
||||
- The two `co-op-game-intro-generator/references/` templates retain the upstream
|
||||
creative structure with an added portable contract and explicit user overrides.
|
||||
Their instructions are normalized to English, and the video template expresses
|
||||
visible UI copy as localized variables with English defaults. It also removes
|
||||
an unrequested gender restriction and a fixed yellow/black environment
|
||||
transformation.
|
||||
- The five 3D animation references are condensed adaptations of their upstream
|
||||
counterparts: shot table, storyboards, model selection, fallback, and review.
|
||||
- Product, papercraft, collage, and music references extract reusable direction
|
||||
from their corresponding upstream skill. The shared portable workflow, catalog,
|
||||
and UI metadata are additions for this repository.
|
||||
|
||||
No upstream `LICENSE`, `COPYING`, or `NOTICE` file was present in the Git tree at
|
||||
the pinned snapshot. Its README links to the MiniMax H3 Community License on the
|
||||
model release. Because that license has redistribution conditions, this package
|
||||
does not include the upstream prompt guides verbatim. This provenance record
|
||||
preserves attribution and does not assert that this repository's code license
|
||||
relicenses upstream material.
|
||||
|
||||
## Runtime changes
|
||||
|
||||
Canvas documents become local files and asset manifests; choice-card UI becomes
|
||||
the host's normal clarification mechanism when a decision is actually missing.
|
||||
Available generation/editing tools replace proprietary dispatch calls. Previous
|
||||
user choices and authorization persist across steps. Providers, prices, resolution
|
||||
support, and long-video features are checked rather than hardcoded as guarantees.
|
||||
|
||||
User intent overrides style presets, including instrumental/no-text work, selected
|
||||
language, final morph shape, no-cut sequences, and custom player designs. Native
|
||||
generation and external processing remain separately identified in delivery.
|
||||
@@ -1,96 +0,0 @@
|
||||
# Portable MiniMax H3 Skills
|
||||
|
||||
This directory contains one general prompt-writing skill and eight creative
|
||||
skills adapted from
|
||||
[MiniMax-AI/MiniMax-H3](https://github.com/MiniMax-AI/MiniMax-H3/tree/main/skills).
|
||||
See [H3-SOURCES.md](H3-SOURCES.md) for the source revision, attribution, and
|
||||
adaptation scope.
|
||||
|
||||
## Skill catalog
|
||||
|
||||
| Skill | Intended use |
|
||||
| --- | --- |
|
||||
| [h3-prompt-writing](h3-prompt-writing/SKILL.md) | Structured prompts for T2VA, I2VA, FL2VA, L2VA, and Ref2VA |
|
||||
| [minimalist-product-ad-generator](minimalist-product-ad-generator/SKILL.md) | Minimal product ads, isolated product references, copy, and pacing |
|
||||
| [3d-animation-short-generator](3d-animation-short-generator/SKILL.md) | 3D narrative shorts, character and environment design, shot tables, storyboards, and editing |
|
||||
| [papercraft-stop-motion-explainer](papercraft-stop-motion-explainer/SKILL.md) | Layered paper art, puppets, mechanisms, and miniature educational scenes |
|
||||
| [brand-promo-video-generator](brand-promo-video-generator/SKILL.md) | Brand promos, verified assets and claims, product interaction, and calls to action |
|
||||
| [music-video-subtitle-generator](music-video-subtitle-generator/SKILL.md) | Music videos, instrumental visual narratives, beat planning, and optional lyric typography |
|
||||
| [co-op-game-intro-generator](co-op-game-intro-generator/SKILL.md) | Two-player game menus, confirmation images, and animated world entry |
|
||||
| [paper-collage-explainer-generator](paper-collage-explainer-generator/SKILL.md) | Halftone photo collage, incremental paper assembly, and visual metaphors |
|
||||
| [handdrawn-live-video-generator](handdrawn-live-video-generator/SKILL.md) | Live action with rough luminous drawings, physical contact, continuous morphing, and delayed pursuit |
|
||||
|
||||
## Usage
|
||||
|
||||
Codex discovers repository skills from `.agents/skills/`. Clients that scan
|
||||
skills only when a session starts may need a new session. Claude Code reads the
|
||||
same files through the same-named relative symlinks in `.claude/skills/`. Other
|
||||
agents that support Markdown skills can load the relevant `SKILL.md` directly
|
||||
and follow its links to supporting references.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
Use $music-video-subtitle-generator to design a 30-second two-dimensional
|
||||
fluid-motion graphic for this instrumental track. Do not add lyrics or
|
||||
subtitles. Preserve leftward motion and initially return only a storyboard and
|
||||
three H3 prompts.
|
||||
```
|
||||
|
||||
```text
|
||||
Use $co-op-game-intro-generator to create a two-player game intro. The players
|
||||
are Lin and Mei, the style is layered paper sculpture, and the title is
|
||||
Together. Create the confirmation image first and wait for my approval before
|
||||
generating video.
|
||||
```
|
||||
|
||||
## MV smoke test
|
||||
|
||||
Test `music-video-subtitle-generator` together with `h3-prompt-writing` using a
|
||||
user-owned song with timestamped lyrics and an original adult-character
|
||||
reference image:
|
||||
|
||||
```text
|
||||
Use $music-video-subtitle-generator and $h3-prompt-writing to turn this song
|
||||
into a historical-romance music video about a woman and a man. Intercut
|
||||
narrative acting with the woman's singing performance, do not display lyric
|
||||
text, split the work according to the backend's verified clip duration, and
|
||||
preserve the complete original song in the final master.
|
||||
```
|
||||
|
||||
Verify that the global timeline matches the source audio duration, lyrics remain
|
||||
verbatim, and each generation window records its incoming state, outgoing state,
|
||||
and local timing. Ref2VA prompts must preserve the six-section field order and
|
||||
resolve every reference label to a real input. The assembled video must decode
|
||||
completely, and its final audio must come from the supplied source rather than a
|
||||
model reconstruction.
|
||||
|
||||
When moving the collection to another repository or a user-level skill folder,
|
||||
copy all nine skill directories and preserve their sibling layout. The eight
|
||||
creative skills share `h3-prompt-writing/references/portable-workflow.md` and the
|
||||
H3 format guides, so copying only one `SKILL.md` is insufficient. Keep each
|
||||
skill's `references/` and `agents/` directories with it. `agents/openai.yaml`
|
||||
provides optional Codex UI metadata and does not restrict the workflow to an
|
||||
OpenAI-based agent or require a dedicated plugin.
|
||||
|
||||
## Execution boundaries
|
||||
|
||||
Prompt writing, storyboarding, and planning require only an agent that can read
|
||||
and write files. Actual image, video, speech, music, and editing work uses tools
|
||||
or services available in the current environment. When a capability is missing,
|
||||
deliver the complete usable production package and identify the unfinished media
|
||||
step.
|
||||
|
||||
The user's style, language, assets, audio, model selection, and existing
|
||||
authorization take priority. An MV template does not add lyrics to an
|
||||
instrumental track, and a clean motion-graphics request does not inherit film
|
||||
grain. A prompt-only request does not start generation. When the user requests
|
||||
approval of a confirmation image before rendering, preserve that checkpoint.
|
||||
|
||||
Treat vLLM-Omni long-video extensions as capabilities of the checked-out branch
|
||||
and active backend. Do not present a development branch's continuation parameters
|
||||
as a universal H3 API. Preserve real requests and raw outputs, distinguish model
|
||||
results from post-production and upscaling, and inspect actual media properties.
|
||||
|
||||
These skills contain no model weights and do not automatically start a service
|
||||
or invoke a paid generation endpoint.
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
name: brand-promo-video-generator
|
||||
description: Plan, prompt, or produce brand, product, website, app, and service promotional shorts using traceable assets, accurate claims, timed beats, and clear calls to action. Use for launch and campaign videos, not invented product evidence or long narrative films.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing and available research, media generation, inspection, and editing tools.
|
||||
---
|
||||
|
||||
# Brand Promo Video Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Use supplied brand assets, source links, and existing session decisions. No
|
||||
proprietary tool allowlist or canvas is required. A completed promo requires real
|
||||
generation/editing capability; otherwise deliver an explicitly incomplete production
|
||||
package with finished prompts and a concrete generation handoff.
|
||||
|
||||
## Establish facts and assets
|
||||
|
||||
Inspect product images, UI screenshots, logo exports, typography, colors, and
|
||||
campaign material. Resolve duration, ratio, audience, campaign focus, exact claims,
|
||||
CTA, copy language, narration, and required deliverable from context. Ask for
|
||||
missing identity-bearing assets when they are essential; do not fabricate logos,
|
||||
product UI, screenshots, results, or performance claims.
|
||||
|
||||
Build a compact truth sheet from user-supplied originals and official sources.
|
||||
Record asset ID, role, local path, source URL/note, and verification status. Do not
|
||||
confuse a retrieved search thumbnail with an original brand asset. Distinguish
|
||||
verified facts from clearly labeled concept copy. Preserve the requested language;
|
||||
chat language alone does not justify translating supplied campaign text.
|
||||
|
||||
## Design a product-specific story
|
||||
|
||||
Choose a spine that exposes real cause and effect:
|
||||
|
||||
- Software: user intent → actual interaction → useful output → evidence → CTA.
|
||||
- Physical product: reveal → interaction → mechanism/detail → use → benefit → CTA.
|
||||
- Service: situation → process → evidence → outcome → promise.
|
||||
- Image-led brand: authentic imagery → repeated visual motif → emotional payoff.
|
||||
|
||||
Plan time/frame ranges, asset IDs, one main action, product evidence, exact copy,
|
||||
readable hold, color state, camera, transition, and rhythm. Use the actual output
|
||||
fps for frame timing. Five to eight major beats can fit a 15-second promo, but
|
||||
readability takes priority over a fixed beat count.
|
||||
|
||||
Let product edges, UI movement, light, cursor paths, or matched geometry motivate
|
||||
transitions. Use a few energy peaks and deliberate settling moments. Preserve logo
|
||||
clear space and readable screenshots. Avoid invented HUDs or metrics that look like
|
||||
product proof. Summarize the truth sheet and plan before rendering; continue under
|
||||
existing authorization unless the user requested approval at that stage.
|
||||
|
||||
## Produce and inspect
|
||||
|
||||
Use authentic still/UI plates for exact content. Generated material can supply
|
||||
conceptual scenes, atmosphere, and transition geometry with clear provenance.
|
||||
Compile actual H3 references and shots using the appropriate bundled guide.
|
||||
Check backend resolution, duration, and audio support before calling it.
|
||||
|
||||
Use native instrumental/SFX direction when appropriate; do not generate a second
|
||||
score automatically. Separate narration or music is useful when requested for
|
||||
control or remixing. Preserve original SFX during assembly when they matter, and
|
||||
duck music under speech. Do not add subtitles unless requested.
|
||||
|
||||
Check claims and exact product names against sources, inspect logo geometry and
|
||||
copy, verify real UI states, and review transitions, aspect ratio, duration, audio,
|
||||
and decoding. For wrong identity-bearing content, use the correct supplied/source
|
||||
asset rather than polishing an imitation. Deliver the video, prompts/timeline,
|
||||
asset manifest, and any remaining factual or visual limitations.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Brand Promo Videos"
|
||||
short_description: "Build brand promos from verified assets and claims"
|
||||
default_prompt: "Use $brand-promo-video-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
name: co-op-game-intro-generator
|
||||
description: Design or generate a two-player co-op game menu and opening animation from player names, title, style, and optional identity references. Use for character-led game intros, not playable game implementation or complex multi-page interfaces.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing and bundled image/video templates. Rendering uses available generation tools.
|
||||
---
|
||||
|
||||
# Co-op Game Intro Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Before writing a confirmation-image prompt, read
|
||||
[h3-confirmation-image-template.md](references/h3-confirmation-image-template.md).
|
||||
Before writing the video prompt, read
|
||||
[h3-video-prompt-template.md](references/h3-video-prompt-template.md).
|
||||
Both templates are required package resources, not remote tools.
|
||||
|
||||
## Resolve player and style bindings
|
||||
|
||||
Reuse or collect the two player names, game title, visual style, desired duration,
|
||||
ratio, and optional identity references. The template defaults to a 15-second,
|
||||
16:9 menu-to-world sequence; adapt it to explicit user requirements. Do not assume
|
||||
the players' gender, body shape, outfits, or equipment from a generic example.
|
||||
|
||||
Use each portrait only for recognizable identity anchors: face silhouette, hair,
|
||||
glasses, proportions, and distinguishing traits. Reinterpret rendering, skin
|
||||
texture, lighting, and costume in the selected visual style. If a portrait or body
|
||||
comparison reference is absent, write a text design rather than inventing an asset
|
||||
label. Maintain player-to-name mapping across all references and frames.
|
||||
|
||||
## Confirmation image
|
||||
|
||||
Fill the template's style, palette, characters, title, and name variables. Keep
|
||||
the two characters central, player cards toward the upper left, menu on the right,
|
||||
and the continue action as the main interaction focus. The title establishes
|
||||
identity without competing with that focus. Buttons, icons, typography, and all
|
||||
player/equipment panels share one palette and style. Keep menu labels on one line.
|
||||
|
||||
The style changes material, shapes, fonts, and decoration; it should not randomly
|
||||
rearrange the menu hierarchy. User-requested layout or ratio changes take priority
|
||||
over template defaults. Preserve the exact supplied title/names and requested copy.
|
||||
|
||||
For image-only/prompt-only work, stop at that deliverable. For video production,
|
||||
generate and inspect the confirmation image if within scope. Honor a request to
|
||||
approve the image first; otherwise use the established end-to-end authorization
|
||||
and continue after reporting the selected image and fixing material defects.
|
||||
|
||||
## Video event plan
|
||||
|
||||
Fill the video template from the current image and player bindings. Its baseline
|
||||
sequence is menu → first player's light equipment → second player's heavy equipment
|
||||
→ shared confirmation → world loading → both enter the world. Equipment, world,
|
||||
body contrast, and timing are editable design choices; preserve requested identity
|
||||
and style instead of forcing a cyberpunk example onto another setting.
|
||||
|
||||
Bind UI reference, each actual portrait, and any real body-comparison image to
|
||||
distinct roles. Compile this creative template into the H3 mode guide's fields;
|
||||
the mixed-language template is not itself an API request. Preserve literal visible
|
||||
UI text in its required language, and remove all unfilled variables before dispatch.
|
||||
Do not treat a composition reference as an exact first-frame anchor unless the
|
||||
selected backend actually supports that role.
|
||||
|
||||
## Review and repair
|
||||
|
||||
Check both names, exact menu strings, layout, player identity, side assignment,
|
||||
equipment distinction, and the transition into the requested world. Match UI
|
||||
hover/click, mechanical locks, loading, and footsteps to the visible events.
|
||||
For identity swaps, strengthen name/position/reference bindings; for weak style,
|
||||
revise palette, material, character, and typography together. Reduce excessive
|
||||
UI text if necessary without silently changing required player names.
|
||||
|
||||
Deliver the requested prompts/assets or the inspected intro video, with reference
|
||||
roles and any text/timing limitations. Missing templates block template execution;
|
||||
missing generation capability still permits a complete prompt handoff.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Co-op Game Intros"
|
||||
short_description: "Design two-player menus and world-entry animation"
|
||||
default_prompt: "Use $co-op-game-intro-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
-397
@@ -1,397 +0,0 @@
|
||||
# H3 Confirmation Image Template
|
||||
|
||||
## Portable adaptation
|
||||
|
||||
This is a creative template, not a tool call. User requirements override its
|
||||
default ratio, layout, poses, and wording. Fill every relevant variable from the
|
||||
brief; set `{character_count}` to two for this skill. When portraits are absent,
|
||||
replace portrait references with explicit character designs. Do not claim to
|
||||
reference an image that was not supplied. Use the image-generation capability
|
||||
available in the host environment; no specific image provider is required.
|
||||
|
||||
## Core Rule
|
||||
Generate a polished 16:9 landscape confirmation image for a cooperative game main menu.
|
||||
The image consists of two independent systems:
|
||||
• UI Framework (fixed)
|
||||
• Visual Style (dynamic)
|
||||
The UI Framework defines the layout, composition, information hierarchy, interaction logic, reading flow, menu structure, and readability. These elements must remain unchanged.
|
||||
{visual_style} only controls the artistic appearance, including rendering medium, materials, color language, lighting mood, environment style, character design, costume language, UI materials, icon language, typography appearance, decorative motifs, and overall atmosphere.
|
||||
Changing {visual_style} should feel like applying a different art direction to the same professionally designed game menu, never redesigning the interface itself.
|
||||
If any default description conflicts with {visual_style}, prioritize {visual_style} while preserving the UI framework.
|
||||
|
||||
---
|
||||
|
||||
## Design Goal
|
||||
Create a premium commercial console game main menu mockup in 16:9 landscape format. The image should combine the quality of a polished game promotional artwork with the clarity of a professional game UI. Characters and interface must feel naturally integrated, with strong visual hierarchy, excellent readability, clean composition, and a clear interaction focus.
|
||||
|
||||
---
|
||||
|
||||
## Overall Style
|
||||
Follow the user-selected visual style: {visual_style}.
|
||||
The selected style controls the rendering medium, illustration style, materials, palette language, environment, character appearance, outfit design, UI appearance, icon language, typography treatment, decorative elements, and atmosphere.
|
||||
All visual elements must belong to one unified artistic language.
|
||||
Do not mix incompatible visual styles within the same image.
|
||||
The UI framework, layout, hierarchy, composition, and interaction logic must remain consistent regardless of style.
|
||||
|
||||
---
|
||||
|
||||
## Color Palette
|
||||
Derive the entire color system from {visual_style}.
|
||||
Use:
|
||||
• {main_background_color} as the primary background color.
|
||||
• {ui_body_color} as the main UI color.
|
||||
• {text_color} as the primary text color.
|
||||
• {functional_accent_color} as the interaction highlight color.
|
||||
• Red only for warning, danger, or exit actions.
|
||||
Limit the overall palette to five primary colors.
|
||||
All buttons, player cards, icons, borders, highlights, decorative elements, typography, and visual effects must follow the same color system.
|
||||
Avoid introducing unrelated colors outside the selected palette.
|
||||
|
||||
---
|
||||
|
||||
## Composition
|
||||
Use a fixed 16:9 landscape layout.
|
||||
Place {character_count} playable characters in the center as the visual subject.
|
||||
The UI surrounds the characters without blocking them.
|
||||
Place the player information cards in the upper-left corner.
|
||||
Place the main menu vertically on the right.
|
||||
Place a decorative horizontal strip along the bottom, adapting naturally to {visual_style} (such as ribbon, warning strip, wooden plank, stone slab, energy bar, scroll, etc.).
|
||||
Keep a clear Z-shaped reading flow:
|
||||
Player Cards → Characters → Menu → Continue Button.
|
||||
The Continue button must always remain the primary visual focus.
|
||||
Maintain generous negative space and excellent readability.
|
||||
|
||||
---
|
||||
|
||||
## Background
|
||||
Generate a background naturally derived from {visual_style}.
|
||||
The background supports the interface rather than competing with it.
|
||||
Keep enough clean negative space behind all important UI elements.
|
||||
Use subtle textures, environmental details, and decorative motifs only where appropriate for the selected style.
|
||||
Avoid excessive visual noise, dense decoration, heavy dirt, scratches, paint splatter, or distracting textures.
|
||||
The overall background should feel polished, premium, modern, and suitable for a commercial game menu.
|
||||
|
||||
---
|
||||
|
||||
## Character Style
|
||||
All characters strictly follow {visual_style}.
|
||||
The selected style determines:
|
||||
• rendering medium
|
||||
• body proportion
|
||||
• facial stylization
|
||||
• costume language
|
||||
• material treatment
|
||||
• lighting response
|
||||
• animation style
|
||||
• overall character quality
|
||||
Only the character identity anchors and pose remain fixed. Facial rendering, eye design, nose simplification, mouth shape, skin treatment, expression style, and head proportion must be fully reinterpreted in {visual_style}.
|
||||
No default character style may override {visual_style}.
|
||||
|
||||
---
|
||||
|
||||
## Character A / PLAYER 1
|
||||
### Identity
|
||||
Use character reference image 1 only for identity mapping.
|
||||
The reference image donates identity anchors only: recognizable face silhouette, hairstyle, glasses if present, relative facial proportions, and distinctive personal traits.
|
||||
It does not donate photographic realism, real skin texture, real-world lighting, camera quality, or the original image style.
|
||||
Do not lose the character's identity anchors, but the face must be redesigned into {visual_style}.
|
||||
All facial rendering, eye design, nose simplification, mouth shape, skin treatment, expression style, and head proportion must follow {visual_style}.
|
||||
Do not swap identity with other characters.
|
||||
|
||||
### Appearance
|
||||
Character rendering, expression, outfit, materials, and accessories are fully derived from {visual_style} while maintaining a recognizable playable-game-character silhouette.
|
||||
|
||||
### Action
|
||||
Character A sits cross-legged on the ground.
|
||||
Both hands support the body on the floor.
|
||||
The upper body leans slightly backward.
|
||||
Character A looks upward toward the camera.
|
||||
|
||||
---
|
||||
|
||||
## Character B / PLAYER 2
|
||||
### Identity
|
||||
Use character reference image 2 only for identity mapping.
|
||||
The reference image donates identity anchors only: recognizable face silhouette, hairstyle, glasses if present, relative facial proportions, and distinctive personal traits.
|
||||
It does not donate photographic realism, real skin texture, real-world lighting, camera quality, or the original image style.
|
||||
Do not lose the character's identity anchors, but the face must be redesigned into {visual_style}.
|
||||
All facial rendering, eye design, nose simplification, mouth shape, skin treatment, expression style, and head proportion must follow {visual_style}.
|
||||
Do not swap identity with other characters.
|
||||
|
||||
### Appearance
|
||||
Character rendering, expression, outfit, materials, and accessories are fully derived from {visual_style} while maintaining a recognizable playable-game-character silhouette.
|
||||
|
||||
### Action
|
||||
Character B sits cross-legged on the ground.
|
||||
Both hands rest naturally in front of the legs.
|
||||
The upper body leans slightly forward.
|
||||
Character B looks directly toward the camera.
|
||||
|
||||
---
|
||||
|
||||
## Lighting
|
||||
Lighting follows {visual_style} while preserving clear readability.
|
||||
Use a primary top light, secondary fill light, soft ambient bounce, contact shadows, gentle rim lighting, and high-quality global illumination where appropriate.
|
||||
Faces should remain clearly visible.
|
||||
Characters must integrate naturally into the environment while maintaining strong separation from the background.
|
||||
|
||||
---
|
||||
|
||||
## Game UI
|
||||
Create a professional commercial console game main menu.
|
||||
The menu structure, interaction hierarchy, spacing, composition, reading flow, and information architecture remain fixed.
|
||||
{visual_style} only changes the artistic appearance of the interface.
|
||||
The selected style may affect:
|
||||
• UI materials
|
||||
• surface textures
|
||||
• border treatment
|
||||
• corner shape
|
||||
• decorative motifs
|
||||
• color rendering
|
||||
• lighting treatment
|
||||
• icon language
|
||||
• typography appearance
|
||||
• interaction effects
|
||||
It must NOT change:
|
||||
• menu hierarchy
|
||||
• button order
|
||||
• button placement
|
||||
• player-card placement
|
||||
• composition
|
||||
• interaction logic
|
||||
• information architecture
|
||||
• readability
|
||||
The entire interface should feel like one professionally designed game UI system rather than separate elements assembled together.
|
||||
|
||||
---
|
||||
|
||||
## UI Visual Language
|
||||
Derive the complete UI appearance from {visual_style}.
|
||||
The selected style determines:
|
||||
• button materials
|
||||
• border language
|
||||
• panel materials
|
||||
• decorative details
|
||||
• edge treatment
|
||||
• visual weight
|
||||
• highlight style
|
||||
• interaction feedback
|
||||
• shadow treatment
|
||||
• texture language
|
||||
Possible materials include wood, stone, paper, leather, cloth, glass, crystal, hologram, neon, metal, ceramic, carved surfaces, painted surfaces, magical energy, pixel blocks, watercolor paper, ink brush, comic graphics or any other material naturally belonging to {visual_style}.
|
||||
Every UI element must share the same material language and artistic style.
|
||||
|
||||
---
|
||||
|
||||
## Button Layout Rules
|
||||
Place all menu buttons vertically on the right side.
|
||||
Maintain a consistent vertical rhythm.
|
||||
Every button uses the same design family.
|
||||
Button width automatically adapts to text length.
|
||||
Keep all menu titles on a single line.
|
||||
No automatic wrapping.
|
||||
No stacked text.
|
||||
Text remains horizontally centered.
|
||||
Keep consistent padding.
|
||||
Maintain equal spacing between buttons.
|
||||
Avoid visual clutter.
|
||||
|
||||
---
|
||||
|
||||
## Continue Button
|
||||
The Continue button is always the primary visual focus.
|
||||
Make it visually dominant using the emphasis method naturally belonging to {visual_style}.
|
||||
Possible emphasis methods include:
|
||||
• larger size
|
||||
• stronger contrast
|
||||
• brighter color
|
||||
• unique material
|
||||
• distinctive border
|
||||
• lighting
|
||||
• glow
|
||||
• engraving
|
||||
• magical energy
|
||||
• holographic effect
|
||||
• embossed frame
|
||||
• animated visual cue
|
||||
Do not force glow or neon if they conflict with {visual_style}.
|
||||
The Continue button should immediately attract the player's attention.
|
||||
|
||||
---
|
||||
|
||||
## Start New Game Button
|
||||
Place above Continue.
|
||||
Use the standard secondary button appearance.
|
||||
Keep the visual language consistent with the selected style.
|
||||
Add one small style-appropriate icon if suitable.
|
||||
Maintain single-line text.
|
||||
|
||||
---
|
||||
|
||||
## Settings Button
|
||||
Use the standard secondary button appearance.
|
||||
Use one settings icon naturally derived from {visual_style}.
|
||||
Maintain consistent proportions, spacing, and hierarchy.
|
||||
|
||||
---
|
||||
|
||||
## Exit Game Button
|
||||
Use the standard secondary button appearance.
|
||||
Use red only as the danger indicator.
|
||||
Red should remain an accent rather than the dominant color.
|
||||
Add one style-appropriate exit icon if suitable.
|
||||
|
||||
---
|
||||
|
||||
## Player Cards
|
||||
Create an upper-left player information system for {character_count} players.
|
||||
The appearance follows {visual_style}.
|
||||
Card materials, borders, corners, textures, shadows, decorations, and icons are automatically derived from the selected style.
|
||||
Each player card contains:
|
||||
• Player label
|
||||
• Player nickname
|
||||
• READY status
|
||||
Player names:
|
||||
PLAYER 1
|
||||
{player1_name}
|
||||
PLAYER 2
|
||||
{player2_name}
|
||||
READY always uses the functional accent color.
|
||||
The cards should remain highly readable and visually connected to the rest of the interface.
|
||||
|
||||
---
|
||||
|
||||
## Icon System
|
||||
Derive all icons from {visual_style}.
|
||||
Use one unified icon family throughout the entire interface.
|
||||
Possible icon languages include:
|
||||
• minimal outline
|
||||
• solid filled
|
||||
• geometric
|
||||
• comic
|
||||
• pixel
|
||||
• engraved
|
||||
• carved
|
||||
• painted
|
||||
• brush
|
||||
• rune
|
||||
• glyph
|
||||
• holographic
|
||||
• crystal
|
||||
• paper-cut
|
||||
• origami
|
||||
• embossed
|
||||
• low-poly
|
||||
• flat design
|
||||
• stylized 3D
|
||||
All icons must share:
|
||||
• the same artistic style
|
||||
• the same material language
|
||||
• the same perspective
|
||||
• the same lighting treatment
|
||||
• the same stroke language
|
||||
• the same color hierarchy
|
||||
Icons should remain simple, recognizable, and functional.
|
||||
Each interface block may contain at most one horizontal row of icons.
|
||||
No wrapping.
|
||||
No vertical stacking.
|
||||
Icons should support the interface rather than dominate it.
|
||||
|
||||
---
|
||||
|
||||
## Typography
|
||||
Typography hierarchy remains fixed.
|
||||
Typography appearance follows {visual_style}.
|
||||
The selected style controls:
|
||||
• font family
|
||||
• decorative treatment
|
||||
• texture
|
||||
• material
|
||||
• outline
|
||||
• lighting
|
||||
• shadow
|
||||
• wear pattern
|
||||
Maintain:
|
||||
• excellent readability
|
||||
• strong visual hierarchy
|
||||
• single-line menu buttons
|
||||
• consistent spacing
|
||||
• high contrast
|
||||
Button width should always adapt to text length.
|
||||
Never allow menu titles to wrap onto multiple lines.
|
||||
|
||||
---
|
||||
|
||||
## Game Title
|
||||
Display the game title:
|
||||
{game_title}
|
||||
The title should become the primary branding element of the menu.
|
||||
Its material, decorative style, typography, and rendering all follow {visual_style}.
|
||||
Avoid imitating existing commercial game logos.
|
||||
Create an original title treatment inspired by the selected style.
|
||||
Keep the title highly readable.
|
||||
Avoid random symbols or meaningless letters.
|
||||
|
||||
---
|
||||
|
||||
## Interaction States
|
||||
Represent the interface as an active game menu.
|
||||
Buttons may display appropriate interaction feedback naturally belonging to {visual_style}.
|
||||
Possible interaction feedback includes:
|
||||
• hover
|
||||
• pressed state
|
||||
• magical glow
|
||||
• energy pulse
|
||||
• holographic highlight
|
||||
• carved illumination
|
||||
• ink spread
|
||||
• brush bloom
|
||||
• metallic reflection
|
||||
• paper emboss
|
||||
• animated border
|
||||
• pixel flashing
|
||||
Only use interaction effects that naturally belong to the selected style.
|
||||
Avoid forcing modern UI effects into incompatible artistic styles.
|
||||
|
||||
---
|
||||
|
||||
## Rendering
|
||||
Generate a polished AAA-quality commercial game menu artwork.
|
||||
Characters, environment, UI, typography, icons, and decorative elements should feel like one cohesive production.
|
||||
Maintain premium rendering quality.
|
||||
Strong visual hierarchy.
|
||||
Professional commercial presentation.
|
||||
High readability.
|
||||
Balanced composition.
|
||||
Clean finish.
|
||||
|
||||
---
|
||||
|
||||
## Negative Constraints
|
||||
No split screen.
|
||||
No duplicated characters.
|
||||
No identity swap.
|
||||
No loss of identity anchors. No photorealistic face unless {visual_style} explicitly requires photorealism. No realistic photo face pasted onto a stylized body. No mismatched face style.
|
||||
No hairstyle changes.
|
||||
No unreadable UI.
|
||||
No distorted buttons.
|
||||
No overlapping interface.
|
||||
No random text.
|
||||
No random symbols.
|
||||
No watermark.
|
||||
No copied game logo.
|
||||
No branded interface.
|
||||
No inconsistent icon styles.
|
||||
No mixed UI materials.
|
||||
No excessive decoration.
|
||||
No overwhelming background.
|
||||
No visual clutter.
|
||||
No broken layout.
|
||||
No low readability.
|
||||
|
||||
---
|
||||
|
||||
## Medium Lock
|
||||
Render as a polished commercial game key art combined with a professional console game main menu mockup.
|
||||
The interface remains fixed.
|
||||
Only the artistic appearance changes according to {visual_style}.
|
||||
@@ -1,249 +0,0 @@
|
||||
# H3 Video Prompt Template
|
||||
|
||||
## Portable adaptation
|
||||
|
||||
This is a creative planning template. User requirements override its default
|
||||
15-second timeline, body proportions, equipment, environment, ratio, and palette.
|
||||
Retiming must preserve readable event order. Use only supplied reference assets:
|
||||
omit `{height_ref}` or identity-image bindings when absent and replace their role
|
||||
with text design constraints. Compile the filled plan into the H3 base or Ref2VA
|
||||
format before rendering; preserve exact visible UI strings in the user's language.
|
||||
|
||||
Use available tools and the actual backend's parameter schema. Audio directions
|
||||
describe desired content rather than asserting that a particular API supports it.
|
||||
|
||||
## Template completion
|
||||
|
||||
Resolve the palette variables `{background_color}`, `{ui_color}`, `{text_color}`,
|
||||
`{functional_accent_color}`, and `{power_accent_color}` from the selected style.
|
||||
Keep the five values distinct enough for readable UI contrast. Use the power
|
||||
accent for danger, exit, and warning states unless the user supplies a separate
|
||||
semantic-color requirement.
|
||||
|
||||
Fill visible copy in the user's requested language. The following English values
|
||||
are defaults only, not mandatory output:
|
||||
|
||||
- `{player1_label}` / `{player2_label}`: `PLAYER 1` / `PLAYER 2`
|
||||
- `{ready_label}`: `READY`
|
||||
- `{start_label}`, `{continue_label}`, `{settings_label}`, `{exit_label}`:
|
||||
`START NEW GAME`, `CONTINUE`, `SETTINGS`, `EXIT GAME`
|
||||
- `{player1_equipment_label}`: `RIGHT ARM EQUIPMENT`
|
||||
- `{player1_option_a}` / `{player1_option_b}`: `PHANTOM GRIP` / `CHRONOS CLAW`
|
||||
- `{player2_equipment_label}`: `ARMAMENT CUSTOMIZATION`
|
||||
- `{hand_label}`, `{forearm_label}`, `{elbow_label}`, `{upper_arm_label}`:
|
||||
`HAND`, `FOREARM`, `ELBOW`, `UPPER ARM`
|
||||
- `{confirm_label}` / `{loading_label}`: `CONFIRM CONFIG` / `LOADING`
|
||||
|
||||
Before compiling the rendering prompt, search the filled template for every
|
||||
remaining token enclosed in braces. Replace required values. For an absent optional
|
||||
reference, remove its binding and rewrite every dependent clause as a text-only
|
||||
constraint. Do not dispatch a prompt while any template token remains.
|
||||
|
||||
## Prompt principle
|
||||
Use the same method as the GPT confirmation-image prompt:
|
||||
|
||||
**Fixed video event framework + dynamic user-style fill + locked character identity + palette-linked UI system.**
|
||||
|
||||
The video prompt must not blindly preserve the source prompt's default style. It must preserve the timeline, UI events, player positions, equipment logic, and text structure, while dynamically rewriting visual treatment, palette, character rendering, lighting, UI surface, icons, and city-world style according to the user's selected style.
|
||||
|
||||
## Priority order
|
||||
1. User-selected style: {visual_style}
|
||||
2. Confirmed image / UI reference: {ui_ref}
|
||||
3. PLAYER 1 identity reference: {player1_ref}
|
||||
4. PLAYER 2 identity reference: {player2_ref}
|
||||
5. Height/body comparison reference: {height_ref}
|
||||
6. Fixed source video event framework
|
||||
|
||||
## Reference roles
|
||||
- {ui_ref}: confirmed first image. Use it to lock UI layout, menu hierarchy, color system, typography scale, button structure, character-game integration, and overall composition logic.
|
||||
- {player1_ref}: PLAYER 1 identity anchor. Lock exact face, hairstyle, glasses if present, facial proportions, body identity, and nickname mapping to {player1_name}.
|
||||
- {player2_ref}: PLAYER 2 identity anchor. Lock exact face, hairstyle, facial proportions, body identity, and nickname mapping to {player2_name}.
|
||||
- {height_ref}: body comparison anchor. Lock the visible contrast between the two players and prevent identical body proportions.
|
||||
|
||||
## Global style baseline
|
||||
The overall art direction must prioritize the user's selected {visual_style}.
|
||||
Preserve these fixed qualities: a game main-menu interface, premium game-trailer
|
||||
finish, deep integration between the UI and characters, modern commercial game
|
||||
UI design, strong visual impact, clean composition, and restrained decoration.
|
||||
Derive the remaining visual treatment from {visual_style}: character rendering,
|
||||
expressions, costume language, color system, lighting temperature, UI materials,
|
||||
button icons, typography texture, and the world revealed after loading.
|
||||
|
||||
## Palette system
|
||||
Derive the complete video palette from {visual_style}, with linked UI colors:
|
||||
|
||||
- Use {background_color} as the primary background and world color.
|
||||
- Use {ui_color} as the main UI color.
|
||||
- Use {text_color} as the text color.
|
||||
- Use {functional_accent_color} as PLAYER 1's functional accent color.
|
||||
- Use {power_accent_color} as PLAYER 2's power accent and for danger, exit,
|
||||
and warning states.
|
||||
- Limit the complete palette to five colors.
|
||||
- Use a vivid, modern, high-contrast color language consistent with
|
||||
{visual_style}.
|
||||
|
||||
Menus, equipment panels, player cards, buttons, the HUD, loading bars, icons,
|
||||
and text must share this color system. Do not introduce unrelated colors.
|
||||
|
||||
## Character identity and style lock
|
||||
PLAYER 1 follows the facial identity in {player1_ref}. Preserve the face,
|
||||
facial proportions, hairstyle, glasses when present, personal identity, and the
|
||||
mapping to the nickname {player1_name}. Adapt expression, rendering, clothing,
|
||||
and mechanical equipment to {visual_style}. PLAYER 1 always stays on the left,
|
||||
with a taller, slender, agile build. Equipment uses the functional accent color,
|
||||
and the mechanical claw is light, narrow, and flexible.
|
||||
|
||||
PLAYER 2 follows the facial identity in {player2_ref}. Preserve the face,
|
||||
facial proportions, hairstyle, personal identity, and the mapping to the
|
||||
nickname {player2_name}. Adapt expression, rendering, clothing, and mechanical
|
||||
equipment to {visual_style}. PLAYER 2 always stays on the right, with a shorter,
|
||||
broader, power-oriented build. Equipment uses {power_accent_color}, and the
|
||||
mechanical fist is broad, heavy, and visibly weighty.
|
||||
|
||||
Do not swap character identities, merge faces, swap nicknames, or converge the
|
||||
two body types.
|
||||
|
||||
## Fixed timeline framework
|
||||
|
||||
### [0-2 seconds] - Two-player main menu
|
||||
|
||||
Composition and camera: A high-angle extreme wide shot follows the composition
|
||||
logic of {ui_ref}. The camera tilts slightly downward and slowly pushes in.
|
||||
|
||||
Visuals: PLAYER 1 ({player1_name}) and PLAYER 2 ({player2_name}) sit side by
|
||||
side at the center. PLAYER 1 remains on the left and PLAYER 2 on the right. They
|
||||
look up toward the camera with only natural breathing, blinking, and subtle body
|
||||
movement.
|
||||
|
||||
UI: The upper-left player card displays `{player1_label}`, `{player1_name}`, and
|
||||
`{ready_label}` exactly as filled. The second card in the upper-center or
|
||||
upper-left two-card system displays `{player2_label}`, `{player2_name}`, and
|
||||
`{ready_label}` exactly as filled. The vertical menu on the right displays
|
||||
`{start_label}`, `{continue_label}`, `{settings_label}`, and `{exit_label}`
|
||||
exactly as filled. `{continue_label}` is the visual center and primary
|
||||
highlighted button.
|
||||
|
||||
Dynamic style fill: Adapt the menu background, ground texture, button shapes,
|
||||
icons, typography, borders, glow, and sticker treatment to {visual_style}, while
|
||||
preserving the layout and hierarchy in {ui_ref}.
|
||||
|
||||
Sound: Menu ambience, subtle UI-hover sounds, and a low electronic atmosphere
|
||||
before the click. Ignore this direction when generating silent video.
|
||||
|
||||
### [2-4 seconds] - PLAYER 1 right-arm configuration
|
||||
|
||||
Composition and camera: In a medium shot, the camera pushes smoothly from the
|
||||
main menu toward PLAYER 1's right arm. PLAYER 2 remains visible and stable in the
|
||||
background.
|
||||
|
||||
UI: The right-side menu contracts and slides away. A panel with functional-accent
|
||||
guide lines slides in from the left and displays `{player1_label}` and
|
||||
`{player1_equipment_label}` exactly as filled. The equipment list first
|
||||
highlights `{player1_option_a}`, then moves the selection to
|
||||
`{player1_option_b}`.
|
||||
|
||||
Action: PLAYER 1's right cuff opens automatically and a lightweight mechanism
|
||||
unfolds beneath the forearm. The fingers spread as long claw-like joints slide
|
||||
into position and lock one by one, briefly revealing fine wiring, miniature
|
||||
pistons, and metal connectors. Functional-accent LEDs illuminate sequentially
|
||||
when configuration completes.
|
||||
|
||||
Dynamic style fill: Adapt the mechanism, panel material, icons, lines, locking
|
||||
animation, and light effects to {visual_style}. Keep the result light, precise,
|
||||
and flexible, proportioned to PLAYER 1 without changing the face, hairstyle, or
|
||||
main clothing design.
|
||||
|
||||
Sound: Precise mechanical unfolding, light UI switching tones, and small locking
|
||||
clicks.
|
||||
|
||||
### [4-7 seconds] - PLAYER 2 heavy-arm configuration
|
||||
|
||||
Composition and camera: A medium shot tracks smoothly between the players and
|
||||
arcs toward PLAYER 2's left side. PLAYER 1 remains in the background, quietly
|
||||
examining the configured mechanical hand.
|
||||
|
||||
UI: A {power_accent_color} panel slides in and displays `{player2_label}` and
|
||||
`{player2_equipment_label}` exactly as filled. Its grid contains `{hand_label}`,
|
||||
`{forearm_label}`, `{elbow_label}`, and `{upper_arm_label}`, with the selection
|
||||
moving quickly but legibly through all four.
|
||||
|
||||
Action: PLAYER 2's left sleeve opens in sections. Heavy forearm plates spring
|
||||
outward, the previous components detach, and new armor slides into place along
|
||||
guide rails. A thick mechanical bearing replaces the elbow joint, and the broad
|
||||
mechanical hand reassembles and locks. Thick wiring, hydraulic pistons, and a
|
||||
dark metal skeleton appear briefly during the replacement. A restrained warm
|
||||
indicator illuminates as each component locks.
|
||||
|
||||
Dynamic style fill: Adapt the heavy arm, UI panel, component icons, materials,
|
||||
and light effects to {visual_style}. Keep it broad, weighty, and powerful, in
|
||||
clear contrast with PLAYER 1's lightweight claw.
|
||||
|
||||
Sound: Low motor movement, heavy mechanical engagement, and weighty locking
|
||||
feedback.
|
||||
|
||||
### [7-8.5 seconds] - Shared configuration confirmation
|
||||
|
||||
Composition and camera: Pull back smoothly to a medium two-player composition,
|
||||
with PLAYER 1 on the left and PLAYER 2 on the right.
|
||||
|
||||
UI: The two equipment panels converge at center to form a shared button that
|
||||
displays `{confirm_label}` exactly as filled. Adapt its border, glow, icon, and
|
||||
sticker treatment to {visual_style}, while keeping the hierarchy and text legible.
|
||||
|
||||
Action: The cursor clicks the button. A functional-accent energy pulse travels
|
||||
through PLAYER 1's mechanical claw, while a {power_accent_color} pulse travels
|
||||
through PLAYER 2's fist. All UI panels contract inward and disappear. Both
|
||||
players uncross their legs and adjust their posture: PLAYER 1 lifts one knee
|
||||
lightly and articulates the claw fingers in sequence; PLAYER 2 plants one foot
|
||||
firmly and slowly closes the heavy fist.
|
||||
|
||||
Sound: Confirmation tone, two-color energy pulses, and UI contraction.
|
||||
|
||||
### [8.5-10 seconds] - Shared world loading
|
||||
|
||||
Composition and camera: A wide shot reveals a shared loading bar at the bottom.
|
||||
|
||||
UI: The loading bar displays `{loading_label}` exactly as filled and fills
|
||||
rapidly from 0% to 100%. Its left half uses {functional_accent_color} and its
|
||||
right half uses {power_accent_color}. Adapt the HUD and loading-bar shape,
|
||||
border, texture, and typography to {visual_style}, while keeping them legible.
|
||||
|
||||
Environment transformation: Continuously transform the confirmed
|
||||
{visual_style} menu background into a game world in the same style. Menu strips,
|
||||
color blocks, and patterns become corresponding roads, structures, and world
|
||||
elements. Preserve the confirmed palette rather than forcing a yellow-and-black
|
||||
or cyberpunk city.
|
||||
|
||||
Key constraint: The transformation remains continuous and natural, with no hard
|
||||
cut, smoke occlusion, or identity change.
|
||||
|
||||
Sound: Rising loading tone and an ambience transition from menu to game world.
|
||||
|
||||
### [10-15 seconds] - Two-player world entry
|
||||
|
||||
Composition and camera: Move from an extreme wide shot into a third-person
|
||||
tracking view. At 100% loading, both characters stand simultaneously. The camera
|
||||
descends smoothly and arcs behind them into a stable two-player cooperative view.
|
||||
|
||||
World style: Generate the complete world from {visual_style}. Preserve the game
|
||||
opening structure: dense buildings, roads, signal lights, signs, crowds or other
|
||||
moving background elements, passing vehicles, overhead wires, industrial pipes,
|
||||
and a distant skyline. Materials, architecture, signage, lighting, and motion
|
||||
must follow {visual_style}. Do not let the source template's cyberpunk default
|
||||
override the user unless the selected style is cyberpunk.
|
||||
|
||||
Character relationship: Clearly show the players from behind and preserve their
|
||||
body contrast. PLAYER 1 is on the left, tall and slender, with the lightweight
|
||||
claw hanging naturally. PLAYER 2 is on the right, shorter and broader, with the
|
||||
heavy fist slightly raised. PLAYER 1 steps forward first, PLAYER 2 follows, and
|
||||
they enter the street side by side.
|
||||
|
||||
HUD: The HUD fades in. A mini-map appears in the upper right. Separate status
|
||||
bars labeled `{player1_name}` and `{player2_name}` appear in the lower left.
|
||||
The first uses {functional_accent_color}; the second uses {power_accent_color}.
|
||||
A shared objective marker appears in the street ahead.
|
||||
|
||||
Sound: City ambience, distant vehicles, footsteps, and a HUD fade-in cue.
|
||||
|
||||
## Negative constraints
|
||||
No extra player, duplicated character, identity or username swapping, merged bodies, unintended face/hair changes, missing player, unrequested split screen or cuts, random camera shake, floating body parts, gruesome dismemberment, unwanted weapons, style-incompatible neon, unreadable UI, random letters, misspelled usernames, extra menu options, copied game logo, or watermark. Preserve the user's chosen character designs and palette.
|
||||
@@ -1,42 +0,0 @@
|
||||
---
|
||||
name: h3-prompt-writing
|
||||
description: Write MiniMax H3 video generation prompts for T2VA, I2VA, FL2VA, L2VA, and Ref2VA. Use when rewriting multimodal requests into H3 prompt structures, composing integrated_multimodal_description, overall_soundscape, and non_diegetic_music, aligning keyframes, or defining reference labels for images, videos, and audio.
|
||||
metadata:
|
||||
compatibility: Portable to any agent that can read local files — no external API calls, MiniMax Hub tools, or proprietary runtime required. The agents/openai.yaml file only adds optional ChatGPT/Codex UI metadata; it does not restrict the skill to OpenAI agents.
|
||||
---
|
||||
|
||||
# H3 Prompt Writing
|
||||
|
||||
## Workflow
|
||||
|
||||
0. Read [portable-workflow.md](references/portable-workflow.md) for scope, available tools, durable artifacts, and generation handoff. For prompt-only work, no generation tools are required.
|
||||
1. Identify the input mode: T2VA, I2VA, FL2VA, L2VA, or full-reference Ref2VA.
|
||||
2. For base text/keyframe modes, read `references/base-format.md` and follow its final prompt structure.
|
||||
3. For full-reference mode, read `references/ref-format.md` and follow its six-section rewrite format.
|
||||
4. Preserve the exact field names, section order, labels, and timing notation from the selected guide.
|
||||
|
||||
## Base Modes
|
||||
|
||||
- T2VA: build the full audiovisual timeline from text.
|
||||
- I2VA: start from the first frame and develop forward from it.
|
||||
- FL2VA: describe the continuous path between the first and last frames.
|
||||
- L2VA: infer a plausible opening and converge to the supplied last frame.
|
||||
|
||||
Use `integrated_multimodal_description`, `overall_soundscape`, and `non_diegetic_music` in the order shown in `references/base-format.md`.
|
||||
|
||||
## Full-Reference Mode
|
||||
|
||||
Ref2VA rewrites use `subject_definitions`, `summary`, `retention_analysis`, `detailed_description`, `overall_soundscape`, and `non_diegetic_music` in that order. Reference labels stay consistent across all sections.
|
||||
|
||||
Read `references/ref-format.md` for label rules and retention analysis.
|
||||
|
||||
## Output Rules
|
||||
|
||||
- Write rewrite sections in English; preserve dialogue, lyrics, and visible scene text in their original language.
|
||||
- Describe each shot by composition, subjects, environment, actions, camera, sound, and the exact point where referenced content appears.
|
||||
- Avoid plot summaries, unresolved reference labels, and timing that does not match the requested duration.
|
||||
## Tips for Better Results
|
||||
- Match the description to the effective duration of the current clip or generation window. The upstream guides describe 4–15-second clips; longer work requires a verified backend extension or an explicit assembly plan, as described in the portable workflow.
|
||||
- Keep reference labels consistent (e.g. `<Picture 1>`, `<Video 1>`, `<Audio 1>`) across every section.
|
||||
- Prefer concrete visual and audio details over abstract words like "cinematic" or "beautiful".
|
||||
- When using keyframes (I2VA / FL2VA / L2VA), clearly state how the first and/or last frame connects to the timeline.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "H3 Prompt Writing"
|
||||
short_description: "Write structured H3 prompts for each input mode"
|
||||
default_prompt: "Use $h3-prompt-writing to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,86 +0,0 @@
|
||||
# H3 base-mode prompt format
|
||||
|
||||
Use this reference for T2VA, I2VA, FL2VA, and L2VA prompt writing. It is a
|
||||
compact, repository-authored description of the required output contract, not a
|
||||
copy of an upstream prompt guide.
|
||||
|
||||
## Select the anchor mode
|
||||
|
||||
- T2VA has no image anchor. Build the audiovisual sequence from the brief.
|
||||
- I2VA binds `<Picture 1>` to the first frame at 0.00 seconds.
|
||||
- FL2VA binds `<Picture 1>` to the first frame and `<Picture 2>` to the final
|
||||
frame. Describe the observable path between them.
|
||||
- L2VA binds `<Picture 1>` to the final frame. Infer a compatible opening and
|
||||
converge to the supplied image.
|
||||
|
||||
For an image-anchored mode, put a plain-English alignment statement before the
|
||||
structured fields. State each picture label, its shot, and its exact target time.
|
||||
The ending time must match the effective clip duration, formatted to two decimal
|
||||
places. Do not invent a picture label when no image was supplied.
|
||||
|
||||
## Emit three fields in order
|
||||
|
||||
```text
|
||||
integrated_multimodal_description: [Shot 1] ...
|
||||
|
||||
overall_soundscape: ...
|
||||
|
||||
non_diegetic_music: ...
|
||||
```
|
||||
|
||||
`integrated_multimodal_description` is the playback-ordered visual and diegetic
|
||||
audio plan. Establish the initial style, framing, subjects, environment, and
|
||||
spatial relationships, then describe actions and state changes. For keyframe
|
||||
modes, explicitly preserve or approach the relevant composition, identity,
|
||||
clothing, colors, objects, and geometry.
|
||||
|
||||
`overall_soundscape` summarizes ambience, physical effects, and non-verbal human
|
||||
sound. Do not repeat dialogue or music already described elsewhere. Use `N/A`
|
||||
only for explicitly silent output.
|
||||
|
||||
`non_diegetic_music` describes music heard by the audience but not the characters.
|
||||
Specify audible properties such as instrumentation, tempo, rhythm, dynamics, and
|
||||
ending behavior. Put source-visible music, singing, radio, or instrument playing
|
||||
in the main description. Use `N/A` when no score is wanted.
|
||||
|
||||
## Timeline notation
|
||||
|
||||
- Start the opening with `[Shot 1]` and no timestamp.
|
||||
- Mark each later cut with a sequential label and a strictly increasing time,
|
||||
for example `[Shot 2] At 00:03.500, ...`.
|
||||
- Use a new shot for an actual edit or meaningful viewpoint/time change. Keep a
|
||||
continuous action or morph in one shot and describe its internal timing.
|
||||
- Describe camera motion as an action: type, direction, useful amplitude, and
|
||||
speed. Do not append a disconnected list of camera keywords.
|
||||
- Keep every cut and event within the requested clip duration.
|
||||
|
||||
## Speech, text, and sound
|
||||
|
||||
Give each speaking or singing source a stable ID such as `(S1)` or `(S2)` and
|
||||
reuse it across shots. Put only the language tag and literal words inside the
|
||||
dialogue wrapper:
|
||||
|
||||
```text
|
||||
The courier (S1) says softly: <d>[English] The gate is open.</d>
|
||||
```
|
||||
|
||||
Preserve supplied dialogue, lyrics, punctuation, and visible text verbatim. State
|
||||
when a voice is off-screen and keep the visible character's lips closed when that
|
||||
is required. If a line spans a cut, describe the audio continuity on both sides
|
||||
of the cut. Place visible signs, labels, and interface copy in double quotes.
|
||||
|
||||
Tie diegetic sounds to the event that produces them. Avoid describing the same
|
||||
sound as both diegetic and non-diegetic.
|
||||
|
||||
## Anchor-specific checks
|
||||
|
||||
- I2VA: begin from the supplied frame rather than merely borrowing its style.
|
||||
- FL2VA: describe intermediate motion; do not provide two unrelated still-image
|
||||
descriptions. Reach Picture 2 at the actual end time.
|
||||
- L2VA: keep the final picture assigned to the last shot, not automatically to
|
||||
Shot 1, and make the preceding action physically plausible.
|
||||
- All modes: reference labels must map to supplied assets and remain stable.
|
||||
|
||||
Before delivery, verify the field order, shot numbering, timestamps, reference
|
||||
labels, dialogue fidelity, requested language, duration, and absence of unresolved
|
||||
template variables.
|
||||
@@ -1,121 +0,0 @@
|
||||
# Portable H3 production workflow
|
||||
|
||||
This is the runtime contract for the nine H3 skills in this repository. Read it
|
||||
alongside the selected creative skill. The bundle uses local files and available
|
||||
tools; no proprietary canvas, choice-card widget, planner, or executor is required.
|
||||
|
||||
## Scope and decisions
|
||||
|
||||
- Resolve the requested deliverable first: prompt, storyboard, assets, or finished
|
||||
video. A prompt-only request ends with the prompt; it does not start generation.
|
||||
- Reuse the user's duration, language, style, model, audio policy, supplied assets,
|
||||
and existing authorization. State reasonable defaults for optional gaps. Ask
|
||||
only about missing inputs or choices that materially affect the requested work.
|
||||
- Creative checkpoints are reviewable artifacts, not mandatory repeated approvals.
|
||||
Honor an explicit request to approve a preview before rendering. When the user
|
||||
has authorized the complete workflow or delegated creative choices, show the
|
||||
plan and continue within that scope without requiring a special UI control.
|
||||
- A skill never grants permission to publish, send messages, spend outside an
|
||||
authorized budget, or change models against the user's choice. User constraints
|
||||
override a preset's style, language, shot count, timing, or audio defaults.
|
||||
|
||||
## Tools and durable artifacts
|
||||
|
||||
1. Inspect the actual available tools, existing scripts, endpoint documentation,
|
||||
and supplied media. Do not invent a tool name, model parameter, or returned path.
|
||||
2. Write plans and prompts to the user-selected project folder. If none is given,
|
||||
use a new named folder under `outputs/`, outside the skill package. Keep useful
|
||||
artifacts such as `brief.md`, `storyboard.md`, `prompts/`, `references/`,
|
||||
`clips/`, and `final/`; create only those needed for this deliverable.
|
||||
3. Record each reference's role and path: identity, environment, style, typography,
|
||||
first/last frame, or audio. Tie generated clips to the prompt revision, input
|
||||
assets, model, settings, and output paths. Update downstream references when an
|
||||
asset changes; retain previous outputs instead of overwriting user files.
|
||||
4. Use available image generation/editing tools for requested raster assets. Use
|
||||
an available video endpoint or local service for clips, and available audio
|
||||
tooling for requested standalone speech/music. For assembly and inspection,
|
||||
use an installed editor or tools such as FFmpeg, ffprobe, and OpenCV. Follow the
|
||||
host's tool policies, including its preferred image generation mechanism.
|
||||
5. Independent tool jobs may run concurrently when resources permit; this skill
|
||||
does not require separate agents or delegated generation.
|
||||
6. If a required capability is missing, finish the useful in-scope preparation:
|
||||
complete prompts, asset-role manifest, timeline, and verified request example
|
||||
where possible. State the missing capability and unfinished media step. Do not
|
||||
silently substitute another model, claim a render happened, or call a written
|
||||
plan a completed video. Resume generation when the capability becomes available.
|
||||
|
||||
## Compile the creative plan into H3 prompts
|
||||
|
||||
Read [base-format.md](base-format.md) for T2VA / I2VA / FL2VA / L2VA, or
|
||||
[ref-format.md](ref-format.md) for Ref2VA. These are writing modes, not guaranteed API
|
||||
task names. Check the selected backend's input mapping before sending a request.
|
||||
|
||||
- Base prompts use `integrated_multimodal_description`, `overall_soundscape`,
|
||||
`non_diegetic_music`, in that order, with the appropriate keyframe preamble.
|
||||
- Ref2VA uses `subject_definitions`, `summary`, `retention_analysis`,
|
||||
`detailed_description`, `overall_soundscape`, `non_diegetic_music`, in that order.
|
||||
- Every reference label must resolve to a supplied asset, in the same order used
|
||||
by the backend. Omit nonexistent optional references and their dependencies.
|
||||
- Keep the creative brief in the user's language. Structured H3 rendering prompts
|
||||
use English prose and exact field labels, while dialogue, lyrics, and visible
|
||||
text retain their requested language and wording. If the user explicitly wants
|
||||
only a same-language creative prompt, deliver that without silently replacing it
|
||||
with English. Compile a separate rendering prompt when generation is requested.
|
||||
- `[Shot 1]` has no initial timestamp. Later shot labels mark actual cuts. For a
|
||||
continuous morph, describe timed actions inside one shot rather than turning
|
||||
each form into a new shot. Express camera direction, speed, and useful amplitude.
|
||||
- Separate image style from character identity. A style reference must not inject
|
||||
its people, scenery, layout, typography, texture, or lighting into unrelated
|
||||
shots. For literal text fidelity, inspect the rendered result; a prompt is not
|
||||
proof that the exact lettering or beat synchronization was achieved.
|
||||
|
||||
## Longer videos and audio
|
||||
|
||||
Use the actual backend's supported duration, dimensions, frame grid, reference
|
||||
count, audio modes, and model partition. Inspect this checkout's H3 implementation
|
||||
or serving documentation before building requests. Do not assume that a feature
|
||||
branch's long-video extension exists on main or on a hosted service.
|
||||
|
||||
For a verified vLLM-Omni Ref2VA continuation backend, `long_video=true` with
|
||||
`long_video_mode=continuation` can accept one `continuation_prompts` entry per
|
||||
planned window only when the Ref2VA request is executed with its local text
|
||||
encoder. Step execution and stages that use an external text encoder reject
|
||||
`continuation_prompts`; omit the field and use the supported per-step inputs in
|
||||
those configurations. Compute the window count from the aligned total frames,
|
||||
window size, and overlap. Each prompt uses local window time; track global
|
||||
shot/music time separately. Repeated overlap is guidance, not additional output
|
||||
duration. Do not use this mode on a task/partition that does not support it.
|
||||
|
||||
Otherwise split into supported clips and assemble them. Same-scene clips may use
|
||||
tail/head keyframe continuity when supported. Scene changes should have explicit
|
||||
new scene references; a previous scene's latent tail can preserve the wrong
|
||||
setting. Choose cuts, morphing, and transitions from the brief rather than imposing
|
||||
hard cuts whenever a work exceeds 15 seconds. Generation windows and visual shots
|
||||
are different: one window may contain cuts, and one shot may span multiple windows.
|
||||
|
||||
For supplied music, use the selected excerpt as the master timeline unless the
|
||||
user requests replacement. For native audio, write the musical/SFX direction in
|
||||
the prompt and verify that the backend actually returns audio. Do not invent a
|
||||
generic `generate_audio` parameter. If available, locked-source audio latents guide
|
||||
generation but the audio-VAE reconstruction is not bit-identical to the source.
|
||||
Use explicit final audio stream-copy when an exact original track is required.
|
||||
Prompted BPM is a target, not measured timing. Avoid automatically adding lyrics,
|
||||
voiceover, subtitles, or multiple competing scores to an instrumental request.
|
||||
|
||||
## Verification and bounded recovery
|
||||
|
||||
Inspect reference images before using them. After rendering, inspect representative
|
||||
frames, transitions, opening/ending states, text, identities, and scene continuity.
|
||||
Check audio by listening when possible; report any limit on audio review. Check
|
||||
duration, dimensions, fps, frame count, audio presence, and decode validity with
|
||||
media tools. If preserving audio exactly, compare extracted audio stream hashes.
|
||||
|
||||
Correct a demonstrated failure with a targeted change to the affected prompt,
|
||||
reference, or edit. Keep raw generation alongside edited/upscaled versions and
|
||||
label the difference. Use an agreed retry/resource budget; absent one, allow one
|
||||
targeted rerender, then report persistent defects and concrete options rather than
|
||||
entering an unbounded generation loop. Do not switch providers automatically.
|
||||
|
||||
Deliver actual artifact paths, the prompts/settings needed to repeat the work,
|
||||
and material visual/audio limitations. Distinguish requested behavior, observed
|
||||
model output, and post-production repairs.
|
||||
@@ -1,105 +0,0 @@
|
||||
# H3 full-reference prompt format
|
||||
|
||||
Use this reference for Ref2VA prompt writing. It is a compact,
|
||||
repository-authored description of the output contract, not a copy of an
|
||||
upstream prompt guide.
|
||||
|
||||
## Emit six fields in order
|
||||
|
||||
```text
|
||||
subject_definitions: ...
|
||||
|
||||
summary: ...
|
||||
|
||||
retention_analysis: ...
|
||||
|
||||
detailed_description: [Shot 1] ...
|
||||
|
||||
overall_soundscape: ...
|
||||
|
||||
non_diegetic_music: ...
|
||||
```
|
||||
|
||||
Write the structural prose in English. Preserve the requested language and exact
|
||||
wording for dialogue, lyrics, and visible text.
|
||||
|
||||
## Reference labels
|
||||
|
||||
Assign labels only to supplied content and keep every label stable across all six
|
||||
fields:
|
||||
|
||||
- `<Subject N>` identifies reusable visible content such as a person, object,
|
||||
environment, costume, interface, action, or visual treatment.
|
||||
- `<Picture N>` identifies an image used as a concrete frame, composition, or
|
||||
storyboard anchor. If an image only defines a subject, cite it in that subject's
|
||||
definition instead of inventing a separate target-frame role.
|
||||
- `<Video N>` identifies a source video being edited or continued, or a video
|
||||
whose temporal structure is referenced.
|
||||
- `<Audio N>` identifies a supplied audio signal that is copied or referenced.
|
||||
Audio and video label numbers are independent even when they came from one file.
|
||||
|
||||
In `subject_definitions`, give each tracked item a separate line. State its source,
|
||||
role, and the features that later sections must preserve or transform. Do not
|
||||
create labels for assets or roles that do not exist.
|
||||
|
||||
## Summary and retention
|
||||
|
||||
`summary` is one short paragraph describing the target and the principal reference
|
||||
relationships. Use only labels already defined. Distinguish these operations:
|
||||
|
||||
- a concrete image frame anchor;
|
||||
- generation guided by identity, style, action, camera, or audio characteristics;
|
||||
- direct editing of a source video;
|
||||
- continuation from a source video;
|
||||
- copying an audio signal;
|
||||
- referencing audio properties without copying the signal.
|
||||
|
||||
`retention_analysis` has one line per label and states where it is used and what
|
||||
changes. Use consistent machine-readable relationship values:
|
||||
|
||||
- Visual: `fully_preserved`, `partially_preserved`, `attribute_transfer`, or
|
||||
`weak_reference`.
|
||||
- Audio: `fully_copy`, `partially_copy`, `reference`, or `weak_reference`.
|
||||
|
||||
Explain the concrete retained and changed traits after the relationship value.
|
||||
Do not claim full preservation when identity, geometry, timing, or signal content
|
||||
is intentionally altered. Do not count newly requested target action as an
|
||||
unintended loss of reference fidelity.
|
||||
|
||||
## Detailed description
|
||||
|
||||
`detailed_description` is the playback-ordered rendering plan. For each shot,
|
||||
cover the current composition, subject appearance and position, environment,
|
||||
lighting, actions and state changes, camera behavior, diegetic sound, and the
|
||||
point at which each reference becomes relevant.
|
||||
|
||||
- Start with `[Shot 1]` and no timestamp.
|
||||
- Give later cuts sequential labels with strictly increasing times, such as
|
||||
`[Shot 2] At 00:03.500, ...`.
|
||||
- Place relevant reference labels next to the described subject, frame, motion,
|
||||
or sound; do not collect detached labels at the end.
|
||||
- Use stable `(S1)`, `(S2)`, and subsequent IDs for vocal sources. Put literal
|
||||
speech or lyrics inside `<d>[Language] ...</d>`.
|
||||
- Preserve supplied visible text verbatim and place it in double quotes.
|
||||
- Describe continuous transformations inside one shot unless an actual cut is
|
||||
intended. State camera direction, speed, and amplitude when useful.
|
||||
|
||||
For video editing, explicitly identify what stays from the source timeline and
|
||||
what is replaced. For continuation, distinguish the inherited opening state from
|
||||
new content. For style reference, transfer visual properties without accidentally
|
||||
copying people, scenery, layout, or typography.
|
||||
|
||||
## Audio fields
|
||||
|
||||
`overall_soundscape` summarizes ambience, physical action sounds, and non-verbal
|
||||
human sounds. `non_diegetic_music` describes score audible only to the audience.
|
||||
Keep dialogue and diegetic music in `detailed_description`. Use `N/A` only when
|
||||
the corresponding layer is intentionally absent.
|
||||
|
||||
If audio is copied, state whether the whole signal or only a range/layer is used.
|
||||
If it is referenced, describe the specific property—such as voice timbre, tempo,
|
||||
rhythm, or texture—without claiming signal reuse.
|
||||
|
||||
Before delivery, verify the six-field order, one-to-one label definitions,
|
||||
retention coverage, shot timing, dialogue fidelity, requested language, and
|
||||
absence of unresolved template variables.
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
name: handdrawn-live-video-generator
|
||||
description: Write or generate a continuous live-action scene with a rough luminous hand-drawn entity that contacts real objects, morphs, and escapes a slightly delayed handheld camera. Use for gentle surreal mixed-media shorts, not flat-vector-only animation or horror clips.
|
||||
metadata:
|
||||
compatibility: Portable prompt writing with sibling h3-prompt-writing; actual generation requires an available video backend and any referenced media.
|
||||
---
|
||||
|
||||
# Handdrawn Live-Action Fusion
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Keep the user's requested scope and language. A creative-prompt request delivers
|
||||
the prompt, without generating media. For a requested finished video, prepare
|
||||
the prompt and proceed under existing authorization using available tools.
|
||||
|
||||
## Creative contract
|
||||
|
||||
The default is a 15-second, 16:9 continuous scene in an everyday real space. A
|
||||
flat luminous drawn entity visibly contacts a real hand/object early, transforms
|
||||
as one traceable being, escapes through connected space, and the handheld camera
|
||||
reacts slightly late. Duration and ratio follow the user's explicit choices.
|
||||
|
||||
Use rough crayon, chalk, colored pencil, pastel, or brush strokes with uneven fill,
|
||||
frayed edges, line jitter, and frame-by-frame redraw. Keep the live-action setting
|
||||
physically grounded and the drawn presence visibly planar. Uniform neon tubes,
|
||||
plush characters, polished CG, and clean vector strokes would change this style.
|
||||
|
||||
Choose a fresh setting, entity, palette, contact mechanism, transformation chain,
|
||||
escape route, camera reaction, and emotional ending within the user's constraints.
|
||||
Do not ban an otherwise requested motif simply because an upstream example used
|
||||
it. The tone is warm, playful, and gently surprising rather than threatening.
|
||||
|
||||
## Timed action structure
|
||||
|
||||
For a 15-second brief, use these intervals as a starting structure:
|
||||
|
||||
- 0–3s: establish obvious real/drawn contact, such as landing on a palm or slipping
|
||||
around a finger. The real participant reacts.
|
||||
- 3–6s: begin a continuous morph while retaining a color smear, line, curve, or tail
|
||||
from the previous shape. Give it a playful action.
|
||||
- 6–10s: let it escape into an adjacent part of the same space. The camera pans,
|
||||
tilts, or moves only after the entity starts leaving its view.
|
||||
- 10–13s: add another readable interaction or surprise, keeping the same entity.
|
||||
- 13–15s: extend an earlier drawn motif across the space into a gentle larger
|
||||
transformation, then resolve with a small emotional/comic detail.
|
||||
|
||||
Retiming changes the intervals proportionally or reorganizes beats; do not leave
|
||||
15-second timestamps in a shorter prompt. Preserve a user's explicit final form
|
||||
instead of forcing the spatial finale when it conflicts with their ending.
|
||||
|
||||
## Prompt delivery and rendering
|
||||
|
||||
For a same-language creative prompt, begin with duration, ratio, real space, and
|
||||
the live-action/drawing fusion. Then describe phone-camera texture, the timed
|
||||
action intervals, drawn material, camera lag, specific exclusions, and ambient
|
||||
sound. Avoid unrelated headings or an essay when the user asked for copy-ready text.
|
||||
|
||||
For actual H3 generation, create a separate structured rendering prompt using the
|
||||
appropriate base/Ref2VA guide: English prose with literal dialogue/visible text in
|
||||
the requested language. Preserve the creative intent and actual media bindings.
|
||||
Do not send unsupported model settings or promise exact first-frame conditioning
|
||||
merely because an image is present.
|
||||
|
||||
## Review
|
||||
|
||||
Check clear early physical contact, one continuous entity, connected geography,
|
||||
retained shape motifs, delayed camera reaction, rough planar drawing, and a
|
||||
non-threatening ending. Avoid jump scares, sudden blackouts, menacing anatomy, and
|
||||
unrequested extra characters. Deliver the actual prompt/video and state any
|
||||
remaining contact, continuity, texture, or camera-timing defects.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Handdrawn Live-Action Fusion"
|
||||
short_description: "Direct drawn morphs and real-world interactions"
|
||||
default_prompt: "Use $handdrawn-live-video-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,71 +0,0 @@
|
||||
---
|
||||
name: minimalist-product-ad-generator
|
||||
description: Plan, prompt, or produce minimalist product advertising shorts from product images, preserving variants, material, color, concise typography, and rhythmic camera motion. Use for product launches and ecommerce films, not presenter-led ads or complex UI demonstrations.
|
||||
metadata:
|
||||
compatibility: File-based agent workflow. Requires the sibling h3-prompt-writing package; rendering uses available image, video, audio, and editing tools.
|
||||
---
|
||||
|
||||
# Minimalist Product Ad Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
Use [product direction](references/product-direction.md) when planning anchors,
|
||||
copy, and timed beats. Respect the requested output: prompt-only work needs no
|
||||
generation tool, while a finished film includes generation, inspection, and assembly.
|
||||
|
||||
## Resolve the product brief
|
||||
|
||||
Inspect supplied product photos, packaging, renders, and detail shots. Record the
|
||||
product category, main variant, actual body/accent colors, material finish, usable
|
||||
mechanisms, and requested copy. Preserve identity and product facts; visual polish
|
||||
does not authorize recoloring the product or inventing a feature.
|
||||
|
||||
Reuse the specified duration, ratio, language, and model. With no constraints,
|
||||
suggest a 10-second H3 film and one of white studio, dark rim light, brand color
|
||||
field, or light lifestyle staging. Ask for a missing product reference when exact
|
||||
identity is required; distinguish a speculative concept from an existing product.
|
||||
If the user delegates direction, record the adopted choices and proceed.
|
||||
|
||||
## Design the film
|
||||
|
||||
1. Choose a product-specific spine: launch reveal, mechanism interaction, or color
|
||||
family. Describe concrete actions such as a lid opening or a crown turning.
|
||||
2. Write short sensory/benefit copy grounded in supplied facts. Preserve the user's
|
||||
exact approved wording and language. For newly requested English copy, 3–5 words
|
||||
usually fit one readable line; do not force English on other-language campaigns.
|
||||
3. Plan three independent anchors: hero silhouette, material/mechanism detail, and
|
||||
final product/copy composition. Generate only the assets needed or requested.
|
||||
Do not combine them into a panel sheet that the video model might reproduce.
|
||||
4. Keep product color, light direction, ratio, and finish consistent across anchors.
|
||||
Original photos establish identity; generated anchors must pass that identity
|
||||
check before they can replace them as rendering references.
|
||||
5. Build the beat table from the reference. Include setup, action, impact, and a
|
||||
stable closing composition; vary shot distance only when it reveals something.
|
||||
|
||||
## Prompt and render
|
||||
|
||||
Compile each beat into the appropriate H3 fields using the sibling writing guide.
|
||||
Bind every anchor to its actual role. Include exact visible copy, entry/exit times,
|
||||
colors, and same-line text motion in the rendering prompt, not only in the plan.
|
||||
Keep typography in intentional negative space without covering the product. A
|
||||
copy-reference image does not mean its line must persist throughout the whole film.
|
||||
|
||||
For a short ad, prefer one coherent full-frame film when the backend supports it.
|
||||
Keep variants ordered: main variant first, supporting variants later, full family
|
||||
only when useful at the close. Avoid product walls, grids, or an anchor-sheet ending.
|
||||
|
||||
Use native audio when appropriate to the brief. A restrained electronic score may
|
||||
use crisp plucks, airy pads, kick/sub accents and tactile wooden percussion. Set
|
||||
tempo and energy from the user's direction. Generate a separate score only if
|
||||
requested or needed for a disclosed audio repair, using an available tool.
|
||||
|
||||
## Review and deliver
|
||||
|
||||
Verify product geometry, variant colors, functional action, final copy, aspect
|
||||
ratio, and closing readability. Listen for excessive bass, dropouts, and unfinished
|
||||
endings. If editing to supplied music, select a suitable continuous excerpt and
|
||||
align visual impacts to observed beats; do not claim synchronization from text alone.
|
||||
|
||||
For bad typography, first reduce simultaneous text or correct its reference. Exact
|
||||
lettering may need a disclosed overlay when consistent with the user's goal; keep
|
||||
the raw output. Deliver the film, prompt/beat plan, used copy and music excerpt,
|
||||
and any remaining product/text fidelity issues.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Minimalist Product Ads"
|
||||
short_description: "Direct product anchors, copy, motion, and audio"
|
||||
default_prompt: "Use $minimalist-product-ad-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,52 +0,0 @@
|
||||
# Product anchors, typography, and beats
|
||||
|
||||
## Narrative choices
|
||||
|
||||
- Launch: negative space → hero product → material detail → real mechanism action
|
||||
→ supporting variants → stable product and copy.
|
||||
- Feature touch: stillness → interaction trigger → visible mechanism → detail
|
||||
magnification → useful result → closing.
|
||||
- Color family: main variant alone → supporting colors enter → ordered relation
|
||||
→ unified material finish → full set and copy.
|
||||
|
||||
## Independent anchor briefs
|
||||
|
||||
| Anchor | What it locks | Check |
|
||||
| --- | --- | --- |
|
||||
| Hero | Main variant, recognizable silhouette, attractive opening view | Match supplied body color and product structure |
|
||||
| Detail | Material, reflection, texture, visible mechanism | No invented ports, controls, or functionality |
|
||||
| Closing | Product position, negative space, typography treatment | Real copy, readable contrast, no panel/grid layout |
|
||||
|
||||
Each is a standalone full-frame image with the project ratio. Reference roles
|
||||
protect appearance and layout; they do not replace the storyboard or require three
|
||||
sequential static-image segments. Simplify the number of anchors if the backend
|
||||
cannot bind them all, and update the prompt labels accordingly.
|
||||
|
||||
## Typography
|
||||
|
||||
Use one readable line at a time, with a typeface appropriate to the brand. A clean
|
||||
semibold sans-serif is a useful default, not a requirement to copy a specific brand.
|
||||
For a two-part color treatment, use dark text on a light field or light text on a
|
||||
dark field, then emphasize the second part with a specific real product color.
|
||||
Check contrast rather than blindly coloring text like the product.
|
||||
|
||||
The first phrase may enter gently; the second follows while the first shifts
|
||||
slightly within the same line. Do not accidentally turn this into stacked lines.
|
||||
Use actual copy in the prompt and spell out its timing. Avoid arrows, separators,
|
||||
or placeholder marks that the model may reproduce. For a 10-second ad, one middle
|
||||
copy moment and one stable closing line are often enough. Omit copy when requested.
|
||||
|
||||
## Beat table
|
||||
|
||||
| Time range | Shot purpose | Visual lead and camera | Variant state | Exact copy | Text color/motion | Handoff | Rhythm |
|
||||
| --- | --- | --- | --- | --- | --- | --- | --- |
|
||||
|
||||
Use one main action per beat. For 5/10/15-second films, roughly 3–4/5–7/6–9 beats
|
||||
are planning suggestions, not required cuts. Record the ending product position
|
||||
and motion vector so the next beat can inherit them. Ensure the closing line has
|
||||
time to be read rather than using every second for motion.
|
||||
|
||||
If replacing music, keep useful SFX unless replacement of the whole track was
|
||||
requested. Use a continuous excerpt with a clear entry, stable middle, and suitable
|
||||
ending. A short tail fade may resolve a click; do not hide an incomplete phrase
|
||||
behind a long fade. Confirm the entire result remains audible.
|
||||
@@ -1,74 +0,0 @@
|
||||
---
|
||||
name: music-video-subtitle-generator
|
||||
description: Plan, write, audit, or produce music videos with coordinated music, imagery, camera motion, optional performance, and lyric typography. Use for beat-aware MVs and instrumental visual narratives, not ordinary subtitle cleanup or unrelated video edits.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing; audio analysis, generation, and final assembly use available tools and backends.
|
||||
---
|
||||
|
||||
# Music Video and Lyric Typography
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md)
|
||||
and, when building a timeline, [music timeline](references/music-timeline.md).
|
||||
Deliver the requested scope: audit, prompt, storyboard, or complete video.
|
||||
|
||||
## Lock the musical and visual intent
|
||||
|
||||
Identify duration, ratio, output resolution, music source/excerpt, visual style,
|
||||
character/scene references, performance mode, and whether text is wanted. Treat a
|
||||
supplied song as the master track unless the user requests replacement. Preserve
|
||||
provided lyrics exactly; do not translate or rewrite them unasked. For instrumental
|
||||
work, do not invent singing, lyrics, a performer, or typography. Generate original
|
||||
lyrics only when requested or clearly within an authorized songwriting brief.
|
||||
|
||||
Measure supplied audio duration and inspect/listen for beat, phrase, breath, drop,
|
||||
and energy changes using available tools. Label estimated or planned timing when
|
||||
measurement is unavailable. If the requested excerpt exceeds the audio, resolve
|
||||
the discrepancy without silently stretching or looping it.
|
||||
|
||||
## Reference roles
|
||||
|
||||
- Character references control identity, silhouette, wardrobe, and presence.
|
||||
- Scene references control environment, time of day, spatial anchors, and lighting.
|
||||
- Typography references control letter treatment, layout, and motion, not people
|
||||
or scenery visible in those references.
|
||||
- Music references control the requested track or sound style; distinguish copying
|
||||
a supplied recording from generating similar instrumentation.
|
||||
|
||||
Do not apply a preset's grain, darkness, hard cuts, or constant close-ups to an
|
||||
incompatible brief. A bright nighttime scene remains night. A continuous flat
|
||||
vector morph keeps smooth color fields and uses elastic shape motion for accents.
|
||||
Use hard cuts, slow holds, or dissolves only when appropriate to the user's intent.
|
||||
|
||||
## Compose the timeline and H3 prompts
|
||||
|
||||
Map visual actions, camera moves, scene changes, and optional text to the music's
|
||||
global time. Build a causal visual progression instead of defaulting to one person
|
||||
singing throughout. Allow room to read a shape, gesture, or lyric before changing it.
|
||||
|
||||
For longer work, choose a supported continuation or clip-assembly approach through
|
||||
the portable workflow. Keep visual shot boundaries distinct from generation
|
||||
windows. Local window prompts must account for overlap without replaying the full
|
||||
previous action. A new scene may require a new reference rather than inheritance
|
||||
of the old scene. Record entry/exit states for each planned segment.
|
||||
|
||||
Compile the timeline into H3's base or Ref2VA fields, preserving dialogue, lyrics,
|
||||
and visible text verbatim. Include concrete instrumentation and rhythmic accents,
|
||||
but treat requested BPM and beat alignment as targets until checked in the output.
|
||||
|
||||
## Typography and finishing
|
||||
|
||||
When requested, give each shot one main readable text event. Keep words away from
|
||||
eyes and critical lip motion. Match sung text to the active phrase, and distinguish
|
||||
spatial typography from subtitle bars. Reduce text complexity if the model cannot
|
||||
render it, or use a disclosed overlay consistent with the requested deliverable.
|
||||
|
||||
For supplied audio, assemble clips to one global master track, maintaining phrase
|
||||
and lip continuity; avoid cuts inside a vowel. For native audio, inspect actual
|
||||
continuity across generation windows. Preserve useful SFX and avoid layered duplicate
|
||||
music. Retiming, grain, grading, interpolation, and upscaling are deliberate editing
|
||||
choices, not automatic finishing requirements.
|
||||
|
||||
Verify transformation/story order, identities, scene changes, requested text,
|
||||
musical continuity, timing, and full media decode. Label raw and processed versions.
|
||||
Deliver the final file and current prompt/timeline files, with measured properties
|
||||
and remaining limitations rather than claims of perfect synchronization.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Music Video Direction"
|
||||
short_description: "Plan music-driven scenes and optional lyric text"
|
||||
default_prompt: "Use $music-video-subtitle-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,40 +0,0 @@
|
||||
# Global music and local video timing
|
||||
|
||||
Use this planning skeleton, omitting vocal/text fields when not requested:
|
||||
|
||||
```text
|
||||
Global: music/excerpt, duration, ratio, character/style/scene locks, cut policy.
|
||||
|
||||
Segment S01 — global start/end; local generation start/end
|
||||
Incoming state:
|
||||
Music cue: measured beat/phrase timestamp, or clearly labeled planned accent
|
||||
Vocal line and speaker (if present):
|
||||
Visible typography (if requested):
|
||||
Visual action and camera:
|
||||
Outgoing state and transition:
|
||||
References with exact roles:
|
||||
```
|
||||
|
||||
Keep one global music clock. For measured tempo B, the nominal quarter-note spacing
|
||||
is 60/B seconds, but real tracks may have pickups, swing, tempo changes, or inaccurate
|
||||
automatic detections. Validate important cut points against the waveform/listening.
|
||||
An onset detector also finds SFX and melodic attacks; it does not prove drum beats.
|
||||
|
||||
Choose a phrase-complete excerpt when possible. Track breaths, lyric pauses, and
|
||||
vowel continuity for a performance MV. For an instrumental morph, accent the
|
||||
physical preparation, transformation impact, and release rather than introducing
|
||||
unrequested cuts. The final shape or story payoff needs a readable ending.
|
||||
|
||||
For continuation, maintain a table of absolute output frames and local window
|
||||
times. Use the verified backend's native alignment and overlap behavior. Opening
|
||||
overlap continues the inherited state; new actions belong after it. Do not sum
|
||||
overlap twice or describe every window as a fresh opening shot.
|
||||
|
||||
For separately generated clips, use a previous tail frame only when the next shot
|
||||
continues the same scene and the backend supports that keyframe role. For an
|
||||
intentional cut, retain appropriate identity references and select the new scene.
|
||||
Preserve screen direction or a useful match element when it helps the transition.
|
||||
|
||||
When text is requested, track exact string, language, entry/exit time, screen
|
||||
region, and motion. Typography style cards must not donate their depicted characters
|
||||
or settings. Preserve user text; do not silently shorten lyrics to fit a template.
|
||||
@@ -1,69 +0,0 @@
|
||||
---
|
||||
name: paper-collage-explainer-generator
|
||||
description: Translate narration, ideas, or knowledge points into editorial halftone paper-collage metaphors, still prompts, storyboards, or stop-motion clips. Use for photographic cutout collage and tactile assembly, distinct from paper-puppet dioramas or generic cartoons.
|
||||
metadata:
|
||||
compatibility: File-based workflow with sibling h3-prompt-writing; media creation uses available image/video tools and an editor when needed.
|
||||
---
|
||||
|
||||
# Paper Collage Explainer Generator
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md)
|
||||
and [collage design](references/collage-design.md) when constructing scenes.
|
||||
Honor prompt-only and asset-only requests as well as finished video requests.
|
||||
|
||||
## Meaning and media policy
|
||||
|
||||
Extract each source line's meaning, emotion, action verb, visual metaphor, and
|
||||
three to six readable object groups. Preserve the user's topic and wording as
|
||||
context; do not put narration on screen merely because it is available.
|
||||
|
||||
Default to tactile paper SFX, with no BGM, voiceover, or subtitles unless requested.
|
||||
An explainer does not inherently require speech. Existing user requests for those
|
||||
layers already establish the choice; do not ask again. A short 16:9 clip of roughly
|
||||
four seconds is a useful default for one assembly beat, subject to backend limits.
|
||||
|
||||
## Plan the image and motion together
|
||||
|
||||
Create a concise plan of metaphor, palette, object groups, assembly order, duration,
|
||||
and sound cues. For a series, keep paper texture, halftone treatment, keylines,
|
||||
shadows, and rhythmic language consistent; vary the metaphor and color field only
|
||||
where the meaning benefits. A user's no-text or no-music requirement applies to
|
||||
every downstream asset, prompt, clip, and edit.
|
||||
|
||||
Design each still as the completed final composition. Use black-and-white
|
||||
photographic halftone cutouts, selective colored cardstock, warm cream keylines,
|
||||
subtle fiber/torn edges, and soft physical layer shadows on a bold color field.
|
||||
Keep layered depth and negative space; avoid excessive age, dirt, wrinkles, or
|
||||
automatic brown/kraft backgrounds. Exact editable layers require a compositing
|
||||
workflow, not a claim that a generated raster video is a layered project.
|
||||
|
||||
Generate final-composition stills when requested/needed and inspect them. Follow
|
||||
any requested approval gate; otherwise continue within the authorized workflow.
|
||||
|
||||
## Animate the assembly
|
||||
|
||||
Start from a clean paper field matching the planned final color. Base pieces enter,
|
||||
then the main metaphor, then supporting objects. Each piece slides or pops in,
|
||||
bounces slightly, presses flat, pauses, and locks. End with a brief hold on the
|
||||
completed composition. Use stop-motion assembly rather than global fades, smooth
|
||||
digital drifting, fast spinning, or liquid morphs unless explicitly requested.
|
||||
|
||||
Use the supplied final still as a last-frame anchor only when the backend supports
|
||||
it. Otherwise identify it honestly as composition/style guidance; do not describe
|
||||
a reference image as a guaranteed final-frame constraint. Compile the actual H3
|
||||
mode with the sibling guide and bind only real inputs.
|
||||
|
||||
Map short paper slides, pops, taps, rustles, and snaps to motion. Preserve those
|
||||
SFX during assembly. When the user adds music or narration later, mix with the
|
||||
useful SFX unless they request full replacement.
|
||||
|
||||
## Review and deliver
|
||||
|
||||
Check the metaphor without relying on text, object count/readability, matching
|
||||
opening color, visible piece-by-piece construction, final-frame resemblance,
|
||||
controlled paper texture, and no unwanted UI/letters/music/voiceover. If the
|
||||
assembly is weak, simplify the groups and make their arrival order concrete.
|
||||
|
||||
For multiple clips, preserve the planned order and SFX. Deliver prompts, current
|
||||
stills, and actual clips/assembly as requested. Explain any missing generation
|
||||
capability or remaining drift; keep raw output separate from edited fixes.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Paper Collage Explainers"
|
||||
short_description: "Animate editorial paper metaphors with tactile SFX"
|
||||
default_prompt: "Use $paper-collage-explainer-generator to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,33 +0,0 @@
|
||||
# Editorial collage design
|
||||
|
||||
For each source idea record meaning, emotion, one active verb, a concrete visual
|
||||
metaphor, three to six object groups, and a tactile sound cue. For example, a
|
||||
collaboration concept can become separate cutout hands fitting paper routes
|
||||
together; it need not become a screen full of the word "collaboration".
|
||||
|
||||
Style signature: flat bold color field, black-and-white halftone photographic
|
||||
cutouts, selective colored cardstock accents, warm cream keylines, soft physical
|
||||
paper shadows, fine uncoated fiber, refined irregular torn edges, layered seams.
|
||||
Control texture so the video feels handmade without becoming dirty or distressed.
|
||||
|
||||
Palette choices may support meaning: orange/red for urgency, mustard for tools or
|
||||
warning, green for reflection, purple for memory, teal for collaboration, rose for
|
||||
theatrical contrast. These are suggestions, not universal semantic rules. Use the
|
||||
user's palette and avoid an unrelated kraft-paper opening.
|
||||
|
||||
Still specification: source meaning → metaphor → ratio → background/accent colors
|
||||
→ object roles → foreground/midground/background → final composition → exclusions.
|
||||
|
||||
Motion specification: matching empty color field → base paper structures → main
|
||||
subject → secondary groups → light bounce/press-flat/pause → completed composition.
|
||||
Explicitly describe material boundaries and which piece moves at each beat. A
|
||||
static camera is usually enough to reveal assembly; add movement only when useful.
|
||||
|
||||
Sound specification: synchronize restrained slides, pops, taps, rustles, and snaps.
|
||||
Set `non_diegetic_music: N/A` for the default SFX-only H3 prompt. Do not use silence
|
||||
as a synonym for no background music. Dialogue and subtitles are optional layers.
|
||||
|
||||
Review one established clip as a style/cadence reference for subsequent clips.
|
||||
Change the specific failure: fewer groups for clutter, more physical arrival
|
||||
steps for digital drifting, less aging for dirty paper, stronger final state for
|
||||
composition drift. Do not apply an unrelated aesthetic preset to fix one defect.
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
name: papercraft-stop-motion-explainer
|
||||
description: Turn knowledge topics into layered papercraft stop-motion explainers, including visual metaphors, miniature sets, prompts, storyboards, motion and sound plans, or rendered video. Use for tactile paper dioramas and mechanisms, not ordinary vector cartoons or photographic collage.
|
||||
metadata:
|
||||
compatibility: Portable file-based workflow with sibling h3-prompt-writing; actual media generation uses tools available in the host environment.
|
||||
---
|
||||
|
||||
# Papercraft Stop-Motion Explainer
|
||||
|
||||
Read the [portable workflow](../h3-prompt-writing/references/portable-workflow.md).
|
||||
For scene/prompt planning read [paper staging](references/paper-staging.md).
|
||||
Route a single image prompt, image series, short animation prompt, or storyboard
|
||||
request directly to that artifact; do not force a full film workflow.
|
||||
|
||||
## Knowledge and direction
|
||||
|
||||
Identify the exact topic, source facts, audience, desired learning outcome, and
|
||||
output type. Preserve technical terms and causal meaning. A visual metaphor helps
|
||||
explain a mechanism; it must not turn into a false scientific claim.
|
||||
|
||||
Reuse requested duration and ratio. If unspecified, a 30-second, 16:9 general-audience
|
||||
concept is a reasonable starting point. Offer distinct directions only when useful:
|
||||
pop-up-book journey, paper laboratory, layered cross-section, miniature nature
|
||||
theater, or mechanism board. Each should identify one learning outcome, a paper
|
||||
mechanism that conveys it, and the signature visual moment.
|
||||
|
||||
## Build the paper world
|
||||
|
||||
Design characters only if they help the explanation. Specify paper construction,
|
||||
silhouette, joint/hinge design, replaceable facial parts, gestures, and repeatable
|
||||
identity cues. Keep the central knowledge object readable.
|
||||
|
||||
Plan foreground occluders, midground mechanism, background scenery, and a distant
|
||||
paper plane when the shot benefits from depth. Record each layer's material,
|
||||
thickness, cut/fold edges, shadow, movement, and educational role. Use real paper
|
||||
mechanics such as pull-tabs, sliders, rotating discs, page folds, and pinned joints.
|
||||
Motion is stepped with small pauses and rebounds, not a smooth CG melt.
|
||||
|
||||
Create an asset list by function: characters, scientific objects, information cards,
|
||||
environment, and moving mechanisms. For a complete production, prepare one to three
|
||||
preview briefs or images when useful and authorized. Respect any preview approval
|
||||
requested by the user; otherwise continue under the existing end-to-end scope.
|
||||
|
||||
## Prompts and timeline
|
||||
|
||||
Write a compact storyboard with columns: time, knowledge beat, visual action,
|
||||
paper movement, camera/transition, sound cue. Each shot explains one knowledge
|
||||
step, with a readable pause for a necessary label. Keep ratio, material, light
|
||||
direction, identity, and domain vocabulary consistent across image-series prompts.
|
||||
|
||||
For a short image-to-video prompt, begin from the actual image and animate a few
|
||||
paper mechanisms with restrained push-in or lateral parallax. Preserve thickness,
|
||||
cut edges, texture, and layer shadows. Compile final rendering prompts using the
|
||||
bundled H3 mode guide, with truthful references and local clip timing.
|
||||
|
||||
## Sound, generation, and review
|
||||
|
||||
Map paper flips, cardboard slides, joint taps, soft pops, and rustles to actual
|
||||
motion. Match any requested music to the topic and cultural context rather than
|
||||
automatically using the same whimsical score. Keep speech intelligible. If narration
|
||||
is requested, preserve its approved wording, measure its duration, and adjust the
|
||||
timeline or script within the user's scope; do not silently speed up the voice.
|
||||
|
||||
Use available generation/editing tools, then inspect both educational accuracy
|
||||
and paper construction: readable mechanism, separate layers/shadows, hinged or
|
||||
stepped motion, plausible camera movement, and a clear ending takeaway. Check
|
||||
narration tails, clicks, audio gaps, and music balance. Deliver requested artifacts
|
||||
and state missing rendering capabilities or unresolved scientific/visual issues.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Papercraft Explainers"
|
||||
short_description: "Explain ideas with layered paper mechanisms"
|
||||
default_prompt: "Use $papercraft-stop-motion-explainer to prepare the requested H3 prompts or video with the available tools."
|
||||
@@ -1,48 +0,0 @@
|
||||
# Paper staging and prompt building blocks
|
||||
|
||||
## Materials and depth
|
||||
|
||||
Use layered matte cardboard, visible thickness, fibers, folds, seams, tabs, brads,
|
||||
cut edges, and physical shadows. Separate foreground, mechanism, background, and
|
||||
far background with spacing. Roughly 4–7 layers often suffice; remove layers that
|
||||
hide the explanation. Place reading-heavy labels on stable planes. Use miniature
|
||||
macro photography and moderate depth of field without blurring necessary facts.
|
||||
|
||||
| Plane | Useful content | Motion | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| Foreground | Paper leaves, frame, curtain, props | Small parallax or hinged reveal | Depth without concealing the mechanism |
|
||||
| Midground | Knowledge model, host, arrow/cards | Pull-tab, slider, joint, rotating disc | Main cause and effect |
|
||||
| Background | Diorama set or schematic layers | Slow paper rail shift | Context and hierarchy |
|
||||
| Far plane | Paper sky/board | Restrained change | Stable visual support |
|
||||
|
||||
## Prompt ingredients
|
||||
|
||||
Image: topic and core object → miniature diorama → layered paper construction →
|
||||
specific material edges → physical light/shadows → clear depth → necessary labels.
|
||||
One frame should communicate one main idea, not the entire lesson at once.
|
||||
|
||||
Image series: repeat the style/identity anchors, then state what changes and why
|
||||
that state explains the next knowledge step. Keep the same ratio and light direction.
|
||||
|
||||
Short animation: preserve the supplied image, make a small paper mechanism move,
|
||||
include tiny pauses/rebounds, and select one restrained camera move. Small puppet
|
||||
gestures and background rails can provide life without overwhelming the model.
|
||||
|
||||
## Camera and transitions
|
||||
|
||||
Use slow push-in, lateral parallax, static demonstration, macro detail, or a slight
|
||||
top-down cross-section view. A full orbit requires a set built for that view.
|
||||
Transitions may use a page flip, pop-up unfold, pull-tab, paper cloud, circular
|
||||
mask, cut-paper door, layer separation, confetti, or sticker reveal. Avoid liquid
|
||||
morphing, neon glitches, and glass shatter unless explicitly requested as contrast.
|
||||
|
||||
## Sound and exclusions
|
||||
|
||||
Use a few motion-specific accents: paper flip, scissors, cardboard slide, joint
|
||||
tap, tape peel, or box opening. A score, when appropriate, can use light percussion,
|
||||
plucked instruments, or topic-specific instrumentation. Do not infer narration
|
||||
merely because the video teaches something.
|
||||
|
||||
Keep constraints concise and relevant: no glossy plastic, smooth CG morphs, metal
|
||||
surfaces replacing paper, missing thickness/shadows, or frantic camera movement.
|
||||
Include them in supported prompt fields; do not invent a negative-prompt API field.
|
||||
@@ -1,7 +1,4 @@
|
||||
#!/bin/bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
# vllm-omni customized version
|
||||
# Based on: https://github.com/vllm-project/ci-infra/blob/main/buildkite/bootstrap-amd.sh
|
||||
# Last synced: 2025-12-15
|
||||
@@ -9,13 +6,8 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# The bootstrap runs from the repository root; ShellCheck does not follow
|
||||
# sourced files unless invoked with -x.
|
||||
# shellcheck disable=SC1091
|
||||
source .buildkite/common/scripts/resolve_skip_ci.sh
|
||||
|
||||
AMD_SUITE_SELECTOR=".buildkite/amd/scripts/select_test_suites.py"
|
||||
|
||||
if [[ -z "${RUN_ALL:-}" ]]; then
|
||||
RUN_ALL=0
|
||||
fi
|
||||
@@ -32,20 +24,31 @@ if [[ -z "${AMD_MIRROR_HW:-}" ]]; then
|
||||
AMD_MIRROR_HW="amdproduction"
|
||||
fi
|
||||
|
||||
PR_LABELS=""
|
||||
# Explicit debug suites do not depend on PR labels.
|
||||
if [[ "${BUILDKITE_PULL_REQUEST:-false}" != "false" && -z "${DEBUG_TEST_YAML:-}" ]]; then
|
||||
if ! PR_LABELS=$(curl -fsSL \
|
||||
"https://api.github.com/repos/vllm-project/vllm-omni/pulls/$BUILDKITE_PULL_REQUEST" \
|
||||
| jq -r '.labels[].name'); then
|
||||
echo "ERROR: Could not read PR labels; refusing to select an incorrect AMD test tier." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
fail_fast() {
|
||||
if [[ "${BUILDKITE_PULL_REQUEST:-false}" != "false" ]]; then
|
||||
echo true
|
||||
DISABLE_LABEL="ci-no-fail-fast"
|
||||
# If BUILDKITE_PULL_REQUEST != "false", then we check the PR labels using curl and jq
|
||||
if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
|
||||
PR_LABELS=$(curl -s "https://api.github.com/repos/vllm-project/vllm-omni/pulls/$BUILDKITE_PULL_REQUEST" | jq -r '.labels[].name')
|
||||
if [[ $PR_LABELS == *"$DISABLE_LABEL"* ]]; then
|
||||
echo false
|
||||
else
|
||||
echo true
|
||||
fi
|
||||
else
|
||||
echo false # not a PR or BUILDKITE_PULL_REQUEST not set
|
||||
fi
|
||||
}
|
||||
|
||||
check_run_all_label() {
|
||||
RUN_ALL_LABEL="ready-run-all-tests"
|
||||
# If BUILDKITE_PULL_REQUEST != "false", then we check the PR labels using curl and jq
|
||||
if [ "$BUILDKITE_PULL_REQUEST" != "false" ]; then
|
||||
PR_LABELS=$(curl -s "https://api.github.com/repos/vllm-project/vllm-omni/pulls/$BUILDKITE_PULL_REQUEST" | jq -r '.labels[].name')
|
||||
if [[ $PR_LABELS == *"$RUN_ALL_LABEL"* ]]; then
|
||||
echo true
|
||||
else
|
||||
echo false
|
||||
fi
|
||||
else
|
||||
echo false # not a PR or BUILDKITE_PULL_REQUEST not set
|
||||
fi
|
||||
@@ -55,71 +58,11 @@ if [[ -z "${COV_ENABLED:-}" ]]; then
|
||||
COV_ENABLED=0
|
||||
fi
|
||||
|
||||
resolve_test_specs() {
|
||||
local selected_specs
|
||||
if ! selected_specs=$(python3 "$AMD_SUITE_SELECTOR" \
|
||||
--branch "$BUILDKITE_BRANCH" \
|
||||
--labels "$PR_LABELS" \
|
||||
--debug-test-yaml "${DEBUG_TEST_YAML:-}" \
|
||||
--nightly "$NIGHTLY"); then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TEST_SPECS=()
|
||||
while IFS= read -r suite_spec; do
|
||||
[[ -z "$suite_spec" ]] && continue
|
||||
TEST_SPECS+=("$suite_spec")
|
||||
done <<< "$selected_specs"
|
||||
}
|
||||
|
||||
filter_test_specs_by_skip_ci() {
|
||||
local suite_spec level decision
|
||||
local skip_all=0
|
||||
local -a runnable_specs=()
|
||||
|
||||
for suite_spec in "${TEST_SPECS[@]}"; do
|
||||
case "$suite_spec" in
|
||||
READY_TESTS:*) level="l2" ;;
|
||||
MERGE_TESTS:*) level="l3" ;;
|
||||
NIGHTLY_TESTS:*)
|
||||
# Explicit PR and scheduled nightly selections must survive
|
||||
# L2/L3 diff gating, including docs-only main commits.
|
||||
runnable_specs+=("$suite_spec")
|
||||
continue
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: unknown AMD test suite spec '$suite_spec'" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
if decision=$(python3 "$SKIP_CI_PY" gate amd "$level"); then
|
||||
echo "Skipping AMD $level suite due to $decision decision."
|
||||
[[ "$decision" == "skip-all" ]] && skip_all=1
|
||||
else
|
||||
runnable_specs+=("$suite_spec")
|
||||
fi
|
||||
done
|
||||
|
||||
TEST_SPECS=("${runnable_specs[@]}")
|
||||
if [[ ${#TEST_SPECS[@]} -eq 0 ]]; then
|
||||
if [[ $skip_all -eq 1 ]]; then
|
||||
buildkite-agent annotate \
|
||||
":memo: CI skipped — docs or pytest skip-mark changes only" \
|
||||
--style "info" 2>/dev/null || true
|
||||
fi
|
||||
echo "No AMD suites remain after skip-ci filtering."
|
||||
exit 0
|
||||
fi
|
||||
}
|
||||
|
||||
upload_pipeline() {
|
||||
echo "Uploading pipeline..."
|
||||
# Install minijinja
|
||||
ls .buildkite || buildkite-agent annotate --style error 'Please merge upstream main branch for buildkite CI'
|
||||
curl -sSfL https://github.com/mitsuhiko/minijinja/releases/download/2.3.1/minijinja-cli-installer.sh | sh
|
||||
# Installed by the minijinja bootstrap above and only present on the CI agent.
|
||||
# shellcheck disable=SC1091
|
||||
source /var/lib/buildkite-agent/.cargo/env
|
||||
|
||||
if [[ $BUILDKITE_PIPELINE_SLUG == "fastcheck" ]]; then
|
||||
@@ -133,7 +76,7 @@ upload_pipeline() {
|
||||
# (WIP) Use pipeline generator instead of jinja template
|
||||
if [ -e ".buildkite/amd/pipeline_generator/pipeline_generator.py" ]; then
|
||||
python -m pip install click pydantic
|
||||
python .buildkite/amd/pipeline_generator/pipeline_generator.py --run_all="$RUN_ALL" --list_file_diff="$LIST_FILE_DIFF" --nightly="$NIGHTLY" --mirror_hw="$AMD_MIRROR_HW"
|
||||
python .buildkite/amd/pipeline_generator/pipeline_generator.py --run_all=$RUN_ALL --list_file_diff="$LIST_FILE_DIFF" --nightly="$NIGHTLY" --mirror_hw="$AMD_MIRROR_HW"
|
||||
buildkite-agent pipeline upload .buildkite/amd/pipeline.yaml
|
||||
exit 0
|
||||
fi
|
||||
@@ -146,46 +89,28 @@ upload_pipeline() {
|
||||
|
||||
cd .buildkite/amd
|
||||
|
||||
# Multiple label-selected or debug-selected suites share one image build.
|
||||
if [[ ${#TEST_SPECS[@]} -eq 1 ]]; then
|
||||
TEST_YAML="${TEST_SPECS[0]#*:}"
|
||||
# Select test definition file: merge suite for main, ready suite for PRs.
|
||||
# For debugging, DEBUG_TEST_YAML can override the selection — accepts
|
||||
# "merge" or "ready" (case-insensitive).
|
||||
if [[ -n "${DEBUG_TEST_YAML:-}" ]]; then
|
||||
case "${DEBUG_TEST_YAML,,}" in
|
||||
merge)
|
||||
TEST_YAML="test-amd-merge.yml"
|
||||
;;
|
||||
ready)
|
||||
TEST_YAML="test-amd-ready.yml"
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: DEBUG_TEST_YAML must be 'merge' or 'ready', got '$DEBUG_TEST_YAML'" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
echo "DEBUG_TEST_YAML override: using $TEST_YAML"
|
||||
elif [[ $BUILDKITE_BRANCH == "main" ]]; then
|
||||
TEST_YAML="test-amd-merge.yml"
|
||||
else
|
||||
TEST_YAML=$(mktemp "${TMPDIR:-/tmp}/amd-selected-tests.XXXXXX.yml")
|
||||
python - "$TEST_YAML" "${TEST_SPECS[@]}" <<'PY'
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
output_path, *suite_specs = sys.argv[1:]
|
||||
combined = {"env": {}, "steps": []}
|
||||
|
||||
for suite_spec in suite_specs:
|
||||
group_name, input_path = suite_spec.split(":", 1)
|
||||
with open(input_path, encoding="utf-8") as test_file:
|
||||
suite = yaml.safe_load(test_file)
|
||||
|
||||
for name, value in (suite.get("env") or {}).items():
|
||||
previous = combined["env"].get(name, value)
|
||||
if previous != value:
|
||||
raise ValueError(
|
||||
f"Conflicting environment value for {name}: {previous!r} != {value!r}"
|
||||
)
|
||||
combined["env"][name] = value
|
||||
|
||||
suite_steps = []
|
||||
for entry in suite.get("steps") or []:
|
||||
if "group" in entry:
|
||||
suite_steps.extend(entry.get("steps") or [])
|
||||
else:
|
||||
suite_steps.append(entry)
|
||||
combined["steps"].append({"group": group_name, "steps": suite_steps})
|
||||
|
||||
with open(output_path, "w", encoding="utf-8") as output_file:
|
||||
yaml.safe_dump(combined, output_file, sort_keys=False)
|
||||
PY
|
||||
TEST_YAML="test-amd-ready.yml"
|
||||
fi
|
||||
echo "AMD test suites: ${TEST_SPECS[*]}"
|
||||
|
||||
(
|
||||
set -x
|
||||
@@ -205,20 +130,19 @@ PY
|
||||
> pipeline.yaml
|
||||
)
|
||||
cat pipeline.yaml
|
||||
if [[ "$TEST_YAML" == "${TMPDIR:-/tmp}/amd-selected-tests."*.yml ]]; then
|
||||
rm -f -- "$TEST_YAML"
|
||||
fi
|
||||
buildkite-agent artifact upload pipeline.yaml
|
||||
buildkite-agent pipeline upload pipeline.yaml
|
||||
exit 0
|
||||
}
|
||||
|
||||
get_diff() {
|
||||
git diff --name-only --diff-filter=ACMDR "$(git merge-base origin/main HEAD)"
|
||||
$(git add .)
|
||||
echo $(git diff --name-only --diff-filter=ACMDR $(git merge-base origin/main HEAD))
|
||||
}
|
||||
|
||||
get_diff_main() {
|
||||
git diff --name-only --diff-filter=ACMDR HEAD~1
|
||||
$(git add .)
|
||||
echo $(git diff --name-only --diff-filter=ACMDR HEAD~1)
|
||||
}
|
||||
|
||||
file_diff=$(get_diff)
|
||||
@@ -226,10 +150,12 @@ if [[ $BUILDKITE_BRANCH == "main" ]]; then
|
||||
file_diff=$(get_diff_main)
|
||||
fi
|
||||
|
||||
# Resolve PR tier labels before skip-ci so ready/merge suites can be filtered
|
||||
# independently while an explicitly selected nightly suite passes through.
|
||||
resolve_test_specs
|
||||
filter_test_specs_by_skip_ci
|
||||
# Early exit: unified skip-ci (docs / skip marks) and CI-yaml-only level targeting.
|
||||
if [[ $BUILDKITE_BRANCH == "main" ]]; then
|
||||
gate_bootstrap_ci amd l3
|
||||
else
|
||||
gate_bootstrap_ci amd l2
|
||||
fi
|
||||
|
||||
patterns=(
|
||||
"docker/Dockerfile"
|
||||
@@ -255,7 +181,7 @@ for file in $file_diff; do
|
||||
# First check if file matches any pattern
|
||||
matches_pattern=0
|
||||
for pattern in "${patterns[@]}"; do
|
||||
if [[ $file == "$pattern"* ]] || [[ $file == "$pattern" ]]; then
|
||||
if [[ $file == $pattern* ]] || [[ $file == $pattern ]]; then
|
||||
matches_pattern=1
|
||||
break
|
||||
fi
|
||||
@@ -265,7 +191,7 @@ for file in $file_diff; do
|
||||
if [[ $matches_pattern -eq 1 ]]; then
|
||||
matches_ignore=0
|
||||
for ignore in "${ignore_patterns[@]}"; do
|
||||
if [[ $file == "$ignore"* ]] || [[ $file == "$ignore" ]]; then
|
||||
if [[ $file == $ignore* ]] || [[ $file == $ignore ]]; then
|
||||
matches_ignore=1
|
||||
break
|
||||
fi
|
||||
@@ -279,6 +205,14 @@ for file in $file_diff; do
|
||||
fi
|
||||
done
|
||||
|
||||
# Check for ready-run-all-tests label
|
||||
LABEL_RUN_ALL=$(check_run_all_label)
|
||||
if [[ $LABEL_RUN_ALL == true ]]; then
|
||||
RUN_ALL=1
|
||||
NIGHTLY=1
|
||||
echo "Found 'ready-run-all-tests' label. Running all tests including optional tests."
|
||||
fi
|
||||
|
||||
# Decide whether to use precompiled wheels
|
||||
# Relies on existing patterns array as a basis.
|
||||
if [[ -n "${VLLM_USE_PRECOMPILED:-}" ]]; then
|
||||
|
||||
@@ -1,69 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Build and push the ROCm CI image with a registry-backed BuildKit cache. The
|
||||
# AMD builders are ephemeral, so a local Docker cache cannot survive between
|
||||
# jobs. Keeping the cache beside the runtime image also reuses the registry
|
||||
# credentials already configured on the amd-cpu agents.
|
||||
|
||||
REGISTRY="${ROCM_CI_IMAGE_REGISTRY:-rocm/vllm-omni}"
|
||||
DOCKERFILE="docker/Dockerfile.rocm"
|
||||
BUILDER_NAME="vllm-omni-rocm-builder"
|
||||
ROCM_ARCH="${PYTORCH_ROCM_ARCH:-gfx942}"
|
||||
|
||||
if [[ -z "${BUILDKITE_COMMIT:-}" ]]; then
|
||||
echo "ERROR: BUILDKITE_COMMIT is not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
dependency_files=(
|
||||
"$DOCKERFILE"
|
||||
"${DOCKERFILE}.dockerignore"
|
||||
pyproject.toml
|
||||
setup.py
|
||||
tools/install_torchcodec_rocm.sh
|
||||
)
|
||||
while IFS= read -r dependency_file; do
|
||||
dependency_files+=("$dependency_file")
|
||||
done < <(find requirements -maxdepth 1 -type f -name '*.txt' -print | LC_ALL=C sort)
|
||||
|
||||
for dependency_file in "${dependency_files[@]}"; do
|
||||
if [[ ! -f "$dependency_file" ]]; then
|
||||
echo "ERROR: cache-key input does not exist: $dependency_file" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
CACHE_KEY=$(
|
||||
{
|
||||
printf 'rocm-arch=%s\n' "$ROCM_ARCH"
|
||||
for dependency_file in "${dependency_files[@]}"; do
|
||||
printf 'path=%s\n' "$dependency_file"
|
||||
sha256sum "$dependency_file"
|
||||
done
|
||||
} | sha256sum | cut -c1-16
|
||||
)
|
||||
CACHE_REF="${REGISTRY}:rocm-deps-cache-${CACHE_KEY}"
|
||||
IMAGE_REF="${REGISTRY}:${BUILDKITE_COMMIT}"
|
||||
|
||||
echo "ROCm dependency cache: ${CACHE_REF}"
|
||||
echo "ROCm CI image: ${IMAGE_REF}"
|
||||
|
||||
# The docker-container driver supports exporting the full cache to a registry;
|
||||
# the default docker driver on CI does not.
|
||||
if ! docker buildx inspect "$BUILDER_NAME" >/dev/null 2>&1; then
|
||||
docker buildx create --name "$BUILDER_NAME" --driver docker-container
|
||||
fi
|
||||
docker buildx use "$BUILDER_NAME"
|
||||
docker buildx inspect --bootstrap
|
||||
|
||||
docker buildx build --push --pull --progress=plain \
|
||||
--target test \
|
||||
--build-arg "ARG_PYTORCH_ROCM_ARCH=${ROCM_ARCH}" \
|
||||
--cache-from "type=registry,ref=${CACHE_REF}" \
|
||||
--cache-to "type=registry,ref=${CACHE_REF},mode=max,compression=zstd,ignore-error=true" \
|
||||
--file "$DOCKERFILE" \
|
||||
--tag "$IMAGE_REF" .
|
||||
@@ -1,78 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""Prebuild the BF16 AITER MHA modules used by Cosmos3 on ROCm."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.attention.backends.utils import fa
|
||||
from vllm_omni.platforms import current_omni_platform
|
||||
|
||||
ATTENTION_VARIANTS = (
|
||||
(True, "mha_fwd_bf16_nbias_mask_nlse_ndropout_nqscale"),
|
||||
(False, "mha_fwd_bf16_nbias_nmask_nlse_ndropout_nqscale"),
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not current_omni_platform.is_rocm():
|
||||
raise RuntimeError("The Cosmos3 AITER prewarm must run on ROCm")
|
||||
if fa.flash_attn_func is None:
|
||||
raise RuntimeError("ROCm FlashAttention/AITER is unavailable")
|
||||
|
||||
cache_dir_value = os.environ.get("AITER_JIT_DIR")
|
||||
if not cache_dir_value:
|
||||
raise RuntimeError("AITER_JIT_DIR must identify the job-local cache")
|
||||
cache_dir = Path(cache_dir_value)
|
||||
|
||||
device = torch.device(current_omni_platform.device_type)
|
||||
dtype = torch.bfloat16
|
||||
batch_size = 1
|
||||
sequence_length = 64
|
||||
num_heads = 8
|
||||
head_dim = 64
|
||||
|
||||
torch.manual_seed(42)
|
||||
query = torch.randn(
|
||||
batch_size,
|
||||
sequence_length,
|
||||
num_heads,
|
||||
head_dim,
|
||||
device=device,
|
||||
dtype=dtype,
|
||||
)
|
||||
key = torch.randn_like(query)
|
||||
value = torch.randn_like(query)
|
||||
|
||||
for causal, expected_module in ATTENTION_VARIANTS:
|
||||
output = fa.flash_attn_func(
|
||||
query,
|
||||
key,
|
||||
value,
|
||||
dropout_p=0.0,
|
||||
softmax_scale=head_dim**-0.5,
|
||||
causal=causal,
|
||||
)
|
||||
if isinstance(output, tuple):
|
||||
output = output[0]
|
||||
current_omni_platform.synchronize()
|
||||
|
||||
if output.shape != query.shape:
|
||||
raise RuntimeError(f"Unexpected AITER output shape: {output.shape}")
|
||||
if not torch.isfinite(output).all():
|
||||
raise RuntimeError("AITER prewarm produced non-finite output")
|
||||
|
||||
module_path = cache_dir / f"{expected_module}.so"
|
||||
if not module_path.is_file():
|
||||
raise RuntimeError(f"Expected AITER module was not built: {module_path}")
|
||||
print(f"Prebuilt Cosmos3 AITER module: {module_path}", flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,100 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Select AMD ready/merge/nightly suites after Buildkite creates a build.
|
||||
|
||||
This repository-side selector does not decide which GitHub label events start
|
||||
the external ``vllm-omni-amd-ci`` pipeline. Its Buildkite PR build condition
|
||||
must admit the relevant label event before this code can inspect the labels.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sys
|
||||
from collections.abc import Iterable
|
||||
|
||||
SUITE_SPECS = {
|
||||
"ready": "READY_TESTS:test-amd-ready.yml",
|
||||
"merge": "MERGE_TESTS:test-amd-merge.yml",
|
||||
"nightly": "NIGHTLY_TESTS:test-amd-nightly.yml",
|
||||
}
|
||||
|
||||
|
||||
def _parse_debug_suites(value: str) -> tuple[str, ...]:
|
||||
suites = tuple(item.strip().lower() for item in value.split(",") if item.strip())
|
||||
if not suites:
|
||||
raise ValueError("DEBUG_TEST_YAML did not contain a test suite")
|
||||
|
||||
seen: set[str] = set()
|
||||
for suite in suites:
|
||||
if suite not in SUITE_SPECS:
|
||||
raise ValueError(f"DEBUG_TEST_YAML entries must be 'merge', 'nightly', or 'ready', got {suite!r}")
|
||||
if suite in seen:
|
||||
raise ValueError(f"duplicate DEBUG_TEST_YAML suite {suite!r}")
|
||||
seen.add(suite)
|
||||
return suites
|
||||
|
||||
|
||||
def select_amd_test_suites(
|
||||
*,
|
||||
branch: str,
|
||||
labels: Iterable[str],
|
||||
debug_test_yaml: str = "",
|
||||
nightly: bool = False,
|
||||
) -> tuple[str, ...]:
|
||||
"""Return ordered suite names while preserving legacy unlabeled PRs."""
|
||||
|
||||
if debug_test_yaml:
|
||||
return _parse_debug_suites(debug_test_yaml)
|
||||
if branch == "main":
|
||||
return ("nightly",) if nightly else ("merge",)
|
||||
|
||||
label_set = {label.strip() for label in labels if label.strip()}
|
||||
selected = tuple(
|
||||
suite
|
||||
for label, suite in (
|
||||
("ready", "ready"),
|
||||
("merge-test", "merge"),
|
||||
("nightly-test", "nightly"),
|
||||
)
|
||||
if label in label_set
|
||||
)
|
||||
# AMD historically uploaded the ready suite for every admitted PR build.
|
||||
# Preserve that fallback when no explicit tier label is present.
|
||||
return selected or ("ready",)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--branch", required=True)
|
||||
parser.add_argument("--labels", default="")
|
||||
parser.add_argument("--debug-test-yaml", default="")
|
||||
parser.add_argument("--nightly", type=int, choices=(0, 1), default=0)
|
||||
args = parser.parse_args()
|
||||
|
||||
labels = tuple(args.labels.splitlines())
|
||||
try:
|
||||
suites = select_amd_test_suites(
|
||||
branch=args.branch,
|
||||
labels=labels,
|
||||
debug_test_yaml=args.debug_test_yaml,
|
||||
nightly=bool(args.nightly),
|
||||
)
|
||||
except ValueError as exc:
|
||||
parser.error(str(exc))
|
||||
|
||||
label_set = {label.strip() for label in labels if label.strip()}
|
||||
if args.branch != "main" and not args.debug_test_yaml and not ({"ready", "merge-test", "nightly-test"} & label_set):
|
||||
print(
|
||||
"No AMD ready/merge/nightly label found; preserving the legacy ready-suite fallback.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
|
||||
for suite in suites:
|
||||
print(SUITE_SPECS[suite])
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -9,63 +9,27 @@ steps:
|
||||
steps:
|
||||
- label: "Simple · Model Executor Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 3
|
||||
parallelism: 2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu and not omni' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Omni Processor Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- "pytest -sv tests/model_executor/models/test_omni_processing.py -m 'core_model and cpu and omni'"
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Diffusion Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 4
|
||||
parallelism: 2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 45
|
||||
# Buildkite's status-based rule cannot prove every exit 134 is a GPU hang.
|
||||
# It retries 134 once; the Kubernetes job retry starts a fresh pod.
|
||||
# Assertions (1), timeouts (124), and OOMs (137) remain non-retryable.
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 134
|
||||
limit: 1
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
# Keep multi-GPU tests out of this one-GPU pool even when their module
|
||||
# also carries the cpu marker for unit tests.
|
||||
# Keep π0.5 CPU units out of these shards: CUDA READY/MERGE already run
|
||||
# the same suite mandatorily, and AMD covers it non-blocking in nightly.
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu and not (cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)' --ignore=tests/diffusion/cache/test_teacache_extractors.py --ignore=tests/diffusion/models/pi05/test_pi05_units.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Diffusion · AuK VAE GPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 35m
|
||||
pytest -sv tests/diffusion/models/auk/test_auk_vae_fastpath.py
|
||||
-m 'core_model and rocm and cards_1' --run-level core_model
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Engine&Entrypoints Test"
|
||||
agent_pool: mi300_1
|
||||
@@ -89,25 +53,6 @@ steps:
|
||||
|
||||
- group: ":card_index_dividers: Diffusion Test"
|
||||
steps:
|
||||
# AR paged attention is real GPU work, including cold AITER compilation.
|
||||
# Keep its complete numerical matrix out of the CPU-marked Simple shards.
|
||||
- label: "ROCm · AR Diffusion Paged Attention GPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=1
|
||||
- export VLLM_OMNI_AR_FA_REQUIRED=1
|
||||
- export AITER_JIT_DIR="/tmp/vllm-omni-ar-aiter-$$BUILDKITE_JOB_ID"
|
||||
- export TORCH_EXTENSIONS_DIR="/tmp/vllm-omni-ar-torch-extensions-$$BUILDKITE_JOB_ID"
|
||||
- mkdir -p "$$AITER_JIT_DIR" "$$TORCH_EXTENSIONS_DIR"
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 15m
|
||||
pytest -sv tests/diffusion/ar_diffusion/test_paged_attention.py
|
||||
-m 'core_model and rocm and MI325 and cards_1' --run-level core_model
|
||||
|
||||
- label: "Diffusion · LoRA Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
@@ -126,46 +71,26 @@ steps:
|
||||
commands:
|
||||
- export VLLM_LOGGING_LEVEL=DEBUG
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- export DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA
|
||||
- pytest -s -v tests/diffusion/offloader/test_diffusion_cpu_offload.py
|
||||
- >-
|
||||
pytest -s -v tests/diffusion/offloader/test_diffusion_layerwise_offload.py
|
||||
--deselect='tests/diffusion/offloader/test_diffusion_layerwise_offload.py::test_layerwise_offload_diffusion_model[stabilityai/stable-audio-open-1.0]'
|
||||
- |
|
||||
bash -c '
|
||||
set +e
|
||||
pytest -s -v tests/diffusion/offloader/test_diffusion_cpu_offload.py
|
||||
EXIT1=\$?
|
||||
pytest -s -v tests/diffusion/offloader/test_diffusion_layerwise_offload.py
|
||||
EXIT2=\$?
|
||||
exit \$((EXIT1 | EXIT2))
|
||||
'
|
||||
|
||||
# TODO: Diagnose unstable Stable Audio layerwise-offload memory savings on ROCm.
|
||||
- label: "Diffusion Stable Audio Layerwise Offload Test (quarantined)"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
commands:
|
||||
- export VLLM_LOGGING_LEVEL=DEBUG
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- >-
|
||||
pytest -s -v
|
||||
'tests/diffusion/offloader/test_diffusion_layerwise_offload.py::test_layerwise_offload_diffusion_model[stabilityai/stable-audio-open-1.0]'
|
||||
|
||||
- label: "Diffusion Sequence Parallelism Test (Need 4 GPUs) · Shard 1/2"
|
||||
- label: "Diffusion Sequence Parallelism Test (Need 4 GPUs)"
|
||||
agent_pool: mi300_4
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
timeout_in_minutes: 40
|
||||
grade: Blocking
|
||||
commands:
|
||||
- export VLLM_LOGGING_LEVEL=DEBUG
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v tests/diffusion/distributed/test_sequence_parallel.py
|
||||
|
||||
- label: "Diffusion Sequence Parallelism Test (Need 4 GPUs) · Shard 2/2"
|
||||
agent_pool: mi300_4
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
timeout_in_minutes: 40
|
||||
grade: Blocking
|
||||
commands:
|
||||
- export VLLM_LOGGING_LEVEL=DEBUG
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v tests/diffusion/distributed/test_ulysses_uaa_perf.py
|
||||
- timeout 20m pytest -s -v tests/diffusion/distributed/test_sequence_parallel.py
|
||||
- timeout 20m pytest -s -v tests/diffusion/distributed/test_ulysses_uaa_perf.py
|
||||
|
||||
# merge-only tests
|
||||
- label: "Diffusion Tensor Parallelism Test"
|
||||
@@ -175,11 +100,6 @@ steps:
|
||||
grade: Blocking
|
||||
commands:
|
||||
- pytest -s -v tests/diffusion/distributed/test_tensor_parallel.py
|
||||
- >-
|
||||
timeout 20m pytest -s -v
|
||||
tests/diffusion/attention/test_ulysses_uaa.py::test_ulysses_uaa_2d_mask_layout_matches_baseline
|
||||
-m 'core_model and cards_2'
|
||||
--run-level "core_model"
|
||||
|
||||
- label: "Diffusion Model Test"
|
||||
agent_pool: mi300_1
|
||||
@@ -187,9 +107,7 @@ steps:
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
commands:
|
||||
# A cold Z-Image run spends about six minutes loading weights before
|
||||
# the first-request AITER compilation, which can itself exceed 23m.
|
||||
- timeout 55m pytest -s -v tests/e2e/offline_inference/test_z_image.py
|
||||
- timeout 30m pytest -s -v tests/e2e/offline_inference/test_z_image.py
|
||||
|
||||
- label: "Diffusion GPU Worker Test"
|
||||
agent_pool: mi300_2
|
||||
@@ -253,45 +171,21 @@ steps:
|
||||
grade: Blocking
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py tests/e2e/online_serving/test_qwen3_omni.py tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
- pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py tests/e2e/online_serving/test_qwen3_omni.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
|
||||
- label: "Qwen3-TTS CustomVoice E2E Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 3
|
||||
parallelism: 2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 80
|
||||
commands:
|
||||
- |
|
||||
timeout --signal=TERM --kill-after=2m 80m bash -c '
|
||||
bash -c '
|
||||
export VLLM_LOGGING_LEVEL=DEBUG
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
TESTS=(
|
||||
"tests/e2e/online_serving/test_qwen3_tts_customvoice.py::test_default_cuda_graph_startup[async_chunk]"
|
||||
"tests/e2e/online_serving/test_qwen3_tts_customvoice.py::test_text_to_audio_002[async_chunk]"
|
||||
"tests/e2e/offline_inference/test_qwen3_tts_customvoice.py::test_text_to_audio_001[no_cuda_graph]"
|
||||
)
|
||||
TEST="$${TESTS[$$BUILDKITE_PARALLEL_JOB]}"
|
||||
pytest -s -v "$$TEST" -m "advanced_model" --run-level "advanced_model"
|
||||
'
|
||||
|
||||
# TODO: Restore this case to the blocking sharded step once its long-running
|
||||
# Whisper validation on AMD CI is resolved.
|
||||
- label: "Qwen3-TTS CustomVoice async_chunk Test (quarantined)"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 80
|
||||
commands:
|
||||
- |
|
||||
timeout --signal=TERM --kill-after=2m 75m bash -c '
|
||||
export VLLM_LOGGING_LEVEL=DEBUG
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
pytest -s -v "tests/e2e/online_serving/test_qwen3_tts_customvoice.py::test_text_to_audio_001[async_chunk]" -m "advanced_model" --run-level "advanced_model"
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py tests/e2e/offline_inference/test_qwen3_tts_customvoice.py -m "advanced_model" --run-level "advanced_model" --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
|
||||
'
|
||||
|
||||
- label: "Qwen3-TTS Base E2E Test"
|
||||
@@ -301,10 +195,13 @@ steps:
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- export VLLM_LOGGING_LEVEL=DEBUG
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- timeout 40m pytest -s -v tests/e2e/online_serving/test_qwen3_tts_base.py tests/e2e/offline_inference/test_qwen3_tts_base.py -m "advanced_model and cuda" --run-level "advanced_model"
|
||||
- |
|
||||
timeout 40m bash -c '
|
||||
export VLLM_LOGGING_LEVEL=DEBUG
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen3_tts_base.py tests/e2e/offline_inference/test_qwen3_tts_base.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
'
|
||||
|
||||
# TODO: Bagel test on ROCm is very unstable. @tjtanaa
|
||||
# Need to debug before reneable numerical changes across large PRs
|
||||
|
||||
@@ -1,170 +0,0 @@
|
||||
# AMD MI300 `full_model` nightly coverage during burn-in.
|
||||
# All jobs are non-blocking until the lane has stable runtime evidence.
|
||||
# Historical build #11646 was red in Function Expansion and Accuracy (one
|
||||
# async_chunk similarity mismatch; Seed-TTS WER 0.376480 > 0.35).
|
||||
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
HF_HUB_DOWNLOAD_TIMEOUT: 300
|
||||
HF_HUB_ETAG_TIMEOUT: 60
|
||||
|
||||
steps:
|
||||
# MI325 is the repository's legacy ROCm pytest marker. The AMD template maps
|
||||
# agent_pool to the current MI300 Buildkite queues independently of that mark.
|
||||
- group: ":full_moon: Qwen3-Omni MI300 Nightly"
|
||||
steps:
|
||||
- label: "Qwen3-Omni Function Expansion"
|
||||
agent_pool: mi300_2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_expansion.py -m "full_model and rocm and MI325 and cards_2" --run-level "full_model"
|
||||
|
||||
- label: "Qwen3-Omni Accuracy"
|
||||
agent_pool: mi300_2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc/*.json
|
||||
commands:
|
||||
- export SEED_TTS_WER_EVAL=1
|
||||
# PyTorch ROCm uses the torch.cuda device namespace.
|
||||
- export SEED_TTS_EVAL_DEVICE=cuda:1
|
||||
- pytest -s -v tests/e2e/accuracy/qwen3_omni/test_qwen3_omni.py -m "full_model and rocm and MI325 and cards_2" --run-level "full_model"
|
||||
# Native tests run from the image copy under /app/vllm-omni, while
|
||||
# Buildkite's artifact hook searches the checkout workspace.
|
||||
- |
|
||||
if [[ "$$PWD" != "$${BUILDKITE_BUILD_CHECKOUT_PATH:?}" ]]; then
|
||||
artifact_dir="$${BUILDKITE_BUILD_CHECKOUT_PATH}/tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc"
|
||||
mkdir -p "$$artifact_dir"
|
||||
cp -v tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc/*.json "$$artifact_dir/"
|
||||
fi
|
||||
|
||||
- label: "Qwen3-Omni Documentation Examples"
|
||||
agent_pool: mi300_2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/examples/online_serving/test_qwen3_omni.py -m "full_model and rocm and MI325 and cards_2" --run-level "full_model"
|
||||
|
||||
- label: "Qwen3-Omni AITER-on Smoke"
|
||||
agent_pool: mi300_2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=1
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_expansion.py::test_text_to_text_audio_001 -m "full_model and rocm and MI325 and cards_2" --run-level "full_model"
|
||||
|
||||
- group: ":speaker: CosyVoice3 MI300 Nightly"
|
||||
steps:
|
||||
# Exercise all sync/async and Chinese/English cases with the platform's
|
||||
# default attention backend. On ROCm this retains end-to-end AITER
|
||||
# coverage without putting its cold JIT build on the READY critical path.
|
||||
- label: "CosyVoice3-TTS E2E Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 90
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 134
|
||||
limit: 1
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py -m "slow" --run-level "core_model"
|
||||
|
||||
# π0.5 CPU units are mandatory on CUDA READY/MERGE. On AMD they exercise the
|
||||
# ROCm container/import path with CPU tensors only, so keep them off the
|
||||
# blocking READY/MERGE critical path and retain non-blocking nightly coverage.
|
||||
- group: ":robot_face: Pi0.5 CPU Nightly"
|
||||
steps:
|
||||
- label: "Simple · Pi0.5 CPU Test"
|
||||
agent_pool: mi300_1
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- "pytest -sv tests/diffusion/models/pi05/test_pi05_units.py -m 'core_model and cpu'"
|
||||
|
||||
- group: ":full_moon: LingBot-Video MI300 Nightly"
|
||||
steps:
|
||||
- label: "LingBot-Video Dense CFG-off Smoke"
|
||||
agent_pool: mi300_1
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 90
|
||||
artifact_paths:
|
||||
- artifacts/rocm-lingbot-nightly/**/*
|
||||
commands:
|
||||
- pytest --collect-only -q tests/e2e/online_serving/test_lingbot_video_expansion.py::test_cfg_off -m "full_model and diffusion and rocm and MI325 and cards_1" --run-level full_model
|
||||
- pytest -s -v tests/e2e/online_serving/test_lingbot_video_expansion.py::test_cfg_off -m "full_model and diffusion and rocm and MI325 and cards_1" --run-level full_model --junitxml=/tmp/lingbot-pytest.xml
|
||||
- |
|
||||
artifact_dir="$${BUILDKITE_BUILD_CHECKOUT_PATH:?}/artifacts/rocm-lingbot-nightly"
|
||||
mkdir -p "$$artifact_dir"
|
||||
cp -v /tmp/lingbot-pytest.xml "$$artifact_dir/"
|
||||
|
||||
- group: ":full_moon: Cosmos3 MI300 Nightly"
|
||||
steps:
|
||||
- label: "Cosmos3 T2I Function"
|
||||
agent_pool: mi300_1
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 90
|
||||
artifact_paths:
|
||||
- artifacts/rocm-cosmos3-nightly/**/*
|
||||
commands:
|
||||
# Cosmos3 uses both masked and unmasked BF16 AITER MHA modules. Build
|
||||
# them before server startup, then reuse the same job-local cache
|
||||
# during the smoke test.
|
||||
- export VLLM_ROCM_USE_AITER=1
|
||||
- export AITER_JIT_DIR="/tmp/vllm-omni-cosmos3-aiter-$$BUILDKITE_JOB_ID"
|
||||
- export TORCH_EXTENSIONS_DIR="/tmp/vllm-omni-cosmos3-torch-extensions-$$BUILDKITE_JOB_ID"
|
||||
- mkdir -p "$$AITER_JIT_DIR" "$$TORCH_EXTENSIONS_DIR"
|
||||
- pytest --collect-only -q tests/e2e/online_serving/test_cosmos3.py::test_text_to_image_001 -m "core_model and rocm and MI325 and cards_1" --run-level full_model
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 55m
|
||||
python .buildkite/amd/scripts/prewarm_cosmos3_aiter.py
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 25m
|
||||
pytest -s -v
|
||||
tests/e2e/online_serving/test_cosmos3.py::test_text_to_image_001
|
||||
-m "core_model and rocm and MI325 and cards_1"
|
||||
--run-level full_model
|
||||
--junitxml=/tmp/cosmos3-pytest.xml
|
||||
- |
|
||||
artifact_dir="$${BUILDKITE_BUILD_CHECKOUT_PATH:?}/artifacts/rocm-cosmos3-nightly"
|
||||
mkdir -p "$$artifact_dir"
|
||||
cp -v /tmp/cosmos3-pytest.xml "$$artifact_dir/"
|
||||
|
||||
- group: ":full_moon: HunyuanImage3 MI300 Nightly"
|
||||
steps:
|
||||
- label: "HunyuanImage3 Offline Pixel Accuracy"
|
||||
agent_pool: mi300_4
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 180
|
||||
env:
|
||||
HUNYUAN_IMAGE3_MODEL: "tencent/HunyuanImage-3.0-Instruct"
|
||||
HUNYUAN_IMAGE3_DEVICES: "0,1,2,3"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
artifact_paths:
|
||||
- artifacts/rocm-hunyuanimage3/**/*
|
||||
commands:
|
||||
- pytest --collect-only -q tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py::test_hunyuan_image3_pixel_accuracy_offline -m "full_model and rocm and MI325 and cards_4" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py::test_hunyuan_image3_pixel_accuracy_offline -m "full_model and rocm and MI325 and cards_4" --run-level full_model --junitxml=/tmp/hunyuanimage3-pytest.xml
|
||||
- |
|
||||
artifact_dir="$${BUILDKITE_BUILD_CHECKOUT_PATH:?}/artifacts/rocm-hunyuanimage3"
|
||||
mkdir -p "$$artifact_dir"
|
||||
cp -v tests/e2e/accuracy/artifacts/HunyuanImage-3_0-Instruct/*.png "$$artifact_dir/"
|
||||
cp -v /tmp/hunyuanimage3-pytest.xml "$$artifact_dir/"
|
||||
@@ -9,63 +9,26 @@ steps:
|
||||
steps:
|
||||
- label: "Simple · Model Executor Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 3
|
||||
parallelism: 2
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu and not omni' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Simple · Omni Processor Test"
|
||||
- label: "Simple · Diffusion Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- "pytest -sv tests/model_executor/models/test_omni_processing.py -m 'core_model and cpu and omni'"
|
||||
|
||||
- label: "Simple · Diffusion Test · Shard %N/%t"
|
||||
agent_pool: mi300_1
|
||||
parallelism: 4
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 45
|
||||
# Buildkite's status-based rule cannot prove every exit 134 is a GPU hang.
|
||||
# It retries 134 once; the Kubernetes job retry starts a fresh pod.
|
||||
# Assertions (1), timeouts (124), and OOMs (137) remain non-retryable.
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 134
|
||||
limit: 1
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
# ignore test_teacache_extractors.py because it use rocm gemm kernel from vLLM
|
||||
# that is not supported on CPU
|
||||
# Keep multi-GPU tests out of this one-GPU pool even when their module
|
||||
# also carries the cpu marker for unit tests.
|
||||
# Keep π0.5 CPU units out of these shards: CUDA READY/MERGE already run
|
||||
# the same suite mandatorily, and AMD covers it non-blocking in nightly.
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu and not (cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)' --ignore=tests/diffusion/cache/test_teacache_extractors.py --ignore=tests/diffusion/models/pi05/test_pi05_units.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
|
||||
- label: "Diffusion · AuK VAE GPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=0
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 35m
|
||||
pytest -sv tests/diffusion/models/auk/test_auk_vae_fastpath.py
|
||||
-m 'core_model and rocm and cards_1' --run-level core_model
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu' --ignore=tests/diffusion/cache/test_teacache_extractors.py"
|
||||
|
||||
- label: "Simple · Engine&Entrypoints Test"
|
||||
agent_pool: mi300_1
|
||||
@@ -106,55 +69,6 @@ steps:
|
||||
|
||||
- group: ":card_index_dividers: Diffusion Test"
|
||||
steps:
|
||||
# Keep real FlashAttention execution out of the CPU-marked Simple lane.
|
||||
# AITER may need to build several kernels on a cold worker, so run one
|
||||
# pytest worker with job-local build directories and a bounded timeout.
|
||||
- label: "ROCm · FlashAttention/AITER GPU Coverage"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=1
|
||||
- export AITER_JIT_DIR="/tmp/vllm-omni-aiter-$$BUILDKITE_JOB_ID"
|
||||
- export TORCH_EXTENSIONS_DIR="/tmp/vllm-omni-torch-extensions-$$BUILDKITE_JOB_ID"
|
||||
- mkdir -p "$$AITER_JIT_DIR" "$$TORCH_EXTENSIONS_DIR"
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 55m
|
||||
pytest -s -v -n 1
|
||||
tests/diffusion/attention/
|
||||
-m 'core_model and rocm and MI325 and cards_1'
|
||||
--run-level "core_model"
|
||||
# MammothModa2 exercises the same two AITER MHA modules built by the
|
||||
# preceding suite. Keep its default-backend run in this job so a cold
|
||||
# worker compiles each module once instead of timing out the generic
|
||||
# model job.
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 10m
|
||||
pytest -s -v -n 1
|
||||
tests/diffusion/models/mammoth_moda2/test_dit_attention_cuda.py
|
||||
--run-level "core_model"
|
||||
|
||||
# AR paged attention is real GPU work, including cold AITER compilation.
|
||||
# Keep its complete numerical matrix out of the CPU-marked Simple shards.
|
||||
- label: "ROCm · AR Diffusion Paged Attention GPU Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- export VLLM_ROCM_USE_AITER=1
|
||||
- export VLLM_OMNI_AR_FA_REQUIRED=1
|
||||
- export AITER_JIT_DIR="/tmp/vllm-omni-ar-aiter-$$BUILDKITE_JOB_ID"
|
||||
- export TORCH_EXTENSIONS_DIR="/tmp/vllm-omni-ar-torch-extensions-$$BUILDKITE_JOB_ID"
|
||||
- mkdir -p "$$AITER_JIT_DIR" "$$TORCH_EXTENSIONS_DIR"
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=2m 15m
|
||||
pytest -sv tests/diffusion/ar_diffusion/test_paged_attention.py
|
||||
-m 'core_model and rocm and MI325 and cards_1' --run-level core_model
|
||||
|
||||
- label: "Diffusion · Batch Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
@@ -193,16 +107,6 @@ steps:
|
||||
- export VLLM_LOGGING_LEVEL=DEBUG
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- timeout 20m pytest -s -v tests/diffusion/distributed/test_sequence_parallel.py -m core_model
|
||||
- >-
|
||||
timeout 20m pytest -s -v
|
||||
tests/diffusion/attention/test_ulysses_uaa.py::test_ulysses_uaa_2d_mask_layout_matches_baseline
|
||||
-m 'core_model and cards_2'
|
||||
--run-level "core_model"
|
||||
- >-
|
||||
timeout 20m pytest -s -v
|
||||
tests/diffusion/models/ltx2/test_ltx2_transformer_ulysses.py
|
||||
-m 'core_model and rocm and cards_2'
|
||||
--run-level "core_model"
|
||||
|
||||
- label: "Diffusion GPU Worker Test"
|
||||
agent_pool: mi300_2
|
||||
@@ -218,22 +122,7 @@ steps:
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
commands:
|
||||
# Preserve legacy one-GPU tests without cards_1 while excluding every
|
||||
# registered multi-GPU marker from this one-GPU job.
|
||||
- >-
|
||||
timeout 15m pytest -sv tests/diffusion/models/
|
||||
--ignore=tests/diffusion/models/auk/test_auk_vae_fastpath.py
|
||||
--ignore=tests/diffusion/models/mammoth_moda2/test_dit_attention_cuda.py
|
||||
-m 'core_model and cuda and not
|
||||
(cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)'
|
||||
--run-level "core_model"
|
||||
# Keep the model-specific numerical coverage blocking without paying
|
||||
# the default ROCm backend's cold AITER compilation cost in this lane.
|
||||
- >-
|
||||
timeout 10m env DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA
|
||||
pytest -sv
|
||||
tests/diffusion/models/mammoth_moda2/test_dit_attention_cuda.py
|
||||
--run-level "core_model"
|
||||
- timeout 15m pytest -sv tests/diffusion/models/ -m 'core_model and cuda' --run-level "core_model"
|
||||
|
||||
- label: "Diffusion · Wan22 Test"
|
||||
agent_pool: mi300_1
|
||||
@@ -260,38 +149,6 @@ steps:
|
||||
# - export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
# - timeout 20m pytest -s -v tests/e2e/offline_inference/test_stable_audio_expansion.py -m "advanced_model and diffusion and L4" --run-level advanced_model
|
||||
|
||||
- label: "ROCm · Model Executor GPU Coverage (R2-01)"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 20
|
||||
artifact_paths:
|
||||
- "artifacts/rocm-r2-01/**/*"
|
||||
commands:
|
||||
# Native AMD tests run from /app/vllm-omni, while Buildkite uploads
|
||||
# artifacts relative to its checkout directory.
|
||||
- export R2_01_ARTIFACT_DIR="$$BUILDKITE_BUILD_CHECKOUT_PATH/artifacts/rocm-r2-01"
|
||||
- mkdir -p "$$R2_01_ARTIFACT_DIR"
|
||||
# The dedicated Engine Test below owns the only selected tests/engine case;
|
||||
# do not duplicate its model startup in this time-bounded coverage lane.
|
||||
# The CUDA ready job also covers tests/worker/test_batched_omni_output.py.
|
||||
# Keep the initial ROCm burn-in scoped to tests/model_executor/.
|
||||
# Verbose pytest output and JUnit retain node IDs without a second
|
||||
# collection-only pass over the same GPU-backed test tree.
|
||||
- >-
|
||||
pytest -s -v -ra
|
||||
tests/model_executor/
|
||||
-m 'core_model and cuda and not nvidia_only and not
|
||||
(cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)'
|
||||
--run-level "core_model"
|
||||
--durations=0
|
||||
--junitxml="$$R2_01_ARTIFACT_DIR/pytest.xml"
|
||||
2>&1 | tee "$$R2_01_ARTIFACT_DIR/pytest.log"
|
||||
- >-
|
||||
tail -n 100 "$$R2_01_ARTIFACT_DIR/pytest.log" >
|
||||
"$$R2_01_ARTIFACT_DIR/pytest-summary.txt"
|
||||
|
||||
- label: "Engine Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
@@ -304,71 +161,6 @@ steps:
|
||||
pytest -s -v tests/engine/test_async_omni_engine_abort.py
|
||||
'
|
||||
|
||||
- label: "ROCm · Entrypoints GPU Coverage (R2-02)"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: NonBlocking
|
||||
timeout_in_minutes: 30
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
artifact_paths:
|
||||
- "artifacts/rocm-r2-02/**/*"
|
||||
commands:
|
||||
# Native AMD tests run from /app/vllm-omni, while Buildkite uploads
|
||||
# artifacts relative to its checkout directory.
|
||||
- export R2_02_ARTIFACT_DIR="$$BUILDKITE_BUILD_CHECKOUT_PATH/artifacts/rocm-r2-02"
|
||||
- mkdir -p "$$R2_02_ARTIFACT_DIR"
|
||||
- >-
|
||||
ps -eo pid=,ppid=,lstart=,stat=,comm= >
|
||||
"$$R2_02_ARTIFACT_DIR/processes-before.txt"
|
||||
# Verbose pytest output and JUnit retain selected node IDs without a
|
||||
# second collection-only pass over the same GPU-backed test tree.
|
||||
# The inner timeout leaves five minutes for teardown verification and
|
||||
# artifact generation before the Buildkite step timeout.
|
||||
- >-
|
||||
timeout --signal=TERM --kill-after=1m 25m
|
||||
pytest -s -v -ra
|
||||
tests/entrypoints/
|
||||
-m 'core_model and cuda and not
|
||||
(cards_2 or cards_3 or cards_4 or cards_5 or cards_6 or cards_7 or cards_8)'
|
||||
--run-level "core_model"
|
||||
--durations=0
|
||||
--junitxml="$$R2_02_ARTIFACT_DIR/pytest.xml"
|
||||
2>&1 | tee "$$R2_02_ARTIFACT_DIR/pytest.log"
|
||||
- >-
|
||||
tail -n 100 "$$R2_02_ARTIFACT_DIR/pytest.log" >
|
||||
"$$R2_02_ARTIFACT_DIR/pytest-summary.txt"
|
||||
- |
|
||||
sleep 5
|
||||
ps -eo pid=,ppid=,lstart=,stat=,comm= > "$$R2_02_ARTIFACT_DIR/processes-after.txt"
|
||||
# Compare PID plus process start time so PID reuse cannot hide a leak.
|
||||
# Double dollars preserve awk field references through Buildkite interpolation.
|
||||
awk '
|
||||
NR == FNR {
|
||||
if ($$9 != "ps") {
|
||||
baseline[$$1 "|" $$3 "|" $$4 "|" $$5 "|" $$6 "|" $$7] = 1
|
||||
}
|
||||
next
|
||||
}
|
||||
$$9 != "ps" {
|
||||
identity = $$1 "|" $$3 "|" $$4 "|" $$5 "|" $$6 "|" $$7
|
||||
if (!(identity in baseline)) {
|
||||
print
|
||||
leaked = 1
|
||||
}
|
||||
}
|
||||
END {
|
||||
if (leaked) {
|
||||
print "status=FAIL leaked_processes_detected=1"
|
||||
exit 1
|
||||
}
|
||||
print "status=PASS leaked_processes=0"
|
||||
}
|
||||
' "$$R2_02_ARTIFACT_DIR/processes-before.txt" \
|
||||
"$$R2_02_ARTIFACT_DIR/processes-after.txt" \
|
||||
2>&1 | tee "$$R2_02_ARTIFACT_DIR/process-cleanup.txt"
|
||||
|
||||
# - label: "Omni Model Test Qwen3-Omni"
|
||||
# agent_pool: mi300_2
|
||||
# depends_on: amd-build
|
||||
@@ -393,23 +185,16 @@ steps:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- timeout 30m pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py -m "core_model" --run-level "core_model"
|
||||
|
||||
# Keep a fast model/server integration check in the blocking lane. The
|
||||
# full default-backend suite remains in the nightly pipeline because a
|
||||
# cold ROCm AITER MHA build can add more than 25 minutes to this job.
|
||||
- label: "CosyVoice3-TTS E2E Smoke (SDPA)"
|
||||
- label: "CosyVoice3-TTS E2E Test"
|
||||
agent_pool: mi300_1
|
||||
depends_on: amd-build
|
||||
mirror_hardwares: [amdproduction]
|
||||
grade: Blocking
|
||||
# A ROCm hardware hang aborts the interpreter with 134. Let Buildkite
|
||||
# create one fresh pod instead of retrying on the poisoned GPU process.
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: 134
|
||||
limit: 1
|
||||
commands:
|
||||
- export DIFFUSION_ATTENTION_BACKEND=TORCH_SDPA
|
||||
- pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py::test_voice_clone_zh_002 -m "slow" --run-level "core_model"
|
||||
- |
|
||||
timeout 20m bash -c '
|
||||
pytest -s -v tests/e2e/online_serving/test_cosyvoice3_tts_expansion.py -m "slow" --run-level "core_model"
|
||||
'
|
||||
|
||||
# TODO: Bagel test on ROCm is very unstable. @tjtanaa
|
||||
# Need to debug before reneable numerical changes across large PRs
|
||||
|
||||
@@ -3,8 +3,7 @@
|
||||
Last synced: 2026-08-04
|
||||
Modifications: Use the shared MI300 queues and Omni command-passing contract
|
||||
#}
|
||||
{% set docker_registry_amd = "rocm/vllm-omni" %}
|
||||
{% set docker_image_amd = docker_registry_amd ~ ":$BUILDKITE_COMMIT" %}
|
||||
{% set docker_image_amd = "rocm/vllm-omni:$BUILDKITE_COMMIT" %}
|
||||
{% set default_working_dir = "/app/vllm-omni" %}
|
||||
|
||||
{# MI300 runs tests natively in Kubernetes; Docker-in-Docker is unavailable. #}
|
||||
@@ -75,12 +74,11 @@ steps:
|
||||
depends_on: ~
|
||||
soft_fail: false
|
||||
commands:
|
||||
- "bash .buildkite/amd/scripts/build-ci-image.sh"
|
||||
- "docker build -f docker/Dockerfile.rocm -t {{ docker_image_amd }} --target test --build-arg ARG_PYTORCH_ROCM_ARCH=gfx942 --progress plain ."
|
||||
- "docker push {{ docker_image_amd }}"
|
||||
key: "amd-build"
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
PYTORCH_ROCM_ARCH: "gfx942"
|
||||
ROCM_CI_IMAGE_REGISTRY: "{{ docker_registry_amd }}"
|
||||
retry:
|
||||
automatic:
|
||||
- exit_status: -1 # Agent was lost
|
||||
@@ -158,14 +156,6 @@ steps:
|
||||
{% for path in step.artifact_paths %}
|
||||
- "{{ path }}"
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.retry %}
|
||||
retry:
|
||||
automatic:
|
||||
{% for retry_rule in step.retry.automatic %}
|
||||
- exit_status: {{ retry_rule.exit_status }}
|
||||
limit: {{ retry_rule.limit }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.timeout_in_minutes %}
|
||||
timeout_in_minutes: {{ step.timeout_in_minutes }}
|
||||
@@ -234,14 +224,6 @@ steps:
|
||||
{% for path in step.artifact_paths %}
|
||||
- "{{ path }}"
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.retry %}
|
||||
retry:
|
||||
automatic:
|
||||
{% for retry_rule in step.retry.automatic %}
|
||||
- exit_status: {{ retry_rule.exit_status }}
|
||||
limit: {{ retry_rule.limit }}
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if step.timeout_in_minutes %}
|
||||
timeout_in_minutes: {{ step.timeout_in_minutes }}
|
||||
|
||||
@@ -1,36 +1,19 @@
|
||||
# Uploader-only hardware presets for CUDA / NPU test YAML.
|
||||
# Referenced as ``mirror_hardwares: l4_1`` on a step; ``upload_pipeline.py`` expands to
|
||||
# ``agents`` (+ optional top-level ``image`` for NPU) + ``plugins`` (+ ``retry``
|
||||
# for ``l4_*``), then strips ``mirror_hardwares`` before Buildkite upload.
|
||||
#
|
||||
# Omit the key: compose {chip}_{n} from pytest -m SKU + cards_n (max of several
|
||||
# positives; not cards_1 with no positive cards_* → that chip's highest existing preset).
|
||||
# Unset MIRROR_HW matches H100 or L4 in -m (both → H100); no match skips the step.
|
||||
# MIRROR_HW=b200 matches B200 in -m (else the step is skipped).
|
||||
# MIRROR_HW must be empty or b200 (case-insensitive); unknown values fail the upload.
|
||||
# CUDA preset strings (h100_4, l4_1) are omitted when MIRROR_HW=b200.
|
||||
# Unset MIRROR_HW keeps string presets. NPU presets ignore MIRROR_HW.
|
||||
# ``agents`` (+ optional top-level ``image`` for NPU) + ``plugins``, then strips
|
||||
# ``mirror_hardwares`` before Buildkite upload.
|
||||
#
|
||||
# CUDA naming: {chip}_{count}
|
||||
# l4_1, l4_2, l4_3, l4_4, h100_1 … h100_4, b200_1 … b200_4
|
||||
# l4_1, l4_4, h100_1 … h100_4
|
||||
# NPU naming: {queue}_{resource_class}
|
||||
# a2b3_npu_1 — ascend-a2b3, resource_class npu-1 (A2/B3 CI image)
|
||||
# a2b3_npu_4 — ascend-a2b3, resource_class npu-4
|
||||
# a3_npu_2 — ascend-a3, resource_class npu-2 (A3 CI image tag …:$BUILDKITE_COMMIT-a3)
|
||||
# a5_npu_2 — ascend-a5, resource_class npu-2
|
||||
# 310p_npu_1 — ascend-310p, resource_class npu-1
|
||||
# a3_npu_2 — ascend-a3, resource_class npu-2
|
||||
#
|
||||
# B200 (b200-k8s): matches vLLM device ``b200-k8s`` from ci-infra
|
||||
# (AgentQueue.B200_K8S + b200_plugin_template). Queue routes the K8s pool;
|
||||
# podSpec uses hostNetwork + /raid|/mnt/shared (no H100-style nodeSelector).
|
||||
#
|
||||
# The ``l4_*`` presets run on the EKS ``l4-k8s`` queue (agent-stack-k8s),
|
||||
# packed onto g6.12xlarge L4 nodes (``vllm.ci/gpu-pool=l4x4``). GPU / CPU /
|
||||
# memory / shm requests scale with card count (1 GPU = 10 CPU / 40Gi / 24Gi shm).
|
||||
# Retry covers lost agents / killed pods, not pytest or OOM (exit 1 / 137).
|
||||
# The Kubernetes HF secret deliberately uses VLLM_CI_HF_TOKEN. Scheduled
|
||||
# build-level HF_TOKEN values can override pod env entries with the same name;
|
||||
# upload_pipeline.py restores the secret-backed value inside the job shell.
|
||||
# The ``l4_*`` presets run under the ``docker`` plugin, which does not put
|
||||
# ``buildkite-agent`` on PATH inside the container the way the ``kubernetes``
|
||||
# presets do, so they set ``mount-buildkite-agent: true``; without it any step
|
||||
# calling ``buildkite-agent`` (artifact upload, annotate, meta-data) fails there.
|
||||
|
||||
x-npu-k8s-plugins: &npu_k8s_plugins
|
||||
- kubernetes:
|
||||
@@ -38,224 +21,39 @@ x-npu-k8s-plugins: &npu_k8s_plugins
|
||||
imagePullSecrets:
|
||||
- name: swr-secret
|
||||
|
||||
# Infra-only retry. Omit exit 1 (pytest failed) and 137 (cgroup OOM / SIGKILL
|
||||
# of the test process). ci-infra ``K8S_RETRY`` still retries exit 1.
|
||||
x-l4-k8s-retry: &l4_k8s_retry
|
||||
automatic:
|
||||
- exit_status: -1
|
||||
limit: 1
|
||||
- exit_status: 128
|
||||
limit: 1
|
||||
- signal_reason: agent_stop
|
||||
limit: 1
|
||||
- signal_reason: agent_refused
|
||||
limit: 1
|
||||
|
||||
|
||||
mirror_hardwares:
|
||||
l4_1:
|
||||
agents:
|
||||
queue: l4-k8s
|
||||
retry: *l4_k8s_retry
|
||||
queue: gpu_1_queue
|
||||
plugins:
|
||||
- kubernetes:
|
||||
metadata:
|
||||
annotations:
|
||||
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
|
||||
karpenter.sh/do-not-disrupt: "true"
|
||||
podSpec:
|
||||
priorityClassName: ci
|
||||
serviceAccountName: buildkite-gpu-jobs
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
vllm.ci/gpu-pool: l4x4
|
||||
containers:
|
||||
- image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
requests:
|
||||
nvidia.com/gpu: 1
|
||||
cpu: "10"
|
||||
memory: 40Gi
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /fsx/hf_cache
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /fsx/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 24Gi
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /fsx/hf_cache
|
||||
type: DirectoryOrCreate
|
||||
|
||||
l4_2:
|
||||
agents:
|
||||
queue: l4-k8s
|
||||
retry: *l4_k8s_retry
|
||||
plugins:
|
||||
- kubernetes:
|
||||
metadata:
|
||||
annotations:
|
||||
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
|
||||
karpenter.sh/do-not-disrupt: "true"
|
||||
podSpec:
|
||||
priorityClassName: ci
|
||||
serviceAccountName: buildkite-gpu-jobs
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
vllm.ci/gpu-pool: l4x4
|
||||
containers:
|
||||
- image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 2
|
||||
requests:
|
||||
nvidia.com/gpu: 2
|
||||
cpu: "20"
|
||||
memory: 80Gi
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /fsx/hf_cache
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /fsx/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 48Gi
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /fsx/hf_cache
|
||||
type: DirectoryOrCreate
|
||||
|
||||
l4_3:
|
||||
agents:
|
||||
queue: l4-k8s
|
||||
retry: *l4_k8s_retry
|
||||
plugins:
|
||||
- kubernetes:
|
||||
metadata:
|
||||
annotations:
|
||||
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
|
||||
karpenter.sh/do-not-disrupt: "true"
|
||||
podSpec:
|
||||
priorityClassName: ci
|
||||
serviceAccountName: buildkite-gpu-jobs
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
vllm.ci/gpu-pool: l4x4
|
||||
containers:
|
||||
- image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 3
|
||||
requests:
|
||||
nvidia.com/gpu: 3
|
||||
cpu: "30"
|
||||
memory: 120Gi
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /fsx/hf_cache
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /fsx/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 72Gi
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /fsx/hf_cache
|
||||
type: DirectoryOrCreate
|
||||
- docker#v5.2.0:
|
||||
image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
always-pull: true
|
||||
propagate-environment: true
|
||||
mount-buildkite-agent: true
|
||||
shm-size: "8gb"
|
||||
environment:
|
||||
- HF_HOME=/fsx/hf_cache
|
||||
- HF_TOKEN
|
||||
volumes:
|
||||
- /fsx/hf_cache:/fsx/hf_cache
|
||||
|
||||
l4_4:
|
||||
agents:
|
||||
queue: l4-k8s
|
||||
retry: *l4_k8s_retry
|
||||
queue: gpu_4_queue
|
||||
plugins:
|
||||
- kubernetes:
|
||||
metadata:
|
||||
annotations:
|
||||
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
|
||||
karpenter.sh/do-not-disrupt: "true"
|
||||
podSpec:
|
||||
priorityClassName: ci
|
||||
serviceAccountName: buildkite-gpu-jobs
|
||||
tolerations:
|
||||
- key: nvidia.com/gpu
|
||||
operator: Exists
|
||||
effect: NoSchedule
|
||||
nodeSelector:
|
||||
vllm.ci/gpu-pool: l4x4
|
||||
containers:
|
||||
- image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 4
|
||||
requests:
|
||||
nvidia.com/gpu: 4
|
||||
cpu: "40"
|
||||
memory: 160Gi
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: hf-cache
|
||||
mountPath: /fsx/hf_cache
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /fsx/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
sizeLimit: 96Gi
|
||||
- name: hf-cache
|
||||
hostPath:
|
||||
path: /fsx/hf_cache
|
||||
type: DirectoryOrCreate
|
||||
- docker#v5.2.0:
|
||||
image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
always-pull: true
|
||||
propagate-environment: true
|
||||
mount-buildkite-agent: true
|
||||
shm-size: "8gb"
|
||||
environment:
|
||||
- HF_HOME=/fsx/hf_cache
|
||||
- HF_TOKEN
|
||||
volumes:
|
||||
- /fsx/hf_cache:/fsx/hf_cache
|
||||
|
||||
h100_1:
|
||||
agents:
|
||||
@@ -276,7 +74,7 @@ mirror_hardwares:
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
@@ -311,7 +109,7 @@ mirror_hardwares:
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
@@ -346,7 +144,7 @@ mirror_hardwares:
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
@@ -381,7 +179,7 @@ mirror_hardwares:
|
||||
env:
|
||||
- name: HF_HOME
|
||||
value: /root/.cache/huggingface
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
- name: HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
@@ -397,211 +195,6 @@ mirror_hardwares:
|
||||
path: /mnt/hf-cache
|
||||
type: DirectoryOrCreate
|
||||
|
||||
# vLLM device: b200-k8s → agents.queue b200-k8s + Buildkite K8s agent-stack podSpec
|
||||
b200_1:
|
||||
agents:
|
||||
queue: b200-k8s
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
runtimeClassName: nvidia
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
imagePullSecrets:
|
||||
- name: k8s-ecr-login-renew-docker-secret
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 1
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- IPC_LOCK
|
||||
- SYS_RESOURCE
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: raid
|
||||
mountPath: /raid
|
||||
- name: shared
|
||||
mountPath: /mnt/shared
|
||||
env:
|
||||
- name: NCCL_CUMEM_HOST_ENABLE
|
||||
value: "0"
|
||||
- name: HF_HOME
|
||||
value: /raid/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: raid
|
||||
hostPath:
|
||||
path: /raid
|
||||
type: DirectoryOrCreate
|
||||
- name: shared
|
||||
hostPath:
|
||||
path: /mnt/shared
|
||||
type: DirectoryOrCreate
|
||||
|
||||
b200_2:
|
||||
agents:
|
||||
queue: b200-k8s
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
runtimeClassName: nvidia
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
imagePullSecrets:
|
||||
- name: k8s-ecr-login-renew-docker-secret
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 2
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- IPC_LOCK
|
||||
- SYS_RESOURCE
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: raid
|
||||
mountPath: /raid
|
||||
- name: shared
|
||||
mountPath: /mnt/shared
|
||||
env:
|
||||
- name: NCCL_CUMEM_HOST_ENABLE
|
||||
value: "0"
|
||||
- name: HF_HOME
|
||||
value: /raid/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: raid
|
||||
hostPath:
|
||||
path: /raid
|
||||
type: DirectoryOrCreate
|
||||
- name: shared
|
||||
hostPath:
|
||||
path: /mnt/shared
|
||||
type: DirectoryOrCreate
|
||||
|
||||
b200_3:
|
||||
agents:
|
||||
queue: b200-k8s
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
runtimeClassName: nvidia
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
imagePullSecrets:
|
||||
- name: k8s-ecr-login-renew-docker-secret
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 3
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- IPC_LOCK
|
||||
- SYS_RESOURCE
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: raid
|
||||
mountPath: /raid
|
||||
- name: shared
|
||||
mountPath: /mnt/shared
|
||||
env:
|
||||
- name: NCCL_CUMEM_HOST_ENABLE
|
||||
value: "0"
|
||||
- name: HF_HOME
|
||||
value: /raid/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: raid
|
||||
hostPath:
|
||||
path: /raid
|
||||
type: DirectoryOrCreate
|
||||
- name: shared
|
||||
hostPath:
|
||||
path: /mnt/shared
|
||||
type: DirectoryOrCreate
|
||||
|
||||
b200_4:
|
||||
agents:
|
||||
queue: b200-k8s
|
||||
plugins:
|
||||
- kubernetes:
|
||||
podSpec:
|
||||
runtimeClassName: nvidia
|
||||
hostNetwork: true
|
||||
dnsPolicy: ClusterFirstWithHostNet
|
||||
imagePullSecrets:
|
||||
- name: k8s-ecr-login-renew-docker-secret
|
||||
containers:
|
||||
- image: 936637512419.dkr.ecr.us-west-2.amazonaws.com/vllm-ci-pull-through-cache/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 4
|
||||
securityContext:
|
||||
capabilities:
|
||||
add:
|
||||
- IPC_LOCK
|
||||
- SYS_RESOURCE
|
||||
volumeMounts:
|
||||
- name: devshm
|
||||
mountPath: /dev/shm
|
||||
- name: raid
|
||||
mountPath: /raid
|
||||
- name: shared
|
||||
mountPath: /mnt/shared
|
||||
env:
|
||||
- name: NCCL_CUMEM_HOST_ENABLE
|
||||
value: "0"
|
||||
- name: HF_HOME
|
||||
value: /raid/hf_cache
|
||||
- name: VLLM_CI_HF_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: hf-token-secret
|
||||
key: token
|
||||
volumes:
|
||||
- name: devshm
|
||||
emptyDir:
|
||||
medium: Memory
|
||||
- name: raid
|
||||
hostPath:
|
||||
path: /raid
|
||||
type: DirectoryOrCreate
|
||||
- name: shared
|
||||
hostPath:
|
||||
path: /mnt/shared
|
||||
type: DirectoryOrCreate
|
||||
|
||||
a2b3_npu_1:
|
||||
agents:
|
||||
queue: "ascend-a2b3"
|
||||
@@ -651,52 +244,3 @@ mirror_hardwares:
|
||||
resource_class: "npu-16"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a3"
|
||||
plugins: *npu_k8s_plugins
|
||||
|
||||
a5_npu_2:
|
||||
agents:
|
||||
queue: "ascend-a5"
|
||||
resource_class: "npu-2"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
|
||||
plugins: *npu_k8s_plugins
|
||||
|
||||
a5_npu_4:
|
||||
agents:
|
||||
queue: "ascend-a5"
|
||||
resource_class: "npu-4"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
|
||||
plugins: *npu_k8s_plugins
|
||||
|
||||
a5_npu_8:
|
||||
agents:
|
||||
queue: "ascend-a5"
|
||||
resource_class: "npu-8"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
|
||||
plugins: *npu_k8s_plugins
|
||||
|
||||
a5_npu_16:
|
||||
agents:
|
||||
queue: "ascend-a5"
|
||||
resource_class: "npu-16"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-a5"
|
||||
plugins: *npu_k8s_plugins
|
||||
|
||||
310p_npu_1:
|
||||
agents:
|
||||
queue: "ascend-310p"
|
||||
resource_class: "npu-1"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-310p"
|
||||
plugins: *npu_k8s_plugins
|
||||
|
||||
310p_npu_2:
|
||||
agents:
|
||||
queue: "ascend-310p"
|
||||
resource_class: "npu-2"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-310p"
|
||||
plugins: *npu_k8s_plugins
|
||||
|
||||
310p_npu_4:
|
||||
agents:
|
||||
queue: "ascend-310p"
|
||||
resource_class: "npu-4"
|
||||
image: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry/vllm-omni-ci-npu:${BUILDKITE_COMMIT}-310p"
|
||||
plugins: *npu_k8s_plugins
|
||||
@@ -1,525 +0,0 @@
|
||||
# Uploader-only path-prefix presets for CUDA L2–L5 and NPU L4 test YAML.
|
||||
# Referenced as ``source_file_dependencies: omni_qwen3_omni_function`` (or a list of
|
||||
# keys) on a step; ``upload_pipeline.py`` expands the key(s) only.
|
||||
#
|
||||
# YAML anchors above the registry hold model business-code paths. Job keys
|
||||
# ({model}_function / _perf / _accuracy / _reliability / _doc / _cov) alias
|
||||
# those anchors and add the scripts the job actually runs. Feature keys below
|
||||
# are unchanged. ``source_filter_fallback`` is not a job key: when no job-key
|
||||
# prefix matches, a matching diff keeps every job for that upload.
|
||||
#
|
||||
# YAML anchors are nested lists; the loader flattens them.
|
||||
|
||||
x-qwen3-tts: &qwen3_tts
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/entrypoints/omni_base.py
|
||||
- vllm_omni/worker/gpu_model_runner.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/fused_sampling.py
|
||||
- vllm_omni/model_executor/models/common/short_kv_attention.py
|
||||
- vllm_omni/model_executor/models/common/talker_first_audio.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/common/alias_free_activation.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
- vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2_single_gpu.yaml
|
||||
- vllm_omni/entrypoints/openai/serving_speech.py
|
||||
- vllm_omni/entrypoints/openai/tts_adapters/
|
||||
- vllm_omni/engine/stage_pool.py
|
||||
- vllm_omni/worker_v2/
|
||||
- vllm_omni/core/sched/
|
||||
- vllm_omni/distributed/omni_connectors/
|
||||
- vllm_omni/config/stage_config.py
|
||||
- vllm_omni/config/omni_config.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
- vllm_omni/engine/stage_engine_core_proc.py
|
||||
- vllm_omni/engine/cuda_mps.py
|
||||
- vllm_omni/engine/stage_runtime.py
|
||||
- vllm_omni/engine/stage_init_utils.py
|
||||
- vllm_omni/deploy/qwen3_tts_mrv2.yaml
|
||||
- vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2.yaml
|
||||
- vllm_omni/deploy/qwen3_tts_high_concurrency_mrv2_b4.yaml
|
||||
|
||||
x-minicpmo-4-5: &minicpmo_4_5
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/entrypoints/omni_base.py
|
||||
- vllm_omni/model_executor/models/minicpmo_4_5/
|
||||
- vllm_omni/model_executor/models/step_audio2/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/utils.py
|
||||
- vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
|
||||
- vllm_omni/deploy/minicpmo_4_5.yaml
|
||||
- vllm_omni/transformers_utils/chat_templates/minicpmo45_native.jinja
|
||||
|
||||
x-minicpmo-4-5-duplex: &minicpmo_4_5_duplex
|
||||
- vllm_omni/deploy/minicpmo_4_5_duplex.yaml
|
||||
- vllm_omni/engine/duplex/
|
||||
- vllm_omni/engine/duplex_omni_engine.py
|
||||
- vllm_omni/engine/duplex_orchestrator.py
|
||||
- vllm_omni/engine/omni_engine_base.py
|
||||
- vllm_omni/entrypoints/duplex/
|
||||
- vllm_omni/entrypoints/duplex_omni.py
|
||||
- vllm_omni/entrypoints/async_omni_base.py
|
||||
- vllm_omni/protocol/
|
||||
- vllm_omni/clients/
|
||||
- vllm_omni/outputs/duplex.py
|
||||
- vllm_omni/model_executor/duplex_sampling.py
|
||||
- vllm_omni/experimental/fullduplex/
|
||||
- vllm_omni/core/sched/
|
||||
- vllm_omni/outputs/multimodal_accumulation.py
|
||||
- vllm_omni/engine/stage_init_utils.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
- vllm_omni/entrypoints/cli/serve.py
|
||||
- vllm_omni/entrypoints/openai/api_server.py
|
||||
- vllm_omni/entrypoints/openai/duplex_capability.py
|
||||
- vllm_omni/config/stage_config.py
|
||||
|
||||
x-qwen2-5-omni: &qwen2_5_omni
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/entrypoints/omni_base.py
|
||||
- vllm_omni/model_executor/models/qwen2_5_omni/
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen2_5_omni.py
|
||||
- vllm_omni/deploy/qwen2_5_omni.yaml
|
||||
|
||||
x-qwen3-omni: &qwen3_omni
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/entrypoints/omni_base.py
|
||||
- vllm_omni/entrypoints/openai/serving_chat.py
|
||||
- vllm_omni/model_executor/models/qwen3_omni/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/fused_sampling.py
|
||||
- vllm_omni/model_executor/models/common/short_kv_attention.py
|
||||
- vllm_omni/model_executor/models/common/talker_first_audio.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_omni.py
|
||||
- vllm_omni/deploy/qwen3_omni_moe.yaml
|
||||
- vllm_omni/deploy/qwen3_omni_moe_mrv2.yaml
|
||||
- vllm_omni/deploy/qwen3_omni_moe_mrv2_h200.yaml
|
||||
- vllm_omni/deploy/qwen3_omni_moe_thinking.yaml
|
||||
|
||||
x-wan22: &wan22
|
||||
- vllm_omni/diffusion/models/wan2_2/
|
||||
- vllm_omni/diffusion/models/dmd2/
|
||||
|
||||
x-cosmos3: &cosmos3
|
||||
- vllm_omni/diffusion/models/cosmos3/
|
||||
- vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_wan.py
|
||||
- vllm_omni/experimental/world_models/
|
||||
- vllm_omni/model_extras/cosmos3.py
|
||||
|
||||
x-qwen-image: &qwen_image
|
||||
- vllm_omni/diffusion/models/qwen_image/
|
||||
- vllm_omni/diffusion/models/dmd2/
|
||||
- vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_qwenimage.py
|
||||
|
||||
x-joy-image: &joy_image
|
||||
- vllm_omni/diffusion/models/joy_image/
|
||||
- vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_wan.py
|
||||
|
||||
x-hunyuan-image3: &hunyuan_image3
|
||||
- vllm_omni/diffusion/models/hunyuan_image3/
|
||||
- vllm_omni/model_executor/models/hunyuan_image3/
|
||||
- vllm_omni/deploy/hunyuan_image3_dit.yaml
|
||||
|
||||
x-minimax-h3: &minimax_h3
|
||||
- vllm_omni/diffusion/models/minimax_h3/
|
||||
- vllm_omni/diffusion/layers/indexed_modulation.py
|
||||
|
||||
x-hunyuan-video: &hunyuan_video
|
||||
- vllm_omni/diffusion/models/hunyuan_video/
|
||||
- vllm_omni/model_executor/models/hunyuan_video/
|
||||
|
||||
x-ltx2: <x2
|
||||
- vllm_omni/diffusion/models/ltx2/
|
||||
- vllm_omni/model_extras/ltx2.py
|
||||
|
||||
x-sana-video: &sana_video
|
||||
- vllm_omni/diffusion/models/sana_video/
|
||||
- vllm_omni/model_extras/sana_video.py
|
||||
|
||||
x-z-image: &z_image
|
||||
- vllm_omni/diffusion/models/z_image/
|
||||
|
||||
x-stable-audio: &stable_audio
|
||||
- vllm_omni/diffusion/models/stable_audio/
|
||||
|
||||
x-lingbot-video: &lingbot_video
|
||||
- vllm_omni/diffusion/models/lingbot_video/
|
||||
- vllm_omni/model_extras/lingbot_video.py
|
||||
|
||||
x-lingbot-world: &lingbot_world
|
||||
- vllm_omni/diffusion/models/lingbot_world/
|
||||
|
||||
x-kandinsky6: &kandinsky6
|
||||
- vllm_omni/diffusion/models/kandinsky6/
|
||||
- vllm_omni/model_extras/kandinsky6.py
|
||||
|
||||
|
||||
source_file_dependencies:
|
||||
# Hardware presets and shared test helpers. Not a job key — do not attach this
|
||||
# to a step. Used only when no job-key prefix matches: then a hit keeps every job.
|
||||
source_filter_fallback:
|
||||
- .buildkite/common/ci_mirror_hardwares.yml
|
||||
- tests/helpers/
|
||||
|
||||
# --- features ---
|
||||
diffusion_distributed_attention:
|
||||
- tests/diffusion/distributed/
|
||||
- tests/diffusion/attention/test_attention_sp.py
|
||||
- vllm_omni/diffusion/distributed/
|
||||
- vllm_omni/diffusion/attention/
|
||||
- vllm_omni/diffusion/data.py
|
||||
- vllm_omni/diffusion/models/
|
||||
|
||||
custom_pipeline:
|
||||
- tests/e2e/features/custom_pipeline/
|
||||
- tests/e2e/features/helpers/custom_pipeline.py
|
||||
- vllm_omni/diffusion/
|
||||
- vllm_omni/engine/
|
||||
- vllm_omni/entrypoints/
|
||||
- vllm_omni/inputs/
|
||||
- vllm_omni/outputs/
|
||||
|
||||
rlhf_verl_omni:
|
||||
- tests/e2e/features/rlhf_test/test_verl_omni_e2e.py
|
||||
- tests/e2e/features/helpers/verl_omni_server.py
|
||||
- tests/e2e/features/helpers/custom_pipeline.py
|
||||
- vllm_omni/diffusion/
|
||||
- vllm_omni/engine/
|
||||
- vllm_omni/entrypoints/
|
||||
- vllm_omni/inputs/
|
||||
- vllm_omni/outputs/
|
||||
|
||||
# Tiny-model diffusion jobs (shared across many model trees under models/).
|
||||
diffusion_tiny_model:
|
||||
- tests/model_tests/diffusion/
|
||||
- vllm_omni/diffusion/models/
|
||||
|
||||
diffusion_offloader:
|
||||
- tests/diffusion/offloader/
|
||||
- vllm_omni/diffusion/offloader/
|
||||
|
||||
diffusion_quantization:
|
||||
- tests/diffusion/quantization/
|
||||
- vllm_omni/diffusion/quantization/
|
||||
|
||||
# --- models ---
|
||||
# --- omni / tts jobs ---
|
||||
entrypoints_function:
|
||||
- tests/entrypoints/
|
||||
- vllm_omni/entrypoints/
|
||||
|
||||
omni_qwen3_omni_function:
|
||||
- *qwen3_omni
|
||||
- tests/e2e/offline_inference/test_qwen3_omni.py
|
||||
- tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py
|
||||
- tests/e2e/offline_inference/test_qwen3_omni_autoround_w4a16_expansion.py
|
||||
- tests/e2e/offline_inference/test_qwen3_omni_modelopt_nvfp4_w4a4.py
|
||||
- tests/e2e/online_serving/test_qwen3_omni.py
|
||||
- tests/e2e/online_serving/test_qwen3_omni_expansion.py
|
||||
- tests/e2e/online_serving/test_qwen3_omni_multi_replicas.py
|
||||
|
||||
omni_qwen3_omni_accuracy:
|
||||
- *qwen3_omni
|
||||
- tests/e2e/accuracy/qwen3_omni/
|
||||
|
||||
omni_qwen3_omni_perf:
|
||||
- *qwen3_omni
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_qwen3_omni_vllm_text.json
|
||||
- tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json
|
||||
- tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json
|
||||
- tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json
|
||||
|
||||
omni_qwen3_omni_reliability:
|
||||
- *qwen3_omni
|
||||
- tests/dfx/reliability/test_reliability_qwen3_omni.py
|
||||
|
||||
omni_qwen3_omni_doc:
|
||||
- *qwen3_omni
|
||||
- examples/online_serving/qwen3_omni/
|
||||
- examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py
|
||||
- tests/examples/online_serving/test_qwen3_omni.py
|
||||
|
||||
omni_qwen2_5_omni_doc:
|
||||
- *qwen2_5_omni
|
||||
- examples/online_serving/qwen2_5_omni/
|
||||
- examples/online_serving/openai_chat_completion_client_for_multimodal_generation.py
|
||||
- tests/examples/online_serving/test_qwen2_5_omni.py
|
||||
|
||||
omni_minicpmo_4_5_function:
|
||||
- *minicpmo_4_5
|
||||
- tests/e2e/offline_inference/test_minicpmo_4_5.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_expansion.py
|
||||
|
||||
omni_minicpmo_4_5_accuracy:
|
||||
- *minicpmo_4_5
|
||||
- tests/e2e/accuracy/minicpmo_4_5/
|
||||
|
||||
omni_duplex_eval_ci:
|
||||
- *minicpmo_4_5
|
||||
- *minicpmo_4_5_duplex
|
||||
- tests/e2e/accuracy/omni_duplex_eval/**
|
||||
- tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
|
||||
- vllm_omni/benchmarks/duplex/**
|
||||
- vllm_omni/entrypoints/cli/benchmark/omni_duplex_eval.py
|
||||
|
||||
omni_minicpmo_4_5_perf:
|
||||
- *minicpmo_4_5
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_minicpmo_4_5_ready.json
|
||||
- tests/dfx/perf/tests/test_minicpmo_4_5.json
|
||||
|
||||
omni_minicpmo_4_5_doc:
|
||||
- *minicpmo_4_5
|
||||
- examples/online_serving/minicpmo/realtime_web/
|
||||
- examples/online_serving/minicpmo/realtime_duplex_demo.py
|
||||
- tests/examples/test_minicpmo_realtime_web_server.py
|
||||
- tests/examples/test_minicpmo_realtime_web_static.py
|
||||
- tests/examples/test_minicpmo_realtime_duplex_simple_demo.py
|
||||
|
||||
omni_minicpmo_4_5_duplex_function:
|
||||
- *minicpmo_4_5
|
||||
- *minicpmo_4_5_duplex
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_window.py
|
||||
- tests/e2e/online_serving/helpers/minicpmo_window_e2e.py
|
||||
- tests/e2e/online_serving/test_duplex_client_live.py
|
||||
- tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py
|
||||
- tests/e2e/online_serving/helpers/minicpmo_realtime_duplex_scenarios.py
|
||||
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_multi_session.py
|
||||
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_server_vad.py
|
||||
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py
|
||||
- tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
|
||||
- tests/assets/minicpmo_4_5/
|
||||
- tests/clients/
|
||||
- tests/engine/duplex/
|
||||
- tests/engine/test_duplex_omni_engine.py
|
||||
- tests/engine/test_duplex_orchestrator.py
|
||||
- tests/entrypoints/duplex/
|
||||
- tests/entrypoints/test_duplex_omni.py
|
||||
- tests/entrypoints/openai_api/test_duplex_api_server.py
|
||||
- tests/model_executor/models/minicpmo_4_5/duplex/
|
||||
- tests/protocol/
|
||||
|
||||
omni_minicpmo_4_5_duplex_perf:
|
||||
- *minicpmo_4_5
|
||||
- *minicpmo_4_5_duplex
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json
|
||||
|
||||
tts_qwen3_tts_function:
|
||||
- *qwen3_tts
|
||||
- tests/e2e/features/prefill_phase/
|
||||
- tests/e2e/offline_inference/test_qwen3_tts_base.py
|
||||
- tests/e2e/offline_inference/test_qwen3_tts_customvoice.py
|
||||
- tests/e2e/online_serving/test_qwen3_tts_base.py
|
||||
- tests/e2e/online_serving/test_qwen3_tts_base_expansion.py
|
||||
- tests/e2e/online_serving/test_qwen3_tts_customvoice.py
|
||||
- tests/e2e/online_serving/test_qwen3_tts_customvoice_expansion.py
|
||||
|
||||
tts_qwen3_tts_perf:
|
||||
- *qwen3_tts
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_tts.json
|
||||
|
||||
tts_qwen3_tts_cov:
|
||||
- *qwen3_tts
|
||||
- .buildkite/common/scripts/run_cov_split.sh
|
||||
- pyproject.toml
|
||||
- tests/e2e/offline_inference/test_qwen3_tts_base.py
|
||||
- tests/e2e/online_serving/test_qwen3_tts_base.py
|
||||
|
||||
# --- diffusion jobs ---
|
||||
diffusion_wan22_function:
|
||||
- *wan22
|
||||
- tests/e2e/offline_inference/test_wan22_t2v.py
|
||||
- tests/e2e/offline_inference/test_wan22_autoround_w4a16_expansion.py
|
||||
- tests/e2e/offline_inference/test_device_postprocess_equivalence.py
|
||||
- tests/e2e/online_serving/test_wan22_t2v.py
|
||||
- tests/e2e/online_serving/test_wan22_expansion.py
|
||||
|
||||
diffusion_wan22_accuracy:
|
||||
- *wan22
|
||||
- tests/e2e/accuracy/wan22_i2v/
|
||||
- tests/e2e/accuracy/test_diffusers_backend_similarity.py
|
||||
|
||||
diffusion_wan22_perf:
|
||||
- *wan22
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json
|
||||
|
||||
diffusion_wan22_reliability:
|
||||
- *wan22
|
||||
- tests/dfx/reliability/test_reliability_wan22.py
|
||||
|
||||
diffusion_image_to_video_doc:
|
||||
- *wan22
|
||||
- *hunyuan_video
|
||||
- *ltx2
|
||||
- *sana_video
|
||||
- *lingbot_video
|
||||
- examples/offline_inference/image_to_video/
|
||||
- tests/examples/offline_inference/test_image_to_video.py
|
||||
|
||||
diffusion_cosmos3_function:
|
||||
- *cosmos3
|
||||
- tests/e2e/online_serving/test_cosmos3.py
|
||||
|
||||
diffusion_cosmos3_accuracy:
|
||||
- *cosmos3
|
||||
- tests/e2e/accuracy/test_cosmos3_similarity.py
|
||||
|
||||
diffusion_cosmos3_perf:
|
||||
- *cosmos3
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_cosmos3_vllm_omni.json
|
||||
|
||||
diffusion_qwen_image_function:
|
||||
- *qwen_image
|
||||
- tests/e2e/online_serving/test_qwen_image.py
|
||||
- tests/e2e/online_serving/test_qwen_image_expansion.py
|
||||
- tests/e2e/offline_inference/test_qwen_image_autoround_w4a16_expansion.py
|
||||
|
||||
diffusion_qwen_image_accuracy:
|
||||
- *qwen_image
|
||||
- tests/e2e/accuracy/test_qwen_image.py
|
||||
- tests/e2e/accuracy/test_diffusers_backend_similarity.py
|
||||
|
||||
diffusion_joy_image_accuracy:
|
||||
- *joy_image
|
||||
- tests/diffusion/models/joy_image/
|
||||
- tests/e2e/accuracy/test_joy_image_edit.py
|
||||
- recipes/JD/JoyAI-Image-Edit.md
|
||||
|
||||
diffusion_qwen_image_perf:
|
||||
- *qwen_image
|
||||
- tests/dfx/perf/scripts/run_diffusion_benchmark.py
|
||||
- tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
|
||||
diffusion_text_to_image_doc:
|
||||
- *qwen_image
|
||||
- *z_image
|
||||
- examples/offline_inference/text_to_image/
|
||||
- examples/online_serving/text_to_image/
|
||||
- tests/examples/offline_inference/test_text_to_image.py
|
||||
- tests/examples/online_serving/test_text_to_image.py
|
||||
- vllm_omni/diffusion/models/nextstep_1_1/
|
||||
- vllm_omni/diffusion/models/longcat_image/
|
||||
- vllm_omni/diffusion/models/ovis_image/
|
||||
- vllm_omni/diffusion/models/omnigen2/
|
||||
- vllm_omni/diffusion/models/sd3/
|
||||
- vllm_omni/diffusion/models/flux/
|
||||
- vllm_omni/diffusion/models/flux2/
|
||||
- vllm_omni/diffusion/models/flux2_klein/
|
||||
- vllm_omni/diffusion/models/hidream_image/
|
||||
- vllm_omni/diffusion/models/krea2/
|
||||
- vllm_omni/diffusion/models/lingbot_video/
|
||||
|
||||
diffusion_hunyuan_image3_function:
|
||||
- *hunyuan_image3
|
||||
- tests/e2e/offline_inference/test_hunyuanimage3_i2t_expansion.py
|
||||
|
||||
diffusion_hunyuan_image3_accuracy:
|
||||
- *hunyuan_image3
|
||||
- tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py
|
||||
- tests/e2e/accuracy/test_hunyuan_image3_lora.py
|
||||
- vllm_omni/diffusion/lora/
|
||||
|
||||
diffusion_hunyuan_image3_perf:
|
||||
- *hunyuan_image3
|
||||
- tests/dfx/perf/scripts/run_diffusion_benchmark.py
|
||||
- tests/dfx/perf/tests/test_hunyuan_image_tp2_cfgp2.json
|
||||
- tests/dfx/perf/tests/test_hunyuan_image_tp2_sp2.json
|
||||
- tests/dfx/perf/tests/test_hunyuan_image_tp4.json
|
||||
|
||||
diffusion_hunyuan_image3_reliability:
|
||||
- *hunyuan_image3
|
||||
- tests/dfx/reliability/test_reliability_hunyuan_image.py
|
||||
|
||||
diffusion_minimax_h3_function:
|
||||
- *minimax_h3
|
||||
- tests/helpers/assertions.py
|
||||
- tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py
|
||||
- tests/e2e/online_serving/minimax_h3/
|
||||
- tests/diffusion/models/minimax_h3/
|
||||
- vllm_omni/diffusion/lora/
|
||||
- vllm_omni/diffusion/offloader/
|
||||
- vllm_omni/diffusion/distributed/
|
||||
- vllm_omni/entrypoints/openai/
|
||||
|
||||
diffusion_minimax_h3_accuracy:
|
||||
- *minimax_h3
|
||||
- tests/e2e/accuracy/minimax_h3/
|
||||
|
||||
diffusion_minimax_h3_perf:
|
||||
- *minimax_h3
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json
|
||||
|
||||
diffusion_hunyuan_video_function:
|
||||
- *hunyuan_video
|
||||
- tests/e2e/online_serving/test_hunyuan_video_15_expansion.py
|
||||
|
||||
diffusion_hunyuan_video_accuracy:
|
||||
- *hunyuan_video
|
||||
- tests/e2e/accuracy/hunyuanvideo15_t2v/
|
||||
- tests/e2e/accuracy/hunyuanvideo15_i2v/
|
||||
|
||||
diffusion_hunyuan_video_perf:
|
||||
- *hunyuan_video
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json
|
||||
- tests/dfx/perf/tests/test_hunyuanvideo15_i2v_vllm_omni.json
|
||||
|
||||
diffusion_z_image_function:
|
||||
- *z_image
|
||||
- tests/e2e/online_serving/test_z_image.py
|
||||
- tests/e2e/online_serving/test_zimage_expansion.py
|
||||
|
||||
diffusion_sana_video_function:
|
||||
- *sana_video
|
||||
- tests/e2e/online_serving/test_sana_video_expansion.py
|
||||
- tests/e2e/accuracy/sana_video/
|
||||
|
||||
diffusion_stable_audio_function:
|
||||
- *stable_audio
|
||||
- tests/e2e/offline_inference/test_stable_audio_expansion.py
|
||||
- tests/e2e/online_serving/test_stable_audio_online_expansion.py
|
||||
|
||||
diffusion_lingbot_function:
|
||||
- *lingbot_video
|
||||
- *lingbot_world
|
||||
- tests/e2e/online_serving/test_lingbot_video.py
|
||||
- tests/e2e/online_serving/test_lingbot_video_expansion.py
|
||||
- tests/e2e/online_serving/test_lingbot_video_moe.py
|
||||
- tests/e2e/offline_inference/test_lingbot_world_v2.py
|
||||
|
||||
diffusion_lingbot_perf:
|
||||
- *lingbot_video
|
||||
- tests/dfx/perf/scripts/run_benchmark.py
|
||||
- tests/dfx/perf/tests/test_lingbot_video_vllm_omni.json
|
||||
|
||||
diffusion_kandinsky6_function:
|
||||
- *kandinsky6
|
||||
- tests/e2e/offline_inference/test_kandinsky6.py
|
||||
- tests/e2e/online_serving/test_kandinsky6.py
|
||||
|
||||
|
||||
# Invalid-param jobs: model business code only, plus the reliability scripts.
|
||||
reliability_invalid_param_h100:
|
||||
- *qwen3_omni
|
||||
- *minicpmo_4_5
|
||||
- *qwen_image
|
||||
- *wan22
|
||||
- tests/dfx/reliability/invalid_param_test/
|
||||
|
||||
reliability_invalid_param_l4:
|
||||
- *qwen3_tts
|
||||
- *stable_audio
|
||||
- tests/dfx/reliability/invalid_param_test/
|
||||
@@ -1,6 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Shared Buildkite skip-ci and CI-YAML level targeting logic.
|
||||
|
||||
``resolve_ci_decision(changed_files, ...)`` buckets explicit file lists (tests, dry-run).
|
||||
@@ -11,6 +10,12 @@ CLI (for bootstrap scripts; exit code is the signal):
|
||||
python3 skip_ci.py gate <platform> <level>
|
||||
Exit 0 when bootstrap should stop (skip-all, or that L2/L3 target is off).
|
||||
Prints ``skip-all`` or ``skip-l23`` on stdout; logs the decision once.
|
||||
python3 skip_ci.py check-skip-all
|
||||
Exit 0 when only docs/skip-mark changes → bootstrap skips entire CI upload.
|
||||
python3 skip_ci.py check-skip-l2-l3 <platform> <level>
|
||||
Exit 0 when that L2/L3 pipeline should be skipped (CI config YAML-only diffs).
|
||||
python3 skip_ci.py print-annotate
|
||||
Print decision message for Buildkite annotation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -44,7 +49,6 @@ L3_YAML_FILES: dict[str, str] = {
|
||||
L45_YAML_FILES: dict[str, str] = {
|
||||
".buildkite/cuda/test-nightly.yml": "cuda",
|
||||
".buildkite/cuda/test-weekly.yml": "cuda",
|
||||
".buildkite/amd/test-amd-nightly.yml": "amd",
|
||||
".buildkite/npu/test-npu-nightly.yml": "npu",
|
||||
}
|
||||
|
||||
@@ -254,10 +258,11 @@ class CiDecision:
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CiContext:
|
||||
"""Skip-ci decision plus changed files used for pipeline filtering."""
|
||||
"""Skip-ci decision plus git diff inputs used to compute it."""
|
||||
|
||||
decision: CiDecision
|
||||
changed_files: list[str] | None
|
||||
diff_range: str | None
|
||||
|
||||
|
||||
def _log(message: str) -> None:
|
||||
@@ -327,7 +332,7 @@ def resolve_ci_context_from_git() -> CiContext:
|
||||
diff_range = _resolve_diff_range()
|
||||
changed_files = _changed_files_for_diff_range(diff_range)
|
||||
decision = resolve_ci_decision(changed_files, diff_range=diff_range)
|
||||
return CiContext(decision=decision, changed_files=changed_files)
|
||||
return CiContext(decision=decision, changed_files=changed_files, diff_range=diff_range)
|
||||
|
||||
|
||||
# --- Doc / skip-mark helpers ---
|
||||
@@ -566,6 +571,23 @@ def main() -> int:
|
||||
gate_parser.add_argument("platform", help="cuda, amd, intel, or npu")
|
||||
gate_parser.add_argument("level", choices=("l2", "l3"), help="test level")
|
||||
|
||||
subparsers.add_parser(
|
||||
"check-skip-all",
|
||||
help="exit 0 when docs/skip-mark-only diff should skip entire CI upload",
|
||||
)
|
||||
|
||||
skip_parser = subparsers.add_parser(
|
||||
"check-skip-l2-l3",
|
||||
help="exit 0 when PLATFORM l2/l3 pipeline should be skipped (CI config YAML-only diffs)",
|
||||
)
|
||||
skip_parser.add_argument("platform", help="cuda, amd, intel, or npu")
|
||||
skip_parser.add_argument("level", choices=("l2", "l3"), help="test level")
|
||||
|
||||
subparsers.add_parser(
|
||||
"print-annotate",
|
||||
help="print human-readable skip reason for Buildkite annotation",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
decision = resolve_ci_context_from_git().decision
|
||||
|
||||
@@ -578,6 +600,16 @@ def main() -> int:
|
||||
return 0
|
||||
return 1
|
||||
|
||||
if args.command == "check-skip-all":
|
||||
return 0 if decision.skip_all else 1
|
||||
|
||||
if args.command == "check-skip-l2-l3":
|
||||
return 0 if not decision.is_run(args.platform, args.level) else 1
|
||||
|
||||
if args.command == "print-annotate":
|
||||
print(decision.message or "CI will run normally")
|
||||
return 0
|
||||
|
||||
return 1
|
||||
|
||||
|
||||
|
||||
@@ -10,30 +10,10 @@ Bootstrap mode (``bootstrap-upload-steps.yml``):
|
||||
- Detect docs-only, pytest skip-mark-only, or combined skip-ci from git diff.
|
||||
- When only CI level YAML changes, enable **L2/L3** upload steps for affected levels only.
|
||||
|
||||
Test pipeline mode (e.g. test-merge.yml, test-nightly.yml, test-weekly.yml):
|
||||
- Drop steps whose ``source_file_dependencies`` do not match changed files
|
||||
(string/list keys from ci_source_file_dependencies.yml, or inline path
|
||||
prefixes). Filtering applies on **PR label** uploads and on post-merge
|
||||
``main`` L3 uploads. Scheduled ``main`` uploads that pass ``--all``
|
||||
(NIGHTLY/WEEKLY) or ``--e2e`` keep every selected job and still strip
|
||||
the field. If no job-key prefix matches, a change to the pipeline YAML
|
||||
being uploaded or to a path under the ``source_filter_fallback`` registry
|
||||
key keeps every job so command, env, and hardware edits can be validated
|
||||
before merge. When any job-key prefix already matches, normal filtering
|
||||
wins.
|
||||
- Expand uploader-only ``mirror_hardwares`` into ``agents`` (+ optional ``image``
|
||||
Test pipeline mode (e.g. test-merge.yml):
|
||||
- Drop steps whose ``source_file_dependencies`` do not match changed files.
|
||||
- Expand uploader-only ``mirror_hardwares: l4_1`` into ``agents`` (+ optional ``image``
|
||||
for NPU) + ``plugins`` (see ci_mirror_hardwares.yml).
|
||||
- Omit ``mirror_hardwares`` to compose ``{chip}_{n}`` from pytest ``-m`` SKU
|
||||
markers plus ``cards_n`` (max of several positives; ``not cards_1`` with no
|
||||
positive ``cards_*`` uses that chip's highest existing preset). An explicit
|
||||
preset string (``h100_2``) is used as-is (marks are ignored). Names that
|
||||
are not keys in ``ci_mirror_hardwares.yml`` fail the upload.
|
||||
|
||||
Inferred chip: unset ``MIRROR_HW`` matches ``H100`` or ``L4`` in ``-m``
|
||||
(both → H100); no match skips the step. ``MIRROR_HW=b200`` matches
|
||||
``B200`` in ``-m`` (otherwise skipped). ``-m`` is not rewritten.
|
||||
``MIRROR_HW`` must be empty or ``b200``; unknown values fail the upload.
|
||||
A CUDA preset string such as ``h100_4`` is omitted when ``MIRROR_HW=b200``.
|
||||
|
||||
Usage:
|
||||
python3 upload_pipeline.py [--upload] [--all | --e2e] <pipeline.yml>
|
||||
@@ -45,13 +25,11 @@ from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any, Literal
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
import yaml
|
||||
@@ -67,27 +45,11 @@ from skip_ci import (
|
||||
resolve_ci_context_from_git,
|
||||
)
|
||||
|
||||
if str(ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
from tests.helpers.mark import ( # noqa: E402
|
||||
get_skus_for_platform,
|
||||
get_supported_card_counts,
|
||||
)
|
||||
|
||||
# --- Constants ---
|
||||
|
||||
LOG = "upload_pipeline"
|
||||
BOOTSTRAP_STEPS_FILENAME = "bootstrap-upload-steps.yml"
|
||||
BOOTSTRAP_IMAGE_BUILD_KEYS = frozenset(
|
||||
{
|
||||
"image-build",
|
||||
"image-build-a2",
|
||||
"image-build-a3",
|
||||
"image-build-a5",
|
||||
"image-build-310p",
|
||||
}
|
||||
)
|
||||
BOOTSTRAP_IMAGE_BUILD_KEYS = frozenset({"image-build", "image-build-a2", "image-build-a3"})
|
||||
BOOTSTRAP_UPLOAD_IF_KEYS = {
|
||||
"upload-ready-pipeline": "ready",
|
||||
"upload-merge-pipeline": "merge",
|
||||
@@ -96,20 +58,20 @@ BOOTSTRAP_UPLOAD_IF_KEYS = {
|
||||
}
|
||||
E2E_GROUP_MARKER = "E2E Test"
|
||||
CI_MIRROR_HARDWARES_PATH = ROOT / ".buildkite/common/ci_mirror_hardwares.yml"
|
||||
CI_SOURCE_FILE_DEPENDENCIES_PATH = ROOT / ".buildkite/common/ci_source_file_dependencies.yml"
|
||||
# Registry key whose prefixes bypass source filtering. Paths live in
|
||||
# ci_source_file_dependencies.yml; do not attach this key to a job.
|
||||
SOURCE_FILTER_FALLBACK_KEY = "source_filter_fallback"
|
||||
CUDA_HF_TOKEN_ENV = "VLLM_CI_HF_TOKEN"
|
||||
CUDA_HF_TOKEN_EXPORT = f'if [ -n "$${{{CUDA_HF_TOKEN_ENV}:-}}" ]; then export HF_TOKEN="$${{{CUDA_HF_TOKEN_ENV}}}"; fi'
|
||||
CUDA_RUNTIME_CHECK = "python3 .buildkite/cuda/scripts/check_vllm_runtime.py"
|
||||
|
||||
# Bootstrap Buildkite ``if`` expressions.
|
||||
# ``*_MAIN_IF``: main + env schedule. ``*_LABEL_IF``: PR label (and/or composed with MAIN).
|
||||
# ``*_UPLOAD_IF``: full gate for uploading that child pipeline.
|
||||
NIGHTLY_MAIN_IF = 'build.branch == "main" && build.env("NIGHTLY") == "1"'
|
||||
NIGHTLY_LABEL_IF = (
|
||||
f'({NIGHTLY_MAIN_IF}) || (build.branch != "main" && build.pull_request.labels includes "nightly-test")'
|
||||
f"({NIGHTLY_MAIN_IF}) || "
|
||||
'(build.branch != "main" && ('
|
||||
'build.pull_request.labels includes "nightly-test" || '
|
||||
'build.pull_request.labels includes "omni-test" || '
|
||||
'build.pull_request.labels includes "tts-test" || '
|
||||
'build.pull_request.labels includes "diffusion-x2iat-test" || '
|
||||
'build.pull_request.labels includes "diffusion-x2v-test"'
|
||||
"))"
|
||||
)
|
||||
WEEKLY_E2E_IF = 'build.branch == "main" && build.env("WEEKLY") == "1"'
|
||||
WEEKLY_MAIN_IF = 'build.branch == "main" && (build.env("WEEKLY") == "1" || build.env("NON_CRITICAL") == "1")'
|
||||
@@ -255,7 +217,7 @@ def _render_bootstrap_pipeline(
|
||||
return yaml.safe_dump(doc, sort_keys=False)
|
||||
|
||||
|
||||
# --- Test pipeline (test-ready.yml, test-merge.yml, test-nightly.yml) ---
|
||||
# --- Test pipeline (test-ready.yml, test-merge.yml) ---
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
@@ -271,365 +233,32 @@ def _load_mirror_hardwares() -> dict[str, dict[str, Any]]:
|
||||
return presets
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _load_source_file_dependencies() -> dict[str, list[str]]:
|
||||
def flatten(value: Any, *, key: str) -> list[str]:
|
||||
"""Flatten YAML-anchor nested lists into a de-duplicated prefix list."""
|
||||
if isinstance(value, str):
|
||||
if not value.strip():
|
||||
raise ValueError(f"empty path in source_file_dependencies[{key!r}]")
|
||||
return [value]
|
||||
if isinstance(value, list):
|
||||
flattened: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for item in value:
|
||||
for path in flatten(item, key=key):
|
||||
if path not in seen:
|
||||
seen.add(path)
|
||||
flattened.append(path)
|
||||
return flattened
|
||||
def _expand_mirror_hardwares(step: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Replace uploader-only ``mirror_hardwares`` with preset fields from ci_mirror_hardwares.yml."""
|
||||
hardware = step.get("mirror_hardwares")
|
||||
if hardware is None:
|
||||
return step
|
||||
|
||||
if not isinstance(hardware, str) or not hardware.strip():
|
||||
raise ValueError(
|
||||
f"source_file_dependencies[{key!r}] must be a list of path prefixes, got {type(value).__name__}",
|
||||
f"mirror_hardwares must be a non-empty string in step {_get_step_label(step)!r}",
|
||||
)
|
||||
|
||||
if not CI_SOURCE_FILE_DEPENDENCIES_PATH.is_file():
|
||||
raise FileNotFoundError(
|
||||
f"missing CI source_file_dependencies registry: {CI_SOURCE_FILE_DEPENDENCIES_PATH}",
|
||||
)
|
||||
doc = yaml.safe_load(CI_SOURCE_FILE_DEPENDENCIES_PATH.read_text(encoding="utf-8"))
|
||||
if not isinstance(doc, dict):
|
||||
raise ValueError(f"invalid CI source_file_dependencies registry: {CI_SOURCE_FILE_DEPENDENCIES_PATH}")
|
||||
presets = doc.get("source_file_dependencies")
|
||||
if not isinstance(presets, dict):
|
||||
raise ValueError(f"source_file_dependencies must be a mapping in {CI_SOURCE_FILE_DEPENDENCIES_PATH}")
|
||||
loaded: dict[str, list[str]] = {}
|
||||
for key, value in presets.items():
|
||||
if not isinstance(key, str) or not key.strip():
|
||||
raise ValueError(
|
||||
f"source_file_dependencies keys must be non-empty strings in {CI_SOURCE_FILE_DEPENDENCIES_PATH}",
|
||||
)
|
||||
loaded[key] = flatten(value, key=key)
|
||||
return loaded
|
||||
|
||||
|
||||
_REGISTRY_KEY_RE = re.compile(r"^[a-z][a-z0-9_]*$")
|
||||
|
||||
|
||||
def _parse_command_text(step: dict[str, Any]) -> str:
|
||||
"""Parse step ``commands`` / ``command`` into a single string."""
|
||||
raw = step.get("commands", step.get("command"))
|
||||
if raw is None:
|
||||
return ""
|
||||
if isinstance(raw, str):
|
||||
return raw
|
||||
if isinstance(raw, list):
|
||||
return "\n".join(str(item) for item in raw if item is not None)
|
||||
return str(raw)
|
||||
|
||||
|
||||
def _resolve_source_file_dependencies(step: dict[str, Any]) -> list[str] | None:
|
||||
"""Expand a registry key (or list of keys / inline path prefixes)."""
|
||||
deps = step.get("source_file_dependencies")
|
||||
if deps is None:
|
||||
return None
|
||||
|
||||
def dedupe(*groups: list[str]) -> list[str]:
|
||||
merged: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for group in groups:
|
||||
for path in group:
|
||||
if path not in seen:
|
||||
seen.add(path)
|
||||
merged.append(path)
|
||||
return merged
|
||||
|
||||
def lookup(key: str) -> list[str]:
|
||||
if key == SOURCE_FILTER_FALLBACK_KEY:
|
||||
raise ValueError(
|
||||
f"source_file_dependencies {key!r} bypasses filtering and cannot be attached to a step "
|
||||
f"({_get_step_label(step)!r})",
|
||||
)
|
||||
registry = _load_source_file_dependencies()
|
||||
paths = registry.get(key)
|
||||
if paths is None:
|
||||
known = ", ".join(sorted(registry))
|
||||
raise ValueError(
|
||||
f"unknown source_file_dependencies {key!r} in step {_get_step_label(step)!r}; known: {known}",
|
||||
)
|
||||
return list(paths)
|
||||
|
||||
prefixes: list[str]
|
||||
if isinstance(deps, str):
|
||||
prefixes = lookup(deps) if _REGISTRY_KEY_RE.fullmatch(deps) else [deps]
|
||||
elif isinstance(deps, list):
|
||||
if not all(isinstance(item, str) for item in deps):
|
||||
raise ValueError(
|
||||
f"source_file_dependencies must be a string key or list of strings in step {_get_step_label(step)!r}",
|
||||
)
|
||||
keyish = [bool(_REGISTRY_KEY_RE.fullmatch(item)) for item in deps]
|
||||
if deps and all(keyish):
|
||||
prefixes = dedupe(*(lookup(key) for key in deps))
|
||||
elif any(keyish):
|
||||
raise ValueError(
|
||||
f"source_file_dependencies in step {_get_step_label(step)!r} mixes registry keys and path prefixes",
|
||||
)
|
||||
else:
|
||||
prefixes = list(deps)
|
||||
else:
|
||||
raise ValueError(
|
||||
f"source_file_dependencies must be a string key or list in step {_get_step_label(step)!r}",
|
||||
)
|
||||
return prefixes
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _cuda_mirror_chips() -> tuple[str, ...]:
|
||||
"""Lowercase CUDA SKUs that have at least one ``{chip}_{n}`` preset.
|
||||
|
||||
Longer tokens first so ``h100`` is not matched as a prefix of ``h200``.
|
||||
"""
|
||||
skus = get_skus_for_platform("cuda")
|
||||
known = _load_mirror_hardwares()
|
||||
chips = {
|
||||
sku.lower() for sku in skus if any(name == sku.lower() or name.startswith(f"{sku.lower()}_") for name in known)
|
||||
}
|
||||
return tuple(sorted(chips, key=lambda chip: (-len(chip), chip)))
|
||||
|
||||
|
||||
def _parse_cards_marks(expr: str) -> tuple[set[int], bool]:
|
||||
"""Parse registered ``cards_n`` / ``not cards_n`` from a pytest ``-m`` expr.
|
||||
|
||||
Returns positive card counts and whether any ``not cards_*`` is present.
|
||||
"""
|
||||
counts = get_supported_card_counts()
|
||||
alt = "|".join(str(n) for n in sorted(counts, key=lambda n: (-len(str(n)), -n))) if counts else "(?!)"
|
||||
positives: set[int] = set()
|
||||
has_not_cards = False
|
||||
for match in re.finditer(rf"\b(?:(not)\s+)?cards_({alt})\b", expr):
|
||||
if match.group(1):
|
||||
has_not_cards = True
|
||||
else:
|
||||
positives.add(int(match.group(2)))
|
||||
return positives, has_not_cards
|
||||
|
||||
|
||||
def _parse_hardware_marks(expr: str) -> set[str]:
|
||||
"""Parse lowercase positive CUDA SKUs that have a mirror preset.
|
||||
|
||||
``not H100`` is ignored. SKUs with no ``{chip}_*`` preset (e.g. H200) are dropped.
|
||||
"""
|
||||
skus = sorted(get_skus_for_platform("cuda"), key=lambda s: (-len(s), s))
|
||||
if not skus:
|
||||
return set()
|
||||
alt = "|".join(re.escape(sku) for sku in skus)
|
||||
mirror = set(_cuda_mirror_chips())
|
||||
chips: set[str] = set()
|
||||
for match in re.finditer(rf"\b(?:(not)\s+)?({alt})\b", expr):
|
||||
if match.group(1):
|
||||
continue
|
||||
chip = match.group(2).lower()
|
||||
if chip in mirror:
|
||||
chips.add(chip)
|
||||
return chips
|
||||
|
||||
|
||||
# CI policy: not pytest-mark registry.
|
||||
_SUPPORTED_MIRROR_HW_SELECTORS = frozenset({"b200"})
|
||||
# Unset MIRROR_HW: match H100 then L4 in -m (skip if neither).
|
||||
_DEFAULT_INFER_CHIPS = ("h100", "l4")
|
||||
_PYTEST_MARKER_ARG = re.compile(r"-m\s+(?:\"([^\"]*)\"|'([^']*)'|(\S+))")
|
||||
_CARDS_CHIP_MAX: Literal["max"] = "max"
|
||||
|
||||
|
||||
def _get_mirror_hw_selector() -> str:
|
||||
"""Return lowercase ``MIRROR_HW``, or empty to keep string presets / marker chips.
|
||||
|
||||
A set value must be ``b200`` (case-insensitive). Unknown values fail closed
|
||||
so a typo cannot silently drop the CUDA pipeline.
|
||||
"""
|
||||
selector = os.environ.get("MIRROR_HW", "").strip().lower()
|
||||
if not selector:
|
||||
return ""
|
||||
if selector not in _SUPPORTED_MIRROR_HW_SELECTORS:
|
||||
allowed = " or ".join(repr(s) for s in sorted(_SUPPORTED_MIRROR_HW_SELECTORS))
|
||||
raise ValueError(f"unsupported MIRROR_HW={selector!r}; expected empty or {allowed}")
|
||||
return selector
|
||||
|
||||
|
||||
def _parse_pytest_marks(step: dict[str, Any]) -> tuple[set[str], int | Literal["max"] | None]:
|
||||
"""Parse pytest ``-m`` from step commands: CUDA SKU chips and cards count.
|
||||
|
||||
*chips* are lowercase SKUs from positive markers (``not H100`` is ignored).
|
||||
*cards* is the max positive ``cards_n``, ``"max"`` when only ``not cards_*``
|
||||
is present, or ``None``.
|
||||
"""
|
||||
text = _parse_command_text(step)
|
||||
|
||||
chips: set[str] = set()
|
||||
positives: set[int] = set()
|
||||
has_not_cards = False
|
||||
for match in _PYTEST_MARKER_ARG.finditer(text):
|
||||
expr = match.group(1) or match.group(2) or match.group(3) or ""
|
||||
chips |= _parse_hardware_marks(expr)
|
||||
card_counts, not_cards = _parse_cards_marks(expr)
|
||||
positives.update(card_counts)
|
||||
has_not_cards = has_not_cards or not_cards
|
||||
|
||||
if positives:
|
||||
cards: int | Literal["max"] | None = max(positives)
|
||||
elif has_not_cards:
|
||||
cards = _CARDS_CHIP_MAX
|
||||
else:
|
||||
cards = None
|
||||
return chips, cards
|
||||
|
||||
|
||||
def _compose_mirror_hardware_name(
|
||||
chips: set[str],
|
||||
cards: int | Literal["max"] | None,
|
||||
*,
|
||||
step_label: str,
|
||||
) -> str | None:
|
||||
"""Assemble ``{chip}_{n}`` from pytest ``-m`` chips/cards, or None to skip.
|
||||
|
||||
Unset ``MIRROR_HW``: ``H100`` if in ``-m``, else ``L4``, else skip.
|
||||
Set ``MIRROR_HW``: that chip if in ``-m``, else skip.
|
||||
``cards`` is a registered ``cards_n``, ``"max"`` for that chip's highest
|
||||
existing preset, or None (error after a chip matched — no ``cards_*``).
|
||||
"""
|
||||
selector = _get_mirror_hw_selector()
|
||||
if selector:
|
||||
chip = selector if selector in chips else None
|
||||
skip_reason = f"MIRROR_HW={selector!r} not in -m SKUs {sorted(chips)}"
|
||||
else:
|
||||
chip = next((c for c in _DEFAULT_INFER_CHIPS if c in chips), None)
|
||||
skip_reason = f"no H100 or L4 in -m SKUs {sorted(chips)}"
|
||||
if chip is None:
|
||||
_log(f"skip {step_label}: {skip_reason}")
|
||||
return None
|
||||
if cards is None:
|
||||
raise ValueError(
|
||||
f"step {step_label!r} has a pytest -m SKU marker but no cards_* "
|
||||
f"(or not cards_*); add cards_n or set mirror_hardwares to force a preset",
|
||||
)
|
||||
|
||||
count_source = "not cards_*" if cards == _CARDS_CHIP_MAX else f"cards_{cards}"
|
||||
count = None if cards == _CARDS_CHIP_MAX else cards
|
||||
registered = get_supported_card_counts()
|
||||
if count is not None and count not in registered:
|
||||
supported = ", ".join(f"cards_{n}" for n in sorted(registered)) or "(none)"
|
||||
raise ValueError(
|
||||
f"{count_source} in step {step_label!r} is not a registered cards_* mark; supported: {supported}",
|
||||
)
|
||||
known = _load_mirror_hardwares()
|
||||
if count is None:
|
||||
available = [n for n in sorted(registered) if f"{chip}_{n}" in known]
|
||||
if not available:
|
||||
raise ValueError(
|
||||
f"{count_source} in step {step_label!r} has no CUDA count presets for {chip}",
|
||||
)
|
||||
count = max(available)
|
||||
chosen = f"{chip}_{count}"
|
||||
if chosen not in known:
|
||||
raise ValueError(
|
||||
f"{count_source} in step {step_label!r} has no preset {chosen!r}",
|
||||
)
|
||||
_log(f"{step_label}: {count_source} → {chosen!r}")
|
||||
return chosen
|
||||
|
||||
|
||||
def _resolve_mirror_hardware_name(hardware: Any, *, step_label: str) -> str | None:
|
||||
"""Map ``mirror_hardwares`` to a ``ci_mirror_hardwares.yml`` preset, or None to skip.
|
||||
|
||||
The value is a preset name (``h100_2``). Unknown names fail later when the
|
||||
registry is loaded. CUDA names are omitted when ``MIRROR_HW`` names a
|
||||
different chip; NPU strings ignore ``MIRROR_HW``.
|
||||
"""
|
||||
if isinstance(hardware, bool):
|
||||
raise ValueError(f"mirror_hardwares must not be a boolean in step {step_label!r}")
|
||||
if isinstance(hardware, str):
|
||||
name = hardware.strip()
|
||||
elif isinstance(hardware, int):
|
||||
name = str(hardware)
|
||||
else:
|
||||
raise ValueError(
|
||||
f"mirror_hardwares must be a preset string in step {step_label!r}",
|
||||
)
|
||||
if not name:
|
||||
raise ValueError(f"mirror_hardwares must be a non-empty string in step {step_label!r}")
|
||||
|
||||
selector = _get_mirror_hw_selector()
|
||||
token = name.lower()
|
||||
chip = next(
|
||||
(c for c in _cuda_mirror_chips() if token == c or token.startswith(f"{c}_")),
|
||||
None,
|
||||
)
|
||||
if selector and chip is not None and chip != selector:
|
||||
_log(
|
||||
f"skip {step_label}: preset {name!r} is {chip} hardware; MIRROR_HW={selector!r}",
|
||||
)
|
||||
return None
|
||||
return name
|
||||
|
||||
|
||||
def _expand_mirror_hardwares(step: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Replace ``mirror_hardwares`` (or infer it from ``-m``) with preset agents/plugins.
|
||||
|
||||
Explicit preset string: used as-is; pytest marks are ignored.
|
||||
Omitted key: match H100/L4 (or ``MIRROR_HW``) in ``-m``, then compose
|
||||
``{chip}_{n}`` from ``cards_n``. No match skips the step. Steps with no
|
||||
SKU/cards and no ``mirror_hardwares`` are left unchanged (CPU jobs).
|
||||
|
||||
Returns None when the current ``MIRROR_HW`` selector does not apply.
|
||||
"""
|
||||
step_label = _get_step_label(step)
|
||||
has_pool = step.get("agents") is not None or step.get("plugins") is not None or step.get("image") is not None
|
||||
|
||||
if "mirror_hardwares" not in step:
|
||||
if has_pool:
|
||||
return step
|
||||
chips, cards = _parse_pytest_marks(step)
|
||||
if not chips and cards is None:
|
||||
return step
|
||||
preset_name = _compose_mirror_hardware_name(chips, cards, step_label=step_label)
|
||||
else:
|
||||
if has_pool:
|
||||
raise ValueError(
|
||||
f"step {step_label!r} sets mirror_hardwares together with agents/plugins/image; "
|
||||
f"use mirror_hardwares only",
|
||||
)
|
||||
preset_name = _resolve_mirror_hardware_name(
|
||||
step.get("mirror_hardwares"),
|
||||
step_label=step_label,
|
||||
)
|
||||
|
||||
if preset_name is None:
|
||||
return None
|
||||
|
||||
preset = _load_mirror_hardwares().get(preset_name)
|
||||
preset = _load_mirror_hardwares().get(hardware)
|
||||
if preset is None:
|
||||
known = ", ".join(sorted(_load_mirror_hardwares()))
|
||||
raise ValueError(
|
||||
f"unknown mirror_hardwares {preset_name!r} in step {step_label!r}; known: {known}",
|
||||
f"unknown mirror_hardwares {hardware!r} in step {_get_step_label(step)!r}; known: {known}",
|
||||
)
|
||||
|
||||
if step.get("agents") is not None or step.get("plugins") is not None or step.get("image") is not None:
|
||||
raise ValueError(
|
||||
f"step {_get_step_label(step)!r} sets mirror_hardwares together with agents/plugins/image; "
|
||||
"use mirror_hardwares only",
|
||||
)
|
||||
|
||||
expanded = copy.deepcopy(preset)
|
||||
merged = {key: value for key, value in step.items() if key != "mirror_hardwares"} | expanded
|
||||
# A scheduled build's HF_TOKEN may override a pod env entry with the same
|
||||
# name. The preset injects the Kubernetes secret under a collision-proof
|
||||
# alias; restore the conventional name after Buildkite has merged env.
|
||||
if any(preset_name == chip or preset_name.startswith(f"{chip}_") for chip in _cuda_mirror_chips()):
|
||||
commands = merged.get("commands")
|
||||
if isinstance(commands, list):
|
||||
merged["commands"] = [CUDA_HF_TOKEN_EXPORT, CUDA_RUNTIME_CHECK, *commands]
|
||||
elif commands is None:
|
||||
merged["commands"] = [CUDA_HF_TOKEN_EXPORT, CUDA_RUNTIME_CHECK]
|
||||
else:
|
||||
merged["commands"] = [CUDA_HF_TOKEN_EXPORT, CUDA_RUNTIME_CHECK, commands]
|
||||
# Preset retry (K8S_RETRY on l4_*) must not clobber a step that opted out.
|
||||
if "retry" in step:
|
||||
merged["retry"] = step["retry"]
|
||||
return merged
|
||||
return {key: value for key, value in step.items() if key != "mirror_hardwares"} | expanded
|
||||
|
||||
|
||||
def _match_source_file(changed_files: list[str], prefixes: list[str]) -> bool:
|
||||
@@ -656,7 +285,11 @@ def _process_test_steps(
|
||||
processed.append(step)
|
||||
continue
|
||||
|
||||
deps = _resolve_source_file_dependencies(step)
|
||||
deps = step.get("source_file_dependencies")
|
||||
if deps is not None and not isinstance(deps, list):
|
||||
raise ValueError(
|
||||
f"source_file_dependencies must be a list in step {_get_step_label(step)!r}",
|
||||
)
|
||||
if changed_files is not None and deps is not None and not _match_source_file(changed_files, deps):
|
||||
_log(f"skip {_get_step_label(step)!r} (no changes under {deps})")
|
||||
continue
|
||||
@@ -664,7 +297,7 @@ def _process_test_steps(
|
||||
nested = step.get("steps")
|
||||
if nested is not None:
|
||||
kept_nested = _process_test_steps(nested, changed_files)
|
||||
if not kept_nested:
|
||||
if changed_files is not None and not kept_nested:
|
||||
_log(f"omit empty group {_get_step_label(step)!r}")
|
||||
continue
|
||||
new_step = {key: value for key, value in step.items() if key != "source_file_dependencies"}
|
||||
@@ -672,11 +305,14 @@ def _process_test_steps(
|
||||
processed.append(new_step)
|
||||
continue
|
||||
|
||||
leaf = {key: value for key, value in step.items() if key != "source_file_dependencies"}
|
||||
expanded = _expand_mirror_hardwares(leaf)
|
||||
if expanded is None:
|
||||
continue
|
||||
processed.append(expanded)
|
||||
if deps is not None:
|
||||
processed.append(
|
||||
_expand_mirror_hardwares(
|
||||
{key: value for key, value in step.items() if key != "source_file_dependencies"},
|
||||
),
|
||||
)
|
||||
else:
|
||||
processed.append(_expand_mirror_hardwares(step))
|
||||
|
||||
return processed
|
||||
|
||||
@@ -695,44 +331,18 @@ def _select_e2e_group_steps(steps: list[Any]) -> list[Any]:
|
||||
return selected
|
||||
|
||||
|
||||
def _any_source_dependency_match(steps: list[Any], changed_files: list[str]) -> bool:
|
||||
"""True when any step with ``source_file_dependencies`` matches *changed_files*."""
|
||||
for step in steps:
|
||||
if not isinstance(step, dict):
|
||||
continue
|
||||
deps = _resolve_source_file_dependencies(step)
|
||||
if deps is not None and _match_source_file(changed_files, deps):
|
||||
return True
|
||||
nested = step.get("steps")
|
||||
if isinstance(nested, list) and _any_source_dependency_match(nested, changed_files):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _render_test_pipeline(
|
||||
doc: dict[str, Any],
|
||||
changed_files: list[str] | None,
|
||||
*,
|
||||
pipeline_path: Path | None = None,
|
||||
e2e_only: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Filter steps by PR diff and strip uploader-only ``source_file_dependencies`` metadata."""
|
||||
# Validate MIRROR_HW once up front. Per-step skip must not run for typos
|
||||
# (e.g. b20o), or a pipeline can shrink to leftover CPU-only steps and
|
||||
# still upload successfully.
|
||||
_get_mirror_hw_selector()
|
||||
steps = doc.get("steps")
|
||||
if not isinstance(steps, list):
|
||||
return doc
|
||||
if e2e_only:
|
||||
steps = _select_e2e_group_steps(steps)
|
||||
# Bypass is a fallback: only when no job-key prefix matched.
|
||||
if changed_files is not None and pipeline_path is not None:
|
||||
if not _any_source_dependency_match(steps, changed_files):
|
||||
bypass = _source_filter_fallback_reason(changed_files, pipeline_path)
|
||||
if bypass is not None:
|
||||
_log(f"keep all jobs (no source_file_dependencies match; bypassed by {bypass})")
|
||||
changed_files = None
|
||||
steps = _process_test_steps(steps, changed_files)
|
||||
return {**doc, "steps": steps}
|
||||
|
||||
@@ -740,60 +350,6 @@ def _render_test_pipeline(
|
||||
# --- Entry (read file → bootstrap or test render → YAML string) ---
|
||||
|
||||
|
||||
def _repo_relative_posix(path: Path) -> str:
|
||||
resolved = path.resolve()
|
||||
try:
|
||||
return resolved.relative_to(ROOT).as_posix()
|
||||
except ValueError:
|
||||
return path.as_posix().replace("\\", "/")
|
||||
|
||||
|
||||
def _source_filter_fallback_prefixes() -> list[str]:
|
||||
"""Shared uploader/registry prefixes from ``source_filter_fallback``. Not a job key."""
|
||||
registry = _load_source_file_dependencies()
|
||||
prefixes = registry.get(SOURCE_FILTER_FALLBACK_KEY)
|
||||
if not prefixes:
|
||||
raise ValueError(
|
||||
f"source_file_dependencies[{SOURCE_FILTER_FALLBACK_KEY!r}] must list shared uploader paths "
|
||||
f"in {CI_SOURCE_FILE_DEPENDENCIES_PATH}",
|
||||
)
|
||||
return prefixes
|
||||
|
||||
|
||||
def _source_filter_fallback_reason(changed_files: list[str] | None, pipeline_path: Path) -> str | None:
|
||||
"""Return the changed path that should disable source filtering, if any."""
|
||||
if not changed_files:
|
||||
return None
|
||||
changed = set(changed_files)
|
||||
pipeline_rel = _repo_relative_posix(pipeline_path)
|
||||
if pipeline_rel in changed:
|
||||
return pipeline_rel
|
||||
prefixes = _source_filter_fallback_prefixes()
|
||||
for path in changed_files:
|
||||
if _match_source_file([path], prefixes):
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
def _changed_files_for_source_filter(
|
||||
ctx,
|
||||
*,
|
||||
force_all: bool,
|
||||
e2e_only: bool,
|
||||
) -> list[str] | None:
|
||||
"""Return the diff to filter against, or None to keep every step.
|
||||
|
||||
``--all`` / ``--e2e`` disable filtering (scheduled NIGHTLY/WEEKLY full
|
||||
uploads and weekly E2E sweeps). Post-merge ``main`` L3 and PR-label
|
||||
uploads filter against the commit/PR diff. Pipeline YAML /
|
||||
``source_filter_fallback`` matches are applied later in
|
||||
``_render_test_pipeline`` only when no job-key prefix already matched.
|
||||
"""
|
||||
if force_all or e2e_only:
|
||||
return None
|
||||
return ctx.changed_files
|
||||
|
||||
|
||||
def _render_pipeline(
|
||||
path: Path,
|
||||
*,
|
||||
@@ -811,18 +367,16 @@ def _render_pipeline(
|
||||
|
||||
text = path.read_text(encoding="utf-8")
|
||||
ctx = resolve_ci_context_from_git()
|
||||
changed_files = _changed_files_for_source_filter(ctx, force_all=force_all, e2e_only=e2e_only)
|
||||
if force_all or e2e_only:
|
||||
changed_files = None
|
||||
else:
|
||||
changed_files = ctx.changed_files
|
||||
|
||||
doc = yaml.safe_load(text)
|
||||
if not isinstance(doc, dict):
|
||||
raise ValueError(f"invalid pipeline YAML: {path}")
|
||||
|
||||
doc = _render_test_pipeline(
|
||||
doc,
|
||||
changed_files,
|
||||
pipeline_path=path,
|
||||
e2e_only=e2e_only,
|
||||
)
|
||||
doc = _render_test_pipeline(doc, changed_files, e2e_only=e2e_only)
|
||||
return yaml.safe_dump(doc, sort_keys=False)
|
||||
|
||||
|
||||
|
||||
@@ -22,8 +22,7 @@ steps:
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
|
||||
# L3 Test — PR merge-test label or post-merge on main (both diff-aware);
|
||||
# main+WEEKLY=1 uploads the E2E group only (--e2e).
|
||||
# L3 Test — PR merge-test label (diff-aware) or main+WEEKLY=1 (--e2e, full E2E group)
|
||||
- label: "Upload Merge Pipeline"
|
||||
depends_on: image-build
|
||||
key: upload-merge-pipeline
|
||||
@@ -37,30 +36,20 @@ steps:
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
|
||||
# L4 Test — main+NIGHTLY=1 (full pipeline), or PR nightly-test label (diff-aware)
|
||||
# L4 Test — main+NIGHTLY=1 (scheduled), or PR with specific label (e.g. add label then Rebuild)
|
||||
- label: "Upload Nightly Pipeline"
|
||||
depends_on: image-build
|
||||
key: upload-nightly-pipeline
|
||||
commands:
|
||||
- |
|
||||
EXTRA=""
|
||||
if [ "${NIGHTLY:-}" = "1" ] && [ "${BUILDKITE_BRANCH}" = "main" ]; then
|
||||
EXTRA="--all"
|
||||
fi
|
||||
python3 .buildkite/common/scripts/upload_pipeline.py --upload $$EXTRA .buildkite/cuda/test-nightly.yml
|
||||
- python3 .buildkite/common/scripts/upload_pipeline.py --upload .buildkite/cuda/test-nightly.yml
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
|
||||
# L5 Test — main+WEEKLY=1 / NON_CRITICAL=1 (full pipeline), or PR weekly-test label (diff-aware)
|
||||
# L5 Test — main+WEEKLY=1 / NON_CRITICAL=1 (scheduled), or PR with weekly-test label
|
||||
- label: "Upload Weekly Pipeline"
|
||||
depends_on: image-build
|
||||
key: upload-weekly-pipeline
|
||||
commands:
|
||||
- |
|
||||
EXTRA=""
|
||||
if [ "${BUILDKITE_BRANCH}" = "main" ]; then
|
||||
EXTRA="--all"
|
||||
fi
|
||||
python3 .buildkite/common/scripts/upload_pipeline.py --upload $$EXTRA .buildkite/cuda/test-weekly.yml
|
||||
- python3 .buildkite/common/scripts/upload_pipeline.py --upload .buildkite/cuda/test-weekly.yml
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
@@ -1,38 +0,0 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
"""Verify the pinned vLLM and compiled CUDA imports on a GPU CI worker."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def main() -> None:
|
||||
dockerfile = Path(__file__).resolve().parents[3] / "docker" / "Dockerfile.ci"
|
||||
match = re.search(
|
||||
r"^ARG VLLM_VERSION=([0-9]+\.[0-9]+\.[0-9]+)$",
|
||||
dockerfile.read_text(),
|
||||
re.MULTILINE,
|
||||
)
|
||||
if match is None:
|
||||
raise RuntimeError(f"Missing vLLM release version in {dockerfile}")
|
||||
expected_version = match.group(1)
|
||||
|
||||
import torch
|
||||
import vllm
|
||||
|
||||
assert vllm.__version__ == expected_version, vllm.__version__
|
||||
assert torch.cuda.is_available(), "vLLM runtime check requires a GPU worker"
|
||||
for module in ("flashinfer", "vllm._C_stable_libtorch"):
|
||||
importlib.import_module(module)
|
||||
print(
|
||||
f"Verified vLLM CUDA runtime: vllm={vllm.__version__} "
|
||||
f"expected={expected_version} torch={torch.__version__} "
|
||||
f"cuda={torch.version.cuda} device={torch.cuda.get_device_name()}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+85
-131
@@ -1,9 +1,6 @@
|
||||
# CUDA CI L3 (`advanced_model` e2e) plus L1 CPU Simple Test (`core_model and cpu`).
|
||||
# Uploaded by Upload Merge Pipeline: PR label `merge-test`, or post-merge on main
|
||||
# (both diff-aware via source_file_dependencies).
|
||||
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_USE_FLASHINFER_MOE_FP16: 0
|
||||
HF_HUB_DOWNLOAD_TIMEOUT: 300
|
||||
HF_HUB_ETAG_TIMEOUT: 60
|
||||
|
||||
@@ -18,7 +15,7 @@ steps:
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Simple · Diffusion Test"
|
||||
timeout_in_minutes: 30
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu'"
|
||||
mirror_hardwares: l4_1
|
||||
@@ -38,17 +35,6 @@ steps:
|
||||
- group: ":card_index_dividers: Diffusion Test"
|
||||
depends_on: upload-merge-pipeline
|
||||
steps:
|
||||
- label: "Diffusion · AR Paged Attention GPU Test"
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- export VLLM_OMNI_AR_FA_REQUIRED=1
|
||||
- >-
|
||||
timeout 15m pytest -sv
|
||||
tests/diffusion/ar_diffusion/test_paged_attention.py
|
||||
-m 'core_model and cuda and L4 and cards_1'
|
||||
--run-level "core_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · LoRA Test"
|
||||
timeout_in_minutes: 30
|
||||
commands:
|
||||
@@ -71,50 +57,55 @@ steps:
|
||||
depends_on: upload-merge-pipeline
|
||||
steps:
|
||||
- label: "Entrypoint Test with H100"
|
||||
timeout_in_minutes: 60
|
||||
# TEMPORARY 60 -> 90: engine startup/sleep-wake is ~13-15 min slower per
|
||||
# run on the vLLM 0.26 image (model loading ~2x, init ~2x; see builds
|
||||
# 2915 vs 2916), which pushed this suite past 60 min. Revert once the
|
||||
# 0.26 startup regression is understood and fixed.
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
# Turn-based server VAD over /v1/realtime is deferred to RFC #7181 PR 5
|
||||
# (see the P0 thread on #7413); the test stays in the tree and runs again there.
|
||||
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and H100" --run-level "advanced_model" --deselect tests/entrypoints/openai_api/test_qwen3_omni_realtime_websocket.py::TestQwen3OmniRealtimeWebSocket::test_server_vad_multi_turn_without_client_commit
|
||||
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and H100" --run-level "advanced_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Entrypoint Test with L4"
|
||||
timeout_in_minutes: 60
|
||||
# Workaround https://github.com/vllm-project/vllm-omni/issues/8016: shutdown()
|
||||
# leaves inline UniProc DiffusionWorker VRAM mapped in the pytest parent.
|
||||
# This job does not collect the ready leftover
|
||||
# (test_diffusion_generate_request_id, core_model); isolate the
|
||||
# advanced_model DiT handshake instead. Revert to a single sweep after
|
||||
# the product shutdown fix.
|
||||
commands:
|
||||
- |
|
||||
set -eu
|
||||
pytest -s -v tests/entrypoints/test_omni_sleep_mode.py::TestOmniDiffusionSleepMode::test_diffusion_sleep_handshake -m "advanced_model and cuda and L4 and cards_1" --run-level "advanced_model"
|
||||
pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and L4 and cards_1" --run-level "advanced_model" --deselect tests/entrypoints/test_omni_sleep_mode.py::TestOmniDiffusionSleepMode::test_diffusion_sleep_handshake
|
||||
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and L4" --run-level "advanced_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Entrypoint Test with L4 · 2-GPU"
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -s -v tests/entrypoints/ -m "advanced_model and cuda and L4 and cards_2" --run-level "advanced_model"
|
||||
mirror_hardwares: l4_2
|
||||
|
||||
- group: ":card_index_dividers: E2E Test"
|
||||
depends_on: upload-merge-pipeline
|
||||
steps:
|
||||
- label: "TTS · Qwen3-TTS CustomVoice Test"
|
||||
source_file_dependencies: tts_qwen3_tts_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_tts_customvoice.py
|
||||
- tests/e2e/offline_inference/test_qwen3_tts_customvoice.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/common/alias_free_activation.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_LOGGING_LEVEL=DEBUG
|
||||
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py tests/e2e/offline_inference/test_qwen3_tts_customvoice.py tests/e2e/features/prefill_phase/test_chunked_prefill_tail.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py tests/e2e/offline_inference/test_qwen3_tts_customvoice.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Qwen3-TTS Base Test"
|
||||
source_file_dependencies: tts_qwen3_tts_cov
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_tts_base.py
|
||||
- tests/e2e/offline_inference/test_qwen3_tts_base.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/common/alias_free_activation.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
- .buildkite/common/scripts/run_cov_split.sh
|
||||
- pyproject.toml
|
||||
# 40 -> 60: the coverage split runs the model twice.
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
@@ -129,23 +120,10 @@ steps:
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Omni · Qwen3-Omni Test"
|
||||
source_file_dependencies: omni_qwen3_omni_function
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py tests/e2e/online_serving/test_qwen3_omni.py tests/e2e/offline_inference/test_qwen3_omni_colocate_async.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Omni · Qwen3-Omni Async-Chunk SHM Leak"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_omni_shm_leak.py
|
||||
- tests/e2e/offline_inference/test_qwen3_omni.py
|
||||
- tests/e2e/online_serving/test_qwen3_omni.py
|
||||
- tests/helpers/runtime.py
|
||||
- tests/helpers/stage_config.py
|
||||
- vllm_omni/core/sched/omni_ar_scheduler.py
|
||||
- vllm_omni/distributed/omni_connectors/connectors/shm_connector.py
|
||||
- vllm_omni/distributed/omni_connectors/transfer_adapter/chunk_transfer_adapter.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
- vllm_omni/engine/stage_engine_core_proc.py
|
||||
- vllm_omni/engine/stage_pool.py
|
||||
- vllm_omni/entrypoints/openai/serving_chat.py
|
||||
- vllm_omni/model_executor/models/qwen3_omni/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
@@ -154,33 +132,70 @@ steps:
|
||||
- vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_omni.py
|
||||
- vllm_omni/deploy/qwen3_omni_moe.yaml
|
||||
timeout_in_minutes: 40
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_shm_leak.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
- pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py tests/e2e/online_serving/test_qwen3_omni.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/offline_inference/test_minicpmo_4_5.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5.py
|
||||
- vllm_omni/model_executor/models/minicpmo_4_5/
|
||||
- vllm_omni/model_executor/models/step_audio2/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/utils.py
|
||||
- vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
|
||||
- vllm_omni/deploy/minicpmo_4_5.yaml
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_minicpmo_4_5.py tests/e2e/online_serving/test_minicpmo_4_5.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
# h100_2, not h100_1: advanced_model asserts transcribe every audio
|
||||
# response with Whisper, and tests/helpers/media.py requires at least
|
||||
# 16 GiB free VRAM on a device to run Whisper on GPU. When the model
|
||||
# server leaves insufficient memory, Whisper falls back to CPU:
|
||||
# ~8-10 min per clip vs ~30 s, which overruns the 50-minute job timeout
|
||||
# (build 2956). h100_2 guarantees a second GPU with ample free VRAM.
|
||||
# response with Whisper, and tests/helpers/media.py only uses an
|
||||
# accelerator for Whisper when a spare GPU exists. On a single card it
|
||||
# grades on CPU: ~8-10 min per clip vs ~30 s, which alone overruns the
|
||||
# 50-minute job timeout (build 2956).
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Duplex Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_function
|
||||
source_file_dependencies:
|
||||
- tests/assets/minicpmo_4_5/response_required_16k.wav
|
||||
- tests/assets/minicpmo_4_5/soft_interrupt_16k.wav
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
- tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py
|
||||
- tests/e2e/online_serving/helpers/minicpmo_realtime_duplex_scenarios.py
|
||||
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_multi_session.py
|
||||
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py
|
||||
- vllm_omni/deploy/minicpmo_4_5.yaml
|
||||
- tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
|
||||
- vllm_omni/deploy/minicpmo_4_5_duplex.yaml
|
||||
- vllm_omni/experimental/fullduplex/
|
||||
- vllm_omni/model_executor/models/minicpmo_4_5/
|
||||
- vllm_omni/model_executor/models/step_audio2/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/utils.py
|
||||
- vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
|
||||
- vllm_omni/core/sched/
|
||||
- vllm_omni/outputs/multimodal_accumulation.py
|
||||
- vllm_omni/engine/stage_init_utils.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
- vllm_omni/entrypoints/cli/serve.py
|
||||
- vllm_omni/entrypoints/openai/api_server.py
|
||||
- vllm_omni/entrypoints/openai/duplex_capability.py
|
||||
- vllm_omni/config/stage_config.py
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_minicpmo_4_5_window.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py -m 'advanced_model and cuda' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Qwen Image Test"
|
||||
source_file_dependencies: diffusion_qwen_image_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen_image.py
|
||||
- vllm_omni/diffusion/models/qwen_image/
|
||||
- vllm_omni/diffusion/models/dmd2/
|
||||
- vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_qwenimage.py
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c '
|
||||
@@ -189,17 +204,12 @@ steps:
|
||||
'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Kandinsky6 Test"
|
||||
source_file_dependencies: diffusion_kandinsky6_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
pytest -s -v tests/e2e/offline_inference/test_kandinsky6.py tests/e2e/online_serving/test_kandinsky6.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Wan22 Test"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/offline_inference/test_wan22_t2v.py
|
||||
- tests/e2e/online_serving/test_wan22_t2v.py
|
||||
- vllm_omni/diffusion/models/wan2_2/
|
||||
- vllm_omni/diffusion/models/dmd2/
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
@@ -207,59 +217,3 @@ steps:
|
||||
pytest -s -v tests/e2e/offline_inference/test_wan22_t2v.py tests/e2e/online_serving/test_wan22_t2v.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · HunyuanVideo-1.5 Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · LingBot-Video Test"
|
||||
source_file_dependencies: diffusion_lingbot_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_lingbot_video.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · MiniMax-H3 L3 DLO DP2 and FastH3"
|
||||
source_file_dependencies: diffusion_minimax_h3_function
|
||||
# Keep four server lifecycles in one Buildkite job, but never in one
|
||||
# pytest process. DLO uses multiprocessing shared-memory queues; the
|
||||
# parent's resource_tracker can retain a stale /psm_* entry during
|
||||
# teardown and strand the next Ref2VA collective.
|
||||
# A per-file process plus a hard timeout also makes a regression fail
|
||||
# instead of leaving the merge queue stuck indefinitely.
|
||||
timeout_in_minutes: 150
|
||||
commands:
|
||||
- |
|
||||
set -eu
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
if [ -f vllm_omni/diffusion/models/minimax_h3/lora.py ]; then
|
||||
export VLLM_TEST_MINIMAX_H3_TURBO_LORA=$(python3 -c 'from huggingface_hub import hf_hub_download; print(hf_hub_download(repo_id="lightx2v/Minimax-h3-Turbo", filename="minimax_h3_fl2v_turbo_4step_v1.0_768p_bf16.safetensors"))')
|
||||
fi
|
||||
if [ -f vllm_omni/diffusion/models/minimax_h3/fasth3.py ]; then
|
||||
export VLLM_TEST_MINIMAX_H3_FASTH3_LORA=$(python3 -c 'from huggingface_hub import hf_hub_download; print(hf_hub_download(repo_id="FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA", filename="dense-datafree/adapter_model.safetensors"))')
|
||||
fi
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3_dlo_dp2_fl2va.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3_dlo_dp2_ref2va.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3_turbo_lora.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3_fasth3_lora.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · MiniMax-H3 Latent Upscale and Refine"
|
||||
source_file_dependencies: diffusion_minimax_h3_function
|
||||
timeout_in_minutes: 40
|
||||
commands:
|
||||
- |
|
||||
set -eu
|
||||
VLLM_TEST_MINIMAX_H3_LATENT_UPSCALER=$(python3 -c 'from huggingface_hub import hf_hub_download; print(hf_hub_download(repo_id="LBH-123-AI/Minimax_h3_latent_Upscaler", filename="minimax_h3_latent_upscaler_3d_conv_v1/minimax_h3_latent_upscaler_3d_conv_v1_bf16.safetensors"))')
|
||||
export VLLM_TEST_MINIMAX_H3_LATENT_UPSCALER
|
||||
timeout 35m pytest -s -v tests/e2e/online_serving/minimax_h3/test_minimax_h3.py -m 'advanced_model and cuda and diffusion' --run-level 'advanced_model'
|
||||
mirror_hardwares: h100_2
|
||||
+174
-326
@@ -1,9 +1,6 @@
|
||||
# CUDA CI L4 (`full_model`): nightly and release share this file.
|
||||
# Uploaded by Upload Nightly Pipeline: NIGHTLY=1, or PR labels (nightly-test / omni-test / ...).
|
||||
# Hardware: omit mirror_hardwares to compose from -m SKU + cards_n; or set a preset string to force.
|
||||
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_USE_FLASHINFER_MOE_FP16: 0
|
||||
HF_HUB_DOWNLOAD_TIMEOUT: 300
|
||||
HF_HUB_ETAG_TIMEOUT: 60
|
||||
|
||||
@@ -12,184 +9,146 @@ steps:
|
||||
- group: ":card_index_dividers: Omni Model Test"
|
||||
key: nightly-omni-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test"
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test" || build.pull_request.labels includes "omni-test"
|
||||
steps:
|
||||
|
||||
- label: ":full_moon: Omni · Function Test with H100 · Single-GPU"
|
||||
source_file_dependencies:
|
||||
- omni_qwen3_omni_function
|
||||
- omni_minicpmo_4_5_function
|
||||
- label: ":full_moon: Omni · Function Test with H100"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- pytest -sv tests/e2e --run-level "full_model" -m "full_model and H100 and B200 and omni and cards_1" --ignore=tests/e2e/accuracy --ignore=tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py --ignore=tests/e2e/online_serving/test_minicpmo_4_5_window.py
|
||||
|
||||
- label: ":full_moon: Omni · Function Test with H100 · 2-GPU"
|
||||
source_file_dependencies:
|
||||
- omni_qwen3_omni_function
|
||||
- omni_minicpmo_4_5_function
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -sv tests/e2e --run-level "full_model" -m "full_model and H100 and B200 and omni and cards_2" --ignore=tests/e2e/accuracy --ignore=tests/e2e/online_serving/test_qwen3_omni_multi_replicas.py
|
||||
- pytest -sv tests/e2e --run-level "full_model" -m "full_model and H100 and omni" --ignore=tests/e2e/accuracy --ignore=tests/e2e/online_serving/test_qwen3_omni_multi_replicas.py --ignore=tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 Duplex Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_function
|
||||
timeout_in_minutes: 120
|
||||
timeout_in_minutes: 50
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py -m "full_model and cuda and H100 and B200 and omni and cards_1" --run-level "full_model"
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_window.py -m "full_model and cuda and H100 and B200 and omni and cards_1" --run-level "full_model"
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py -m "full_model and cuda and H100 and omni" --run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: ":full_moon: Omni · Doc Test with L4 · 4-GPU"
|
||||
source_file_dependencies: omni_qwen2_5_omni_doc
|
||||
- label: ":full_moon: Omni · Doc Test with L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/examples/ -m "full_model and omni and L4 and B200 and cards_4" --run-level "full_model"
|
||||
- pytest -s -v tests/examples/ -m "full_model and omni and L4" --run-level "full_model"
|
||||
mirror_hardwares: l4_4
|
||||
|
||||
- label: ":full_moon: Omni · Doc Test with H100 · 2-GPU"
|
||||
source_file_dependencies: omni_qwen3_omni_doc
|
||||
- label: ":full_moon: Omni · Doc Test with H100"
|
||||
timeout_in_minutes: 90
|
||||
artifact_paths:
|
||||
- audio_*.wav
|
||||
commands:
|
||||
- pytest -s -v tests/examples/ -m "full_model and omni and H100 and B200 and cards_2" --run-level "full_model"
|
||||
- pytest -s -v tests/examples/ -m "full_model and omni and H100" --run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · Accuracy Test"
|
||||
source_file_dependencies: omni_qwen3_omni_accuracy
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc/*.json
|
||||
commands:
|
||||
- export SEED_TTS_WER_EVAL=1
|
||||
- export SEED_TTS_EVAL_DEVICE=cuda:1
|
||||
- pytest -s -v tests/e2e/accuracy/qwen3_omni/test_qwen3_omni.py -m "full_model and H100 and B200 and cards_2" --run-level full_model
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Accuracy Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_accuracy
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/e2e/accuracy/minicpmo_4_5/results/*.json
|
||||
commands:
|
||||
- export SEED_TTS_WER_EVAL=1
|
||||
- export SEED_TTS_EVAL_DEVICE=cuda:1
|
||||
- pytest -s -v tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py -m "full_model and H100 and B200 and cuda and (cards_1 or cards_2)" --run-level full_model
|
||||
|
||||
- label: ":full_moon: Omni · Omni-DuplexEval CI Guard"
|
||||
source_file_dependencies: omni_duplex_eval_ci
|
||||
timeout_in_minutes: 240
|
||||
env:
|
||||
VLLM_DUPLEX_EVAL_JUDGE_URL: "http://127.0.0.1:8001"
|
||||
commands:
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/e2e/accuracy/omni_duplex_eval/test_omni_duplex_eval_ci.py \
|
||||
-m "full_model and H100 and B200 and cuda and cards_2" \
|
||||
--run-level full_model \
|
||||
--basetemp /tmp/omni_duplex_ci
|
||||
pytest -s -v tests/e2e/accuracy/qwen3_omni/test_qwen3_omni.py -m "full_model" --run-level full_model
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/omni_duplex_eval/results/*.json"
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/qwen3_omni/results/qwen_omni_acc/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Accuracy Test"
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- export SEED_TTS_WER_EVAL=1
|
||||
- export SEED_TTS_EVAL_DEVICE=cuda:1
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/e2e/accuracy/minicpmo_4_5/test_minicpmo_4_5.py -m "full_model and cuda" --run-level full_model
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/minicpmo_4_5/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · No Async Chunk"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: nightly-omni-performance-no-async-chunk
|
||||
timeout_in_minutes: 300
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json -m "H100 and B200 and cards_2"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_no_async_chunk.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · Async Chunk"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: nightly-omni-performance-async-chunk
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json -m "H100 and B200 and full_model and cards_2"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Perf Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5.json -m "H100 and B200 and cards_1"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Duplex Seed-TTS Perf Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5-duplex-seed-tts
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json -m "H100 and B200 and cards_1"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_duplex_seed_tts.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: ":full_moon: Omni · Multi-Replica Startup Test with 4x H100"
|
||||
source_file_dependencies: omni_qwen3_omni_function
|
||||
timeout_in_minutes: 45
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_multi_replicas.py -m "full_model and H100 and B200 and cards_4" --run-level "core_model"
|
||||
|
||||
- group: ":card_index_dividers: Entrypoints Test"
|
||||
key: nightly-entrypoints-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test"
|
||||
steps:
|
||||
|
||||
- label: ":full_moon: Entrypoint Test with H100"
|
||||
source_file_dependencies: entrypoints_function
|
||||
# Merge collects advanced_model (5 duplex interruptions). Nightly
|
||||
# collects full_model (20 interruptions + BAGEL sleep).
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/entrypoints/ -m "full_model and cuda and H100 and B200 and cards_2" --run-level "full_model"
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni_multi_replicas.py -m "full_model" --run-level "core_model"
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- group: ":card_index_dividers: TTS Model Test"
|
||||
key: nightly-tts-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test"
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test" || build.pull_request.labels includes "tts-test"
|
||||
steps:
|
||||
|
||||
- label: ":full_moon: TTS · Function Test with L4"
|
||||
source_file_dependencies: tts_qwen3_tts_function
|
||||
timeout_in_minutes: 120
|
||||
env:
|
||||
VLLM_OMNI_FAILED_SPEECH_AUDIO_DIR: .
|
||||
artifact_paths:
|
||||
- failed_speech_*.wav
|
||||
- failed_speech_*.json
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- pytest -s -v tests/e2e/ -m "full_model and L4 and B200 and tts and cards_1" --run-level "full_model" --ignore=tests/e2e/accuracy
|
||||
- pytest -s -v tests/e2e/ -m "full_model and L4 and tts" --run-level "full_model" --ignore=tests/e2e/accuracy
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: ":full_moon: TTS · Perf Test · Single-GPU"
|
||||
source_file_dependencies: tts_qwen3_tts_perf
|
||||
key: nightly-tts-performance-single
|
||||
- label: ":full_moon: TTS · Perf Test"
|
||||
key: nightly-tts-performance
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json -m "H100 and B200 and cards_1"
|
||||
|
||||
- label: ":full_moon: TTS · Perf Test · 2-GPU"
|
||||
source_file_dependencies: tts_qwen3_tts_perf
|
||||
key: nightly-tts-performance-2gpu
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json -m "H100 and B200 and cards_2"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_tts.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- group: ":card_index_dividers: Tiny Model Tests [Multi-GPU]"
|
||||
key: nightly-tiny-model-test-group
|
||||
@@ -199,8 +158,7 @@ steps:
|
||||
build.pull_request.labels includes "nightly-test"
|
||||
steps:
|
||||
|
||||
- label: ":full_moon: Tiny Model Tests · Diffusion (Multi-GPU accelerations) · 2-GPU"
|
||||
source_file_dependencies: diffusion_tiny_model
|
||||
- label: ":full_moon: Tiny Model Tests · Diffusion (Multi-GPU accelerations)"
|
||||
timeout_in_minutes: 30
|
||||
# NOTE: Single-GPU (core_model) tiny model tests run on the ready label,
|
||||
# so nightly only adds the multi-GPU parallelism configurations.
|
||||
@@ -210,319 +168,216 @@ steps:
|
||||
# for the CI, since these tests should pass irrespective of whether or not we use
|
||||
# the full weights or not.
|
||||
commands:
|
||||
- pytest -sv tests/model_tests/diffusion -m 'full_model and L4 and B200 and cuda and cards_2' --run-level "core_model"
|
||||
|
||||
- label: ":full_moon: Tiny Model Tests · Diffusion (Multi-GPU accelerations) · 4-GPU"
|
||||
source_file_dependencies: diffusion_tiny_model
|
||||
timeout_in_minutes: 30
|
||||
# CFG+TP extra_test_groups request 2×2=4 devices (see get_required_device_count).
|
||||
commands:
|
||||
- pytest -sv tests/model_tests/diffusion -m 'full_model and L4 and B200 and cuda and cards_4' --run-level "core_model"
|
||||
- timeout 25m pytest -sv tests/model_tests/diffusion -m 'full_model and cuda' --run-level "core_model"
|
||||
mirror_hardwares: l4_4
|
||||
|
||||
- group: ":card_index_dividers: Diffusion X2I(&A&T) Model Test"
|
||||
key: nightly-diffusion-x2iat-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: >-
|
||||
build.env("NIGHTLY") == "1" ||
|
||||
build.pull_request.labels includes "nightly-test"
|
||||
build.pull_request.labels includes "nightly-test" ||
|
||||
build.pull_request.labels includes "diffusion-x2iat-test"
|
||||
steps:
|
||||
# Use explicit file shards so single-card cases do not reserve 2+ GPUs.
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Function Test with H100 · Single-GPU"
|
||||
source_file_dependencies: diffusion_qwen_image_function
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_qwen_image_expansion.py
|
||||
-m "full_model and diffusion and H100 and B200 and cards_1"
|
||||
-m "full_model and diffusion and H100 and not distributed_cuda"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Function Test with H100 · 2-GPU"
|
||||
source_file_dependencies: diffusion_qwen_image_function
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Function Test with H100 · Multi-GPU Qwen-Image"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_qwen_image_expansion.py
|
||||
-m "full_model and diffusion and H100 and B200 and cards_2"
|
||||
-m "full_model and diffusion and H100 and distributed_cuda"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Function Test with L4 · Single-GPU"
|
||||
source_file_dependencies: diffusion_qwen_image_function
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Function Test with L4"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- pytest -sv tests/e2e/offline_inference/test_qwen_image_autoround_w4a16_expansion.py -m "full_model and diffusion and L4 and B200 and cards_1" --run-level "full_model"
|
||||
|
||||
- pytest -sv tests/e2e/ -k "not test_wan and not hunyuan" -m "full_model and diffusion and L4" --run-level "full_model" --ignore=tests/e2e/accuracy
|
||||
mirror_hardwares: l4_4
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Doc Test"
|
||||
source_file_dependencies: diffusion_text_to_image_doc
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -s -v tests/examples/*/test_text_to_image.py -m "full_model and example and H100 and B200 and (cards_1 or cards_2)" --run-level "full_model"
|
||||
- pytest -s -v tests/examples/*/test_text_to_image.py -m "full_model and example and H100" --run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Accuracy Test"
|
||||
source_file_dependencies: diffusion_qwen_image_accuracy
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- export VLLM_HTTP_TIMEOUT_KEEP_ALIVE=120
|
||||
- pytest -s -v tests/e2e/accuracy/test_qwen_image.py -m "H100 and B200 and cards_1" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_diffusers_backend_similarity.py -k '2i_matches_diffusers' -m "full_model and H100 and B200 and cards_1" --run-level full_model
|
||||
|
||||
- label: ":full_moon: JoyAI-Image-Edit · Accuracy Test"
|
||||
source_file_dependencies: diffusion_joy_image_accuracy
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/accuracy/test_joy_image_edit.py -m "full_model and diffusion and H100 and B200 and cards_1" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_qwen_image.py --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_diffusers_backend_similarity.py -k '2i_matches_diffusers' -m full_model --run-level full_model
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: ":full_moon: HunyuanImage3-DIT · Accuracy Test"
|
||||
source_file_dependencies: diffusion_hunyuan_image3_accuracy
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- export HUNYUAN_IMAGE3_MODEL="tencent/HunyuanImage-3.0-Instruct"
|
||||
- export HUNYUAN_IMAGE3_DEVICES="0,1,2,3"
|
||||
- pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py tests/e2e/accuracy/test_hunyuan_image3_lora.py -m "full_model and H100 and B200 and cards_4" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py -m full_model --run-level full_model
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · Single-GPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-single
|
||||
- label: ":full_moon: HiDream-O1-Image · Accuracy Test"
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/accuracy/test_hidream_o1_image.py -m full_model --run-level full_model
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image"
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/diffusion_result_*.json
|
||||
- tests/dfx/perf/results/logs/*.log
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
|
||||
- export CACHE_DIT_VERSION=1.5.0
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · 4-GPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-4gpu
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/diffusion_result_*.json
|
||||
- tests/dfx/perf/results/logs/*.log
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export CACHE_DIT_VERSION=1.5.0
|
||||
# Do not pin DIFFUSION_ATTENTION_BACKEND: H100 auto-selects FLASH_ATTN;
|
||||
# B200 rejects explicit FLASH_ATTN without FA4 and uses CUDNN/TRTLLM.
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json -m "H100 and B200 and cards_4"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- group: ":card_index_dividers: Diffusion X2V Model Test"
|
||||
key: nightly-diffusion-x2v-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: >-
|
||||
build.env("NIGHTLY") == "1" ||
|
||||
build.pull_request.labels includes "nightly-test"
|
||||
build.pull_request.labels includes "nightly-test" ||
|
||||
build.pull_request.labels includes "diffusion-x2v-test"
|
||||
steps:
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 T2V Function Test · Single-GPU"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 T2V Function Test"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_wan22_expansion.py -k "t2v" -m "full_model and cuda and H100 and B200 and cards_1" --run-level "full_model"
|
||||
- pytest -s -v tests/e2e/online_serving/test_wan22_expansion.py -k "t2v" -m "full_model and cuda" --run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 T2V Function Test · 2-GPU"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 I2V/TI2V Function Test"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_wan22_expansion.py -k "t2v" -m "full_model and cuda and H100 and B200 and cards_2" --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 I2V/TI2V Function Test · Single-GPU"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_wan22_expansion.py -k "i2v" -m "full_model and cuda and H100 and B200 and cards_1" --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 I2V/TI2V Function Test · 2-GPU"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_wan22_expansion.py -k "i2v" -m "full_model and cuda and H100 and B200 and cards_2" --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Function Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_function
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m "full_model and H100 and B200 and cuda and (cards_1 or cards_2)" --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · LingBot Function Test"
|
||||
source_file_dependencies: diffusion_lingbot_function
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/e2e/online_serving/test_lingbot_video_expansion.py
|
||||
tests/e2e/online_serving/test_lingbot_video_moe.py
|
||||
tests/e2e/offline_inference/test_lingbot_world_v2.py
|
||||
-m "full_model and diffusion and H100 and B200 and cards_1"
|
||||
--run-level "full_model"
|
||||
- pytest -s -v tests/e2e/online_serving/test_wan22_expansion.py -k "i2v" -m "full_model and cuda" --run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Doc Test"
|
||||
source_file_dependencies: diffusion_image_to_video_doc
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -s -v tests/examples/offline_inference/test_image_to_video.py -m "full_model and example and H100 and B200 and cards_1" --run-level "full_model"
|
||||
- pytest -s -v tests/examples/offline_inference/test_image_to_video.py -m "full_model and example and H100" --run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 Autoround Function Test"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
- label: ":full_moon: Diffusion X2V · Other Function Test"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_wan22_autoround_w4a16_expansion.py -m "full_model and H100 and B200 and cuda and (cards_1 or cards_2)" --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 Device-Postprocess Equivalence Test"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- |
|
||||
VLLM_OMNI_DEVICE_POSTPROCESS_MODEL=Wan-AI/Wan2.2-TI2V-5B-Diffusers \
|
||||
pytest -s -v tests/e2e/offline_inference/test_device_postprocess_equivalence.py -m "full_model and H100 and B200 and cuda and cards_1" --run-level "full_model"
|
||||
- pytest -s -v tests/e2e/offline_inference/test_wan22_autoround_w4a16_expansion.py -m "full_model and cuda" --run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Wan2.2 Accuracy Test"
|
||||
source_file_dependencies: diffusion_wan22_accuracy
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/accuracy/wan22_i2v/test_wan22_i2v_video_similarity.py -m "full_model and H100 and B200 and (cards_1 or cards_2)" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_diffusers_backend_similarity.py -k '2v_matches_diffusers' -m "full_model and H100 and B200 and cards_1" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/wan22_i2v/test_wan22_i2v_video_similarity.py -m full_model --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/test_diffusers_backend_similarity.py -k '2v_matches_diffusers' -m full_model --run-level full_model
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · MiniMax H3 I2VA/Ref2VA Accuracy Test"
|
||||
source_file_dependencies: diffusion_minimax_h3_accuracy
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/accuracy/minimax_h3/test_minimax_h3_i2va_ref2va_similarity.py -m "full_model and H100 and B200 and cards_4" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/minimax_h3/test_minimax_h3_i2va_ref2va_similarity.py -m full_model --run-level full_model
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Accuracy Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_accuracy
|
||||
- label: ":full_moon: Diffusion X2V · Other Accuracy Test"
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/accuracy/hunyuanvideo15_t2v/test_hunyuanvideo15_t2v_video_similarity.py -m "full_model and H100 and B200 and (cards_1 or cards_2)" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/hunyuanvideo15_i2v/test_hunyuanvideo15_i2v_video_similarity.py -m "full_model and H100 and B200 and (cards_1 or cards_2)" --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/hunyuanvideo15_t2v/test_hunyuanvideo15_t2v_video_similarity.py -m full_model --run-level full_model
|
||||
- pytest -s -v tests/e2e/accuracy/hunyuanvideo15_i2v/test_hunyuanvideo15_i2v_video_similarity.py -m full_model --run-level full_model
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test · Single-GPU"
|
||||
source_file_dependencies: diffusion_wan22_perf
|
||||
key: nightly-diffusion-x2v-performance-single
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test"
|
||||
key: nightly-diffusion-x2v-performance
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test · 2-GPU"
|
||||
source_file_dependencies:
|
||||
- diffusion_wan22_perf
|
||||
- diffusion_cosmos3_perf
|
||||
key: nightly-diffusion-x2v-performance-2gpu
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json -m "H100 and B200 and cards_2"
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json
|
||||
EXIT1=$$?
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json -m "H100 and B200 and cards_2"
|
||||
EXIT2=$$?
|
||||
if [ $$EXIT1 -ne 0 ]; then exit $$EXIT1; fi
|
||||
exit $$EXIT2
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test · HunyuanVideo-1.5"
|
||||
source_file_dependencies: diffusion_hunyuan_video_perf
|
||||
key: nightly-diffusion-x2v-performance-hunyuanvideo15
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json -m "H100 and B200 and (cards_1 or cards_2)"
|
||||
EXIT1=$$?
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_i2v_vllm_omni.json -m "H100 and B200 and (cards_1 or cards_2)"
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_cosmos3_vllm_omni.json
|
||||
EXIT2=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
if [ $$EXIT1 -ne 0 ]; then exit $$EXIT1; fi
|
||||
exit $$EXIT2
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test · MiniMax-H3"
|
||||
source_file_dependencies: diffusion_minimax_h3_perf
|
||||
key: nightly-diffusion-x2v-performance-minimax-h3
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json -m "H100 and B200 and cards_4"
|
||||
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test · LingBot"
|
||||
source_file_dependencies: diffusion_lingbot_perf
|
||||
key: nightly-diffusion-x2v-performance-lingbot
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_lingbot_video_vllm_omni.json -m "H100 and B200 and cards_1"
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_ATTENTION_BACKEND=FLASH_ATTN
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py --test-config-file tests/dfx/perf/tests/test_minimax_h3_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- group: ":card_index_dividers: Diffusion Test"
|
||||
key: nightly-diffusion-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test"
|
||||
steps:
|
||||
- label: ":full_moon: Diffusion · Distributed Test with L4 · 2-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
- label: ":full_moon: Diffusion · Distributed Test with L4"
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/distributed/ -m 'full_model and cuda and L4 and B200 and cards_2' --run-level "full_model"
|
||||
- pytest -sv tests/diffusion/distributed/ -m 'full_model and cuda and L4' --run-level "full_model"
|
||||
mirror_hardwares: l4_4
|
||||
|
||||
- label: ":full_moon: Diffusion · Distributed Test with L4 · 3-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
- label: ":full_moon: Diffusion · Distributed Test with H100"
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/distributed/ -m 'full_model and cuda and L4 and B200 and cards_3' --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion · Distributed Test with L4 · 4-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/distributed/ -m 'full_model and cuda and L4 and B200 and cards_4' --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion · Distributed Test with H100 · 2-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/distributed/ -m 'full_model and cuda and H100 and B200 and cards_2' --run-level "full_model"
|
||||
|
||||
- label: ":full_moon: Diffusion · Distributed Test with H100 · 4-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/distributed/ -m 'full_model and cuda and H100 and B200 and cards_4' --run-level "full_model"
|
||||
- pytest -sv tests/diffusion/distributed/ -m 'full_model and cuda and H100' --run-level "full_model"
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- label: ":full_moon: Diffusion · Offloader Test"
|
||||
source_file_dependencies: diffusion_offloader
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/offloader -m "full_model and L4 and B200 and cuda and cards_1"
|
||||
- pytest -sv tests/diffusion/offloader -m "full_model and cuda"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: ":full_moon: Diffusion · Quantization Test with H100 · Single-GPU"
|
||||
source_file_dependencies: diffusion_quantization
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/quantization -m 'full_model and cuda and H100 and B200 and cards_1' --run-level "full_model"
|
||||
- pytest -sv tests/diffusion/quantization -m 'full_model and cuda and H100' --run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: ":full_moon: Diffusion · MiniMax H3 FP8 Quality Test with H100 · 2-GPU"
|
||||
source_file_dependencies: diffusion_minimax_h3_function
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pytest -sv tests/diffusion/models/minimax_h3/test_minimax_h3_quantization_quality.py -m 'full_model and cuda and H100 and B200 and cards_2' --run-level "full_model"
|
||||
- pytest -sv tests/diffusion/models/minimax_h3/test_minimax_h3_quantization_quality.py -m 'full_model and cuda and H100' --run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Diffusion · Quantization Test with L4"
|
||||
source_file_dependencies: diffusion_quantization
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- pip install "vllm-gguf-plugin==0.0.4"
|
||||
- pytest -sv tests/diffusion/quantization -m 'full_model and cuda and L4 and B200 and cards_1' --run-level "full_model"
|
||||
- pip install "vllm-gguf-plugin>=0.0.3"
|
||||
- pytest -sv tests/diffusion/quantization -m 'full_model and cuda and L4' --run-level "full_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
# NOTE: ":email: Nightly Collection & Email" is deprecated and will be removed soon.
|
||||
# Do not add new depends_on entries or extend this step.
|
||||
@@ -531,30 +386,23 @@ steps:
|
||||
depends_on:
|
||||
- nightly-omni-performance-no-async-chunk
|
||||
- nightly-omni-performance-async-chunk
|
||||
- nightly-tts-performance-single
|
||||
- nightly-tts-performance-2gpu
|
||||
- nightly-diffusion-x2iat-performance-qwen-image-single
|
||||
- nightly-diffusion-x2iat-performance-qwen-image-4gpu
|
||||
- nightly-diffusion-x2v-performance-single
|
||||
- nightly-diffusion-x2v-performance-2gpu
|
||||
- nightly-tts-performance
|
||||
- nightly-diffusion-x2iat-performance-qwen-image
|
||||
- nightly-diffusion-x2v-performance
|
||||
if: build.env("NIGHTLY") == "1" && build.env("EMAIL_DISTRIBUTION") == "1"
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.xlsx
|
||||
- tests/dfx/perf/results/*.html
|
||||
commands:
|
||||
- pip install openpyxl
|
||||
- export DEFAULT_INPUT_DIR=tests/dfx/perf/results
|
||||
- export DEFAULT_OUTPUT_DIR=tests/dfx/perf/results
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-tts-performance-single
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-tts-performance-2gpu
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-tts-performance
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-omni-performance-no-async-chunk
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-omni-performance-async-chunk
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image-single
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image-4gpu
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2v-performance-single
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2v-performance-2gpu
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2iat-performance-qwen-image
|
||||
- buildkite-agent artifact download "tests/dfx/perf/results/*.json" . --step nightly-diffusion-x2v-performance
|
||||
- python tools/nightly/generate_nightly_perf_excel.py
|
||||
- python tools/nightly/generate_nightly_perf_html.py
|
||||
- python tools/nightly/send_nightly_email.py --report-file "tests/dfx/perf/results/*.xlsx, tests/dfx/perf/results/*.html"
|
||||
- buildkite-agent artifact upload "tests/dfx/perf/results/*.xlsx"
|
||||
- buildkite-agent artifact upload "tests/dfx/perf/results/*.html"
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
+162
-113
@@ -1,8 +1,6 @@
|
||||
# CUDA CI L2 (`core_model` e2e) plus L1 CPU Simple Test (`core_model and cpu`).
|
||||
# Uploaded by Upload Ready Pipeline: PR label `ready`, or main+WEEKLY=1 with --e2e.
|
||||
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_USE_FLASHINFER_MOE_FP16: 0
|
||||
HF_HUB_DOWNLOAD_TIMEOUT: 300
|
||||
HF_HUB_ETAG_TIMEOUT: 60
|
||||
|
||||
@@ -16,27 +14,8 @@ steps:
|
||||
- "pytest -sv tests/model_executor -m 'core_model and cpu'"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Simple · CosyVoice3 Packed Flow Test"
|
||||
timeout_in_minutes: 15
|
||||
source_file_dependencies:
|
||||
- tests/model_executor/models/cosyvoice3/test_cosyvoice3_full_batch.py
|
||||
- vllm_omni/model_executor/models/cosyvoice3/
|
||||
commands:
|
||||
- "pytest -sv tests/model_executor/models/cosyvoice3/test_cosyvoice3_full_batch.py -k packed -m 'core_model and cuda' --run-level=core_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Simple · MRV2 FA3 Graph Bounds Test"
|
||||
timeout_in_minutes: 15
|
||||
source_file_dependencies:
|
||||
- tests/worker_v2/test_capture_query_bound_gpu.py
|
||||
- tests/worker_v2/test_decode_query_bound_gpu.py
|
||||
- vllm_omni/worker_v2/model_states/omni_model_state.py
|
||||
commands:
|
||||
- "pytest -sv tests/worker_v2/test_capture_query_bound_gpu.py tests/worker_v2/test_decode_query_bound_gpu.py -m 'core_model and cuda and H100' --run-level=core_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Simple · Diffusion Test"
|
||||
timeout_in_minutes: 30
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- "pytest -sv tests/diffusion -m 'core_model and cpu'"
|
||||
mirror_hardwares: l4_1
|
||||
@@ -57,12 +36,6 @@ steps:
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
|
||||
- label: "Diffusion · FlashAttention Test"
|
||||
timeout_in_minutes: 20
|
||||
commands:
|
||||
- timeout 15m pytest -s -v tests/diffusion/attention/ -m 'core_model and cuda and L4 and cards_1' --run-level "core_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · Offloader Test"
|
||||
commands:
|
||||
- timeout 40m pytest -s -v tests/diffusion/offloader -m "core_model and cuda" --run-level "core_model"
|
||||
@@ -78,24 +51,17 @@ steps:
|
||||
- timeout 15m pytest -sv tests/diffusion/batching -m 'core_model and cuda' --run-level "core_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · Distributed Test · Single-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
- label: "Diffusion · Distributed & Attention Test"
|
||||
source_file_dependencies:
|
||||
- tests/diffusion/distributed/
|
||||
- tests/diffusion/attention/test_attention_sp.py
|
||||
- vllm_omni/diffusion/distributed/
|
||||
- vllm_omni/diffusion/attention/
|
||||
- vllm_omni/diffusion/data.py
|
||||
- vllm_omni/diffusion/models/
|
||||
commands:
|
||||
- timeout 20m pytest -sv tests/diffusion/distributed -m 'core_model and cuda and L4 and cards_1' --run-level "core_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · Distributed Test · 2-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
commands:
|
||||
- timeout 20m pytest -sv tests/diffusion/distributed -m 'core_model and cuda and L4 and cards_2' --run-level "core_model"
|
||||
- timeout 10m pytest -sv tests/diffusion/attention/test_ulysses_uaa.py -m 'core_model and cuda and L4 and cards_2' --run-level "core_model"
|
||||
mirror_hardwares: l4_2
|
||||
|
||||
- label: "Diffusion · Distributed & Attention Test · 4-GPU"
|
||||
source_file_dependencies: diffusion_distributed_attention
|
||||
commands:
|
||||
- timeout 20m pytest -sv tests/diffusion/distributed -m 'core_model and cuda and L4 and cards_4' --run-level "core_model"
|
||||
- timeout 20m pytest -sv tests/diffusion/attention/test_attention_sp.py -m 'core_model and cuda and L4 and cards_4' --run-level "core_model"
|
||||
- timeout 20m pytest -sv tests/diffusion/distributed -m 'core_model and cuda' --run-level "core_model"
|
||||
- timeout 20m pytest -sv tests/diffusion/attention/test_attention_sp.py -m 'core_model and cuda' --run-level "core_model"
|
||||
mirror_hardwares: l4_4
|
||||
|
||||
- label: "Diffusion · Model Test"
|
||||
@@ -105,7 +71,7 @@ steps:
|
||||
|
||||
- label: "Diffusion · Other Test"
|
||||
commands:
|
||||
- timeout 15m pytest -sv tests/diffusion/*.py tests/diffusion/ar_diffusion tests/diffusion/layers/test_adalayernorm_*.py -m 'core_model and cuda' --run-level "core_model"
|
||||
- timeout 15m pytest -sv tests/diffusion/*.py -m 'core_model and cuda' --run-level "core_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- group: ":card_index_dividers: Tiny Model Tests [Single GPU]"
|
||||
@@ -122,7 +88,7 @@ steps:
|
||||
commands:
|
||||
- |
|
||||
timeout 15m bash -c "
|
||||
pytest -sv tests/engine/ tests/model_executor/ tests/worker/test_batched_omni_output.py tests/worker/test_gpu_generation_async_output.py tests/worker_v2/ -m 'core_model and cuda' --run-level 'core_model'
|
||||
pytest -sv tests/engine/ tests/model_executor/ -m 'core_model and cuda' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
@@ -137,18 +103,11 @@ steps:
|
||||
|
||||
- label: "Entrypoints Test"
|
||||
depends_on: upload-ready-pipeline
|
||||
# Workaround https://github.com/vllm-project/vllm-omni/issues/8016: shutdown()
|
||||
# leaves inline UniProc DiffusionWorker VRAM mapped in the pytest parent.
|
||||
# Isolate that leftover case in its own process so the driver drops the
|
||||
# mapping; the rest stays a directory sweep with --deselect. Revert to a
|
||||
# single sweep after the product shutdown fix.
|
||||
commands:
|
||||
- |
|
||||
timeout 30m bash -c "
|
||||
set -eu
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
pytest -sv tests/entrypoints/test_async_omni.py::test_diffusion_generate_request_id -m 'core_model and cuda' --run-level 'core_model'
|
||||
pytest -sv tests/entrypoints/ -m 'core_model and cuda' --run-level 'core_model' --deselect tests/entrypoints/test_async_omni.py::test_diffusion_generate_request_id
|
||||
pytest -sv tests/entrypoints/ -m 'core_model and cuda' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
@@ -156,14 +115,53 @@ steps:
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
- label: "Custom Pipeline Test"
|
||||
source_file_dependencies: custom_pipeline
|
||||
source_file_dependencies:
|
||||
- tests/e2e/features/custom_pipeline/
|
||||
- tests/e2e/features/helpers/custom_pipeline.py
|
||||
- vllm_omni/diffusion/
|
||||
- vllm_omni/engine/
|
||||
- vllm_omni/entrypoints/
|
||||
- vllm_omni/inputs/
|
||||
- vllm_omni/outputs/
|
||||
commands:
|
||||
- timeout 20m pytest -s -v tests/e2e/features/custom_pipeline/ -m "core_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
agents:
|
||||
queue: "gpu_1_queue"
|
||||
plugins:
|
||||
- docker#v5.2.0:
|
||||
image: public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
|
||||
always-pull: true
|
||||
propagate-environment: true
|
||||
# vLLM 0.27 added a pre-flight capacity check in
|
||||
# distributed/device_communicators/shm_broadcast.py. Without this
|
||||
# the container gets Docker's 64 MiB default and every test in
|
||||
# this step dies at engine start with
|
||||
# RuntimeError: Insufficient space in /dev/shm:
|
||||
# 240 MiB required, 64 MiB available
|
||||
# (9 failed in 8.7s on build #2949). vLLM 0.26 had no such check
|
||||
# and allocated regardless, so this step passed on main while
|
||||
# being under-provisioned the whole time — 0.27 turned a silent
|
||||
# risk into an explicit error.
|
||||
#
|
||||
# This is the only inline docker plugin block in .buildkite/cuda/;
|
||||
# every other step inherits common/ci_mirror_hardwares.yml, which
|
||||
# already sets 8gb. Matching that rather than inventing a value.
|
||||
shm-size: "8gb"
|
||||
environment:
|
||||
- "HF_HOME=/fsx/hf_cache"
|
||||
volumes:
|
||||
- "/fsx/hf_cache:/fsx/hf_cache"
|
||||
|
||||
- label: "RLHF · VeRL-Omni E2E Test"
|
||||
source_file_dependencies: rlhf_verl_omni
|
||||
source_file_dependencies:
|
||||
- tests/e2e/features/rlhf_test/test_verl_omni_e2e.py
|
||||
- tests/e2e/features/helpers/verl_omni_server.py
|
||||
- tests/e2e/features/helpers/custom_pipeline.py
|
||||
- vllm_omni/diffusion/
|
||||
- vllm_omni/engine/
|
||||
- vllm_omni/entrypoints/
|
||||
- vllm_omni/inputs/
|
||||
- vllm_omni/outputs/
|
||||
commands:
|
||||
- |
|
||||
timeout 20m bash -c "
|
||||
@@ -172,7 +170,17 @@ steps:
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Omni · Qwen3-Omni Test"
|
||||
source_file_dependencies: omni_qwen3_omni_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_omni.py
|
||||
- tests/helpers/runtime.py
|
||||
- vllm_omni/entrypoints/openai/serving_chat.py
|
||||
- vllm_omni/model_executor/models/qwen3_omni/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/qwen2_5_omni/qwen2_5_omni_thinker.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/cuda_graph_decoder_wrapper.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_omni.py
|
||||
- vllm_omni/deploy/qwen3_omni_moe.yaml
|
||||
commands:
|
||||
- |
|
||||
timeout 20m bash -c "
|
||||
@@ -181,7 +189,14 @@ steps:
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Offline Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/offline_inference/test_minicpmo_4_5.py
|
||||
- vllm_omni/model_executor/models/minicpmo_4_5/
|
||||
- vllm_omni/model_executor/models/step_audio2/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/utils.py
|
||||
- vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
|
||||
- vllm_omni/deploy/minicpmo_4_5.yaml
|
||||
commands:
|
||||
- |
|
||||
timeout 20m bash -c "
|
||||
@@ -190,7 +205,14 @@ steps:
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Online Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5.py
|
||||
- vllm_omni/model_executor/models/minicpmo_4_5/
|
||||
- vllm_omni/model_executor/models/step_audio2/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/utils.py
|
||||
- vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
|
||||
- vllm_omni/deploy/minicpmo_4_5.yaml
|
||||
commands:
|
||||
- |
|
||||
timeout 20m bash -c "
|
||||
@@ -199,34 +221,61 @@ steps:
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Duplex Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_function
|
||||
source_file_dependencies:
|
||||
- tests/assets/minicpmo_4_5/response_required_16k.wav
|
||||
- tests/assets/minicpmo_4_5/soft_interrupt_16k.wav
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex.py
|
||||
- tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
- tests/e2e/online_serving/test_minicpmo_realtime_duplex_drivers.py
|
||||
- tests/e2e/online_serving/helpers/minicpmo_realtime_duplex_scenarios.py
|
||||
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_multi_session.py
|
||||
- tests/e2e/online_serving/run_minicpmo_realtime_duplex_soft_interrupt.py
|
||||
- vllm_omni/deploy/minicpmo_4_5.yaml
|
||||
- tests/e2e/online_serving/helpers/minicpmo_4_5_duplex.py
|
||||
- vllm_omni/deploy/minicpmo_4_5_duplex.yaml
|
||||
- vllm_omni/experimental/fullduplex/
|
||||
- vllm_omni/model_executor/models/minicpmo_4_5/
|
||||
- vllm_omni/model_executor/models/step_audio2/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/code2wav_core/
|
||||
- vllm_omni/model_executor/models/cosyvoice3/utils.py
|
||||
- vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
|
||||
- vllm_omni/core/sched/
|
||||
- vllm_omni/outputs/multimodal_accumulation.py
|
||||
- vllm_omni/engine/stage_init_utils.py
|
||||
- vllm_omni/engine/async_omni_engine.py
|
||||
- vllm_omni/engine/orchestrator.py
|
||||
- vllm_omni/entrypoints/cli/serve.py
|
||||
- vllm_omni/entrypoints/openai/api_server.py
|
||||
- vllm_omni/entrypoints/openai/duplex_capability.py
|
||||
- vllm_omni/config/stage_config.py
|
||||
commands:
|
||||
- |
|
||||
timeout 35m bash -c "
|
||||
pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py tests/e2e/online_serving/test_duplex_client_live.py -m 'core_model and cuda' --run-level 'core_model'
|
||||
timeout 20m bash -c "
|
||||
pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py -m 'core_model and cuda' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Perf Test"
|
||||
timeout_in_minutes: 60
|
||||
source_file_dependencies: omni_minicpmo_4_5_perf
|
||||
source_file_dependencies:
|
||||
- tests/dfx/perf/tests/test_minicpmo_4_5_ready.json
|
||||
- vllm_omni/deploy/minicpmo_4_5.yaml
|
||||
- vllm_omni/model_executor/models/minicpmo_4_5/
|
||||
- vllm_omni/model_executor/stage_input_processors/minicpmo_4_5_omni.py
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_ready.json -m 'core_model and cuda'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "TTS · Higgs v3 Full Response Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_higgs_audio_v3.py
|
||||
- vllm_omni/model_executor/models/higgs_audio_v3/
|
||||
- vllm_omni/model_executor/stage_input_processors/higgs_audio_v3.py
|
||||
- vllm_omni/deploy/higgs_multimodal_qwen3.yaml
|
||||
commands:
|
||||
- timeout 20m pytest -s -v tests/e2e/online_serving/test_higgs_audio_v3.py::test_full_response_v1_preserves_audio -m 'core_model and cuda' --run-level 'core_model'
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "TTS · Qwen3-TTS CustomVoice Test"
|
||||
source_file_dependencies: tts_qwen3_tts_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_tts_customvoice.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/common/alias_free_activation.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
commands:
|
||||
- |
|
||||
timeout 20m bash -c "
|
||||
@@ -235,18 +284,45 @@ steps:
|
||||
"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Qwen3-TTS Base Test"
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen3_tts_base.py
|
||||
- vllm_omni/model_executor/models/qwen3_tts/
|
||||
- vllm_omni/model_executor/models/common/qwen3_code_predictor.py
|
||||
- vllm_omni/model_executor/models/common/snake_activation.py
|
||||
- vllm_omni/model_executor/models/common/alias_free_activation.py
|
||||
- vllm_omni/model_executor/stage_input_processors/qwen3_tts.py
|
||||
- vllm_omni/deploy/qwen3_tts.yaml
|
||||
commands:
|
||||
- |
|
||||
timeout 20m bash -c "
|
||||
export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
|
||||
export VLLM_OMNI_USE_V2_RUNNER=1
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen3_tts_base.py -m 'core_model' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · Wan22 Test"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_wan22_t2v.py
|
||||
- vllm_omni/diffusion/models/wan2_2/
|
||||
- vllm_omni/diffusion/models/dmd2/
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_wan22_t2v.py -m 'core_model and cuda' --run-level 'core_model'
|
||||
pytest -s -v tests/e2e/online_serving/test_wan22_t2v.py -m 'core_model' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Cosmos3 Test"
|
||||
source_file_dependencies: diffusion_cosmos3_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_cosmos3.py
|
||||
- vllm_omni/diffusion/models/cosmos3/
|
||||
- vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_wan.py
|
||||
- vllm_omni/experimental/world_models/
|
||||
- vllm_omni/model_extras/cosmos3.py
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
@@ -255,19 +331,12 @@ steps:
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · MiniMax-H3 DLO DP2 T2VA Test"
|
||||
timeout_in_minutes: 90
|
||||
source_file_dependencies: diffusion_minimax_h3_function
|
||||
commands:
|
||||
- |
|
||||
timeout 80m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_minimax_h3_dlo_dp2_t2va.py -m 'core_model and cuda and diffusion' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · Qwen Image Test"
|
||||
source_file_dependencies: diffusion_qwen_image_function
|
||||
source_file_dependencies:
|
||||
- tests/e2e/online_serving/test_qwen_image.py
|
||||
- vllm_omni/diffusion/models/qwen_image/
|
||||
- vllm_omni/diffusion/models/dmd2/
|
||||
- vllm_omni/diffusion/distributed/autoencoders/autoencoder_kl_qwenimage.py
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
@@ -275,23 +344,3 @@ steps:
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · HunyuanVideo-1.5 Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m 'core_model and diffusion' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · LingBot-Video Test"
|
||||
source_file_dependencies: diffusion_lingbot_function
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_lingbot_video.py -m 'core_model and cuda' --run-level 'core_model'
|
||||
"
|
||||
mirror_hardwares: h100_1
|
||||
+34
-501
@@ -1,8 +1,6 @@
|
||||
# CUDA CI L5 (`slow` / reliability / extra coverage).
|
||||
# Uploaded by Upload Weekly Pipeline: WEEKLY=1 / NON_CRITICAL=1, or PR label `weekly-test`.
|
||||
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_USE_FLASHINFER_MOE_FP16: 0
|
||||
HF_HUB_DOWNLOAD_TIMEOUT: 300
|
||||
HF_HUB_ETAG_TIMEOUT: 60
|
||||
|
||||
@@ -12,8 +10,6 @@ steps:
|
||||
depends_on: upload-weekly-pipeline
|
||||
if: build.env("WEEKLY") == "1"
|
||||
timeout_in_minutes: 60
|
||||
artifact_paths:
|
||||
- coverage-core-model-cpu-*.xml.gz
|
||||
commands:
|
||||
- |
|
||||
set +e
|
||||
@@ -21,6 +17,7 @@ steps:
|
||||
pytest -sv tests/ -m 'core_model and cpu' --cov=vllm_omni --cov-report=term-missing:skip-covered --cov-report=xml:$${REPORT}
|
||||
EXIT=$$?
|
||||
gzip -9 -f "$${REPORT}"
|
||||
buildkite-agent artifact upload "$${REPORT}.gz"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
@@ -30,42 +27,30 @@ steps:
|
||||
steps:
|
||||
|
||||
- label: "Reliability Test · qwen3-omni"
|
||||
source_file_dependencies: omni_qwen3_omni_reliability
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/dfx/reliability/test_reliability_qwen3_omni.py -m "slow"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Reliability Test · wan22"
|
||||
source_file_dependencies: diffusion_wan22_reliability
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/dfx/reliability/test_reliability_wan22.py -m "slow"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Reliability Test · hunyuan-image"
|
||||
source_file_dependencies: diffusion_hunyuan_image3_reliability
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/dfx/reliability/test_reliability_hunyuan_image.py -m "slow"
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- label: "Reliability Test · Invalid parameters · H100 · Single-GPU"
|
||||
source_file_dependencies: reliability_invalid_param_h100
|
||||
timeout_in_minutes: 90
|
||||
- label: "Reliability Test · Invalid parameters · H100"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and H100 and cards_1"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Reliability Test · Invalid parameters · H100 · 2-GPU"
|
||||
source_file_dependencies: reliability_invalid_param_h100
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and H100 and cards_2"
|
||||
- pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and H100"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Reliability Test · Invalid parameters · L4"
|
||||
source_file_dependencies: reliability_invalid_param_l4
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and L4"
|
||||
@@ -77,35 +62,29 @@ steps:
|
||||
steps:
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · vLLM Text"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: weekly-omni-performance-vllm-text
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_vllm_text.json
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · Async Chunk · Random"
|
||||
key: weekly-omni-performance-async-chunk-random
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_async_chunk.json -m "H100 and slow"
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_vllm_text.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: ":full_moon: Omni · Perf Test · Multi-Replica"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: weekly-omni-performance-multi-replicas
|
||||
timeout_in_minutes: 180
|
||||
artifact_paths:
|
||||
- tests/dfx/perf/results/*.json
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_qwen3_omni_multi_replicas.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
mirror_hardwares: h100_3
|
||||
|
||||
- group: ":card_index_dividers: E2E Tests"
|
||||
@@ -113,161 +92,37 @@ steps:
|
||||
if: build.env("NON_CRITICAL") == "1"
|
||||
steps:
|
||||
|
||||
# Online Qwen2.5 expansion is 2-card; offline expansion + AutoRound are 4-card.
|
||||
- label: "Omni · L4 · 2-GPU"
|
||||
- label: "Omni · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen2_5_omni_expansion.py -m "slow and L4 and omni and cards_2" --run-level "core_model"
|
||||
mirror_hardwares: l4_2
|
||||
|
||||
- label: "Omni · L4 · 4-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- pytest -s -v tests/e2e/offline_inference/test_qwen2_5_omni_expansion.py -m "slow and L4 and omni and cards_4" --run-level "core_model"
|
||||
- pytest -s -v tests/e2e/*/test_qwen2_5_omni_expansion.py --run-level "core_model"
|
||||
- >-
|
||||
pytest -s -v tests/e2e/
|
||||
--ignore=tests/e2e/online_serving/test_qwen2_5_omni_expansion.py
|
||||
--ignore=tests/e2e/offline_inference/test_qwen2_5_omni_expansion.py
|
||||
-m "slow and L4 and omni and cards_4"
|
||||
-m "slow and L4 and omni"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: l4_4
|
||||
|
||||
- label: "Omni · H100 · Single-GPU"
|
||||
- label: "Omni · H100"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/ -m "slow and H100 and omni and cards_1"
|
||||
tests/e2e/ -m "slow and H100 and omni"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Omni · H100 · 4-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/ -m "slow and H100 and omni and cards_4"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
- label: "TTS · Moss · L4"
|
||||
timeout_in_minutes: 90
|
||||
- label: "TTS · L4"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_moss_tts_expansion.py \
|
||||
tests/e2e/offline_inference/test_moss_tts_nano_expansion.py \
|
||||
tests/e2e/offline_inference/test_moss_tts_realtime.py \
|
||||
tests/e2e/offline_inference/test_moss_tts_realtime_expansion.py \
|
||||
tests/e2e/online_serving/test_moss_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_moss_tts_nano_expansion.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · VoxCPM2 · Voxtral · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_voxcpm2_tts.py \
|
||||
tests/e2e/online_serving/test_voxcpm2_tts.py \
|
||||
tests/e2e/online_serving/test_voxcpm2_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_voxcpm2_tts_lora.py \
|
||||
tests/e2e/offline_inference/test_voxtral_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_voxtral_tts_expansion.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Higgs · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_higgs_audio_v2_expansion.py \
|
||||
tests/e2e/online_serving/test_higgs_audio_v2_expansion.py \
|
||||
tests/e2e/online_serving/test_higgs_audio_v3.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · OmniVoice · Step-Audio · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_omnivoice_expansion.py \
|
||||
tests/e2e/online_serving/test_omnivoice_expansion.py \
|
||||
tests/e2e/online_serving/test_omnivoice_parity.py \
|
||||
tests/e2e/offline_inference/test_step_audio2_expansion.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Ming · GLM · IndexTTS · Breeze · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_ming_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_ming_tts_expansion.py \
|
||||
tests/e2e/offline_inference/test_glm_tts_expansion.py \
|
||||
tests/e2e/online_serving/test_glm_tts_expansion.py \
|
||||
tests/e2e/offline_inference/test_indextts2_expansion.py \
|
||||
tests/e2e/online_serving/test_indextts2_expansion.py \
|
||||
tests/e2e/online_serving/test_breeze_tts_2.py
|
||||
do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "TTS · Audex · CosyVoice · Qwen3-TTS · L4"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- export VLLM_USE_DEEP_GEMM="0"
|
||||
- export VLLM_MOE_USE_DEEP_GEMM="0"
|
||||
- |
|
||||
failed=0
|
||||
for f in \
|
||||
tests/e2e/offline_inference/test_audex_tts.py \
|
||||
tests/e2e/offline_inference/test_audex_s2t.py \
|
||||
tests/e2e/offline_inference/test_audex_tta.py \
|
||||
tests/e2e/offline_inference/test_cosyvoice3_expansion.py \
|
||||
tests/e2e/online_serving/test_qwen3_tts_word_timestamps.py
|
||||
do
|
||||
for f in $(pytest --collect-only -q tests/e2e/ -m "slow and L4 and tts" --run-level "full_model" | grep '\.py::' | sed -E 's|::.*||' | sort -u); do
|
||||
pytest -s -v "$$f" -m "slow and L4 and tts" --run-level "full_model" || failed=1
|
||||
done
|
||||
exit $$failed
|
||||
@@ -290,362 +145,40 @@ steps:
|
||||
# from downgrading transformers; see
|
||||
# examples/offline_inference/text_to_speech/README.md.
|
||||
- label: "TTS · Gepard-1.0 · L4"
|
||||
timeout_in_minutes: 90
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
|
||||
- printf '%s\n' "transformers>=5.5.3" "huggingface-hub>=1.0" > /tmp/gepard-overrides.txt
|
||||
- uv pip install --python "$(python3 -c 'import sys; print(sys.executable)')" "nemo_toolkit[tts]==2.7.3" --overrides /tmp/gepard-overrides.txt
|
||||
- >-
|
||||
pytest -s -v tests/e2e/offline_inference/test_gepard_tts.py
|
||||
tests/e2e/online_serving/test_gepard_tts.py
|
||||
-m "tts and cuda"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · L4 · Single-GPU"
|
||||
- label: "Diffusion · L4"
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/ -m "slow and diffusion and L4 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: l4_1
|
||||
|
||||
- label: "Diffusion · L4 · 2-GPU"
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/ -m "slow and diffusion and L4 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: l4_2
|
||||
|
||||
- label: "Diffusion · L4 · 4-GPU"
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/ -m "slow and diffusion and L4 and cards_4"
|
||||
tests/e2e/ -m "slow and diffusion and L4"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: l4_4
|
||||
|
||||
# Split the old shared H100 Single-GPU sweep: one pytest process collected
|
||||
# every slow+cards_1 diffusion file and exceeded 150 minutes.
|
||||
- label: ":robot_face: Robot Policy OpenPI · π0.5 · H100 · Single-GPU"
|
||||
timeout_in_minutes: 120
|
||||
- label: "Diffusion · H100 · Single-GPU"
|
||||
timeout_in_minutes: 150
|
||||
commands:
|
||||
- pip install --no-deps "openpi-client @ git+https://github.com/Physical-Intelligence/openpi.git@215abfb217dbac7d5f1273282331b9b1866c0479#subdirectory=packages/openpi-client"
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_pi05_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
tests/e2e/ -m "slow and diffusion and H100 and not distributed_cuda"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Qwen-Image · H100 · Single-GPU"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_qwen_image_edit.py
|
||||
tests/e2e/online_serving/test_qwen_image_edit_expansion.py
|
||||
tests/e2e/online_serving/test_qwen_image_layered.py
|
||||
tests/e2e/online_serving/test_qwen_image_layered_expansion.py
|
||||
tests/e2e/accuracy/test_qwen_image_edit.py
|
||||
tests/e2e/accuracy/test_qwen_image_layered.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Wan2.1 VACE · H100 · Single-GPU"
|
||||
- label: "Diffusion · H100 · Multi-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_wan_2_1_vace_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
tests/e2e/ -m "slow and diffusion and H100 and distributed_cuda"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · LongCat · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_longcat_image_expansion.py
|
||||
tests/e2e/online_serving/test_longcat_image_edit_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Bagel · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_bagel.py
|
||||
tests/e2e/online_serving/test_bagel_expansion.py
|
||||
tests/e2e/offline_inference/test_bagel.py
|
||||
tests/e2e/offline_inference/test_bagel_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Boogu · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_boogu_image.py
|
||||
tests/e2e/online_serving/test_boogu_image_edit.py
|
||||
tests/e2e/online_serving/test_boogu_image_expansion.py
|
||||
tests/e2e/online_serving/test_boogu_image_edit_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Flux · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_flux_2_dev_expansion.py
|
||||
tests/e2e/offline_inference/test_flux1_schnell_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · LTX · H100 · Single-GPU"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_ltx.py
|
||||
tests/e2e/online_serving/test_ltx25.py
|
||||
tests/e2e/accuracy/ltx/test_ltx_official_similarity.py
|
||||
tests/e2e/accuracy/ltx/test_ltx25_official_similarity.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Krea2 · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_krea2_expansion.py
|
||||
tests/e2e/offline_inference/test_krea2_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · HiDream-I1 · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_hidream_i1_full_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · SANA-Video · H100 · Single-GPU"
|
||||
source_file_dependencies: diffusion_sana_video_function
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- export SANA_VIDEO_GOLDEN_BASE_URL=https://vllm-public-assets.s3.us-west-2.amazonaws.com/omni-assets/sana-video/v2
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_sana_video_expansion.py
|
||||
tests/e2e/accuracy/sana_video/
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · SenseNova-U1 · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/offline_inference/test_sensenova_u1_expansion.py
|
||||
tests/e2e/offline_inference/test_sensenova_u1_text2img_expansion.py
|
||||
tests/e2e/offline_inference/test_sensenova_u1_img2img_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · MammothModa2 · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/offline_inference/test_mammoth_moda2_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
- label: "Diffusion · Helios · H100 · Single-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/accuracy/test_video_streaming_output_similarity.py
|
||||
-m "slow and diffusion and H100 and cards_1"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_1
|
||||
|
||||
# Split the old shared H100 2-GPU sweep: one pytest process collected every
|
||||
# slow+cards_2 diffusion file and exceeded 90 minutes.
|
||||
- label: "Diffusion · Qwen-Image · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_qwen_image_edit_expansion.py
|
||||
tests/e2e/online_serving/test_qwen_image_layered_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · Wan2.1 VACE · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_wan_2_1_vace_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · LongCat · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_longcat_image_expansion.py
|
||||
tests/e2e/online_serving/test_longcat_image_edit_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · Bagel · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_bagel_expansion.py
|
||||
tests/e2e/offline_inference/test_bagel_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · Boogu · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_boogu_image_expansion.py
|
||||
tests/e2e/online_serving/test_boogu_image_edit_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · Flux · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_flux_2_dev_expansion.py
|
||||
tests/e2e/offline_inference/test_flux1_kontext_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · GLM-Image · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_glm_image_expansion.py
|
||||
tests/e2e/offline_inference/test_glm_image_autoround_w4a16_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · LTX · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_ltx.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · Krea2 · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/offline_inference/test_krea2_expansion.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · DreamZero · H100 · 2-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_dreamzero_expansion.py
|
||||
tests/e2e/accuracy/test_dreamzero.py
|
||||
-m "slow and diffusion and H100 and cards_2"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
- label: "Diffusion · H100 · 3-GPU"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/ -m "slow and diffusion and H100 and cards_3"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_3
|
||||
|
||||
- label: "Diffusion · H100 · 4-GPU"
|
||||
timeout_in_minutes: 90
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/ -m "slow and diffusion and H100 and cards_4"
|
||||
--run-level "full_model"
|
||||
--ignore=tests/e2e/online_serving/test_magi2.py
|
||||
mirror_hardwares: h100_4
|
||||
|
||||
# Dedicated: HiDream-O1 accuracy needs a long TP2 run; the shared H100
|
||||
# 2-GPU sweep is only 90 minutes.
|
||||
- label: "Diffusion · HiDream-O1-Image Accuracy · H100 · 2-GPU"
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/accuracy/test_hidream_o1_image.py -m "slow and diffusion and cuda and H100 and cards_2" --run-level "full_model"
|
||||
mirror_hardwares: h100_2
|
||||
|
||||
# Dedicated: Boogu CFG-parallel size 3 needs 3 GPUs; weekly has no H100 3-GPU sweep.
|
||||
- label: "Diffusion · Boogu Edit CFG · H100 · 3-GPU"
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- >-
|
||||
pytest -sv
|
||||
tests/e2e/online_serving/test_boogu_image_edit_expansion.py
|
||||
-k "test_double_guidance_cfg_parallel and cfg_parallel_3_full"
|
||||
-m "slow and diffusion and cuda and H100 and cards_3"
|
||||
--run-level "full_model"
|
||||
mirror_hardwares: h100_3
|
||||
|
||||
# Dedicated: MAGI-2 Preview serving smoke needs ~30 min server init; the
|
||||
# shared H100 4-GPU sweep is only 90 minutes.
|
||||
- label: "Diffusion · MAGI-2 Preview Serving Smoke · H100 · 4-GPU"
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_magi2.py -m "slow and diffusion and cuda and H100 and cards_4" --run-level "full_model"
|
||||
mirror_hardwares: h100_4
|
||||
@@ -10,7 +10,7 @@ steps:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
# Buildkite will automatically replace this with the actual commit hash
|
||||
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}"
|
||||
VLLM_VERSION: "v0.31.0"
|
||||
VLLM_VERSION: "v0.27.0"
|
||||
priority: 100
|
||||
timeout_in_minutes: 60
|
||||
soft_fail: false
|
||||
@@ -5,12 +5,7 @@ set -ex
|
||||
|
||||
omni_source_dir=$(git rev-parse --show-toplevel)
|
||||
|
||||
: "${VLLM_VERSION:?VLLM_VERSION must be set}"
|
||||
# Official upstream XPU image. It is published from vllm's own
|
||||
# docker/Dockerfile.xpu, which is exactly what build_vllm_base() clones and
|
||||
# builds below, so pulling it is equivalent to building the fallback base.
|
||||
upstream_base_image="vllm/vllm-openai-xpu:${VLLM_VERSION}"
|
||||
local_base_image="xpu/vllm-omni-ci-base:${VLLM_VERSION}"
|
||||
base_image_name="xpu/vllm-omni-ci-base:${VLLM_VERSION:?VLLM_VERSION must be set}"
|
||||
image_name="xpu/vllm-omni-ci:${BUILDKITE_COMMIT:?BUILDKITE_COMMIT must be set}"
|
||||
container_name="xpu_${BUILDKITE_COMMIT}_$(
|
||||
tr -dc A-Za-z0-9 </dev/urandom | head -c 10
|
||||
@@ -54,36 +49,9 @@ build_vllm_base() (
|
||||
"${vllm_source_dir}"
|
||||
)
|
||||
|
||||
# Resolve the vLLM base image, in priority order:
|
||||
# 1. the published upstream image for VLLM_VERSION, pulled fresh (so moving
|
||||
# tags like nightly are refreshed rather than served stale from disk);
|
||||
# 2. a local copy of that same image, when the registry is unreachable;
|
||||
# 3. VLLM_BASE, if that image is already on disk;
|
||||
# 4. otherwise build VLLM_BASE from the matching vLLM source tag (~30min).
|
||||
# VLLM_BASE names the fallback base and defaults to ${local_base_image}; set it
|
||||
# to reuse or produce a base under a different tag.
|
||||
resolve_vllm_base() {
|
||||
if docker pull "${upstream_base_image}"; then
|
||||
base_image_name="${upstream_base_image}"
|
||||
return
|
||||
fi
|
||||
|
||||
# Registry unreachable, but a previous run may have left the image behind.
|
||||
if [ -n "$(docker images -q "${upstream_base_image}")" ]; then
|
||||
echo "WARNING: could not pull ${upstream_base_image}; using local copy" >&2
|
||||
base_image_name="${upstream_base_image}"
|
||||
return
|
||||
fi
|
||||
|
||||
echo "WARNING: ${upstream_base_image} unavailable remotely and locally" >&2
|
||||
base_image_name="${VLLM_BASE:-${local_base_image}}"
|
||||
if [ -z "$(docker images -q "${base_image_name}")" ]; then
|
||||
echo "WARNING: building ${base_image_name} from vLLM ${VLLM_VERSION} source" >&2
|
||||
build_vllm_base
|
||||
fi
|
||||
}
|
||||
|
||||
resolve_vllm_base
|
||||
if [ -z "$(docker images -q "${base_image_name}")" ]; then
|
||||
build_vllm_base
|
||||
fi
|
||||
|
||||
# Try building the docker image
|
||||
docker_build "${image_name}" --build-arg "VLLM_BASE=${base_image_name}" --build-arg "VLLM_VERSION=${VLLM_VERSION}"
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
steps:
|
||||
- label: ":buildkit: Build and Push NPU Test Image - A2"
|
||||
key: image-build-a2
|
||||
agents: &npu-image-build-agents
|
||||
agents:
|
||||
queue: "ascend-a2b3"
|
||||
resource_class: "npu-1"
|
||||
plugins: &npu-image-build-plugins
|
||||
plugins:
|
||||
- kubernetes:
|
||||
metadata:
|
||||
annotations:
|
||||
@@ -46,67 +46,136 @@ steps:
|
||||
IMAGE_NAME: "vllm-omni-ci-npu"
|
||||
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
|
||||
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
|
||||
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}"
|
||||
command: |
|
||||
set -ex
|
||||
|
||||
echo "--- Building and pushing NPU Test Image"
|
||||
echo "Image: $${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
|
||||
echo "buildkitd address: $${BUILDKITD_ADDR}"
|
||||
|
||||
if ! command -v buildctl &> /dev/null; then
|
||||
echo "Installing buildctl..."
|
||||
mkdir -p /tmp/buildkit
|
||||
BUILDKIT_VERSION="v0.29.0"
|
||||
wget -q "https://github.com/moby/buildkit/releases/download/$${BUILDKIT_VERSION}/buildkit-$${BUILDKIT_VERSION}.linux-arm64.tar.gz" -O /tmp/buildkit.tar.gz
|
||||
tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
|
||||
cp /tmp/buildkit/bin/buildctl /usr/local/bin/
|
||||
fi
|
||||
|
||||
export DOCKER_CONFIG=/home/user/.docker
|
||||
buildctl \
|
||||
--addr="$${BUILDKITD_ADDR}" \
|
||||
--tlscacert=/home/user/.docker/ca.pem \
|
||||
--tlscert=/home/user/.docker/cert.pem \
|
||||
--tlskey=/home/user/.docker/key.pem \
|
||||
build \
|
||||
--frontend dockerfile.v0 \
|
||||
--local context=. \
|
||||
--local dockerfile=./docker \
|
||||
--opt filename=Dockerfile.npu.ci \
|
||||
--secret id=dockerconfig,src=/home/user/.docker/config.json \
|
||||
--output type=image,name=$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG},push=true \
|
||||
--progress=plain
|
||||
|
||||
echo "--- Image pushed successfully"
|
||||
echo "$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
|
||||
|
||||
- label: ":buildkit: Build and Push NPU Test Image - A3"
|
||||
key: image-build-a3
|
||||
agents: *npu-image-build-agents
|
||||
plugins: *npu-image-build-plugins
|
||||
agents:
|
||||
queue: "ascend-a2b3"
|
||||
resource_class: "npu-1"
|
||||
plugins:
|
||||
- kubernetes:
|
||||
metadata:
|
||||
annotations:
|
||||
vault.hashicorp.com/agent-init-first: "true"
|
||||
vault.hashicorp.com/agent-inject: "true"
|
||||
vault.hashicorp.com/agent-inject-perms-ca.pem: "0400"
|
||||
vault.hashicorp.com/agent-inject-perms-cert.pem: "0400"
|
||||
vault.hashicorp.com/agent-inject-perms-config.json: "0400"
|
||||
vault.hashicorp.com/agent-inject-perms-key.pem: "0400"
|
||||
vault.hashicorp.com/agent-inject-secret-ca.pem: internal/data/ascend/buildkitd
|
||||
vault.hashicorp.com/agent-inject-secret-cert.pem: internal/data/ascend/buildkitd
|
||||
vault.hashicorp.com/agent-inject-secret-config.json: internal/data/ascend/buildkitd
|
||||
vault.hashicorp.com/agent-inject-secret-key.pem: internal/data/ascend/buildkitd
|
||||
vault.hashicorp.com/agent-inject-template-ca.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.RootCA }}\n{{- end }}"
|
||||
vault.hashicorp.com/agent-inject-template-cert.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaCert }}\n{{- end }}"
|
||||
vault.hashicorp.com/agent-inject-template-config.json: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.dockerConfig }}\n{{- end }}"
|
||||
vault.hashicorp.com/agent-inject-template-key.pem: "{{- with secret \"internal/data/ascend/buildkitd\" -}}\n{{ .Data.data.ClientCaKey }}\n{{- end }}"
|
||||
vault.hashicorp.com/agent-pre-populate-only: "true"
|
||||
vault.hashicorp.com/agent-run-as-group: "1000"
|
||||
vault.hashicorp.com/agent-run-as-user: "1000"
|
||||
vault.hashicorp.com/agent-service-account-token-volume-name: token-vol
|
||||
vault.hashicorp.com/role: ascend-gha-runners
|
||||
vault.hashicorp.com/secret-volume-path: /home/user/.docker/
|
||||
vault.hashicorp.com/tls-skip-verify: "true"
|
||||
podSpecPatch:
|
||||
volumes:
|
||||
- name: token-vol
|
||||
projected:
|
||||
defaultMode: 420
|
||||
sources:
|
||||
- serviceAccountToken:
|
||||
audience: api
|
||||
expirationSeconds: 600
|
||||
path: token
|
||||
env:
|
||||
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}-a3"
|
||||
IMAGE_NAME: "vllm-omni-ci-npu"
|
||||
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
|
||||
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
|
||||
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}" -a3
|
||||
command: |
|
||||
set -ex
|
||||
|
||||
- label: ":buildkit: Build and Push NPU Test Image - A5"
|
||||
key: image-build-a5
|
||||
agents: *npu-image-build-agents
|
||||
plugins: *npu-image-build-plugins
|
||||
env:
|
||||
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}-a5"
|
||||
IMAGE_NAME: "vllm-omni-ci-npu"
|
||||
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
|
||||
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
|
||||
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}" -a5
|
||||
echo "--- Building and pushing NPU Test Image"
|
||||
echo "Image: $${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
|
||||
echo "buildkitd address: $${BUILDKITD_ADDR}"
|
||||
|
||||
- label: ":buildkit: Build and Push NPU Test Image - 310P"
|
||||
key: image-build-310p
|
||||
agents: *npu-image-build-agents
|
||||
plugins: *npu-image-build-plugins
|
||||
env:
|
||||
VLLM_IMAGE_TAG: "${BUILDKITE_COMMIT}-310p"
|
||||
IMAGE_NAME: "vllm-omni-ci-npu"
|
||||
IMAGE_REGISTRY: "swr.cn-southwest-2.myhuaweicloud.com/modelfoundry"
|
||||
BUILDKITD_ADDR: "tcp://buildkitd-service.buildkitd:1234"
|
||||
command: bash .buildkite/npu/scripts/build-npu-image.sh Dockerfile.npu.ci "$${VLLM_IMAGE_TAG}" -310p
|
||||
if ! command -v buildctl &> /dev/null; then
|
||||
echo "Installing buildctl..."
|
||||
mkdir -p /tmp/buildkit
|
||||
BUILDKIT_VERSION="v0.29.0"
|
||||
wget -q "https://github.com/moby/buildkit/releases/download/$${BUILDKIT_VERSION}/buildkit-$${BUILDKIT_VERSION}.linux-arm64.tar.gz" -O /tmp/buildkit.tar.gz
|
||||
tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
|
||||
cp /tmp/buildkit/bin/buildctl /usr/local/bin/
|
||||
fi
|
||||
|
||||
export DOCKER_CONFIG=/home/user/.docker
|
||||
buildctl \
|
||||
--addr="$${BUILDKITD_ADDR}" \
|
||||
--tlscacert=/home/user/.docker/ca.pem \
|
||||
--tlscert=/home/user/.docker/cert.pem \
|
||||
--tlskey=/home/user/.docker/key.pem \
|
||||
build \
|
||||
--frontend dockerfile.v0 \
|
||||
--local context=. \
|
||||
--local dockerfile=./docker \
|
||||
--opt filename=Dockerfile.npu.ci.a3 \
|
||||
--secret id=dockerconfig,src=/home/user/.docker/config.json \
|
||||
--output type=image,name=$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG},push=true \
|
||||
--progress=plain
|
||||
|
||||
echo "--- Image pushed successfully"
|
||||
echo "$${IMAGE_REGISTRY}/$${IMAGE_NAME}:$${VLLM_IMAGE_TAG}"
|
||||
|
||||
- label: "Upload Ready Pipeline"
|
||||
depends_on:
|
||||
- image-build-a2
|
||||
- image-build-a3
|
||||
- image-build-a5
|
||||
- image-build-310p
|
||||
key: upload-ready-pipeline
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
command: |
|
||||
python3 .buildkite/common/scripts/upload_pipeline.py --upload .buildkite/npu/test-npu-ready.yml
|
||||
|
||||
# L4 Test — main+NIGHTLY=1 (scheduled), or PR nightly-test label
|
||||
# L4 Test — main+NIGHTLY=1 (scheduled), or PR with specific label (e.g. add label then Rebuild)
|
||||
- label: "Upload Nightly Pipeline"
|
||||
depends_on:
|
||||
- image-build-a2
|
||||
- image-build-a3
|
||||
- image-build-a5
|
||||
- image-build-310p
|
||||
key: upload-nightly-pipeline
|
||||
commands:
|
||||
- |
|
||||
EXTRA=""
|
||||
if [ "${NIGHTLY:-}" = "1" ] && [ "${BUILDKITE_BRANCH}" = "main" ]; then
|
||||
EXTRA="--all"
|
||||
fi
|
||||
python3 .buildkite/common/scripts/upload_pipeline.py --upload $$EXTRA .buildkite/npu/test-npu-nightly.yml
|
||||
- python3 .buildkite/common/scripts/upload_pipeline.py --upload .buildkite/npu/test-npu-nightly.yml
|
||||
agents:
|
||||
queue: "cpu_queue_premerge"
|
||||
@@ -96,7 +96,13 @@
|
||||
# key: upload-nightly-pipeline
|
||||
# if: >-
|
||||
# (build.branch == "main" && build.env("NIGHTLY") == "1") ||
|
||||
# (build.branch != "main" && build.pull_request.labels includes "nightly-test")
|
||||
# (build.branch != "main" && (
|
||||
# build.pull_request.labels includes "nightly-test" ||
|
||||
# build.pull_request.labels includes "omni-test" ||
|
||||
# build.pull_request.labels includes "tts-test" ||
|
||||
# build.pull_request.labels includes "diffusion-x2iat-test" ||
|
||||
# build.pull_request.labels includes "diffusion-x2v-test"
|
||||
# ))
|
||||
# commands:
|
||||
# - buildkite-agent pipeline upload .buildkite/npu/test-npu-nightly.yml
|
||||
# agents:
|
||||
|
||||
@@ -1,48 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
set -ex
|
||||
|
||||
dockerfile=${1:?Dockerfile is required}
|
||||
image_tag=${2:?Image tag is required}
|
||||
device_suffix=${3:-}
|
||||
|
||||
image_name=${IMAGE_NAME:-vllm-omni-ci-npu}
|
||||
image_registry=${IMAGE_REGISTRY:-swr.cn-southwest-2.myhuaweicloud.com/modelfoundry}
|
||||
buildkitd_addr=${BUILDKITD_ADDR:-tcp://buildkitd-service.buildkitd:1234}
|
||||
|
||||
echo "--- Building and pushing NPU Test Image"
|
||||
echo "Image: ${image_registry}/${image_name}:${image_tag}"
|
||||
echo "Dockerfile: ${dockerfile}"
|
||||
|
||||
if ! command -v buildctl &> /dev/null; then
|
||||
mkdir -p /tmp/buildkit
|
||||
buildkit_version=v0.29.0
|
||||
wget -q \
|
||||
"https://github.com/moby/buildkit/releases/download/${buildkit_version}/buildkit-${buildkit_version}.linux-arm64.tar.gz" \
|
||||
-O /tmp/buildkit.tar.gz
|
||||
tar -xzf /tmp/buildkit.tar.gz -C /tmp/buildkit
|
||||
cp /tmp/buildkit/bin/buildctl /usr/local/bin/
|
||||
fi
|
||||
|
||||
build_args=()
|
||||
if [[ -n "${device_suffix}" ]]; then
|
||||
build_args+=(--opt "build-arg:VLLM_ASCEND_DEVICE_SUFFIX=${device_suffix}")
|
||||
fi
|
||||
|
||||
export DOCKER_CONFIG=/home/user/.docker
|
||||
buildctl \
|
||||
--addr="${buildkitd_addr}" \
|
||||
--tlscacert=/home/user/.docker/ca.pem \
|
||||
--tlscert=/home/user/.docker/cert.pem \
|
||||
--tlskey=/home/user/.docker/key.pem \
|
||||
build \
|
||||
--frontend dockerfile.v0 \
|
||||
--local context=. \
|
||||
--local dockerfile=./docker \
|
||||
--opt "filename=${dockerfile}" \
|
||||
"${build_args[@]}" \
|
||||
--secret id=dockerconfig,src=/home/user/.docker/config.json \
|
||||
--output "type=image,name=${image_registry}/${image_name}:${image_tag},push=true" \
|
||||
--progress=plain
|
||||
|
||||
echo "--- Image pushed successfully"
|
||||
echo "${image_registry}/${image_name}:${image_tag}"
|
||||
@@ -4,10 +4,10 @@ steps:
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: >-
|
||||
build.env("NIGHTLY") == "1" ||
|
||||
build.pull_request.labels includes "nightly-test"
|
||||
build.pull_request.labels includes "nightly-test" ||
|
||||
build.pull_request.labels includes "omni-test"
|
||||
steps:
|
||||
- label: ":full_moon: Omni · Perf Test · No Async Chunk"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: nightly-omni-performance-no-async-chunk
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
@@ -24,7 +24,6 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · Perf Test · Async Chunk"
|
||||
source_file_dependencies: omni_qwen3_omni_perf
|
||||
key: nightly-omni-performance-async-chunk
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
@@ -41,7 +40,6 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_function
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
@@ -50,7 +48,6 @@ steps:
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_minicpmo_4_5.py tests/e2e/online_serving/test_minicpmo_4_5.py tests/e2e/online_serving/test_minicpmo_4_5_expansion.py -m 'full_model' --run-level 'full_model'
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Duplex Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_function
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
@@ -64,13 +61,7 @@ steps:
|
||||
tests/e2e/online_serving/test_minicpmo_4_5_duplex_expansion.py
|
||||
-m "npu and A3 and omni"
|
||||
--run-level full_model
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/e2e/online_serving/test_minicpmo_4_5_window.py
|
||||
-m "full_model and npu and A3 and omni"
|
||||
--run-level full_model
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Accuracy Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_accuracy
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
@@ -90,27 +81,7 @@ steps:
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/minicpmo_4_5/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · Omni-DuplexEval CI Guard"
|
||||
source_file_dependencies: omni_duplex_eval_ci
|
||||
timeout_in_minutes: 240
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
VLLM_DUPLEX_EVAL_JUDGE_URL: "http://127.0.0.1:8001"
|
||||
commands:
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/e2e/accuracy/omni_duplex_eval/test_omni_duplex_eval_ci.py \
|
||||
-m "full_model and npu and A3 and cards_2" \
|
||||
--run-level full_model \
|
||||
--basetemp /tmp/omni_duplex_ci
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/e2e/accuracy/omni_duplex_eval/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Perf Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5-npu
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
@@ -127,7 +98,6 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Perf Test (A2)"
|
||||
source_file_dependencies: omni_minicpmo_4_5_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5-npu-a2
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a2b3_npu_1
|
||||
@@ -145,7 +115,6 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Omni · MiniCPM-o 4.5 · Duplex Seed-TTS Perf Test"
|
||||
source_file_dependencies: omni_minicpmo_4_5_duplex_perf
|
||||
key: nightly-omni-performance-minicpmo-4-5-duplex-seed-tts-npu
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
@@ -164,10 +133,9 @@ steps:
|
||||
- group: ":card_index_dividers: TTS Model Test"
|
||||
key: nightly-tts-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test"
|
||||
if: build.env("NIGHTLY") == "1" || build.pull_request.labels includes "nightly-test" || build.pull_request.labels includes "tts-test"
|
||||
steps:
|
||||
- label: ":full_moon: TTS · Perf Test"
|
||||
source_file_dependencies: tts_qwen3_tts_perf
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
@@ -183,15 +151,15 @@ steps:
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
- group: ":card_index_dividers: Diffusion X2I(&A&T) Model Test"
|
||||
key: nightly-diffusion-x2iat-group
|
||||
- group: ":card_index_dividers: Diffusion I2I Model Test"
|
||||
key: nightly-HunyuanImage3-x2i-test-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: >-
|
||||
build.env("NIGHTLY") == "1" ||
|
||||
build.pull_request.labels includes "nightly-test"
|
||||
build.pull_request.labels includes "nightly-test" ||
|
||||
build.pull_request.labels includes "diffusion-x2iat-test"
|
||||
steps:
|
||||
- label: ":full_moon: HunyuanImage3-DIT · Accuracy Test"
|
||||
source_file_dependencies: diffusion_hunyuan_image3_accuracy
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
@@ -201,8 +169,15 @@ steps:
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/accuracy/test_hunyuan_image3_pixel_accuracy.py -m "full_model and A3" --run-level full_model
|
||||
- group: ":card_index_dividers: Diffusion Benchmark Perf Test"
|
||||
key: nightly-diffusion-benchmark-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: >-
|
||||
build.env("NIGHTLY") == "1" ||
|
||||
build.pull_request.labels includes "nightly-test" ||
|
||||
build.pull_request.labels includes "diffusion-x2iat-test"
|
||||
steps:
|
||||
- label: ":full_moon: Diffusion Perf · HunyuanImage3 TP=2 CFGP=2"
|
||||
source_file_dependencies: diffusion_hunyuan_image3_perf
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
@@ -219,7 +194,6 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion Perf · HunyuanImage3 TP=2 SP=2"
|
||||
source_file_dependencies: diffusion_hunyuan_image3_perf
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
@@ -236,7 +210,6 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion Perf · HunyuanImage3 TP=4"
|
||||
source_file_dependencies: diffusion_hunyuan_image3_perf
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
@@ -252,81 +225,15 @@ steps:
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · Single-NPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-single-npu
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "npu and A3 and cards_1" --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion X2I(&A&T) · Perf Test · Qwen-Image · 4-NPU"
|
||||
source_file_dependencies: diffusion_qwen_image_perf
|
||||
key: nightly-diffusion-x2iat-performance-qwen-image-4npu
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m "npu and A3 and cards_4" --test-config-file tests/dfx/perf/tests/test_qwen_image_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
- group: ":card_index_dividers: Diffusion X2V Model Test"
|
||||
key: nightly-diffusion-x2v-group
|
||||
depends_on: upload-nightly-pipeline
|
||||
if: >-
|
||||
build.env("NIGHTLY") == "1" ||
|
||||
build.pull_request.labels includes "nightly-test"
|
||||
build.pull_request.labels includes "nightly-test" ||
|
||||
build.pull_request.labels includes "diffusion-x2v-test"
|
||||
steps:
|
||||
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Function Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_function
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a2b3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
HF_HUB_OFFLINE: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m "npu and A2" --run-level "full_model"
|
||||
- label: ":full_moon: Diffusion X2V · HunyuanVideo-1.5 Perf Test"
|
||||
source_file_dependencies: diffusion_hunyuan_video_perf
|
||||
key: nightly-diffusion-x2v-performance-hunyuanvideo15
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
HF_HUB_OFFLINE: "1"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m "npu and A3" --test-config-file tests/dfx/perf/tests/test_hunyuanvideo15_t2v_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
exit $$EXIT
|
||||
- label: ":full_moon: Diffusion X2V · Wan 2.2 Function Test"
|
||||
source_file_dependencies: diffusion_wan22_function
|
||||
- label: ":full_moon: Diffusion X2V · Function Test"
|
||||
mirror_hardwares: a2b3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
@@ -334,8 +241,7 @@ steps:
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_wan22_expansion.py -k npu_i2v --run-level "full_model"
|
||||
- label: ":full_moon: Diffusion X2V · Wan 2.2 Perf Test"
|
||||
source_file_dependencies: diffusion_wan22_perf
|
||||
- label: ":full_moon: Diffusion X2V · Perf Test"
|
||||
key: nightly-diffusion-x2v-performance
|
||||
timeout_in_minutes: 180
|
||||
mirror_hardwares: a3_npu_4
|
||||
@@ -345,10 +251,11 @@ steps:
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
DIFFUSION_ATTENTION_BACKEND: TORCH_SDPA
|
||||
commands:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- export DIFFUSION_BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- |
|
||||
set +e
|
||||
pytest -s -v tests/dfx/perf/scripts/run_benchmark.py -m npu --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py -m npu --test-config-file tests/dfx/perf/tests/test_wan22_i2v_vllm_omni.json
|
||||
EXIT=$$?
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/diffusion_result_*.json"
|
||||
buildkite-agent artifact upload "tests/dfx/perf/results/logs/*.log"
|
||||
exit $$EXIT
|
||||
@@ -1,62 +1,17 @@
|
||||
steps:
|
||||
- group: ":card_index_dividers: Simple Test"
|
||||
- label: "Platforms Test"
|
||||
key: ready-platforms-test
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
- label: "Platforms Test"
|
||||
key: ready-platforms-test
|
||||
mirror_hardwares: a3_npu_2
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/platforms/npu/test_minicpmo_code2wav_npugraph.py
|
||||
-m 'core_model and npu and A3 and omni'
|
||||
--run-level 'core_model'
|
||||
mirror_hardwares: a3_npu_2
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/platforms/npu/test_minicpmo_code2wav_npugraph.py
|
||||
-m 'core_model and npu and A3 and omni'
|
||||
--run-level 'core_model'
|
||||
|
||||
- label: "Diffusion NPU layer tests - A2"
|
||||
key: ready-diffusion-npu-layer-tests-a2
|
||||
mirror_hardwares: a2b3_npu_1
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/diffusion/layers/test_activation.py
|
||||
-m 'core_model and npu and A2'
|
||||
--run-level 'core_model'
|
||||
|
||||
- label: "Diffusion NPU layer tests - A3"
|
||||
key: ready-diffusion-npu-layer-tests
|
||||
mirror_hardwares: a3_npu_2
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/diffusion/layers/test_activation.py
|
||||
tests/diffusion/layers/test_fused_qk_norm_rope_npu.py
|
||||
tests/diffusion/models/minimax_h3/test_minimax_h3_qwen3vl_swiglu.py
|
||||
tests/diffusion/models/minimax_h3/test_minimax_h3_modulation.py
|
||||
-m 'core_model and npu and A3'
|
||||
--run-level 'core_model'
|
||||
|
||||
- label: "Diffusion NPU layer tests - A5"
|
||||
key: ready-diffusion-npu-layer-tests-a5
|
||||
mirror_hardwares: a5_npu_2
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/diffusion/layers/test_activation.py
|
||||
-m 'core_model and npu and A5'
|
||||
--run-level 'core_model'
|
||||
|
||||
- label: "Diffusion NPU layer tests - 310P"
|
||||
key: ready-diffusion-npu-layer-tests-310p
|
||||
mirror_hardwares: 310p_npu_1
|
||||
commands:
|
||||
- >-
|
||||
pytest -s -v
|
||||
tests/diffusion/layers/test_activation.py
|
||||
-m 'core_model and npu and 310P'
|
||||
--run-level 'core_model'
|
||||
|
||||
- group: ":card_index_dividers: E2E Test"
|
||||
key: ready-e2e-group
|
||||
- group: ":card_index_dividers: Omni Model Test"
|
||||
key: ready-omni-group
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
- label: "Omni · MiniCPM-o 4.5 Test"
|
||||
@@ -66,7 +21,6 @@ steps:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/offline_inference/test_minicpmo_4_5.py tests/e2e/online_serving/test_minicpmo_4_5.py -m 'core_model and npu and A3 and omni' --run-level 'core_model'
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Duplex Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
@@ -74,7 +28,6 @@ steps:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_minicpmo_4_5_duplex.py -m 'core_model and npu and A3 and omni' --run-level 'core_model'
|
||||
|
||||
- label: "Omni · MiniCPM-o 4.5 Perf Test"
|
||||
timeout_in_minutes: 60
|
||||
mirror_hardwares: a3_npu_2
|
||||
@@ -85,15 +38,10 @@ steps:
|
||||
- export BENCHMARK_DIR=tests/dfx/perf/results
|
||||
- pytest -s -v tests/dfx/perf/scripts/run_benchmark.py --test-config-file tests/dfx/perf/tests/test_minicpmo_4_5_ready.json -m 'core_model and npu and A3 and omni'
|
||||
|
||||
- label: "Omni · Qwen3-Omni Test"
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m 'core_model and npu and A3 and omni' --run-level 'core_model'
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- group: ":card_index_dividers: TTS Model Test"
|
||||
key: ready-tts-group
|
||||
depends_on: upload-ready-pipeline
|
||||
steps:
|
||||
- label: ":sound: TTS · Qwen3-TTS CustomVoice Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
@@ -102,55 +50,3 @@ steps:
|
||||
HF_TOKEN: "${HF_TOKEN}"
|
||||
commands:
|
||||
- pytest -s -v tests/e2e/online_serving/test_qwen3_tts_customvoice.py -m 'core_model and npu' --run-level 'core_model'
|
||||
|
||||
- label: "Diffusion · HunyuanVideo-1.5 Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_hunyuan_video_15_expansion.py -m 'core_model and diffusion and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- label: "Diffusion · Qwen Image Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_qwen_image.py -m 'core_model and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- label: "Diffusion · Wan22 Test"
|
||||
mirror_hardwares: a3_npu_2
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_wan22_t2v.py -m 'core_model and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
|
||||
- label: "Diffusion · Hunyuan Image Test"
|
||||
mirror_hardwares: a3_npu_4
|
||||
env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: spawn
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
commands:
|
||||
- |
|
||||
timeout 40m bash -c "
|
||||
export VLLM_IMAGE_FETCH_TIMEOUT=60
|
||||
pytest -s -v tests/e2e/online_serving/test_hunyuan_image.py -m 'core_model and diffusion and npu' --run-level 'core_model'
|
||||
"
|
||||
timeout_in_minutes: 60
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/3d-animation-short-generator
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: add-diffusion-model
|
||||
description: Add a new diffusion model (text-to-image, text-to-video, image-to-video, text-to-audio, image editing) to vLLM-Omni, including native non-Diffusers ports, reference-parity validation, Cache-DiT, offload, and parallelism support (TP, SP/USP, CFG-Parallel, HSDP). Use when integrating or reviewing a new diffusion model, porting a Diffusers pipeline or custom model repository, creating a DiT adapter, reusing shared examples, or qualifying multi-GPU and memory optimizations.
|
||||
description: Add a new diffusion model (text-to-image, text-to-video, image-to-video, text-to-audio, image editing) to vLLM-Omni, including Cache-DiT acceleration and parallelism support (TP, SP/USP, CFG-Parallel, HSDP). Use when integrating a new diffusion model, porting a diffusers pipeline or a custom model repo to vllm-omni, creating a new DiT transformer adapter, adding diffusion model support, or enabling multi-GPU parallelism and cache acceleration for an existing model.
|
||||
---
|
||||
|
||||
# Adding a Diffusion Model to vLLM-Omni
|
||||
@@ -26,16 +26,9 @@ Check the model's HF repo for `model_index.json`. This determines your path:
|
||||
|----------|----------------|----------------|
|
||||
| **Already supported** | `_class_name` in `model_index.json` matches a key in `_DIFFUSION_MODELS` in `registry.py` | Skip implementation, then validate model-specific examples, tests, and docs as needed |
|
||||
| **Diffusers-based** | Has standard `model_index.json` with `_diffusers_version`, subfolders for `transformer/`, `vae/`, etc. | Follow **Path A** below |
|
||||
| **Native non-Diffusers model** | No Diffusers index, non-standard checkpoint hierarchy, or custom architecture in a separate repo | Follow **Path B** below; port the runtime natively unless an external adapter was explicitly requested |
|
||||
| **Custom/private repo** | No diffusers `model_index.json`, weights in non-standard format, custom model code in a separate git repo | Follow **Path B** below |
|
||||
| **Hybrid** | Has some diffusers components (VAE) but custom transformer/fusion | Mix of Path A and Path B |
|
||||
|
||||
Before coding, write a short integration contract covering runtime ownership,
|
||||
checkpoint discovery, reference revision, I/O geometry, attention semantics,
|
||||
CFG behavior, auxiliary components, target hardware, and the default deployment.
|
||||
For Path B, hybrid models, or any optimization work, read
|
||||
[references/native-model-integration-checklist.md](references/native-model-integration-checklist.md)
|
||||
and use its phase gates.
|
||||
|
||||
## Path A: Diffusers-Based Model
|
||||
|
||||
For models with a standard diffusers layout. See [references/transformer-adaptation.md](references/transformer-adaptation.md) for detailed code patterns.
|
||||
@@ -110,15 +103,9 @@ For custom loop structures, follow `vllm_omni/diffusion/models/progress_bar.py`
|
||||
|
||||
---
|
||||
|
||||
## Path B: Native / Non-Diffusers Model
|
||||
## Path B: Custom/Private Repo Model
|
||||
|
||||
For models without a Diffusers pipeline—weights in custom formats and model
|
||||
code in another public or private repository. Treat that repository as a pinned
|
||||
correctness oracle. A request for native support means the vLLM-Omni runtime
|
||||
must not import the reference implementation; an external adapter is appropriate
|
||||
only when the requested scope explicitly permits that dependency. See
|
||||
[references/custom-model-patterns.md](references/custom-model-patterns.md) for
|
||||
concrete integration patterns.
|
||||
For models without a diffusers pipeline — weights in custom format, model code in a private repo. Real examples: DreamID-Omni, BAGEL, HunyuanImage3. See [references/custom-model-patterns.md](references/custom-model-patterns.md) for concrete custom-model integration patterns.
|
||||
|
||||
### B1. Understand the reference repo
|
||||
|
||||
@@ -139,25 +126,16 @@ This is the key design decision for custom models. Follow these placement rules:
|
||||
| **Pipeline orchestration** (init, forward, denoise loop) | `vllm_omni/diffusion/models/<name>/pipeline_<name>.py` | Always required |
|
||||
| **Custom transformer/backbone** (ported and adapted to vllm-omni) | `vllm_omni/diffusion/models/<name>/<name>_transformer.py` or similar | `wan2_2.py`, `fusion.py`, `bagel_transformer.py` |
|
||||
| **Custom sub-models** (VAE, fusion, autoencoder) | `vllm_omni/diffusion/models/<name>/` as separate files | `autoencoder.py`, `fusion.py` |
|
||||
| **Reference-only code** | Keep outside the runtime; use a pinned revision for golden outputs and architecture analysis | Reference inference script |
|
||||
| **Explicit external adapter dependency** | External package, only when the requested scope and maintainer direction allow it | Compatibility adapter, not native support |
|
||||
| **External dependency code** (original repo utilities) | **External repo**, installed via download script or pip | `dreamid_omni` package via git clone |
|
||||
| **Hardcoded model configs** | Module-level dicts in pipeline file | `VIDEO_CONFIG`, `AUDIO_CONFIG` dicts |
|
||||
| **Download/setup script** | `examples/offline_inference/<name>/download_<name>.py` | `download_<name>.py` |
|
||||
| **Download/setup script** | `examples/offline_inference/<name>/download_<name>.py` | `download_dreamid_omni.py` |
|
||||
| **Custom `model_index.json`** | Generated by download script, placed at model root | Minimal: `{"_class_name": "YourPipeline", ...}` |
|
||||
|
||||
### B3. Handle external dependencies
|
||||
|
||||
If the model's code lives in a separate git repo, first decide whether the
|
||||
requested deliverable is native support or an external adapter. Do not silently
|
||||
choose the adapter path.
|
||||
If the model's code lives in a separate git repo:
|
||||
|
||||
**Option 1: Port the code directly** (default for native support)
|
||||
|
||||
Copy the essential model files into `vllm_omni/diffusion/models/<name>/` and
|
||||
adapt them to shared vLLM-Omni contracts. Keep checkpoint loading strict and
|
||||
use the pinned reference only to generate parity evidence.
|
||||
|
||||
**Option 2: Import with graceful fallback** (adapter scope only)
|
||||
**Option 1: Import with graceful fallback** (recommended for models with external utils)
|
||||
|
||||
```python
|
||||
try:
|
||||
@@ -169,15 +147,17 @@ except ImportError:
|
||||
)
|
||||
```
|
||||
|
||||
Use an external runtime dependency only when the user explicitly requested an
|
||||
adapter or a maintainer approved the exception. Document the dependency,
|
||||
pinned revision, installation path, and unsupported native features.
|
||||
**Option 2: Port the code directly** (preferred when feasible)
|
||||
|
||||
Copy the essential model files into `vllm_omni/diffusion/models/<name>/` and adapt them. This avoids external dependencies. BAGEL does this — `autoencoder.py` and `bagel_transformer.py` are ported directly.
|
||||
|
||||
**Decision criteria**: Port if the code is self-contained and won't diverge. Use external deps if the model repo is actively maintained and the code is complex.
|
||||
|
||||
### B4. Handle custom weight loading
|
||||
|
||||
Custom models have two common patterns for weight loading:
|
||||
|
||||
**Pattern 1: Bypass standard loader** (eager custom init)
|
||||
**Pattern 1: Bypass standard loader** (DreamID-Omni style)
|
||||
|
||||
When the original model has complex custom init functions that load weights in `__init__`:
|
||||
|
||||
@@ -241,8 +221,7 @@ class CustomPipeline(nn.Module):
|
||||
|
||||
### B5. Create the `model_index.json`
|
||||
|
||||
Prefer a `model_index.json` at the model root when vLLM-Omni owns an assembled
|
||||
checkpoint directory. For custom models, this is minimal:
|
||||
Custom models need a `model_index.json` at the model root for vllm-omni to discover them. For custom models, this is minimal:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -251,14 +230,7 @@ checkpoint directory. For custom models, this is minimal:
|
||||
}
|
||||
```
|
||||
|
||||
The `_class_name` must match a key in `_DIFFUSION_MODELS` in `registry.py`.
|
||||
Additional keys are model-specific (accessed via `od_config.model_config`).
|
||||
|
||||
If the released repository is immutable and has neither a root `config.json`
|
||||
nor a Diffusers index, add it to a generic native-checkpoint signature resolver:
|
||||
match the exact Hub ID or a distinctive set of local files and return the
|
||||
pipeline class. Do not use model-name substrings or add parallel one-off
|
||||
predicates in CLI, config, and serving consumers.
|
||||
The `_class_name` must match a key in `_DIFFUSION_MODELS` in `registry.py`. Additional keys are model-specific (accessed via `od_config.model_config`).
|
||||
|
||||
If the model's weights come from multiple HF repos, write a **download script** that:
|
||||
1. Downloads from each repo
|
||||
@@ -320,9 +292,7 @@ Use the appropriate existing example script:
|
||||
| Image-to-Image | `examples/offline_inference/image_to_image/image_edit.py` |
|
||||
| Text-to-Audio | `examples/offline_inference/text_to_audio/text_to_audio.py` |
|
||||
|
||||
Reuse these shared scripts even for custom models when their request and output
|
||||
contracts fit. Create a dedicated model script only when the shared category
|
||||
cannot represent the protocol, and document that gap in the PR.
|
||||
For custom/Omni models that don't fit these categories, create a dedicated example script.
|
||||
|
||||
**Validation**: No errors, output is meaningful, quality matches reference implementation.
|
||||
|
||||
@@ -330,17 +300,13 @@ See [references/troubleshooting.md](references/troubleshooting.md) for common er
|
||||
|
||||
### Step 6: Add Example Scripts
|
||||
|
||||
Only when the shared category scripts cannot represent the model, create:
|
||||
For Omni or custom models, create:
|
||||
- `examples/offline_inference/your_model_name/` — offline script + README
|
||||
- `examples/online_serving/your_model_name/` — server script + client
|
||||
- Download script if weights require assembly from multiple sources
|
||||
|
||||
### Step 7: Update Documentation
|
||||
|
||||
Follow the [`add-recipe` skill](../add-recipe/SKILL.md) to add or update the
|
||||
model-family recipe and its `recipes/README.md` row with verified specifications,
|
||||
hardware, commands, feature links, and qualification evidence.
|
||||
|
||||
Required updates:
|
||||
1. `docs/user_guide/diffusion/parallelism/overview.md` — parallelism support overview/table
|
||||
2. `docs/user_guide/diffusion/cpu_offload.md` — if CPU offload supported (add to supported models table)
|
||||
@@ -372,7 +338,7 @@ Classify the model's **CI priority** first:
|
||||
|
||||
**L2 & L3 online — same file, dual marks on the baseline smoke:** The **first / simplest** case in `test_{slug}.py` (default deploy, minimal steps, single prompt) should carry **both** `@pytest.mark.core_model` and `@pytest.mark.advanced_model` on the **same** function so L2 (`test-ready.yml`) and L3 (`test-merge.yml`) share one smoke test. Heavier deploy variants or API paths in the same file use **`advanced_model` only**. When L3 moves to nightly, migrate those heavier cases into `test_{slug}_expansion.py` with `full_model` and remove the dedicated `test-merge.yml` job (see `test_longcat_image_expansion.py`, `test_qwen_image_expansion.py`).
|
||||
|
||||
**L4 design (high priority):** Combine multiple supported features (Cache-DiT, TP, USP, CFG, HSDP, CPU offload, quantization) into **few parametrized** `OmniServerParams` rows so each feature appears in at least one case without exploding GPU jobs. Shard single-GPU vs multi-GPU cases across the nightly X2I/X2V function steps (`cards_1` vs `not cards_1`).
|
||||
**L4 design (high priority):** Combine multiple supported features (Cache-DiT, TP, USP, CFG, HSDP, CPU offload, quantization) into **few parametrized** `OmniServerParams` rows so each feature appears in at least one case without exploding GPU jobs. Shard single-GPU vs multi-GPU cases across the nightly X2I/X2V function steps (1-GPU vs `distributed_cuda`).
|
||||
|
||||
**L4 design (medium / low):** One or two parametrized rows covering the best quality/perf trade-off; skip perf JSON unless the model is high priority.
|
||||
|
||||
@@ -380,20 +346,107 @@ Classify the model's **CI priority** first:
|
||||
|
||||
**Keep model-specific code inside test modules — not `tests/helpers/{slug}.py`:** deploy constants, prompts, sampling dicts, and inline `request_config` / `form_data` belong in each `test_{slug}.py` and `test_{slug}_expansion.py`. Do not add per-model files under `tests/helpers/`; reuse only repo-wide harness (`mark`, `media`, `runtime`, `stage_config`, `assertions`). **L2+ online/offline e2e:** reuse or add `send_*_request` in `tests/helpers/runtime.py` — tests call the handler, not raw `omni.generate` / HTTP. See [vllm-omni-test skill](../vllm-omni-test/SKILL.md) § **Runtime send helpers**.
|
||||
|
||||
Keep the model suite proportional using the six distinct failure owners in the
|
||||
native integration checklist. Combine supported features into a few
|
||||
parametrized E2E rows instead of creating one test per optimization.
|
||||
|
||||
### Step 9: Add Cache-DiT Acceleration
|
||||
|
||||
Add caching only after uncached single-device correctness. Read
|
||||
[references/cache-dit-patterns.md](references/cache-dit-patterns.md), use the
|
||||
automatic single-block-list path when possible, and add a registered
|
||||
`BlockAdapter` only for genuinely custom block topology.
|
||||
Cache-DiT accelerates inference by caching intermediate computation results across denoising steps. After your model is working correctly on a single GPU, add cache-dit support.
|
||||
|
||||
Verify a real cache hit and compare quality with the uncached baseline. Make a
|
||||
speed claim only at a realistic step count where warmup permits hits; an
|
||||
all-warmup smoke proves integration, not acceleration.
|
||||
See [references/cache-dit-patterns.md](references/cache-dit-patterns.md) for detailed code patterns.
|
||||
|
||||
#### 9a. Determine your model type
|
||||
|
||||
| Model Type | Description | Action |
|
||||
|------------|-------------|--------|
|
||||
| **Standard single-transformer** | One transformer with one `ModuleList` of blocks | No code needed — `CacheDiTBackend` auto-detects via `enable_cache_for_dit()` |
|
||||
| **Multi-block-list** | One transformer with multiple block lists (e.g., `transformer_blocks` + `single_transformer_blocks`) | Write custom enabler with `BlockAdapter` |
|
||||
| **Dual-transformer** | Two transformers (e.g., high-noise + low-noise) | Write custom enabler with `BlockAdapter` wrapping both |
|
||||
|
||||
#### 9b. Standard models — verify automatic support
|
||||
|
||||
For standard single-transformer models, test directly:
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="your-model-name",
|
||||
cache_backend="cache_dit",
|
||||
cache_config={
|
||||
"Fn_compute_blocks": 1,
|
||||
"Bn_compute_blocks": 0,
|
||||
"max_warmup_steps": 4,
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
Check logs for "Cache-dit enabled successfully on xxx". If it works, skip to Step 9e.
|
||||
|
||||
#### 9c. Custom architectures — write a custom enabler
|
||||
|
||||
For multi-block-list or dual-transformer models, write a custom enabler function:
|
||||
|
||||
```python
|
||||
from cache_dit import BlockAdapter, ForwardPattern, ParamsModifier, DBCacheConfig
|
||||
|
||||
def enable_cache_for_your_model(pipeline, cache_config):
|
||||
db_cache_config = DBCacheConfig(
|
||||
num_inference_steps=None,
|
||||
Fn_compute_blocks=cache_config.Fn_compute_blocks,
|
||||
Bn_compute_blocks=cache_config.Bn_compute_blocks,
|
||||
max_warmup_steps=cache_config.max_warmup_steps,
|
||||
max_cached_steps=cache_config.max_cached_steps,
|
||||
max_continuous_cached_steps=cache_config.max_continuous_cached_steps,
|
||||
residual_diff_threshold=cache_config.residual_diff_threshold,
|
||||
)
|
||||
|
||||
cache_dit.enable_cache(
|
||||
BlockAdapter(
|
||||
transformer=pipeline.transformer,
|
||||
blocks=[
|
||||
pipeline.transformer.transformer_blocks,
|
||||
pipeline.transformer.single_transformer_blocks,
|
||||
],
|
||||
forward_pattern=[ForwardPattern.Pattern_1, ForwardPattern.Pattern_1],
|
||||
params_modifiers=[ParamsModifier(...)],
|
||||
),
|
||||
cache_config=db_cache_config,
|
||||
)
|
||||
|
||||
def refresh_cache_context(pipeline, num_inference_steps, verbose=True):
|
||||
cache_dit.refresh_context(
|
||||
pipeline.transformer, num_inference_steps=num_inference_steps, verbose=verbose
|
||||
)
|
||||
return refresh_cache_context
|
||||
```
|
||||
|
||||
#### 9d. Register the custom enabler
|
||||
|
||||
Add your enabler to `CUSTOM_DIT_ENABLERS` in
|
||||
`vllm_omni/diffusion/cache/cachedit/model_specific.py`:
|
||||
|
||||
```python
|
||||
CUSTOM_DIT_ENABLERS = {
|
||||
"Wan22Pipeline": enable_cache_for_wan22,
|
||||
"LongCatImagePipeline": enable_cache_for_longcat_image,
|
||||
"YourModelPipeline": enable_cache_for_your_model, # Add here
|
||||
}
|
||||
```
|
||||
|
||||
#### 9e. Test Cache-DiT
|
||||
|
||||
```python
|
||||
omni = Omni(
|
||||
model="your-model-name",
|
||||
cache_backend="cache_dit",
|
||||
cache_config={
|
||||
"Fn_compute_blocks": 1, "Bn_compute_blocks": 0,
|
||||
"max_warmup_steps": 4, "residual_diff_threshold": 0.24,
|
||||
}
|
||||
)
|
||||
images = omni.generate("a beautiful landscape",
|
||||
OmniDiffusionSamplingParams(num_inference_steps=50))
|
||||
```
|
||||
|
||||
**Verify**: 1) logs show cache enabled, 2) 1.5-2x speedup, 3) output quality acceptable vs baseline.
|
||||
|
||||
If quality degrades, lower `residual_diff_threshold` (try 0.12-0.18) or increase `max_warmup_steps` (try 6-8).
|
||||
|
||||
---
|
||||
|
||||
@@ -407,30 +460,107 @@ See [references/parallelism-patterns.md](references/parallelism-patterns.md) for
|
||||
|
||||
#### 10a. Tensor Parallelism (TP)
|
||||
|
||||
Replace compatible projections with vLLM parallel linears, preserve checkpoint
|
||||
fusion/loading, and use local head counts. Require query/KV head divisibility
|
||||
and compare a multi-rank forward with the one-rank oracle.
|
||||
Shards DiT linear layers across GPUs. Requires code changes in the transformer.
|
||||
|
||||
**What to change in the transformer**:
|
||||
1. Replace `nn.Linear` with `ColumnParallelLinear` / `RowParallelLinear` / `QKVParallelLinear`
|
||||
2. Update `load_weights()` to handle QKV fusion with `stacked_params_mapping`
|
||||
3. Use `self.to_qkv.num_heads` (local heads) instead of total heads for split sizes
|
||||
|
||||
```python
|
||||
from vllm.model_executor.layers.linear import (
|
||||
QKVParallelLinear, RowParallelLinear, ColumnParallelLinear,
|
||||
)
|
||||
|
||||
# Attention: QKV → RowParallel output
|
||||
self.to_qkv = QKVParallelLinear(dim, head_dim, num_heads, num_kv_heads)
|
||||
self.to_out = RowParallelLinear(dim, dim, input_is_parallel=True)
|
||||
|
||||
# FFN: ColumnParallel → RowParallel
|
||||
self.w1 = ColumnParallelLinear(dim, ffn_dim)
|
||||
self.w2 = RowParallelLinear(ffn_dim, dim, input_is_parallel=True)
|
||||
```
|
||||
|
||||
**Constraints**: `num_heads % tp_size == 0` and `num_kv_heads % tp_size == 0`.
|
||||
|
||||
**Test**: `--tensor-parallel-size 2`
|
||||
|
||||
#### 10b. Sequence Parallelism (SP / USP)
|
||||
|
||||
Prefer the declarative `_sp_plan`. For packed variable-length attention or
|
||||
learned-sink LSE correction, keep model math explicit and reuse shared exchange
|
||||
utilities. Validate uneven sequence splits, RoPE coordinates, and outputs
|
||||
against the one-rank oracle.
|
||||
Splits sequence tokens across GPUs. Non-intrusive via `_sp_plan` on the transformer class — no changes to `forward()`.
|
||||
|
||||
**What to change in the transformer**:
|
||||
|
||||
Add `_sp_plan` class attribute:
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.distributed.sp_plan import (
|
||||
SequenceParallelInput, SequenceParallelOutput,
|
||||
)
|
||||
|
||||
class YourTransformer(nn.Module):
|
||||
_sp_plan = {
|
||||
"blocks.0": {
|
||||
"hidden_states": SequenceParallelInput(split_dim=1, expected_dims=3),
|
||||
},
|
||||
"proj_out": SequenceParallelOutput(gather_dim=1, expected_dims=3),
|
||||
}
|
||||
```
|
||||
|
||||
If inline tensor ops (e.g., `torch.cat`) exist between shard/gather points, extract them into `nn.Module` submodules so hooks can intercept them.
|
||||
|
||||
For RoPE that needs splitting, add an entry for the RoPE module with `split_output=True`.
|
||||
|
||||
**Test**: `--ulysses-degree 2` (offline) or `--usp 2` (online serving)
|
||||
|
||||
#### 10c. CFG Parallel
|
||||
|
||||
Confirm the model uses CFG. Then reuse `CFGParallelMixin`, overriding prediction
|
||||
or recombination only for non-standard or multi-output pipelines. Distinguish a
|
||||
packed positive/negative implementation from two independent branches and
|
||||
validate the two-rank result against the packed one-rank oracle.
|
||||
Distributes positive/negative CFG branches across 2 GPUs. Requires the pipeline to inherit `CFGParallelMixin`.
|
||||
|
||||
**What to change in the pipeline**:
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.distributed.cfg_parallel import CFGParallelMixin
|
||||
|
||||
class YourPipeline(nn.Module, CFGParallelMixin):
|
||||
def diffuse(self, ...) -> torch.Tensor:
|
||||
for i, t in enumerate(timesteps):
|
||||
positive_kwargs = {...}
|
||||
negative_kwargs = {...} if do_true_cfg else None
|
||||
noise_pred = self.predict_noise_maybe_with_cfg(
|
||||
do_true_cfg=do_true_cfg, true_cfg_scale=cfg_scale,
|
||||
positive_kwargs=positive_kwargs, negative_kwargs=negative_kwargs,
|
||||
)
|
||||
latents = self.scheduler_step_maybe_with_cfg(
|
||||
noise_pred, t, latents, do_true_cfg
|
||||
)
|
||||
return latents
|
||||
```
|
||||
|
||||
Override `predict_noise()` if your transformer call is non-standard. Override `combine_cfg_noise()` for multi-output models (e.g., video + audio).
|
||||
|
||||
**Constraint**: Exactly 2 GPUs. Only for models using classifier-free guidance.
|
||||
|
||||
**Test**: `--cfg-parallel-size 2`
|
||||
|
||||
#### 10d. HSDP (Hybrid Sharded Data Parallel)
|
||||
|
||||
Declare layer shard conditions and ignored rank-local modules; preserve mixed
|
||||
checkpoint dtypes. HSDP cannot combine with TP. Measure parameter loading,
|
||||
FSDP materialization, warm HBM, and host PSS rather than assuming sharding saves
|
||||
peak memory. Keep the resident layout as default if HSDP is worse.
|
||||
Shards transformer weights via PyTorch FSDP2 to reduce per-GPU VRAM. No code changes to the forward pass — just add a class attribute.
|
||||
|
||||
**What to change in the transformer**:
|
||||
|
||||
```python
|
||||
class YourTransformer(nn.Module):
|
||||
@staticmethod
|
||||
def _is_transformer_block(name: str, module) -> bool:
|
||||
return "blocks" in name and name.split(".")[-1].isdigit()
|
||||
|
||||
_hsdp_shard_conditions = [_is_transformer_block]
|
||||
```
|
||||
|
||||
**Constraint**: Cannot combine with TP. For standalone HSDP, set `hsdp_shard_size` explicitly.
|
||||
|
||||
**Test**: `--use-hsdp` or `DiffusionParallelConfig(use_hsdp=True)`
|
||||
|
||||
#### 10e. Update parallelism documentation
|
||||
|
||||
@@ -469,12 +599,6 @@ Pipelines without `SupportsComponentDiscovery` fall back to scanning
|
||||
well-known attribute names (`transformer`, `text_encoder`, `vae`,
|
||||
etc.), which fails for non-standard names.
|
||||
|
||||
Keep model-specific checkpoint paths, nested block aliases, layout transforms,
|
||||
and component lifecycles in the model package. Change a shared offloader only
|
||||
for a general contract, demonstrate another consumer or a framework-level bug,
|
||||
and add one focused shared regression. Avoid `if ModelName` branches in shared
|
||||
backends.
|
||||
|
||||
### Step 12: Performance Profiling
|
||||
|
||||
After verifying correctness and implementing parallelism/caching, profile the model's performance to identify bottlenecks and ensure optimal execution.
|
||||
@@ -484,12 +608,6 @@ See the [Profiling Single-Stage Diffusion](../../../docs/contributing/profiling.
|
||||
2. Using Nsight Systems (`nsys`) with `profiler: "cuda"` for low-overhead CUDA traces.
|
||||
3. Controlling profiling via `omni.start_profile()` and `omni.stop_profile()`.
|
||||
|
||||
Report cold E2E, warm user latency, steady-state wave time, peak allocated and
|
||||
reserved HBM, host PSS for the full process tree, and stage boundaries. State
|
||||
whether prompt encoding and VAE/audio decoding are included. For multi-device
|
||||
layouts, plot user latency against throughput per device and keep different
|
||||
denoising-step counts on separate Pareto frontiers.
|
||||
|
||||
---
|
||||
|
||||
## Pre-commit conventions
|
||||
@@ -517,7 +635,7 @@ shellcheck). In particular:
|
||||
3. **Use small models**: Test with smaller variants first
|
||||
4. **Check tensor shapes**: Most errors are reshape mismatches in attention
|
||||
5. **Add features incrementally**: Single GPU → TP → SP → CFG → HSDP → Cache-DiT
|
||||
6. **For custom models**: Run the pinned reference separately, then port the runtime natively; do not ship temporary imports from the reference implementation
|
||||
6. **For custom models**: Get the model running with the original code first, then progressively replace components with vllm-omni equivalents
|
||||
7. **Cache-DiT before parallelism tuning**: Cache-DiT is lossy — verify quality at baseline before combining with parallelism
|
||||
8. **Combine lossless + lossy**: e.g., TP + SP + Cache-DiT for maximum throughput
|
||||
|
||||
@@ -526,7 +644,6 @@ shellcheck). In particular:
|
||||
- [vllm-omni-test skill](../vllm-omni-test/SKILL.md) — L1–L4 markers, naming, Buildkite wiring, run commands
|
||||
- [Transformer Adaptation](references/transformer-adaptation.md) — porting transformers from diffusers
|
||||
- [Custom Model Patterns](references/custom-model-patterns.md) — patterns for non-diffusers models
|
||||
- [Native Model Integration Checklist](references/native-model-integration-checklist.md) — ownership boundaries, phase gates, qualification matrix, and review evidence
|
||||
- [Parallelism Patterns](references/parallelism-patterns.md) — TP, SP/USP, CFG parallel, HSDP implementation details
|
||||
- [Cache-DiT Patterns](references/cache-dit-patterns.md) — cache-dit acceleration for standard and custom architectures
|
||||
- [Troubleshooting](references/troubleshooting.md) — common errors and fixes
|
||||
@@ -9,9 +9,7 @@ Three caching strategies:
|
||||
- **TaylorSeer**: Calibration-based prediction using Taylor expansion to estimate block outputs
|
||||
- **SCM** (Step Computation Masking): Dynamic step skipping based on configurable policies
|
||||
|
||||
Speedup is model-, step-, and policy-dependent. Treat a cache-enabled run as
|
||||
integration evidence until logs or a focused test show real hits and repeated
|
||||
timings show an improvement over the uncached baseline.
|
||||
**Typical speedup**: 1.5-2.5x depending on model and configuration.
|
||||
|
||||
**Official docs**: https://docs.vllm.ai/projects/vllm-omni/en/latest/design/feature/cache_dit
|
||||
|
||||
@@ -245,10 +243,8 @@ vllm serve your-model --omni --port 8098 \
|
||||
|
||||
**Verification checklist**:
|
||||
1. Logs show "Cache-dit enabled successfully on xxx"
|
||||
2. A focused test or summary confirms at least one real cache hit
|
||||
3. Quality is compared with `cache_backend=None`
|
||||
4. Performance uses a realistic step count, repeated timings, and a policy
|
||||
whose warmup does not consume the whole run
|
||||
2. Performance: 1.5-2x speedup vs no cache
|
||||
3. Quality: compare output with `cache_backend=None`
|
||||
|
||||
## Excluded Models
|
||||
|
||||
|
||||
@@ -16,22 +16,20 @@ vllm_omni/diffusion/models/wan2_2/
|
||||
|
||||
The transformer is loaded separately via `weights_sources` + `load_weights()`. Non-transformer components (VAE, text encoder) are loaded in `__init__` via `from_pretrained()`.
|
||||
|
||||
### External adapter (not native support)
|
||||
### Custom model with external deps (e.g., DreamID-Omni)
|
||||
|
||||
```
|
||||
vllm_omni/diffusion/models/<name>/
|
||||
vllm_omni/diffusion/models/dreamid_omni/
|
||||
├── __init__.py # Exports pipeline only
|
||||
├── pipeline_<name>.py # Pipeline: loads ALL weights in __init__ via custom helpers
|
||||
└── <backbone>.py # Custom transformer / fusion architecture
|
||||
├── pipeline_dreamid_omni.py # Pipeline: loads ALL weights in __init__ via custom helpers
|
||||
├── fusion.py # Custom fusion architecture (video + audio cross-attention)
|
||||
└── wan2_2.py # Re-implemented Wan backbone with split API
|
||||
|
||||
examples/offline_inference/<task>/
|
||||
└── download_<name>.py # Downloads weights and/or clones code repo
|
||||
examples/offline_inference/x_to_video_audio/
|
||||
└── download_dreamid_omni.py # Downloads weights from 3 HF repos + clones code repo
|
||||
```
|
||||
|
||||
All weights are loaded eagerly in `__init__`. `load_weights()` is a no-op. The
|
||||
external dependency is imported with try/except. Use this pattern only when the
|
||||
requested scope explicitly allows an adapter; it does not satisfy a request for
|
||||
a native vLLM-Omni implementation.
|
||||
All weights loaded eagerly in `__init__`. `load_weights()` is a no-op. External dependency (`dreamid_omni` package) imported with try/except.
|
||||
|
||||
### Custom model with ported code (e.g., BAGEL)
|
||||
|
||||
@@ -78,7 +76,7 @@ self.weights_sources = [
|
||||
]
|
||||
```
|
||||
|
||||
### Pattern 3: Fully custom loading
|
||||
### Pattern 3: Fully custom loading (DreamID-Omni)
|
||||
|
||||
```
|
||||
init → load ALL weights eagerly via custom helpers → load_weights() = no-op
|
||||
@@ -89,7 +87,7 @@ init → load ALL weights eagerly via custom helpers → load_weights() = no-op
|
||||
- `load_weights()` is `pass`
|
||||
- Weights may come from multiple HF repos in different formats (`.pth`, `.safetensors`)
|
||||
|
||||
Use this only for an explicitly approved external adapter when:
|
||||
Use this when:
|
||||
- The original model has complex, well-tested loading code you don't want to rewrite
|
||||
- Weights span multiple HF repos
|
||||
- Weight format is non-standard (e.g., a single `.pth` file, not sharded safetensors)
|
||||
@@ -110,23 +108,16 @@ Standard diffusers `model_index.json`:
|
||||
Custom model `model_index.json` (minimal):
|
||||
```json
|
||||
{
|
||||
"_class_name": "YourCustomPipeline",
|
||||
"fusion": "weights/model.safetensors"
|
||||
"_class_name": "DreamIDOmniPipeline",
|
||||
"fusion": "DreamID-Omni/dreamid_omni.safetensors"
|
||||
}
|
||||
```
|
||||
|
||||
The only **required** field is `_class_name` — it must match a key in
|
||||
`_DIFFUSION_MODELS` in `registry.py`. Other fields are model-specific and
|
||||
accessible via `od_config.model_config`.
|
||||
|
||||
When the released repository cannot be modified and contains neither this file
|
||||
nor a root `config.json`, use the generic native-checkpoint signature resolver.
|
||||
Match the exact Hub ID or a distinctive local file set; never infer support from
|
||||
a model-name substring.
|
||||
The only **required** field is `_class_name` — it must match a key in `_DIFFUSION_MODELS` in `registry.py`. Other fields are model-specific and accessible via `od_config.model_config` dict.
|
||||
|
||||
## External Dependency Management
|
||||
|
||||
### Git clone + .pth injection
|
||||
### Git clone + .pth injection (DreamID-Omni pattern)
|
||||
|
||||
```python
|
||||
def download_dependency():
|
||||
@@ -151,10 +142,6 @@ def download_dependency():
|
||||
|
||||
Copy essential files from the original repo into `vllm_omni/diffusion/models/<name>/`. Adapt imports to use vllm-omni utilities. Benefits: no external dependency, no git clone step. Drawback: must maintain the ported code.
|
||||
|
||||
This is the default for native model support. Keep the reference repository at
|
||||
a pinned revision for architecture analysis and golden outputs, but do not
|
||||
import it from production code.
|
||||
|
||||
## Multi-Modal Input/Output Protocols
|
||||
|
||||
Custom models that handle images, audio, or video I/O should implement protocol classes:
|
||||
@@ -176,7 +163,7 @@ The engine checks `isinstance(pipeline, SupportImageInput)` at startup to config
|
||||
|
||||
Diffusers models use `config.json` in each subfolder. Custom models often use:
|
||||
|
||||
**Module-level config dicts**:
|
||||
**Module-level config dicts** (DreamID-Omni):
|
||||
```python
|
||||
VIDEO_CONFIG = {
|
||||
"patch_size": [1, 2, 2], "model_type": "ti2v",
|
||||
@@ -194,7 +181,7 @@ vae_cfg = bagel_cfg.get("vae_config", {})
|
||||
|
||||
## Custom Architecture Patterns
|
||||
|
||||
### Split forward API
|
||||
### Split forward API (DreamID-Omni)
|
||||
|
||||
When a fusion model needs to interleave blocks from two backbones:
|
||||
|
||||
|
||||
@@ -1,131 +0,0 @@
|
||||
# Native Diffusion Model Integration Checklist
|
||||
|
||||
Use this checklist for native non-Diffusers models, hybrid pipelines, and
|
||||
model-specific optimization work.
|
||||
|
||||
## 1. Freeze the contract before coding
|
||||
|
||||
Record:
|
||||
|
||||
- Reference repository and immutable revision.
|
||||
- Official checkpoint ID, local layout, license, and required auxiliary files.
|
||||
- Pipeline stages: prompt/image encoding, denoising, video/image/audio decoding,
|
||||
and muxing.
|
||||
- Token packing, RoPE coordinates, attention sinks/LSE behavior, MoE layout,
|
||||
scheduler equations, CFG formulation, dtypes, and checkpoint transforms.
|
||||
- Output geometry, frame count/FPS, audio rate/channels, supported tasks, and
|
||||
default generation steps.
|
||||
- Available local hardware versus the official qualification hardware.
|
||||
- Intended default deployment and optional optimization paths.
|
||||
|
||||
State whether the deliverable is a native implementation or an external
|
||||
adapter. For native support, use the reference as an oracle only.
|
||||
|
||||
## 2. Use phase gates
|
||||
|
||||
| Phase | Required exit evidence |
|
||||
|---|---|
|
||||
| Architecture | Component/weight inventory; unsupported reference dependencies identified; ownership decisions documented |
|
||||
| Native correctness | Strict checkpoint load; tiny forward or one-step golden vs pinned reference; output contract validated |
|
||||
| Framework integration | Registry/discovery, shared request types, existing category example, serving route, docs, and recipe work without model-name branching in shared scripts |
|
||||
| Lossless distribution | One-rank oracle vs each supported TP/SP/CFG/HSDP/VAE layout; released-checkpoint smoke for the recommended layout |
|
||||
| Memory optimization | Ordinary/layerwise/DLO behavior, peak HBM, full-tree host PSS, and stage timeline; shared backend unchanged unless a general contract is proven |
|
||||
| Lossy optimization | Cache hit demonstrated; quality delta measured; speed claim made only when warmup permits hits |
|
||||
| Full qualification | Default-resolution E2E at official step count for every supported task; media metadata and hashes recorded |
|
||||
| Review readiness | Focused tests, Ruff/format, strict docs, DCO, current-main rebase, issue/RFC for large changes, concise PR evidence and limitations |
|
||||
|
||||
Do not start performance tuning before native one-step parity. Do not claim a
|
||||
feature from topology validation alone when released weights can exercise it.
|
||||
|
||||
## 3. Keep ownership narrow
|
||||
|
||||
Default placement:
|
||||
|
||||
| Concern | Owner |
|
||||
|---|---|
|
||||
| Architecture, checkpoint remap, packed attention math, nested block alias, auxiliary lifecycle | Model package |
|
||||
| Kernel/version dispatch already useful to multiple models | Shared attention utility |
|
||||
| Request-dependent example defaults | Typed model-extra entry consumed through a registry |
|
||||
| Exact-ID/local-file detection for repositories without standard metadata | Generic native-checkpoint signature table |
|
||||
| General offload/cache/HSDP bug | Shared layer plus one focused shared regression |
|
||||
|
||||
Before changing shared code, answer:
|
||||
|
||||
1. Is this a framework contract or only this checkpoint's layout?
|
||||
2. Can the model expose an existing declarative hook instead?
|
||||
3. Is there a second consumer or a framework-level failing test?
|
||||
4. Can the shared change avoid importing or naming the model?
|
||||
|
||||
If the answers do not justify a shared change, keep it model-local.
|
||||
|
||||
## 4. Reuse examples without leaking model logic
|
||||
|
||||
- Reuse the existing `x_to_y.py` category script.
|
||||
- Resolve the pipeline class through standard metadata or native signature
|
||||
discovery, not path substrings.
|
||||
- Put dynamic model defaults and validation in `model_extras`; let shared
|
||||
examples consume only the generic typed API.
|
||||
- Do not create image/speech/default abstractions without a real consumer.
|
||||
- Add a dedicated example only when the shared request/output protocol cannot
|
||||
express the model.
|
||||
|
||||
## 5. Build a bounded evidence matrix
|
||||
|
||||
Start with the recommended resident layout. Add only supported, useful points:
|
||||
|
||||
- Single request: TP, SP, or hybrid TP/SP.
|
||||
- Throughput: DLO/DP layouts when concurrent requests are valid.
|
||||
- CFG: compare CFG-parallel output with the packed-CFG oracle only if the model
|
||||
actually uses CFG.
|
||||
- HSDP: record loading and materialization peaks; it can use more HBM than a
|
||||
resident SP layout.
|
||||
- VAE patch parallelism: compare decoded tensors or media exactly where the
|
||||
algorithm should be lossless.
|
||||
- Cache: separate cache integration from acceleration evidence.
|
||||
|
||||
With four devices, qualify four-device layouts such as DP4, DP2SP2, SP4, or
|
||||
CFG2xSP2. Do not present CFG2xSP4 or an official eight-device runtime as locally
|
||||
validated.
|
||||
|
||||
For each measured layout record:
|
||||
|
||||
- Cold E2E and warm user latency.
|
||||
- Steady-state wave time and throughput per device.
|
||||
- Prompt, denoise, video decode, audio decode, and mux times.
|
||||
- Peak allocated/reserved HBM and full-process-tree host PSS.
|
||||
- Step count, seed, output geometry, frame/FPS/audio metadata, and parity metric.
|
||||
|
||||
Plot latency versus throughput/device. Keep one-step, four-step, and full-step
|
||||
results separate because they are not quality-equivalent.
|
||||
|
||||
## 6. Keep tests proportional
|
||||
|
||||
Prefer a small set with distinct failure ownership:
|
||||
|
||||
1. Strict weight-key/dtype/load contract.
|
||||
2. Tiny native golden or one-step reference parity.
|
||||
3. One parametrized distributed-parity test covering supported layouts.
|
||||
4. Pipeline/request/output geometry and serving smoke.
|
||||
5. Shared-example/default test when the category script changes.
|
||||
6. One shared regression only for a genuine shared-layer fix.
|
||||
|
||||
Avoid duplicating the same shape or configuration assertion across
|
||||
model-specific files. Keep expensive released-checkpoint evidence in the PR or
|
||||
recipe unless the repository's CI tier explicitly requires it.
|
||||
|
||||
## 7. Write evidence, not marketing
|
||||
|
||||
In the PR description include:
|
||||
|
||||
- A compact architecture/data-flow explanation.
|
||||
- Default deployment and why it is recommended.
|
||||
- Code-change counts by component.
|
||||
- Accuracy and distributed-parity evidence tied to commands/scripts.
|
||||
- Separate Pareto figures for different step counts.
|
||||
- Supported versus validated versus planned optimizations.
|
||||
- Hardware gaps and untested official-runtime claims.
|
||||
|
||||
Call a result a bounded local qualification unless it includes repeated timing
|
||||
and variance. State whether latency includes text/image encoding and all
|
||||
decoders. Preserve the reference revision and exact commands needed to
|
||||
reproduce the evidence.
|
||||
@@ -1,126 +0,0 @@
|
||||
---
|
||||
name: add-recipe
|
||||
description: Add or update an in-repository vLLM-Omni model recipe with verified task, input, output, hardware, command, feature, and validation contracts. Use when creating files under recipes/, restructuring a recipe after review, documenting a newly supported model, or synchronizing recipe claims with support tables and shared feature guides.
|
||||
---
|
||||
|
||||
# Add a vLLM-Omni Recipe
|
||||
|
||||
## Source the contract
|
||||
|
||||
Read [`recipes/TEMPLATE.md`](../../../recipes/TEMPLATE.md) and
|
||||
[`recipes/README.md`](../../../recipes/README.md) before editing. Inspect the
|
||||
canonical model card/repository, implementation, shared examples, tests, and
|
||||
available local qualification evidence.
|
||||
|
||||
Do not infer support from a model name, sibling recipe, registry entry, or
|
||||
topology validator. Mark unexecuted configurations as configuration-only.
|
||||
|
||||
## Route by model family
|
||||
|
||||
Read only the contributor and user documentation relevant to the model:
|
||||
|
||||
| Family | Contributor guide | User-facing documentation to synchronize |
|
||||
|---|---|---|
|
||||
| Diffusion | [`adding a diffusion model`](../add-diffusion-model/SKILL.md) | `docs/user_guide/diffusion_features.md`, applicable diffusion feature guides, and shared image/video/audio examples |
|
||||
| TTS | [`adding a TTS model`](../add-tts-model/SKILL.md) | `examples/offline_inference/text_to_speech/README.md`, `examples/online_serving/text_to_speech/README.md`, and `docs/serving/speech_api.md` |
|
||||
| Omni | [`adding an omni model`](../../../docs/contributing/model/adding_omni_model.md) | model-family offline/online example docs, `docs/serving/chat_completions_api.md`, and `docs/user_guide/feature_compatibility.md` |
|
||||
|
||||
All families must update `docs/models/supported_models.md` and the matching row
|
||||
in `recipes/README.md`. Do not add a modality-specific support document when
|
||||
the repository has no such table; update the closest shared user contract.
|
||||
|
||||
## Structure the recipe
|
||||
|
||||
Keep the recipe task-oriented and use one hardware-specific file named
|
||||
`recipes/<vendor>/<model>-<hardware>.md`. A recipe may contain multiple
|
||||
deployment topologies on that hardware, but each additional accelerator model
|
||||
or platform requires its own suffixed file.
|
||||
|
||||
1. **Summary:** vendor, exact model ID, runtime, modes, named hardware,
|
||||
recommended deployment, and maintainer.
|
||||
2. **Supported model contract:** put task, input, output, and provided-profile
|
||||
tables before setup or commands.
|
||||
3. **References:** link the canonical model source, shared examples, supported
|
||||
model table, and feature matrix.
|
||||
4. **Checkpoint/setup:** pin revisions when possible and state required assets.
|
||||
5. **Hardware:** document the single accelerator named by the file suffix,
|
||||
including every validated device-count/topology profile on it.
|
||||
6. **Commands:** reuse shared `examples/` entrypoints unless the model contract
|
||||
truly requires a dedicated script. Include offline and online paths that
|
||||
were validated.
|
||||
7. **Supported features:** use a compact model-specific topology/status table
|
||||
with links to shared feature guides.
|
||||
8. **Verification:** provide a quick command and exact expected output
|
||||
contract.
|
||||
9. **Qualification evidence:** report correctness, memory, and timing evidence
|
||||
with its measurement scope and caveats.
|
||||
|
||||
## Separate hardware from software
|
||||
|
||||
For every locally validated hardware profile, record:
|
||||
|
||||
- accelerator vendor/model and per-device memory;
|
||||
- number of devices;
|
||||
- interconnect (`NVLink`, `PCIe`, or the platform equivalent);
|
||||
- host memory when CPU staging/offload is material;
|
||||
- whether the profile is runtime-qualified or configuration-only.
|
||||
|
||||
Record OS, Python, driver/runtime, framework versions, and the vLLM-Omni
|
||||
revision in a separate software-environment table. Keep precision, worker
|
||||
count, and DP/TP/SP/PP sizes with the deployment profile or exact command.
|
||||
|
||||
Never generalize a result from one accelerator family to another or combine
|
||||
NVIDIA, AMD, NPU, or distinct accelerator models in one recipe. Create a
|
||||
separate hardware-suffixed recipe and `recipes/README.md` row. Distinguish
|
||||
upstream requirements from hardware exercised by the PR.
|
||||
|
||||
## Keep shared documentation concise
|
||||
|
||||
Link the applicable diffusion, TTS, omni, serving, or design guide for generic
|
||||
feature semantics and launch instructions. The recipe should contain only the
|
||||
model-specific status, valid topology, required flag difference, and evidence
|
||||
boundary.
|
||||
|
||||
Put unsupported combinations in the same feature table. Avoid repeating the
|
||||
same model-specific prose below the global feature matrix; the recipe is the
|
||||
detailed source of truth.
|
||||
|
||||
## Report evidence precisely
|
||||
|
||||
For every performance or memory value, state:
|
||||
|
||||
- checkpoint/revision and workload;
|
||||
- accelerator and device count;
|
||||
- precision and topology;
|
||||
- cold/warm scope, step count, and concurrency;
|
||||
- peak HBM and host-memory metric when measured;
|
||||
- output/quality guard and sample count;
|
||||
- monitoring overhead or other known variance.
|
||||
|
||||
Call a single bounded run qualification evidence, not a benchmark. Do not add
|
||||
a benchmark script unless the contribution explicitly requires one.
|
||||
|
||||
## Synchronize repository documentation
|
||||
|
||||
Update all applicable locations:
|
||||
|
||||
- the row in `recipes/README.md`;
|
||||
- `docs/models/supported_models.md`;
|
||||
- the family-specific documentation selected in **Route by model family**;
|
||||
- shared example documentation when commands or flags change.
|
||||
|
||||
Keep global support tables compact. Link to the recipe instead of duplicating
|
||||
its deployment explanation.
|
||||
|
||||
## Validate
|
||||
|
||||
Run, at minimum:
|
||||
|
||||
```bash
|
||||
pre-commit run --files <changed recipe/docs/skill files>
|
||||
mkdocs build --strict
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Remove generated documentation artifacts after validation. Confirm every
|
||||
recipe link resolves and every claimed profile has matching code or evidence.
|
||||
@@ -345,8 +345,8 @@ Heavier scenarios in the same file use **`advanced_model` only** (streaming, ext
|
||||
@pytest.mark.tts
|
||||
@hardware_test(res={"cuda": "L4"}, num_cards=1)
|
||||
@pytest.mark.parametrize("omni_server", tts_server_params, indirect=True)
|
||||
def test_voice_clone_en_non_streaming_001(omni_server, online_client) -> None:
|
||||
online_client.send_audio_speech_request({...})
|
||||
def test_voice_clone_en_non_streaming_001(omni_server, openai_client) -> None:
|
||||
openai_client.send_audio_speech_request({...})
|
||||
```
|
||||
|
||||
**L4 consolidation:** Prefer parametrized `OmniServerParams` rows (`default`, `async_chunk`, feature flags) in one expansion module rather than many merge-only files ([#1832](https://github.com/vllm-project/vllm-omni/issues/1832)).
|
||||
@@ -363,8 +363,8 @@ def test_voice_clone_en_non_streaming_001(omni_server, online_client) -> None:
|
||||
|
||||
| Path | Fixture | Call |
|
||||
|------|---------|------|
|
||||
| Online `/v1/*` | `online_client` | `online_client.send_*_request(request_config)` |
|
||||
| Offline inference | `offline_client` | `offline_client.send_*_request(request_config)` |
|
||||
| Online `/v1/*` | `openai_client` | `openai_client.send_*_request(request_config)` |
|
||||
| Offline inference | `omni_runner_handler` | `omni_runner_handler.send_*_request(request_config)` |
|
||||
|
||||
1. **Grep `runtime.py` first** — reuse `send_omni_request`, `send_diffusion_request`, `send_audio_speech_request` (online + offline Qwen-style TTS), `send_single_stage_tts_request` (Coqui XTTS / MOSS-TTS-Nano offline), etc.
|
||||
2. **No matching helper** → add `send_<feature>_request` (or `send_<route>_http_request` for negative/dfx) in **`runtime.py`** with general `assert_*` bundled inside, **then** call it from the test.
|
||||
@@ -396,7 +396,7 @@ See [vllm-omni-test skill](../vllm-omni-test/SKILL.md) § **Runtime send helpers
|
||||
`/health` return non-200 until you're actually ready.
|
||||
- **Mark tests per the CI Levels table** — baseline smoke: `core_model` + `advanced_model`; heavier cases: `advanced_model` only; L4 expansion: `full_model`
|
||||
- **No per-model helper modules** — do not create `tests/helpers/{slug}.py`; keep constants and `request_config` payloads in the test file
|
||||
- **Online and offline e2e go through `runtime.py`** — `online_client.send_audio_speech_request` (online); `offline_client.send_audio_speech_request` (Qwen-style offline) or `send_single_stage_tts_request` (single-stage offline). Add a new `send_*_request` in `runtime.py` when none fits; do not embed `omni.generate` or HTTP in tests
|
||||
- **Online and offline e2e go through `runtime.py`** — `openai_client.send_audio_speech_request` (online); `omni_runner_handler.send_audio_speech_request` (Qwen-style offline) or `send_single_stage_tts_request` (single-stage offline). Add a new `send_*_request` in `runtime.py` when none fits; do not embed `omni.generate` or HTTP in tests
|
||||
|
||||
## Phase 4: Async Chunk (Streaming)
|
||||
|
||||
@@ -499,13 +499,6 @@ Common pre-commit failures, recovery commands for missing sign-off, and the
|
||||
full `pre-commit run` invocation for a TTS model:
|
||||
[references/precommit-dco.md](references/precommit-dco.md).
|
||||
|
||||
## Documentation and recipe
|
||||
|
||||
Follow the [`add-recipe` skill](../add-recipe/SKILL.md) to add or update the
|
||||
in-repository model-family recipe, the `recipes/README.md` index, the supported
|
||||
models table, TTS example documentation, and the speech API contract using
|
||||
validated evidence.
|
||||
|
||||
## Integration Checklist
|
||||
|
||||
Use this checklist when integrating a new TTS model:
|
||||
@@ -550,7 +543,6 @@ Use this checklist when integrating a new TTS model:
|
||||
- [ ] Buildkite entries match level: `test-ready.yml` / `test-merge.yml` or nightly TTS job / `test-nightly.yml`
|
||||
- [ ] Gradio demo working
|
||||
- [ ] Documentation added (offline + online docs, nav, supported models)
|
||||
- [ ] In-repository recipe and `recipes/README.md` row added via the `add-recipe` skill
|
||||
|
||||
### Phase 4: Async Chunk
|
||||
- [ ] Pipeline declares the async handoff processor
|
||||
|
||||
@@ -34,7 +34,7 @@ but the changes are correct — stage the modified files and re-commit.
|
||||
| `check-forbidden-imports` | Stdlib `re`/`base64`, pickle, Hugging Face Hub API, or direct Triton/TileLang | `import regex as re` and `pybase64`; use `vllm.transformers_utils.repo_utils`. Do **not** add the file to `allowed_files` without review |
|
||||
| `check-torch-cuda-call` | New `torch.cuda.*` call site | Use `current_omni_platform` / `OmniPlatform`; do not grow `ALLOWED_FILES` without review |
|
||||
| `check-tts-adapter-migration` | New `self._tts_model_type` branch in `serving_speech.py` | Put per-model logic in `tts_adapters/`. Lower `MAX_MODEL_TYPE_BRANCHES` when removing branches; do not raise it without review |
|
||||
| `check-mark` | New `tests/**/test_*.py` missing level/hardware mark, or direct `pytest.mark.H100` | Add `core_model`/`advanced_model`/… plus `cpu`/`cuda`/`hardware_test(`; SKU marks only via helpers |
|
||||
| `check-test-ci-coverage` | New `tests/**/test_*.py` missing level or hardware mark | Add `core_model`/`advanced_model`/… plus `cpu`/`cuda`/`hardware_test(` |
|
||||
| `mypy-3.10` | Type error on changed `vllm_omni/` files | Fix the types; model trees are excluded. Extra versions: `pre-commit run --hook-stage manual mypy-3.12` |
|
||||
| `markdownlint-cli2` | Docs/README markdown lint | Accept auto-fix under `docs/` / `recipes/` / root README |
|
||||
| `check-buildkite` | Invalid `.buildkite/*.yml` | Fix the YAML; do not grow `SKIP_FILES` without review |
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/brand-promo-video-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/co-op-game-intro-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/h3-prompt-writing
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/handdrawn-live-video-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/minimalist-product-ad-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/music-video-subtitle-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/paper-collage-explainer-generator
|
||||
@@ -1 +0,0 @@
|
||||
../../.agents/skills/papercraft-stop-motion-explainer
|
||||
@@ -1,458 +0,0 @@
|
||||
---
|
||||
name: production-add-diffusion-model
|
||||
description: Productionize a vLLM-Omni diffusion model after its Day-0 vertical slice works. Use when work requires official API parity and input limits, feature-combination evidence, online FP8, distributed layerwise offload, per-request Cache-DiT, CUDA/ROCm/NPU/XPU recipes, faster or sparse attention, fused operators, USP or disaggregation analysis, continuous batching and abort handling, long-running RPS stability, or accuracy/performance/reliability CI. For initial architecture porting, registry wiring, and basic weight loading, use add-diffusion-model first.
|
||||
---
|
||||
|
||||
# Productionizing a Diffusion Model
|
||||
|
||||
## Scope
|
||||
|
||||
Use this skill to move a working model integration from Day-0 support to a
|
||||
measured, fail-closed production deployment. It is deliberately separate from
|
||||
[`add-diffusion-model`](../add-diffusion-model/SKILL.md); do not modify or copy
|
||||
that skill when productionizing a model. If registry, pipeline, or basic loader
|
||||
work is incomplete, read that skill first and finish the Day-0 vertical slice.
|
||||
|
||||
MiniMax-H3 [PR #5691](https://github.com/vllm-project/vllm-omni/pull/5691)
|
||||
is the merged Day-0 case study. Its open optimization
|
||||
[issue #5700](https://github.com/vllm-project/vllm-omni/issues/5700) is an
|
||||
actively maintained roadmap and evidence map, not a list of already supported
|
||||
features. It can lag merged implementation: read its current revision, cited
|
||||
PRs, target source, and maintained recipes, then leave every unverified row
|
||||
`not tested` until the target revision has scoped evidence.
|
||||
|
||||
Read these references before the corresponding gate:
|
||||
|
||||
- [API and hardware recipes](references/api-and-recipes.md)
|
||||
- [Production feature patterns](references/feature-patterns.md)
|
||||
- [Performance patterns and code examples](references/performance-patterns.md)
|
||||
- [Serving, soak, and CI validation](references/production-validation.md)
|
||||
|
||||
## Non-negotiable evidence contract
|
||||
|
||||
Create a matrix whose row key is at least:
|
||||
|
||||
```text
|
||||
task x API mode x execution mode/capacity x shape/schedule x attention backend x
|
||||
packed layout x cache policy x quantization x offload mode x topology x
|
||||
hardware x dtype x output representation/transport
|
||||
```
|
||||
|
||||
Use exactly these states:
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| `validated` | The exact row passed correctness and measured deployment tests, with reproducible artifacts. |
|
||||
| `limited` | A bounded subset passed; the limitation and rejection/fallback behavior are explicit. |
|
||||
| `unsupported` | A stable architectural/platform restriction is proven and fails early with an actionable error. |
|
||||
| `not tested` | No adequate evidence exists. This is the default. |
|
||||
|
||||
Never infer compatibility from import success, server startup, another task,
|
||||
another card, or another topology. Record the model/checkpoint revision,
|
||||
vLLM-Omni commit, commands, request assets and hashes, seed/schedule/shape,
|
||||
raw output artifacts, environment, and result for every `validated` row.
|
||||
|
||||
## Workflow
|
||||
|
||||
Follow the gates in order. Keep a dense BF16 single-device path as the
|
||||
correctness oracle until every advertised fast path has parity evidence.
|
||||
|
||||
### Gate 0: Freeze the official contract
|
||||
|
||||
Pin the official implementation and checkpoint revision. Build a matrix for:
|
||||
|
||||
- constructor/call arguments, defaults, scheduler and sigma schedule;
|
||||
- task-to-source modality/count/order rules;
|
||||
- prompt, MIME, bytes, pixels, frames, duration, FPS, steps, seed, guidance,
|
||||
output count, and output format limits;
|
||||
- offline API and every public sync/async serving endpoint;
|
||||
- checkpoint partitions and task selection.
|
||||
|
||||
Normalize offline, sync, and async requests through one validation contract.
|
||||
Reject an invalid task/source combination before downloads, upload persistence,
|
||||
temporary files, async job creation, or engine submission. Use 400 for invalid
|
||||
semantics and 413 for payload limits; clean partial resources on every failure.
|
||||
See [API and hardware recipes](references/api-and-recipes.md).
|
||||
|
||||
### Gate 1: Re-prove correctness under production shapes
|
||||
|
||||
1. Make loading strict: track expected, loaded, unexpected, and intentionally
|
||||
ignored tensors. A fused destination is complete only after every source
|
||||
shard arrives. Missing retained parameters abort startup.
|
||||
2. Fix reference revision, asset hashes, seed, schedule, and shape. Compare
|
||||
component outputs, intermediate denoise latents, and final artifacts.
|
||||
3. Test unequal packed samples. Verify `cu_seqlens`, padding exclusion,
|
||||
modality/CFG ownership, split/gather boundaries, and one output per request.
|
||||
4. Test partial process groups with non-member ranks and group size smaller
|
||||
than world size. Do not construct a private group per component or request.
|
||||
5. Run negative API cases before positive E2E tests to prove side-effect-free
|
||||
rejection and cleanup.
|
||||
|
||||
### Gate 2: Wire shared fast primitives
|
||||
|
||||
Prefer explicit, testable vLLM-Omni/vLLM operators with native fallbacks.
|
||||
`torch.compile` remains a separately validated regional optimization; it is
|
||||
not a substitute for correct operator selection or lifecycle-safe boundaries.
|
||||
|
||||
#### RMSNorm + RoPE baseline
|
||||
|
||||
Use the shared cross-platform operators, preserving the official Q/K norm and
|
||||
RoPE order:
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
||||
from vllm_omni.diffusion.layers.rope import (
|
||||
RotaryEmbedding,
|
||||
apply_rope_to_qk,
|
||||
)
|
||||
|
||||
self.norm_q = RMSNorm(head_dim, eps=eps)
|
||||
self.norm_k = RMSNorm(head_dim, eps=eps)
|
||||
self.rope = RotaryEmbedding(is_neox_style=False)
|
||||
|
||||
query = self.norm_q(query) # [B, S, Hq, D] or packed [T, Hq, D]
|
||||
key = self.norm_k(key) # [B, S, Hkv, D] or packed [T, Hkv, D]
|
||||
query, key = apply_rope_to_qk(
|
||||
self.rope,
|
||||
query,
|
||||
key,
|
||||
(cos, sin),
|
||||
)
|
||||
```
|
||||
|
||||
Verify `is_neox_style`, `half_head_dim`, partial rotary dimensions, cos/sin
|
||||
layout, and packed row ownership against the official model. `False` means
|
||||
interleaved/GPT-J style; do not assume the default is NeoX. Shared RMSNorm and
|
||||
RoPE are two dispatched operators, not automatically one fused kernel.
|
||||
|
||||
For packed non-interleaved RoPE, use the shared fused boundary when its contract
|
||||
matches the model:
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.layers.fused_qk_norm_rope import fused_qk_norm_rope
|
||||
|
||||
query, key = fused_qk_norm_rope(
|
||||
query, # [T, Hq, D]
|
||||
key, # [T, Hkv, D]
|
||||
self.norm_q.weight, # [D]
|
||||
self.norm_k.weight, # [D]
|
||||
rope_table, # [T, rotary_dim] = [cos | sin]
|
||||
self.norm_q.variance_epsilon,
|
||||
)
|
||||
```
|
||||
|
||||
The current CUDA fast path specializes BF16 `head_dim=128`, `rotary_dim=96`;
|
||||
the public function keeps an eager fallback for unsupported inputs. Verify the
|
||||
packed row/frequency contract and trace the actual fast path. Do not reshape an
|
||||
incompatible official RoPE layout merely to enter this kernel.
|
||||
|
||||
MiniMax-H3-style partial NeoX RoPE rotates a prefix and passes the suffix
|
||||
through:
|
||||
|
||||
```python
|
||||
self.q_norm = RMSNorm(head_dim, eps=eps, dtype=torch.bfloat16)
|
||||
self.k_norm = RMSNorm(head_dim, eps=eps, dtype=torch.bfloat16)
|
||||
self.rope = RotaryEmbedding(is_neox_style=True, half_head_dim=False)
|
||||
|
||||
def apply_partial_rope(self, x, freqs, rot_dim):
|
||||
x_rot, x_pass = x[..., :rot_dim], x[..., rot_dim:]
|
||||
cos, sin = torch.cos(freqs).to(x.dtype), torch.sin(freqs).to(x.dtype)
|
||||
return torch.cat((self.rope(x_rot, cos, sin), x_pass), dim=-1)
|
||||
```
|
||||
|
||||
The partial-RoPE snippet assumes `import torch` and lives on the owning
|
||||
attention module; adapt names and validated frequency shapes rather than
|
||||
copying it at module scope.
|
||||
|
||||
Call the modules normally so `CustomOp` selects CUDA/HIP/NPU/XPU/native
|
||||
implementations. Do not call `forward_cuda()` directly. Compare the shared
|
||||
path with native BF16 at operator, block, denoise-trajectory, and artifact
|
||||
levels before claiming either accuracy or speed.
|
||||
|
||||
#### Faster BF16 attention and fused projections
|
||||
|
||||
Use local TP head counts and role-aware shared attention:
|
||||
|
||||
```python
|
||||
from vllm.model_executor.layers.linear import QKVParallelLinear
|
||||
from vllm_omni.diffusion.attention.layer import Attention
|
||||
|
||||
self.qkv_proj = QKVParallelLinear(
|
||||
hidden_size=hidden_size,
|
||||
head_size=head_dim,
|
||||
total_num_heads=num_heads,
|
||||
total_num_kv_heads=num_kv_heads,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj",
|
||||
)
|
||||
self.attn = Attention(
|
||||
num_heads=self.qkv_proj.num_heads,
|
||||
num_kv_heads=self.qkv_proj.num_kv_heads,
|
||||
head_size=head_dim,
|
||||
softmax_scale=head_dim**-0.5,
|
||||
causal=False,
|
||||
qkv_layout="BSND",
|
||||
role="self",
|
||||
prefix=prefix,
|
||||
)
|
||||
```
|
||||
|
||||
Prove the selected backend from runtime logs/trace and compare it with dense
|
||||
SDPA. Packed metadata must express real sample boundaries; do not build masks
|
||||
that the selected backend ignores. A backend that is faster for one shape or
|
||||
platform is not a global default. See [performance patterns](references/performance-patterns.md)
|
||||
for QKV, SwiGLU, AdaLN, sparse attention, redundancy, and fusion examples.
|
||||
|
||||
### Gate 3: Add online FP8 without breaking the loader
|
||||
|
||||
Route the active config to each component, then pass a stable runtime prefix
|
||||
and config into every quantizable vLLM linear:
|
||||
|
||||
```python
|
||||
def resolve_component_quant_config(quant_config, component):
|
||||
return quant_config.resolve(component) if hasattr(quant_config, "resolve") else quant_config
|
||||
|
||||
transformer_quant_config = resolve_component_quant_config(
|
||||
od_config.quantization_config,
|
||||
"transformer",
|
||||
)
|
||||
self.transformer = MyDiT(
|
||||
od_config,
|
||||
quant_config=transformer_quant_config,
|
||||
prefix="transformer",
|
||||
)
|
||||
|
||||
self.qkv_proj = QKVParallelLinear(
|
||||
hidden_size=hidden_size,
|
||||
head_size=head_dim,
|
||||
total_num_heads=num_heads,
|
||||
total_num_kv_heads=num_kv_heads,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj",
|
||||
)
|
||||
```
|
||||
|
||||
Use `build_quant_config()` for direct construction/testing:
|
||||
|
||||
```python
|
||||
from vllm_omni.quantization import build_quant_config
|
||||
|
||||
quant_config = build_quant_config({
|
||||
"transformer": {"method": "fp8"},
|
||||
"text_encoder": None,
|
||||
"vae": None,
|
||||
"default": None,
|
||||
})
|
||||
```
|
||||
|
||||
Preserve each parameter's vLLM `weight_loader`; perform checkpoint layout
|
||||
conversion before invoking it. Account for every QKV/gate-up source shard.
|
||||
Keep precision-sensitive norm/modulation/embedders in BF16 unless separately
|
||||
proven, using the current `safe_quant_config` pattern where applicable.
|
||||
|
||||
Validate resident DiT FP8 first: prove which modules are quantized, HBM saved,
|
||||
BF16-vs-FP8 trajectory/artifact quality, latency, and throughput. Text encoder,
|
||||
VAE, other hardware, TP/cache/DLO combinations are separate rows.
|
||||
Pre-quantized, pruned, rotated, or adapter-modified checkpoints are separate
|
||||
model/loading lanes; compare them with the released BF16 model without calling
|
||||
the result a pure quantization ablation. Read
|
||||
[`../quantization/SKILL.md`](../quantization/SKILL.md) if present and prefer the
|
||||
target revision's implementation/docs when sibling guidance differs.
|
||||
|
||||
### Gate 4: Make every component offloadable
|
||||
|
||||
Declare topology instead of relying on heuristic discovery:
|
||||
|
||||
```python
|
||||
from typing import ClassVar
|
||||
|
||||
import torch.nn as nn
|
||||
|
||||
from vllm_omni.diffusion.models.interface import SupportsComponentDiscovery
|
||||
from vllm_omni.diffusion.offloader import OffloadPlan
|
||||
|
||||
class MyPipeline(nn.Module, SupportsComponentDiscovery):
|
||||
_dit_modules: ClassVar[list[str]] = ["transformer"]
|
||||
_encoder_modules: ClassVar[list[str]] = ["text_encoder"]
|
||||
_vae_modules: ClassVar[list[str]] = ["vae"]
|
||||
_resident_modules: ClassVar[list[str]] = []
|
||||
_offload_plan: ClassVar[OffloadPlan] = OffloadPlan(
|
||||
on_demand_component_paths=frozenset({"text_encoder", "vae"}),
|
||||
block_attrs={"transformer": ("blocks",)},
|
||||
offload_submodules={"token_refiner": "blocks"},
|
||||
resident_dit_paths=frozenset({"transformer"}),
|
||||
encoder_block_attrs={"text_encoder": ("encoder.layers",)},
|
||||
)
|
||||
```
|
||||
|
||||
Add a test that resolves every dotted component/block path and fails if it is
|
||||
missing or is not the expected module/container; discovery may otherwise warn
|
||||
and skip. Validate ordinary layerwise offload and distributed layerwise
|
||||
offload (DLO) independently.
|
||||
|
||||
Keep component lifecycle and host-weight storage separate. `OffloadPlan`
|
||||
declares what can be streamed; the diffusion loader owns any `HostWeightPlan`
|
||||
and hands the exact prevalidated plan to DLO. Do not make the offloader rescan
|
||||
checkpoint files or duplicate loader name/shape/dtype decisions. Direct mmap is
|
||||
a fail-closed optimization: current preflight requires TP1, no HSDP, no online
|
||||
quantization, complete bindings, and represented transforms. Otherwise the
|
||||
ordinary loader remains authoritative.
|
||||
|
||||
If the target revision supports Host Weight Runtime (HWR), treat it as a
|
||||
separate, exact-identity startup/cache path rather than generic offload or
|
||||
zero-copy execution. Qualify `preferred` population/fallback and `required`
|
||||
consume-only failure independently; verify immutable manifests, final-layout
|
||||
restore, lease cleanup, corruption/quarantine, and node-local sharing. Current
|
||||
model contracts may be BF16 no-AllGather-only, so reject AllGather, online
|
||||
quantization, HSDP, LoRA/adapted weights, and unrecognized load formats unless
|
||||
the exact target revision explicitly adds them.
|
||||
|
||||
On small-HBM cards, record load/materialization peak, resident peak, encode,
|
||||
each denoise wave, decode, transient per-rank HBM, host PSS, H2D, and collective
|
||||
time. Test DLO AllGather and `--dlo-no-use-allgather` as different deployments.
|
||||
Sweep the supported resident-block policies and publish the non-dominated
|
||||
latency-HBM frontier; a single minimum-memory or fastest point is not the DLO
|
||||
trade-off. Keep request-specific preparation replica-local. Only weight
|
||||
materialization may enter the declared DLO collective group; never use WORLD
|
||||
for per-request work in a DP deployment.
|
||||
Direct checkpoint mmap preflight is limited to TP1 without HSDP or online
|
||||
quantization; TP>1 falls back to the ordinary TP-aware loader and can still feed
|
||||
DLO AllGather or no-AllGather after scoped E2E. HSDP remains incompatible with
|
||||
DLO AllGather. A target revision may allow finalized per-tensor online FP8
|
||||
through the ordinary loader and AllGather; keep every other online quantizer
|
||||
fail-closed, and require model/card/topology E2E before promotion.
|
||||
Use deploy YAML for DP under `--omni`; vLLM DP CLI flags are rejected.
|
||||
|
||||
### Gate 5: Add request-scoped acceleration safely
|
||||
|
||||
Server-wide Cache-DiT support does not imply a per-request quality API. Only
|
||||
add request policy after defining calibrated quality tiers and a safe hook
|
||||
lifecycle using `SupportsRequestScopedCacheDiT`, `CacheDiTRequestSpec`, and
|
||||
`RequestScopedCacheDiTRuntime`. Validate alternating and concurrent quality
|
||||
tiers, real cache hits, refresh/reset, success, error, disconnect, abort, and
|
||||
the next uncached request. Do not batch different quality policies unless the
|
||||
batch compatibility key and hook ownership make that safe.
|
||||
|
||||
Then test Cache-DiT against FP8, DLO, TP/SP, compile, and batching one
|
||||
combination at a time. See [production feature patterns](references/feature-patterns.md)
|
||||
for the lifecycle skeleton and compatibility matrix.
|
||||
|
||||
### Gate 6: Optimize only from profiles
|
||||
|
||||
Freeze one canonical workload manifest before profiling. Keep lossless runtime
|
||||
A/Bs, accelerated paths with numerical/quality trade-offs, and production
|
||||
topology studies in separate result lanes. Reconcile the client boundary as
|
||||
queue + encode + denoise + video/audio decode + output transport/codec +
|
||||
residual; do not hide a material residual in another stage.
|
||||
|
||||
Profile stage time, GPU kernels, CPU/GPU synchronization, allocations, H2D,
|
||||
output payload/copies, and collectives before choosing work. Apply and A/B one
|
||||
change at a time:
|
||||
|
||||
1. Remove synchronization and redundant denoise-loop work: hoist static prompt
|
||||
conditioning/token refinement, RoPE tables, masks/metadata, bounded AdaLN
|
||||
schedule projections, and reference scans; remove per-layer `.item()` and
|
||||
unused masks/allocations.
|
||||
2. Select the fastest correct BF16 attention backend for each role and shape.
|
||||
3. Prefer explicit fused QKV, gate-up + `SiluAndMul`, RMSNorm/RoPE, AdaLN, and
|
||||
layout/residual operators with guarded native fallbacks. Preserve every
|
||||
materialized dtype/rounding boundary consumed by the unfused graph.
|
||||
4. Enable sparse attention only when platform, shape, topology, realized
|
||||
sparsity, quality, fallback, and end-to-end speedup are all proven.
|
||||
5. Profile USP all-to-all count/bytes, packing, contiguous copies, overlap, and
|
||||
scaling efficiency. Audit replicated cross-attention before using
|
||||
`skip_sequence_parallel`. Qualify regular and accelerated Ulysses transport
|
||||
independently; activation logs, JIT/readiness time, workspace growth,
|
||||
stream ownership, maximum-shape warmup, and numerical drift are gates.
|
||||
Preserve the checkpoint's Q:KV head ratio when padding GQA for Ulysses, and
|
||||
validate strided QKV staging rather than assuming packed contiguity.
|
||||
6. Evaluate text-encoder or VAE disaggregation only if stage share, transfer
|
||||
volume, reuse/concurrency, and independent scaling justify it. Tiling,
|
||||
offload, or patch parallelism is not disaggregation.
|
||||
|
||||
Do not combine independent optimizations into one performance claim. Preserve
|
||||
raw before/after runs with fixed workload and warmup policy.
|
||||
|
||||
### Gate 7: Make serving interruptible and load-stable
|
||||
|
||||
Treat request-mode batching and step-wise execution as separate capabilities:
|
||||
|
||||
- request batching requires `supports_request_batch = True` and
|
||||
`DiffusionRequestBatch -> list[DiffusionOutput]`, exactly one output per
|
||||
request;
|
||||
- step execution requires the complete `SupportsStepExecution` contract:
|
||||
`prepare_encode`, `denoise_step`, `step_scheduler`, and `post_decode`.
|
||||
|
||||
Keep all mutable scheduler, RNG, latent, mask, cache, and step state request
|
||||
scoped. First validate `--step-execution --max-num-seqs 1` and step-boundary
|
||||
abort; only then test heterogeneous multi-request waves. Step continuous
|
||||
batching is experimental and is not automatically compatible with Cache-DiT.
|
||||
Before recommending it, predeclare a useful case and success threshold, then
|
||||
A/B it against request mode under the same arrival process. Structural support
|
||||
or a generic bridge is not latency, throughput, or cancellation evidence for a
|
||||
model.
|
||||
|
||||
Test queued and in-flight abort, client disconnect, timeout, OOM, worker error,
|
||||
and restart. Require one terminal result, idempotent cleanup, no post-decode
|
||||
after abort, no cache/temp/VRAM/request-ID leak, and a successful next request.
|
||||
Inject late worker results/exceptions after cancellation and prove the result
|
||||
pump stays alive instead of raising `InvalidStateError`. For large batched or
|
||||
long-window video outputs, freeze dtype, range, layout, contiguity, ownership,
|
||||
payload size, serializer/codec limits, and offline-versus-HTTP semantics.
|
||||
Validate device-side output preparation, D2H/IPC or shared handles, remote
|
||||
encoding, event-loop responsiveness, and client materialization as separate
|
||||
stages, including abort and consumer failure cleanup.
|
||||
A model/VAE callback that publishes ordered chunks is only a producer contract;
|
||||
it is not transport, backpressure, cancellation, public streaming, or
|
||||
time-to-first-frame evidence. Distinguish complete-response faster-than-playback
|
||||
(`client E2E / output duration <= 1`) from streaming and report first-chunk,
|
||||
steady cadence, finalization, and complete-artifact latency separately.
|
||||
Run below-, near-, and above-saturation mixed-RPS soaks and report success/error
|
||||
rate, throughput, queue time, memory slope, temp growth, and worker health.
|
||||
Report p50/p95/p99 only from a declared arrival model with enough measured
|
||||
requests for the claimed percentile. See
|
||||
[serving and CI validation](references/production-validation.md).
|
||||
|
||||
### Gate 8: Publish recipes and four independent CI tracks
|
||||
|
||||
For every target vendor/card/topology, publish exact environment, serve/deploy
|
||||
command, one complete JSON or multipart curl per supported task, expected
|
||||
output checks, benchmark command, warmup/repeat/concurrency policy, raw metrics,
|
||||
per-rank HBM/host PSS, accuracy artifact, and limitations. Mark each
|
||||
CUDA/ROCm/NPU/XPU row `validated`, `unsupported`, or `not tested`; never inherit
|
||||
support from the platform abstraction or another card. Re-run DLO on each
|
||||
small-HBM recipe.
|
||||
|
||||
Maintain separate CI gates:
|
||||
|
||||
1. Function: strict load, API/offline positive and negative contracts.
|
||||
2. Accuracy: fixed reference revision/assets/seed/schedule and scoped tolerance.
|
||||
3. Performance: fixed best deployment plus memory-constrained/DLO row, raw JSON,
|
||||
explicit warmup, enough samples for each claimed statistic, and owned
|
||||
regression thresholds.
|
||||
4. Reliability: long mixed-RPS soak, abort/disconnect/fault cleanup, memory and
|
||||
temporary-resource trend.
|
||||
|
||||
## Production Definition of Done
|
||||
|
||||
A model is production-ready only when:
|
||||
|
||||
- every official advertised task passes API parity, offline/online parity,
|
||||
strict loading, fixed-reference quality, and negative validation;
|
||||
- the recommended deployment row passes correctness, isolation, abort/error
|
||||
cleanup, benchmark, and soak gates on named hardware;
|
||||
- online FP8 and DLO are each implemented and either validated in scoped rows
|
||||
or explicitly limited/unsupported with fail-fast behavior;
|
||||
- per-request cache quality tiers have measured speed/quality frontiers and
|
||||
request isolation, if advertised;
|
||||
- CUDA/ROCm/NPU/XPU recipes describe evidence rather than inferred support;
|
||||
- Function, Accuracy, Performance, and Reliability CI have owners, artifacts,
|
||||
and actionable failure output;
|
||||
- accelerated paths identify every precision/quality trade-off, and the raw
|
||||
offline plus online encoded output contracts are versioned and tested;
|
||||
- all unverified task/backend/cache/quant/offload/topology/hardware combinations
|
||||
remain `not tested` and are not presented as supported.
|
||||
|
||||
Day-0 support is still a valid milestone, but title, docs, and matrix must say
|
||||
which representative task passed and which production gates remain.
|
||||
@@ -1,4 +0,0 @@
|
||||
interface:
|
||||
display_name: "Production Add Diffusion Model"
|
||||
short_description: "Productionize vLLM-Omni diffusion models"
|
||||
default_prompt: "Use $production-add-diffusion-model to take this integrated diffusion model from Day-0 support to evidence-backed production readiness."
|
||||
@@ -1,315 +0,0 @@
|
||||
# API Contract and Hardware Recipes
|
||||
|
||||
## Contents
|
||||
|
||||
1. [Freeze the upstream contract](#freeze-the-upstream-contract)
|
||||
2. [Validate before side effects](#validate-before-side-effects)
|
||||
3. [Task request templates](#task-request-templates)
|
||||
4. [Serving benchmark template](#serving-benchmark-template)
|
||||
5. [Hardware recipe contract](#hardware-recipe-contract)
|
||||
6. [Best deployment selection](#best-deployment-selection)
|
||||
|
||||
## Freeze the upstream contract
|
||||
|
||||
Pin three revisions independently:
|
||||
|
||||
- official source implementation commit or package version;
|
||||
- checkpoint/model-card revision and partition;
|
||||
- vLLM-Omni target commit.
|
||||
|
||||
Build a table before editing code. One row is one official task/partition.
|
||||
|
||||
| Field | Required content |
|
||||
|---|---|
|
||||
| Task and partition | Official task name, checkpoint subdirectory, task selection rules |
|
||||
| Call contract | Constructor and call arguments, types, required/optional, official defaults |
|
||||
| Prompt/input | Prompt count/length, modality, MIME, order, count, decoded shape |
|
||||
| Size/time | Width/height alignment, pixels, frames, duration, FPS, start offset |
|
||||
| Denoise | Scheduler, sigma schedule, steps, CFG/guidance, seed behavior |
|
||||
| Output | Count, format, codec/sample rate, return type |
|
||||
| API | Offline mapping, sync endpoint, async endpoint/job lifecycle |
|
||||
| Rejection | Status code and stable error class/message for every invalid boundary |
|
||||
|
||||
Official API parity includes limits and defaults, not just names. Do not add a
|
||||
generic default that changes a distilled checkpoint's fixed schedule, or accept
|
||||
an input combination the official task rejects.
|
||||
|
||||
For every accepted boundary, add the neighboring rejected values. Examples:
|
||||
`max_images` and `max_images + 1`, smallest/largest legal duration, aligned and
|
||||
misaligned dimensions, omitted and illegal steps, one and too many outputs.
|
||||
|
||||
## Validate before side effects
|
||||
|
||||
Use one normalization/validation path for offline, sync, and async requests.
|
||||
The ordering must be:
|
||||
|
||||
```text
|
||||
parse -> normalize aliases -> validate task/source matrix -> validate declared
|
||||
limits -> stream with byte caps -> inspect/decode with frame/pixel/duration caps
|
||||
-> persist only if needed -> submit to engine
|
||||
```
|
||||
|
||||
Fail before download, persistence, job creation, or engine submit whenever the
|
||||
available metadata is sufficient. For streamed uploads, enforce:
|
||||
|
||||
- total reference count and per-modality count before reading bodies;
|
||||
- per-file bytes while streaming;
|
||||
- aggregate bytes across all files;
|
||||
- decoded frames, duration, pixels, and media type before engine submission.
|
||||
|
||||
On rejection, close streams and remove every partial temporary resource. Use
|
||||
HTTP 400 for invalid task/field combinations and 413 for payload limits. Test
|
||||
both sync and async paths; an invalid async request must not create a job ID.
|
||||
|
||||
Do not trust client MIME alone. Sniff/validate content and keep ordered repeated
|
||||
multipart fields ordered. Avoid logging prompts, raw references, signed URLs,
|
||||
or base64 payloads.
|
||||
|
||||
## Task request templates
|
||||
|
||||
These are endpoint-shape templates, not model recipes. Replace placeholders
|
||||
with the exact official defaults and validated assets for the target model.
|
||||
Every production recipe must include all advertised tasks and an expected
|
||||
output check.
|
||||
|
||||
### Text to image: JSON
|
||||
|
||||
```bash
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/images/generations \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"model": "<model>",
|
||||
"prompt": "<validated prompt>",
|
||||
"size": "1024x1024",
|
||||
"num_inference_steps": 30,
|
||||
"seed": 42,
|
||||
"response_format": "b64_json"
|
||||
}' | jq -er '.data[0].b64_json' | base64 -d > output.png
|
||||
file output.png
|
||||
```
|
||||
|
||||
### Image edit: multipart
|
||||
|
||||
```bash
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/images/edits \
|
||||
-F 'model=<model>' \
|
||||
-F 'prompt=<validated edit prompt>' \
|
||||
-F 'image=@/path/to/reference.png;type=image/png' \
|
||||
-F 'size=1024x1024' \
|
||||
-F 'num_inference_steps=30' \
|
||||
-F 'seed=42' \
|
||||
-F 'output_format=png' \
|
||||
| jq -er '.data[0].b64_json' | base64 -d > edited.png
|
||||
file edited.png
|
||||
```
|
||||
|
||||
Use repeated `image` fields only if the model's official input matrix allows
|
||||
multiple ordered sources. Never describe a file upload as JSON-only.
|
||||
|
||||
### Public request fields versus internal sampling fields
|
||||
|
||||
Use the field exposed by the target revision's endpoint schema; do not rename
|
||||
it from the pipeline's internal attribute. For example, the current video JSON
|
||||
and multipart contracts expose `extra_params`, while serving translates that
|
||||
object into `DiffusionSamplingParams.extra_args`. Other request paths may
|
||||
expose canonical `extra_args` or nest it under `extra_body`. Inspect the exact
|
||||
Pydantic model/FastAPI form dependency and exercise the generated curl before
|
||||
publishing a recipe. A deprecation warning in a different endpoint is not
|
||||
evidence that an unrecognized multipart form field works here.
|
||||
|
||||
Current endpoint map (re-check it at the target commit):
|
||||
|
||||
| Endpoint | Public model-specific field |
|
||||
|---|---|
|
||||
| `/v1/videos`, `/v1/videos/sync` | Multipart `extra_params` JSON string |
|
||||
| `/v1/images/generations` | JSON `extra_params` object |
|
||||
| `/v1/chat/completions` diffusion | `extra_body.extra_args` |
|
||||
| `/v1/images/edits` | Declared typed form fields only |
|
||||
| `/v1/audio/generate` | Declared typed JSON fields only |
|
||||
|
||||
Prefer declared top-level fields. Use a generic model-specific envelope only
|
||||
when the endpoint actually exposes it.
|
||||
|
||||
### Text to video: multipart
|
||||
|
||||
```bash
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/videos/sync \
|
||||
-H 'Accept: video/mp4' \
|
||||
-F 'model=<model>' \
|
||||
-F 'prompt=<validated prompt>' \
|
||||
-F 'size=1280x720' \
|
||||
-F 'num_frames=81' \
|
||||
-F 'fps=16' \
|
||||
-F 'num_inference_steps=30' \
|
||||
-F 'seed=42' \
|
||||
-F 'extra_params={"task":"<official-task>"}' \
|
||||
-o output.mp4
|
||||
ffprobe -v error -show_entries stream=codec_name,width,height,r_frame_rate \
|
||||
-of json output.mp4
|
||||
```
|
||||
|
||||
### Image/video to video: multipart
|
||||
|
||||
```bash
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/videos/sync \
|
||||
-H 'Accept: video/mp4' \
|
||||
-F 'model=<model>' \
|
||||
-F 'prompt=<validated prompt>' \
|
||||
-F 'num_frames=81' \
|
||||
-F 'fps=16' \
|
||||
-F 'num_inference_steps=30' \
|
||||
-F 'seed=42' \
|
||||
-F 'extra_params={"task":"<official-task>","start_time_seconds":0}' \
|
||||
-F 'input_reference=@/path/to/reference.png;type=image/png' \
|
||||
-o i2v.mp4
|
||||
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/videos/sync \
|
||||
-H 'Accept: video/mp4' \
|
||||
-F 'model=<model>' \
|
||||
-F 'prompt=<validated prompt>' \
|
||||
-F 'num_frames=81' \
|
||||
-F 'num_inference_steps=30' \
|
||||
-F 'seed=43' \
|
||||
-F 'extra_params={"task":"<official-task>","start_time_seconds":0}' \
|
||||
-F 'input_reference=@/path/to/reference.mp4;type=video/mp4' \
|
||||
-o v2v.mp4
|
||||
```
|
||||
|
||||
For multiple references, repeat the model's accepted multipart field in
|
||||
official order. Include complete downloadable or repository fixtures plus
|
||||
SHA256 hashes. A placeholder-only recipe is not runnable evidence.
|
||||
|
||||
### Text to audio: JSON
|
||||
|
||||
```bash
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/audio/generate \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"model": "<model>",
|
||||
"input": "<validated audio prompt>",
|
||||
"audio_length": 10.0,
|
||||
"num_inference_steps": 100,
|
||||
"seed": 42,
|
||||
"response_format": "wav"
|
||||
}' -o output.wav
|
||||
ffprobe -v error -show_entries stream=codec_name,sample_rate,channels \
|
||||
-of json output.wav
|
||||
```
|
||||
|
||||
Use `/v1/audio/generate` for general diffusion audio. Do not substitute the
|
||||
speech API unless the model is actually served through the speech contract.
|
||||
|
||||
### Async video lifecycle
|
||||
|
||||
Alongside `/v1/videos/sync`, include `POST /v1/videos`, status polling,
|
||||
content retrieval, cancellation/abort if public, and cleanup assertions. The
|
||||
same request must normalize to the same engine sampling parameters in both
|
||||
modes.
|
||||
|
||||
## Serving benchmark template
|
||||
|
||||
Run a smoke first, then fixed-concurrency and arrival-rate tests. Video task
|
||||
names for the harness are `t2v`, `i2v`, `ti2v`, or `v2v`; model-specific task
|
||||
selection belongs in `--extra-body` when required.
|
||||
|
||||
```bash
|
||||
python benchmarks/diffusion/diffusion_benchmark_serving.py \
|
||||
--base-url http://127.0.0.1:8091 \
|
||||
--endpoint /v1/videos \
|
||||
--model '<model>' \
|
||||
--dataset random \
|
||||
--task t2v \
|
||||
--num-prompts 100 \
|
||||
--width 1280 --height 720 \
|
||||
--num-frames 81 --fps 16 \
|
||||
--num-inference-steps 30 \
|
||||
--request-rate 0.2 \
|
||||
--max-concurrency 4 \
|
||||
--warmup-requests 4 \
|
||||
--warmup-concurrency 4 \
|
||||
--extra-body '{"extra_params":"{\"task\":\"<official-task>\"}"}' \
|
||||
--output-file result.json
|
||||
```
|
||||
|
||||
The video backend currently stringifies each multipart value. Pre-serialize
|
||||
the nested `extra_params` object as above; passing it as a nested object would
|
||||
produce Python dict syntax rather than valid JSON. If the target revision
|
||||
changes the backend to JSON-encode dict/list values, update and smoke-test this
|
||||
command against that revision.
|
||||
|
||||
For i2v/v2v, use the harness's validated dataset/trace path or provide the
|
||||
documented media source; do not silently benchmark t2v and label it i2v. Keep
|
||||
prompt, assets, seed, dimensions, frames, steps, guidance, warmup, concurrency,
|
||||
and arrival process identical across A/B runs.
|
||||
|
||||
Keep three benchmark lanes distinct:
|
||||
|
||||
1. lossless runtime A/B with the released schedule/precision/attention policy;
|
||||
2. accelerated-path A/B that changes one adapter, quantization, cache, or
|
||||
sparsity policy and adds same-seed quality evidence;
|
||||
3. production-topology studies that vary placement or parallelism and report a
|
||||
latency/throughput/memory frontier rather than a kernel speedup.
|
||||
|
||||
For a fixed single-request A/B, retain every measured run and report median or
|
||||
mean with range after an explicit warmup; do not infer p95/p99 from a few
|
||||
repetitions. Tail latency requires a declared arrival process, request count,
|
||||
concurrency, and enough successful samples. Reconcile client E2E with queue,
|
||||
encoder, denoise, video/audio decode, D2H/IPC, codec, and residual timings.
|
||||
|
||||
For generated media, define `real-time` rather than relying on the label. A
|
||||
complete-response claim uses `client E2E / validated output duration <= 1` and
|
||||
does not imply streaming or low time-to-first-frame. Report first chunk/fragment
|
||||
and cadence separately when a public streaming route exists. Never add gains
|
||||
across benchmark lanes with different schedules, prompts, seeds, artifacts, or
|
||||
topologies.
|
||||
|
||||
The diffusion serving harness currently covers image/video task families, not
|
||||
`/v1/audio/generate`. For text-to-audio, use the target model's task-specific
|
||||
audio benchmark if one exists; otherwise add a serving harness that records
|
||||
request rate, concurrency, latency, generated-audio seconds per second, error
|
||||
rate, and output validation. Do not relabel the TTS harness or a video workload
|
||||
as diffusion-audio evidence.
|
||||
|
||||
## Hardware recipe contract
|
||||
|
||||
Create a separate row and runnable section for each vendor/card/topology:
|
||||
|
||||
| Evidence | Required detail |
|
||||
|---|---|
|
||||
| Identity | Vendor, exact SKU, device count, HBM, interconnect |
|
||||
| Software | Driver, CUDA/ROCm/CANN/oneAPI, framework/container, dependency versions |
|
||||
| Source | vLLM-Omni commit, official/checkpoint revision and hashes |
|
||||
| Deployment | dtype, attention backend per role, TP/SP/CFG/HSDP/DP, cache, quant, offload |
|
||||
| Commands | Complete deploy YAML/serve command, every task curl, benchmark command |
|
||||
| Method | Warmup exclusion, repetitions or request count, concurrency, arrival distribution, duration, statistic definitions |
|
||||
| Results | Success rate, p50/p95/p99, RPS/throughput per device, stage latency |
|
||||
| Memory | Load/materialize, resident, encode, denoise, decode, transient per-rank HBM and host PSS |
|
||||
| Quality | Metric/tolerance, artifact links/hashes, temporal/audio checks |
|
||||
| Output | Raw/offline dtype, range, layout and ownership; online codec, payload bytes, transport route and client boundary |
|
||||
| Limits | `validated`/`limited`/`unsupported`/`not tested`, fallback/rejection behavior |
|
||||
|
||||
Use current paths such as `recipes/<vendor>/`, `recipes/README.md`, supported
|
||||
models, and feature compatibility tables. Top-level documentation must link to
|
||||
the scoped matrix; it must not use an unqualified checkmark that hides a ROCm,
|
||||
NPU, XPU, task, topology, or dtype limitation.
|
||||
|
||||
DLO evidence is per card, especially on small-HBM devices. A DLO result on one
|
||||
CUDA card does not establish another CUDA SKU, ROCm, NPU, or XPU support.
|
||||
|
||||
## Best deployment selection
|
||||
|
||||
Benchmark a correctness-approved candidate matrix, not every theoretical
|
||||
combination. Choose the recommended row by the production objective—latency,
|
||||
throughput, cost, or small-HBM fit—and document that objective.
|
||||
|
||||
Do not recommend a row until it passes:
|
||||
|
||||
- all advertised task/API contracts;
|
||||
- fixed-reference quality;
|
||||
- request isolation and abort/error cleanup;
|
||||
- sustained arrival-rate tests;
|
||||
- per-rank HBM and host PSS limits;
|
||||
- hardware-specific accuracy and performance CI.
|
||||
|
||||
Keep the most memory-constrained DLO row even when it is not the fastest; it is
|
||||
a separate production recipe, not a failed latency candidate.
|
||||
@@ -1,462 +0,0 @@
|
||||
# Production Feature Patterns
|
||||
|
||||
## Contents
|
||||
|
||||
1. [Compatibility matrix](#compatibility-matrix)
|
||||
2. [Online FP8](#online-fp8)
|
||||
3. [Strict fused-weight loading](#strict-fused-weight-loading)
|
||||
4. [Distributed layerwise offload](#distributed-layerwise-offload)
|
||||
5. [Per-request Cache-DiT](#per-request-cache-dit)
|
||||
6. [Combination gates](#combination-gates)
|
||||
|
||||
## Compatibility matrix
|
||||
|
||||
Create rows at the granularity below. Add evidence links and a reason for every
|
||||
non-validated state.
|
||||
|
||||
| Task/shape | Execution mode/capacity | Backend/layout | Quant | Cache policy | Offload | Topology | Hardware/dtype | Output contract/transport | State | Evidence |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| `<task>/<shape>/<schedule>` | serial request / 1 | SDPA, padded | BF16 | none | resident | 1 device | `<card>` BF16 | raw float / local | `not tested` | — |
|
||||
| `<task>/<shape>/<schedule>` | step / N | `<fast>`, packed | BF16 | none | resident | `<TP/SP>` | `<card>` BF16 | encoded / subprocess | `not tested` | — |
|
||||
| `<task>/<shape>/<schedule>` | step / N | `<fast>`, packed | FP8 | high | DLO no-AG | `<TP/SP>` | `<card>` | device uint8 / SHM | `not tested` | — |
|
||||
|
||||
Initialize every row as `not tested`; the table is a work queue, not evidence.
|
||||
Promote the dense BF16 row to the oracle only after its scoped parity passes.
|
||||
Add one axis at a time. If the runtime cannot
|
||||
enforce a known-incompatible combination, add construction/admission
|
||||
validation before writing a recipe.
|
||||
|
||||
## Online FP8
|
||||
|
||||
### Component routing
|
||||
|
||||
The public builder accepts one method or a longest-prefix component map:
|
||||
|
||||
```python
|
||||
from vllm_omni.quantization import build_quant_config
|
||||
|
||||
config = build_quant_config({
|
||||
"transformer": {
|
||||
"method": "fp8",
|
||||
"ignored_layers": ["transformer.final_layer"],
|
||||
},
|
||||
"text_encoder": None,
|
||||
"vae": None,
|
||||
"default": None,
|
||||
})
|
||||
|
||||
assert config.resolve("transformer.blocks.0.attn.qkv_proj") is not None
|
||||
assert config.resolve("text_encoder.layers.0.mlp") is None
|
||||
assert config.resolve("vae.decoder.conv_out") is None
|
||||
```
|
||||
|
||||
Use actual runtime prefixes. Multi-DiT or nested pipelines may not use the
|
||||
generic names `transformer` and `vae`. A prefix that falls through to a default
|
||||
is not component validation. Add tests for every included and ignored prefix.
|
||||
|
||||
For CLI serving, first validate resident DiT-only FP8:
|
||||
|
||||
```bash
|
||||
vllm serve '<model>' --omni --quantization fp8 --max-num-seqs 1
|
||||
```
|
||||
|
||||
For scoped routing:
|
||||
|
||||
```bash
|
||||
vllm serve '<model>' --omni \
|
||||
--diffusion-quantization-config \
|
||||
'{"<dit-runtime-prefix>":{"method":"fp8","ignored_layers":["<exact-linear-prefix>"]},"<vae-runtime-prefix>":null,"default":null}' \
|
||||
--max-num-seqs 1
|
||||
```
|
||||
|
||||
Do not publish these placeholders. A recipe must contain the target model's
|
||||
verified prefixes.
|
||||
|
||||
### Quantizable module wiring
|
||||
|
||||
Pass `quant_config` and a stable prefix through every vLLM linear. Keep
|
||||
precision-sensitive modulation/norm layers safe unless proven otherwise.
|
||||
|
||||
```python
|
||||
from vllm.model_executor.layers.activation import SiluAndMul
|
||||
from vllm.model_executor.layers.linear import (
|
||||
MergedColumnParallelLinear,
|
||||
QKVParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
|
||||
self.qkv_proj = QKVParallelLinear(
|
||||
hidden_size=hidden_size,
|
||||
head_size=head_dim,
|
||||
total_num_heads=num_heads,
|
||||
total_num_kv_heads=num_kv_heads,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.attn.qkv_proj",
|
||||
)
|
||||
self.gate_up_proj = MergedColumnParallelLinear(
|
||||
hidden_size,
|
||||
[intermediate_size, intermediate_size],
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.mlp.gate_up_proj",
|
||||
)
|
||||
self.act_fn = SiluAndMul()
|
||||
self.down_proj = RowParallelLinear(
|
||||
intermediate_size,
|
||||
hidden_size,
|
||||
bias=False,
|
||||
input_is_parallel=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.mlp.down_proj",
|
||||
)
|
||||
```
|
||||
|
||||
Set required checkpoint parameters to fail rather than silently initialize if
|
||||
the current loader supports `missing_param_init = "error"`.
|
||||
|
||||
### FP8 evidence
|
||||
|
||||
For each task/hardware row, capture:
|
||||
|
||||
- startup proves the FP8 quant method is attached to intended linear modules;
|
||||
- no unexpected BF16 fallback and no meta/uninitialized parameters;
|
||||
- same-seed component, trajectory, and final-artifact comparison with BF16;
|
||||
- HBM at load, resident, encode, denoise, decode, and peak;
|
||||
- cold load time; fixed-work warm-latency raw runs; and serving p50/p95/p99 plus
|
||||
throughput only from a declared arrival load with enough samples;
|
||||
- named ignored layers and why their BF16 retention is required.
|
||||
|
||||
DiT FP8 evidence does not cover the text encoder or VAE. CUDA evidence does not
|
||||
cover ROCm/NPU/XPU. A pre-quantized checkpoint is a different loading path from
|
||||
online FP8 and needs its own state. If it is also pruned, rotated, distilled, or
|
||||
adapter-modified, report that model delta explicitly; BF16-versus-artifact
|
||||
quality and speed are deployment comparisons, not pure quantization ablations.
|
||||
|
||||
## Strict fused-weight loading
|
||||
|
||||
Never use `param.data.copy_()` for quantizable vLLM parameters. Transform the
|
||||
checkpoint layout before invoking the parameter loader so the online quant
|
||||
loader remains outermost:
|
||||
|
||||
```python
|
||||
from vllm.model_executor.model_loader.weight_utils import default_weight_loader
|
||||
|
||||
param = params[target_name]
|
||||
weight_loader = getattr(param, "weight_loader", default_weight_loader)
|
||||
|
||||
if target_name.endswith(".qkv_proj.weight"):
|
||||
packed_qkv = reorder_checkpoint_qkv(checkpoint_weight)
|
||||
weight_loader(param, packed_qkv)
|
||||
elif target_name.endswith(".gate_up_proj.weight"):
|
||||
gate, up = checkpoint_weight.chunk(2, dim=0)
|
||||
weight_loader(param, gate, 0)
|
||||
weight_loader(param, up, 1)
|
||||
else:
|
||||
weight_loader(param, checkpoint_weight)
|
||||
```
|
||||
|
||||
Track source shards per fused destination:
|
||||
|
||||
```python
|
||||
expected = {
|
||||
"blocks.0.attn.qkv_proj.weight": {"q", "k", "v"},
|
||||
"blocks.0.mlp.gate_up_proj.weight": {0, 1},
|
||||
}
|
||||
seen = {name: set() for name in expected}
|
||||
|
||||
# After mapping/loading each source:
|
||||
seen[target_name].add(shard_id)
|
||||
|
||||
incomplete = {
|
||||
name: sorted(shards - seen[name], key=str)
|
||||
for name, shards in expected.items()
|
||||
if shards - seen[name]
|
||||
}
|
||||
if incomplete:
|
||||
raise RuntimeError(f"incomplete fused checkpoint parameters: {incomplete}")
|
||||
```
|
||||
|
||||
Also compare retained model parameters with the returned loaded set. Explicit
|
||||
checkpoint ignores need a reason and test; an unknown or missing weight is not
|
||||
an ignore policy.
|
||||
|
||||
Load-time adapter fusion creates a dedicated model, not a request-switchable
|
||||
LoRA. Pin the base and adapter revisions, validate every low-rank and full-rank
|
||||
delta target, freeze the adapter-declared schedule/task, and fuse before
|
||||
sharding only when the loader contract proves identical rank-local weights.
|
||||
Reject offload or host-weight paths that bypass that fusion rather than serving
|
||||
the unfused base model silently.
|
||||
|
||||
## Distributed layerwise offload
|
||||
|
||||
### Declare component topology
|
||||
|
||||
Use `SupportsComponentDiscovery` and `OffloadPlan`. Validate all dotted paths
|
||||
at construction/test time because discovery may otherwise warn and skip.
|
||||
|
||||
Treat a future/requested `--layerwise-offload-components` selector as a public
|
||||
API only if it exists in the target revision. Today, express component topology
|
||||
with `OffloadPlan` (`on_demand_component_paths`, `block_attrs`,
|
||||
`encoder_block_attrs`, `resident_dit_paths`, and `offload_submodules`). Do not
|
||||
invent a CLI flag or claim component-selective DLO from discovery declarations
|
||||
alone; validate the selector separately when it lands.
|
||||
|
||||
```python
|
||||
from operator import attrgetter
|
||||
|
||||
import torch
|
||||
|
||||
def assert_offload_path(root, path):
|
||||
value = attrgetter(path)(root)
|
||||
if not isinstance(value, torch.nn.Module):
|
||||
raise TypeError(f"offload path {path!r} is not an nn.Module")
|
||||
return value
|
||||
|
||||
for path in (
|
||||
*pipeline._dit_modules,
|
||||
*pipeline._encoder_modules,
|
||||
*pipeline._vae_modules,
|
||||
*pipeline._offload_plan.on_demand_component_paths,
|
||||
):
|
||||
assert_offload_path(pipeline, path)
|
||||
```
|
||||
|
||||
For `block_attrs`, resolve the DiT first and then each declared block attribute.
|
||||
Require an indexable block container with at least one `nn.Module`. Validate
|
||||
`offload_submodules`, `resident_dit_paths`, and `encoder_block_attrs` similarly.
|
||||
|
||||
### Keep host storage loader-owned
|
||||
|
||||
Current DLO accepts a loader-produced `HostWeightPlan`. Pipeline authors should
|
||||
not instantiate this plan or add ad hoc mmap capability flags. The loader must
|
||||
preflight complete runtime/checkpoint bindings, names, shapes, dtypes, persistent
|
||||
buffers, and represented transforms before it skips ordinary materialization.
|
||||
The exact accepted plan is consumed once by DLO; a rejected plan records an
|
||||
observable fallback reason and continues through the ordinary loader.
|
||||
|
||||
Direct checkpoint mmap currently requires TP1 without HSDP or online
|
||||
quantization. TP>1 falls back to the ordinary TP-aware loader and may feed DLO
|
||||
AllGather or no-AllGather; the current design records a bounded DP2xTP2
|
||||
AllGather smoke, but that path does not receive direct-mmap host-page sharing.
|
||||
Per-tensor online FP8 must use the ordinary loader. Current revisions can admit
|
||||
its finalized weight/scale layout to DLO AllGather, including transposed runtime
|
||||
weights, while unsupported online quantizers still fail closed. Direct mmap
|
||||
remains unavailable, so capture the temporary full-model host materialization
|
||||
before rank sharding.
|
||||
|
||||
Host Weight Runtime is a distinct opt-in path. Current final-layout consumers
|
||||
use no-AllGather DLO and exact model-declared BF16 representations. In
|
||||
`preferred` mode, consume an exact local hit or canonically load and publish for
|
||||
a future startup; in `required` mode, consume only and fail on a miss or unusable
|
||||
artifact. HWR must not interact with DLO AllGather, online quantization, HSDP,
|
||||
LoRA/adapted weights, or an unrecognized load format unless the target revision
|
||||
adds an exact producer/restore contract. It is immutable node-local CPU backing,
|
||||
not zero-copy GPU execution; the DLO transport still owns staging or direct H2D.
|
||||
Qualify identity inputs, cold population, warm reuse, atomic concurrent
|
||||
publication, source digests, corruption/quarantine, lease cleanup, capacity,
|
||||
node-local PSS, H2D, and E2E latency. A host-memory saving that materially
|
||||
regresses transfer or request latency is a separate memory-first deployment.
|
||||
|
||||
### DLO deployment modes
|
||||
|
||||
Treat these paths separately:
|
||||
|
||||
| Path | Weight/runtime behavior | Required boundaries |
|
||||
|---|---|---|
|
||||
| DLO AllGather | Rank-sharded pinned host tensors; reconstruct layer through collective | TP>1 uses ordinary TP-aware loader output; HSDP and unsupported online quantizers are rejected; finalized per-tensor online FP8 is eligible only when recognized by the target revision; concurrent ranks must participate consistently |
|
||||
| DLO no-AllGather | Loader-approved checkpoint mmap or ordinary runtime tensors; each rank streams a complete block | Direct mmap is TP1/non-HSDP/non-online-quant only; TP/HSDP/online quant use ordinary runtime tensors and require scoped E2E |
|
||||
| SP + DLO | SP group can supply the DLO collective group | Packed boundaries and collective order require E2E |
|
||||
|
||||
Single-stage examples:
|
||||
|
||||
```bash
|
||||
# SP + DLO AllGather candidate; validate exact model/card before publishing.
|
||||
vllm serve '<model>' --omni \
|
||||
--enable-distributed-layerwise-offload \
|
||||
--usp 4 \
|
||||
--max-num-seqs 1
|
||||
|
||||
# TP + DLO rank-local candidate; not a support statement.
|
||||
vllm serve '<model>' --omni \
|
||||
--tensor-parallel-size 2 \
|
||||
--enable-distributed-layerwise-offload \
|
||||
--dlo-no-use-allgather \
|
||||
--max-num-seqs 1
|
||||
```
|
||||
|
||||
Under `--omni`, configure DP in a deploy YAML rather than passing vLLM DP CLI
|
||||
flags:
|
||||
|
||||
```yaml
|
||||
pipeline: <registered-pipeline>
|
||||
async_chunk: false
|
||||
data_parallel_size: 4
|
||||
stages:
|
||||
- stage_id: 0
|
||||
devices: "0,1,2,3"
|
||||
max_num_seqs: 1
|
||||
enable_distributed_layerwise_offload: true
|
||||
dlo_use_allgather: true
|
||||
parallel_config:
|
||||
tensor_parallel_size: 1
|
||||
sequence_parallel_size: 1
|
||||
```
|
||||
|
||||
```bash
|
||||
vllm serve '<model>' --omni --deploy-config /path/to/dlo_dp4.yaml
|
||||
```
|
||||
|
||||
Keep the YAML next to the validated recipe. Match its schema to the target
|
||||
revision; do not assume a model accepts another pipeline's deploy config.
|
||||
|
||||
### DLO validation sequence
|
||||
|
||||
1. Resident BF16 reference.
|
||||
2. Ordinary layerwise offload parity.
|
||||
3. DLO enable, warmup, same-seed generation, disable/restart.
|
||||
4. AllGather and no-AllGather separately.
|
||||
5. Concurrent requests with different prompts and identical collective-affecting
|
||||
parameters; then reject or schedule incompatible steps/shapes safely.
|
||||
6. Queued/in-flight abort, error, idle rank, worker exit, and next request.
|
||||
7. TP, SP, cache, compile, online FP8 one axis at a time.
|
||||
8. Loader-plan selection/fallback reason, cold/warm storage lifecycle, and
|
||||
corruption/stale-entry recovery when mmap/cache backing is advertised.
|
||||
9. Replica-local request preparation plus DLO-group-only weight collectives in
|
||||
DP; exercise concurrent replicas to catch accidental WORLD rendezvous.
|
||||
10. Sweep resident block counts; publish raw latency/HBM pairs, dominated
|
||||
points, and the non-dominated Pareto frontier.
|
||||
11. Per-rank HBM, host PSS for the full process tree, H2D and collective trace.
|
||||
|
||||
On a small-HBM card, measure transient encode/VAE peaks as well as resident DiT
|
||||
weights. If a non-DiT component still exceeds HBM, combine only independently
|
||||
validated on-demand component staging, VAE tiling/patching, or another topology;
|
||||
do not hide the remaining peak behind the phrase “DLO enabled.”
|
||||
|
||||
## Per-request Cache-DiT
|
||||
|
||||
### Policy and lifecycle skeleton
|
||||
|
||||
First verify that the target revision contains the shared protocol/runtime.
|
||||
If not, follow the nearest current model's lifecycle without inventing a new
|
||||
public API.
|
||||
|
||||
The following is a **serial/exclusive request-mode sketch only**. Pipeline-wide
|
||||
hooks are mutable; concurrent requests need an admission/batching key and
|
||||
runner-owned transition serialization before this shape is safe.
|
||||
|
||||
```python
|
||||
import torch.nn as nn
|
||||
|
||||
from vllm_omni.diffusion.cache.cachedit import (
|
||||
CacheDiTBackend,
|
||||
CacheDiTRequestSpec,
|
||||
RequestScopedCacheDiTRuntime,
|
||||
)
|
||||
|
||||
class MyPipeline(nn.Module):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super().__init__()
|
||||
# ... components ...
|
||||
self._cache_dit_runtime = RequestScopedCacheDiTRuntime(self)
|
||||
|
||||
def adopt_cache_dit_backend(self, backend: CacheDiTBackend) -> None:
|
||||
self._cache_dit_runtime.adopt(
|
||||
backend,
|
||||
installation_key="my_model.generic",
|
||||
)
|
||||
|
||||
def is_cache_dit_enabled(self) -> bool:
|
||||
return self._cache_dit_runtime.is_enabled
|
||||
|
||||
def _cache_spec(self, quality, steps):
|
||||
if quality == "lossless":
|
||||
return None
|
||||
if quality == "high":
|
||||
return CacheDiTRequestSpec(
|
||||
installation_key="my_model.high.v1",
|
||||
cache_config=self._validated_high_config,
|
||||
num_inference_steps=steps,
|
||||
)
|
||||
raise ValueError(f"unsupported quality tier: {quality!r}")
|
||||
|
||||
def forward(self, req):
|
||||
quality, steps = validate_and_resolve_request(req)
|
||||
self._cache_dit_runtime.prepare(self._cache_spec(quality, steps))
|
||||
return self._run_validated_request(req)
|
||||
```
|
||||
|
||||
The snippet shows the model-owned transition for exclusive execution. Put
|
||||
disable/restore handling in the runtime/runner lifecycle that owns success,
|
||||
exception, disconnect, and asynchronous abort; a `try/finally` inside
|
||||
`forward()` alone cannot observe every server cancellation boundary.
|
||||
|
||||
The `installation_key` must include every policy dimension that changes hook
|
||||
installation; changes in step count can use runtime refresh. `None` disables
|
||||
installed hooks. Do not mutate the global startup cache config for a request.
|
||||
|
||||
Centralize cleanup in the owning request/runner boundary. An exception, abort,
|
||||
disconnect, or admission failure must not leave a changed hook policy visible
|
||||
to the next request. The exact owner depends on the target revision; add a
|
||||
failure-injection test rather than relying on `forward()` alone to observe
|
||||
asynchronous abort.
|
||||
|
||||
Example model-specific request form after calibration:
|
||||
|
||||
```bash
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/videos/sync \
|
||||
-F 'model=<model>' \
|
||||
-F 'prompt=<prompt>' \
|
||||
-F 'quality=lossless' \
|
||||
-F 'num_inference_steps=<validated-steps>' \
|
||||
-o lossless.mp4
|
||||
|
||||
curl --fail-with-body -sS -X POST http://127.0.0.1:8091/v1/videos/sync \
|
||||
-F 'model=<model>' \
|
||||
-F 'prompt=<prompt>' \
|
||||
-F 'quality=high' \
|
||||
-F 'num_inference_steps=<validated-steps>' \
|
||||
-o high.mp4
|
||||
```
|
||||
|
||||
Do not expose these tier names merely because another model uses them.
|
||||
|
||||
### Cache-DiT evidence
|
||||
|
||||
For every tier/task/shape/schedule:
|
||||
|
||||
- show real cache hits/skipped blocks, not only “backend enabled” logs;
|
||||
- measure fixed-work speed and quality against lossless/native;
|
||||
- publish a speed/quality frontier and chosen tier mapping;
|
||||
- alternate lossless/high/lossless and vary steps to prove restore/refresh;
|
||||
- interleave tasks and shapes; run concurrent requests only when scheduling
|
||||
guarantees exclusive compatible hook state;
|
||||
- inject validation error, denoise error, disconnect, and abort, then prove the
|
||||
next request starts with its requested policy;
|
||||
- bound any cached tensors and clear request-local state.
|
||||
|
||||
## Combination gates
|
||||
|
||||
Start every combination as `not tested`.
|
||||
|
||||
| Combination | Default production posture |
|
||||
|---|---|
|
||||
| Direct checkpoint mmap + TP>1/HSDP/online quant | Preflight falls back to the ordinary loader; do not claim direct-mmap savings |
|
||||
| Per-tensor online FP8 + DLO AllGather | Generic compatibility exists for finalized weights/scales; each model/card/topology remains a candidate until E2E, and other online quantizers fail closed |
|
||||
| Online FP8 + DLO no-AllGather | Candidate only after model/card/topology E2E |
|
||||
| TP>1 + DLO AllGather | Ordinary TP-aware loader only; bounded DP2xTP2 smoke exists, but each new model/card/topology still needs E2E |
|
||||
| TP>1 + DLO no-AllGather | Ordinary TP-aware loader only; candidate with limited generic coverage |
|
||||
| HSDP + DLO AllGather | Reject; it would double-shard HSDP-managed parameters |
|
||||
| HSDP + DLO no-AllGather | Configuration-compatible candidate with limited E2E coverage |
|
||||
| Cache-DiT + online FP8 | Separate trajectory, hit-rate, HBM, latency test |
|
||||
| Cache-DiT + DLO | Verify hooks, streamed blocks, memory peaks, abort cleanup |
|
||||
| Cache-DiT + step execution | Do not advertise until lifecycle/batching support is explicit |
|
||||
| Sparse attention + Ring | Verify backend execution; reject silently ignored sparse settings |
|
||||
| Load-time fused adapter + offload/HWR | Reject unless the selected loader path applies and validates the same fusion before sharding/storage |
|
||||
| Packed attention + any topology | Unequal-sample boundary parity is mandatory |
|
||||
| Device-side uint8 preparation + offline caller | Version the dtype/range/layout change; HTTP MP4 parity does not preserve the raw tensor contract |
|
||||
| Device-side preparation + remote codec | Validate route selection, strided/contiguous layouts, payload bytes, ordering, fallback, and encoded-media parity |
|
||||
|
||||
A startup-only test never upgrades a combination. Require request output,
|
||||
parity/quality, actual fast-path evidence, resource cleanup, and a benchmark on
|
||||
the named hardware.
|
||||
@@ -1,481 +0,0 @@
|
||||
# Performance Patterns and Code Examples
|
||||
|
||||
## Contents
|
||||
|
||||
1. [Measurement loop](#measurement-loop)
|
||||
2. [Faster BF16 attention](#faster-bf16-attention)
|
||||
3. [Sparse attention](#sparse-attention)
|
||||
4. [Fusion-first operator patterns](#fusion-first-operator-patterns)
|
||||
5. [Remove redundant computation](#remove-redundant-computation)
|
||||
6. [USP communication optimization](#usp-communication-optimization)
|
||||
7. [Text encoder and VAE disaggregation](#text-encoder-and-vae-disaggregation)
|
||||
8. [Performance acceptance](#performance-acceptance)
|
||||
|
||||
## Measurement loop
|
||||
|
||||
Do not start with a list of optimizations. Start with a fixed correctness and
|
||||
performance workload:
|
||||
|
||||
```text
|
||||
model/checkpoint revision + prompt/media hashes + seed + dimensions/frames +
|
||||
scheduler/sigma/steps/guidance + task + dtype + backend + topology + hardware
|
||||
```
|
||||
|
||||
Capture:
|
||||
|
||||
- stage time: load, encode, conditioning, denoise, VAE decode/postprocess;
|
||||
- GPU trace: kernel duration/count, launch gaps, allocations, H2D, collectives;
|
||||
- CPU trace: preprocessing, per-layer sync, Python dispatch, temporary files;
|
||||
- memory: per-rank HBM peaks and process-tree host PSS;
|
||||
- output: device preparation, D2H/IPC, payload bytes, codec wall/process CPU,
|
||||
event-loop wait, client materialization, and signed residual;
|
||||
- serving: queue time, throughput, errors, request mix, and distribution samples.
|
||||
|
||||
For a fixed-work single-request A/B, run one explicit warmup policy and at
|
||||
least three measured repetitions, keep every raw run, and report median or mean
|
||||
with range. Do not derive p95/p99 from three samples. For tail latency, declare
|
||||
the arrival model and collect enough successful requests for the percentile and
|
||||
confidence claimed. Implement one optimization at a time, re-run parity, and
|
||||
report both local kernel and end-to-end deltas. A kernel win that moves no E2E
|
||||
metric is not a production speed claim.
|
||||
|
||||
Use the pipeline profiler already exposed by the target revision when
|
||||
available, and use PyTorch/CUDA/ROCm/NPU/XPU profiling tools appropriate to the
|
||||
platform. Do not force every platform through a CUDA-only workflow.
|
||||
|
||||
## Faster BF16 attention
|
||||
|
||||
### Shared attention wiring
|
||||
|
||||
Use the shared role-aware attention layer and give it the true local head
|
||||
counts and QKV layout:
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
from vllm.model_executor.layers.linear import (
|
||||
QKVParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
from vllm_omni.diffusion.attention.layer import Attention
|
||||
|
||||
class SelfAttention(torch.nn.Module):
|
||||
def __init__(
|
||||
self,
|
||||
hidden_size,
|
||||
head_dim,
|
||||
num_heads,
|
||||
num_kv_heads,
|
||||
quant_config=None,
|
||||
prefix="",
|
||||
):
|
||||
super().__init__()
|
||||
self.head_dim = head_dim
|
||||
self.qkv_proj = QKVParallelLinear(
|
||||
hidden_size=hidden_size,
|
||||
head_size=head_dim,
|
||||
total_num_heads=num_heads,
|
||||
total_num_kv_heads=num_kv_heads,
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.qkv_proj",
|
||||
)
|
||||
self.attention = Attention(
|
||||
num_heads=self.qkv_proj.num_heads,
|
||||
num_kv_heads=self.qkv_proj.num_kv_heads,
|
||||
head_size=head_dim,
|
||||
softmax_scale=head_dim**-0.5,
|
||||
causal=False,
|
||||
qkv_layout="BSND",
|
||||
role="self",
|
||||
prefix=prefix,
|
||||
)
|
||||
self.out_proj = RowParallelLinear(
|
||||
num_heads * head_dim,
|
||||
hidden_size,
|
||||
bias=False,
|
||||
input_is_parallel=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.out_proj",
|
||||
)
|
||||
|
||||
def forward(self, x, attn_metadata=None):
|
||||
qkv, _ = self.qkv_proj(x)
|
||||
q_size = self.qkv_proj.num_heads * self.head_dim
|
||||
kv_size = self.qkv_proj.num_kv_heads * self.head_dim
|
||||
q, k, v = qkv.split([q_size, kv_size, kv_size], dim=-1)
|
||||
q = q.unflatten(-1, (self.qkv_proj.num_heads, self.head_dim))
|
||||
k = k.unflatten(-1, (self.qkv_proj.num_kv_heads, self.head_dim))
|
||||
v = v.unflatten(-1, (self.qkv_proj.num_kv_heads, self.head_dim))
|
||||
out = self.attention(q, k, v, attn_metadata)
|
||||
out = out.flatten(-2)
|
||||
out, _ = self.out_proj(out)
|
||||
return out
|
||||
```
|
||||
|
||||
Add Q/K norm and RoPE in the official order before the `Attention` call. Keep
|
||||
checkpoint mapping for the fused QKV and output projection complete.
|
||||
|
||||
### Backend qualification
|
||||
|
||||
For each role (`self`, cross-attention category, token refiner, etc.), record:
|
||||
|
||||
- actual selected backend from logs/trace;
|
||||
- supported dtype, head size, QKV layout, mask/metadata, packed varlen behavior;
|
||||
- platform and compute capability;
|
||||
- TP/SP/Ring/AllGather-KV restrictions;
|
||||
- fallback behavior and whether fallback changes the claimed performance.
|
||||
|
||||
Use unequal packed samples and compare with dense SDPA. Verify `cu_seqlens`,
|
||||
max sequence length, padding exclusion, modality boundaries, CFG ownership, and
|
||||
output slicing. Do not pass an `attn_mask` and assume it is consumed. Inspect
|
||||
the backend contract and trace.
|
||||
|
||||
TRTLLM, Flash Attention, cuDNN, NPU varlen, and other backends are independent
|
||||
qualification rows. The shared `Attention` class does not make every backend
|
||||
packed-safe or faster automatically. Ring and hybrid paths require their own
|
||||
boundary parity. AllGather-KV and specific backend combinations may fail fast;
|
||||
keep that behavior visible in docs.
|
||||
|
||||
## Sparse attention
|
||||
|
||||
Sparse attention is a backend/model/platform optimization, not a synonym for
|
||||
low-precision dense attention. Enable it only after dense BF16 parity.
|
||||
|
||||
For every candidate task/shape/topology:
|
||||
|
||||
1. State the sparsity policy, selected backend/kernel, block/head dimensions,
|
||||
supported hardware, external kernel/package revision, and fallback.
|
||||
2. Prove the sparse kernel executed and report realized sparsity; an enabled
|
||||
flag is insufficient.
|
||||
3. Compare same-seed per-block outputs, denoise trajectory, and final artifact
|
||||
with dense attention.
|
||||
4. Add modality-specific metrics: temporal consistency and audio/video sync for
|
||||
video, spectral/audio quality for audio, and prompt/image metrics as relevant.
|
||||
5. Report attention time and full E2E time. Include preprocessing/index-build
|
||||
overhead, peak memory, and the exact dense control.
|
||||
6. Validate each task and shape. Do not extrapolate a T2V result to I2V/Ref2V,
|
||||
or NPU sparse support to CUDA/ROCm/XPU.
|
||||
|
||||
If Ring bypasses the selected backend or a sparse setting would be silently
|
||||
ignored, reject the combination. Prefer a visible dense fallback only when the
|
||||
recipe labels the row `limited` and does not claim sparse speed.
|
||||
|
||||
## Fusion-first operator patterns
|
||||
|
||||
Prefer current shared/custom operators with explicit guards and native
|
||||
fallbacks. Prove parity at operator, block, trajectory, and artifact levels,
|
||||
then use a trace to prove fewer launches/bytes or lower latency.
|
||||
|
||||
### Q/K RMSNorm + RoPE
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.layers.norm import RMSNorm
|
||||
from vllm_omni.diffusion.layers.rope import RotaryEmbedding, apply_rope_to_qk
|
||||
|
||||
self.norm_q = RMSNorm(head_dim, eps=eps)
|
||||
self.norm_k = RMSNorm(head_dim, eps=eps)
|
||||
# Example only: set these booleans from the official checkpoint contract.
|
||||
use_neox_layout = False
|
||||
cos_sin_are_half_dim = True
|
||||
self.rope = RotaryEmbedding(
|
||||
is_neox_style=use_neox_layout,
|
||||
half_head_dim=cos_sin_are_half_dim,
|
||||
)
|
||||
|
||||
q = self.norm_q(q)
|
||||
k = self.norm_k(k)
|
||||
q, k = apply_rope_to_qk(self.rope, q, k, (cos, sin))
|
||||
```
|
||||
|
||||
Replace the illustrative booleans with the official layout in real code.
|
||||
Preserve partial rotary dimensions by splitting the rotated prefix and concatenating
|
||||
the pass-through suffix. If 3D cos/sin includes a batch axis, the current
|
||||
shared path assumes the values are shared across the batch; per-sample
|
||||
different positions require a proven representation/path.
|
||||
|
||||
For packed `[tokens, heads, head_dim]` tensors with a non-interleaved
|
||||
`[cos | sin]` table, the current shared fused boundary is:
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.layers.fused_qk_norm_rope import fused_qk_norm_rope
|
||||
|
||||
q, k = fused_qk_norm_rope(
|
||||
q,
|
||||
k,
|
||||
self.norm_q.weight,
|
||||
self.norm_k.weight,
|
||||
rope_table,
|
||||
self.norm_q.variance_epsilon,
|
||||
)
|
||||
```
|
||||
|
||||
Its CUDA fast path currently specializes BF16 `head_dim=128` and
|
||||
`rotary_dim=96`; other valid inputs take the eager implementation. The separate
|
||||
shared RMSNorm and RotaryEmbedding still provide the general cross-platform
|
||||
baseline. Treat NPU/MUSA and other vendor fusions as independent qualification
|
||||
rows, and do not generalize a model-specific Triton kernel without compatible
|
||||
shape, layout, frequency partition, dtype, and platform evidence.
|
||||
|
||||
### Fused QKV projection
|
||||
|
||||
Use `QKVParallelLinear` as shown above. Qualification includes:
|
||||
|
||||
- `num_heads` and `num_kv_heads` divisibility by TP;
|
||||
- local split sizes from the layer, never reusing global counts after TP;
|
||||
- Q/K/V checkpoint order conversion before the vLLM loader;
|
||||
- all Q, K, and V shards present;
|
||||
- quantization scale/loader mapping when FP8 or another method is enabled.
|
||||
|
||||
### Gate-up projection + fused SwiGLU
|
||||
|
||||
```python
|
||||
from vllm.model_executor.layers.activation import SiluAndMul
|
||||
from vllm.model_executor.layers.linear import (
|
||||
MergedColumnParallelLinear,
|
||||
RowParallelLinear,
|
||||
)
|
||||
|
||||
self.gate_up_proj = MergedColumnParallelLinear(
|
||||
input_size=hidden_size,
|
||||
output_sizes=[intermediate_size, intermediate_size],
|
||||
bias=False,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.gate_up_proj",
|
||||
)
|
||||
self.act_fn = SiluAndMul()
|
||||
self.down_proj = RowParallelLinear(
|
||||
input_size=intermediate_size,
|
||||
output_size=hidden_size,
|
||||
bias=False,
|
||||
input_is_parallel=True,
|
||||
quant_config=quant_config,
|
||||
prefix=f"{prefix}.down_proj",
|
||||
)
|
||||
|
||||
gate_up, _ = self.gate_up_proj(x)
|
||||
x = self.act_fn(gate_up)
|
||||
x, _ = self.down_proj(x)
|
||||
```
|
||||
|
||||
Map both checkpoint `gate_proj` and `up_proj` shards into the merged parameter
|
||||
and require both. A merged projection followed by `F.silu(gate) * up` is not
|
||||
the same claim as the `SiluAndMul` fused activation kernel; identify what the
|
||||
trace actually uses.
|
||||
|
||||
### AdaLN
|
||||
|
||||
Use a shared layer only when its mathematical/return contract matches the
|
||||
official model:
|
||||
|
||||
```python
|
||||
from vllm_omni.diffusion.layers.adalayernorm import AdaLayerNorm
|
||||
|
||||
self.adaln = AdaLayerNorm(
|
||||
hidden_size,
|
||||
elementwise_affine=False,
|
||||
eps=1e-6,
|
||||
)
|
||||
x = self.adaln(x, scale[:, None, :], shift[:, None, :])
|
||||
```
|
||||
|
||||
`AdaLayerNormZero`, `AdaLayerNormZeroSingle`, and continuous variants include
|
||||
different projections, broadcasts, and tuple outputs; they are not drop-in
|
||||
replacements. A shared AdaLN may have an NPU fast path while using native ops on
|
||||
CUDA/HIP/XPU. Do not call it a universal fused AdaLN kernel.
|
||||
|
||||
For sigma schedules with a small finite set of timesteps, profile whether
|
||||
precomputing the timestep/AdaLN projection once per request is worthwhile.
|
||||
Cache keys must include the exact schedule/config/device/dtype and stay bounded.
|
||||
|
||||
### Layout, residual, and other candidates
|
||||
|
||||
After profiling, inspect:
|
||||
|
||||
- bias/dropout/add or gated residual chains;
|
||||
- norm + linear or norm + modulation boundaries;
|
||||
- repeated transpose/reshape/contiguous around attention;
|
||||
- VAE normalization/activation/convolution boundaries;
|
||||
- MoE routing and fused experts for models that actually use MoE.
|
||||
|
||||
For fused residual-plus-norm or modulation chains, preserve the unfused
|
||||
materialization boundary. If the residual was stored in BF16 before RMSNorm,
|
||||
round to BF16 and promote again inside the fused kernel; using the unrounded
|
||||
FP32 temporary changes the denoising trajectory even when the stored residual
|
||||
looks correct.
|
||||
|
||||
Keep DLO materialize/release hooks, process-group collectives, request cache
|
||||
transitions, and cleanup outside compiled/fused regions unless lifecycle safety
|
||||
is explicitly tested. Compile only stable regions and retain eager fallback.
|
||||
|
||||
## Remove redundant computation
|
||||
|
||||
### Hoist request invariants
|
||||
|
||||
Audit the denoise loop for work that depends only on the request rather than
|
||||
the current timestep:
|
||||
|
||||
```python
|
||||
# Before the denoise loop.
|
||||
condition = self.condition_proj(text_embeddings)
|
||||
refined_condition = self.token_refiner(condition, packed_metadata)
|
||||
rope_tables = self.build_rope_tables(position_ids)
|
||||
attn_metadata = self.build_attention_metadata(layout)
|
||||
|
||||
for step, sigma in enumerate(sigmas):
|
||||
timestep_state = self.prepare_timestep_state(sigma)
|
||||
noise = self.transformer(
|
||||
latents,
|
||||
refined_condition,
|
||||
rope_tables=rope_tables,
|
||||
attn_metadata=attn_metadata,
|
||||
timestep_state=timestep_state,
|
||||
)
|
||||
latents = self.scheduler_step(noise, sigma, latents)
|
||||
```
|
||||
|
||||
Candidates include prompt conditioning, token-refiner output, reference
|
||||
encoding/scans, position IDs/RoPE tables, packed masks/metadata, static CFG
|
||||
branches, and finite-schedule AdaLN projections. First prove they are invariant
|
||||
for all official tasks, LoRA/adapters, prompt-update features, shapes, and
|
||||
schedules. In continuous batching, store request-specific results on request
|
||||
state rather than the pipeline singleton.
|
||||
|
||||
### Eliminate per-layer synchronization
|
||||
|
||||
Avoid device-to-host scalar extraction such as `.item()` inside every layer:
|
||||
|
||||
```python
|
||||
# Bad: forces a GPU/CPU synchronization in every attention layer.
|
||||
max_len = int(cu_seqlens[-1].item())
|
||||
|
||||
# Better: compute validated host metadata once during request packing and pass it.
|
||||
packed = PackedLayout(
|
||||
cu_seqlens=cu_seqlens,
|
||||
max_seqlen=max(sample_lengths),
|
||||
)
|
||||
for block in self.blocks:
|
||||
hidden = block(hidden, packed_layout=packed)
|
||||
```
|
||||
|
||||
Trace to verify the synchronization disappears. Do not replace a correct
|
||||
dynamic value with a stale constant merely to remove `.item()`.
|
||||
|
||||
### Remove unused masks and allocations
|
||||
|
||||
If packed attention consumes `cu_seqlens` and ignores a dense mask, do not
|
||||
construct the dense mask. Reuse immutable request metadata and preallocated
|
||||
buffers when safe. Prefer direct indexed scatter/gather over constructing large
|
||||
temporary tensors only when duplicate-index semantics and gradients (normally
|
||||
disabled for inference) remain correct.
|
||||
|
||||
Any cross-request cache needs a bounded lifetime and a complete key containing
|
||||
shape, layout, device, dtype, schedule, task, adapter/LoRA, and any value that
|
||||
changes the tensor. Otherwise keep it request-scoped.
|
||||
|
||||
### VAE and reference work
|
||||
|
||||
Profile duplicate VAE decode/postprocess, repeated reference decoding/scans,
|
||||
unnecessary host-device round trips, and full-frame materialization. Apply VAE
|
||||
tiling, slicing, patch parallelism, or on-demand staging only with seam,
|
||||
ordering, color/range, and memory parity tests.
|
||||
|
||||
## USP communication optimization
|
||||
|
||||
“USP works” is not evidence that USP communication is optimized. For Ulysses,
|
||||
Ring, hybrid, and AllGather-KV separately, profile:
|
||||
|
||||
- collective count, bytes, duration, stream, and overlap with compute;
|
||||
- Q/K/V and output all-to-all boundaries;
|
||||
- pack/unpack, padding, metadata broadcast, transpose, and contiguous copies;
|
||||
- head divisibility or advanced uneven-head support;
|
||||
- scaling efficiency from 1 to N devices on the same workload/interconnect;
|
||||
- load balance for unequal packed samples.
|
||||
|
||||
Declare `_sp_plan` split/gather boundaries around meaningful module outputs,
|
||||
not arbitrary Python statements. Keep packed metadata sharded/gathered in the
|
||||
same ownership model as tokens.
|
||||
|
||||
When the target revision offers accelerated Ulysses transport such as
|
||||
`--ulysses-a2a-permute`, qualify it separately from regular Ulysses. Prove the
|
||||
strict scatter-heads/gather-sequence layout selected the fast path, record
|
||||
one-time JIT/readiness cost, prewarm the maximum workspace shape before CUDA
|
||||
graph capture, keep the grow-only workspace on one stream, and exercise cleanup
|
||||
before process-group destruction. Compare collective and E2E deltas after the
|
||||
dense backend and topology are frozen; keep unsupported layouts on an explicit
|
||||
regular-Ulysses fallback.
|
||||
|
||||
For GQA, validate `Q_heads % KV_heads == 0`, pad KV heads first, and derive Q
|
||||
padding from the original ratio. Exercise both contiguous and row-strided QKV
|
||||
layouts; a fast copy path must preserve bitwise values and packed boundaries.
|
||||
|
||||
Cross-attention sometimes uses replicated text K/V while only video/audio Q is
|
||||
sequence-sharded. In that specific case, `skip_sequence_parallel=True` can
|
||||
avoid incorrect or wasteful Q/K/V redistribution:
|
||||
|
||||
```python
|
||||
self.cross_attention = Attention(
|
||||
num_heads=local_heads,
|
||||
num_kv_heads=local_kv_heads,
|
||||
head_size=head_dim,
|
||||
softmax_scale=head_dim**-0.5,
|
||||
causal=False,
|
||||
role="cross",
|
||||
skip_sequence_parallel=True,
|
||||
prefix=f"{prefix}.cross_attn",
|
||||
)
|
||||
```
|
||||
|
||||
Use it only after proving K/V are replicated and Q/output ownership is correct.
|
||||
Otherwise it disables needed communication. Report whether the optimization
|
||||
reduces collective bytes/time and improves E2E scaling.
|
||||
|
||||
## Text encoder and VAE disaggregation
|
||||
|
||||
Disaggregation is a measured architecture decision. First collect:
|
||||
|
||||
| Question | Required measurement |
|
||||
|---|---|
|
||||
| Is the stage large enough? | Encode/decode share at target concurrency and shape |
|
||||
| Is transfer economical? | Tensor/media bytes, serialization, connector latency and bandwidth |
|
||||
| Is there reuse? | Prompt/reference reuse rate, cacheability, fan-out |
|
||||
| Can it scale independently? | Queue saturation and resource utilization per stage |
|
||||
| Is the contract stable? | Shape, dtype, mask/tag/position metadata, ordering |
|
||||
|
||||
### Text encoder candidate
|
||||
|
||||
Define an explicit tensor contract containing hidden states, masks, tags or
|
||||
position IDs, dtype, shape, task/checkpoint revision, and request ID. Support
|
||||
disabling the local encoder so the DiT cannot accidentally recompute it. Test
|
||||
all tasks, prompt lengths, partial TP group membership, retries, backpressure,
|
||||
abort, connector cleanup, and independent replica scaling.
|
||||
|
||||
### VAE candidate
|
||||
|
||||
Separate VAE encode and decode contracts if both exist. Define latent shape,
|
||||
scale/shift, dtype/range, tiling/chunk order, output media ordering, and stage
|
||||
metadata. Measure latent/media transfer against local decode cost. Test tiling
|
||||
seams, RNG/determinism where relevant, chunk ordering, codec/postprocess,
|
||||
backpressure, abort, and worker failure.
|
||||
|
||||
Do not label VAE tiling, patch parallelism, compile, or offload as VAE
|
||||
disaggregation. Likewise, a separate text-encoder TP group is not a separate
|
||||
stage. If no reusable generic stage/connector exists in the target revision,
|
||||
produce a feasibility result or RFC and keep the capability `not tested` rather
|
||||
than inventing a model-local production architecture.
|
||||
|
||||
## Performance acceptance
|
||||
|
||||
For each accepted optimization, require:
|
||||
|
||||
- same fixed workload and correctness artifacts before/after;
|
||||
- actual fast-path execution proven by logs/trace/counters;
|
||||
- local metric plus E2E latency/throughput and memory;
|
||||
- at least three raw fixed-work repetitions after explicit warmup, or a
|
||||
declared arrival-load sample set large enough for every tail percentile;
|
||||
- named hardware, backend, topology, dtype, and task scope;
|
||||
- fallback/rejection tests for unsupported shapes/platforms;
|
||||
- no regression in abort/error cleanup or long-run memory trend.
|
||||
|
||||
Update the feature matrix narrowly. Do not convert a profile-driven direction
|
||||
into a generic “supported” feature merely because a model-specific PR exists.
|
||||
@@ -1,464 +0,0 @@
|
||||
# Production Serving and Validation
|
||||
|
||||
## Contents
|
||||
|
||||
1. [Execution-mode decision](#execution-mode-decision)
|
||||
2. [Request-level batching](#request-level-batching)
|
||||
3. [Step execution and continuous batching](#step-execution-and-continuous-batching)
|
||||
4. [Abort and cleanup](#abort-and-cleanup)
|
||||
5. [Mixed-RPS soak](#mixed-rps-soak)
|
||||
6. [Four CI tracks](#four-ci-tracks)
|
||||
7. [PR evidence checklist](#pr-evidence-checklist)
|
||||
|
||||
## Execution-mode decision
|
||||
|
||||
Request batching and step-wise execution solve different problems:
|
||||
|
||||
| Mode | Pipeline contract | Scheduling/abort boundary |
|
||||
|---|---|---|
|
||||
| Serial request | `forward(DiffusionRequestBatch)` for one request | Whole request; in-denoise preemption is unavailable |
|
||||
| Request batching | `supports_request_batch = True`; one forward consumes compatible independent requests | Whole batch/request |
|
||||
| Single-request step | Full `SupportsStepExecution`; `max_num_seqs=1` | Between denoise steps |
|
||||
| Step continuous batching | Full step contract plus heterogeneous batched-step evidence | Between denoise waves/steps |
|
||||
|
||||
Do not expose four stub methods as an opt-out. The current protocol is
|
||||
structural; if a native pipeline is not step-capable, do not define the four
|
||||
step methods. If it is capable, implement and test all four.
|
||||
|
||||
Treat structural support, a generic request-batch bridge, and model benefit as
|
||||
three separate claims. Before recommending step execution, name a hypothesis
|
||||
that request mode cannot satisfy, define a success threshold, and run a matched
|
||||
request-mode control. If throughput or latency regresses, retain step execution
|
||||
as functional/limited and keep request mode as the production recommendation.
|
||||
|
||||
Choose compatibility keys from every value that changes tensor shapes,
|
||||
semantics, hooks, or scheduler behavior: task, output shape/count, frames,
|
||||
steps/schedule, guidance/CFG, dtype/backend, LoRA/adapter, cache quality policy,
|
||||
and model-specific conditions. Reject or schedule separately rather than
|
||||
silently using the first request's settings.
|
||||
|
||||
## Request-level batching
|
||||
|
||||
A request-batched pipeline returns exactly one `DiffusionOutput` per logical
|
||||
request. This is model-owned pseudocode: `validate_batch_compatibility()` and
|
||||
`_generate_batch()` are placeholders to implement from the official contract,
|
||||
not shared APIs to import.
|
||||
|
||||
```python
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.data import DiffusionOutput
|
||||
from vllm_omni.diffusion.worker.request_batch import (
|
||||
DiffusionRequestBatch,
|
||||
split_diffusion_output_by_request,
|
||||
)
|
||||
|
||||
class MyPipeline(torch.nn.Module):
|
||||
supports_request_batch = True
|
||||
|
||||
def forward(self, req: DiffusionRequestBatch) -> list[DiffusionOutput]:
|
||||
validate_batch_compatibility(req.sampling_params_list, req.prompts)
|
||||
common = req.sampling_params_list[0]
|
||||
|
||||
text_prompts: list[str] | None = [
|
||||
prompt
|
||||
if isinstance(prompt, str)
|
||||
else (prompt.get("prompt") or "")
|
||||
for prompt in req.prompts
|
||||
]
|
||||
|
||||
prompt_fields = DiffusionRequestBatch.collate_prompt_field_map(
|
||||
req.prompts,
|
||||
{"prompt_embeds": None, "prompt_mask": None},
|
||||
)
|
||||
if prompt_fields["prompt_embeds"] is not None:
|
||||
text_prompts = None
|
||||
generators = req.collate_request_generators(
|
||||
common.num_outputs_per_prompt,
|
||||
default_generator=None,
|
||||
)
|
||||
output = self._generate_batch(
|
||||
prompt=text_prompts,
|
||||
prompt_fields=prompt_fields,
|
||||
generators=generators,
|
||||
sampling=common,
|
||||
)
|
||||
return split_diffusion_output_by_request(
|
||||
DiffusionOutput(output=output),
|
||||
req,
|
||||
num_outputs_per_prompt=common.num_outputs_per_prompt,
|
||||
)
|
||||
```
|
||||
|
||||
The prompt-field collator only handles tensor fields; extract raw text prompts
|
||||
and model-specific negative/alias fields separately, as current in-tree batched
|
||||
pipelines do. Set a raw prompt field to `None` when the matching precomputed
|
||||
embedding is supplied. The exact aliases and generator device depend on the
|
||||
model. Use current collators rather than manual `torch.cat` when they cover the
|
||||
contract; they reject mixed present/missing tensors and incompatible
|
||||
shape/dtype/device.
|
||||
|
||||
Before reading `sampling_params_list[0]`, validate every field that must be
|
||||
common. Tests must cover:
|
||||
|
||||
- two and N requests, independent seeds/generators, one output each;
|
||||
- mixed prompt lengths and packed/padded boundaries;
|
||||
- multiple outputs per prompt and correct result slicing;
|
||||
- compatible versus incompatible shapes, steps, guidance, LoRA, and quality;
|
||||
- one request failure without cross-request output or state corruption;
|
||||
- output ordering and request IDs under different completion times.
|
||||
|
||||
After the model advertises support, candidate serving is:
|
||||
|
||||
```bash
|
||||
vllm serve '<model>' --omni \
|
||||
--max-num-seqs 4 \
|
||||
--request-batch-max-wait-ms 20
|
||||
```
|
||||
|
||||
The wait window trades admission latency for batch formation. Benchmark at 0
|
||||
and candidate nonzero values under the target arrival process.
|
||||
|
||||
## Step execution and continuous batching
|
||||
|
||||
### Full protocol skeleton
|
||||
|
||||
Use the current shared state types and make every mutable value request-local:
|
||||
|
||||
The helpers beginning with `_` and `validate_step_batch()` below are
|
||||
model-owned placeholders; scheduler initialization is scheduler-specific.
|
||||
|
||||
```python
|
||||
import copy
|
||||
from collections.abc import Sequence
|
||||
from typing import Any, ClassVar
|
||||
|
||||
import torch
|
||||
|
||||
from vllm_omni.diffusion.data import DiffusionOutput
|
||||
from vllm_omni.diffusion.models.interface import SupportsStepExecution
|
||||
from vllm_omni.diffusion.worker.input_batch import InputBatch
|
||||
from vllm_omni.diffusion.worker.utils import StepRequestState
|
||||
|
||||
class MyPipeline(torch.nn.Module, SupportsStepExecution):
|
||||
supports_step_execution: ClassVar[bool] = True
|
||||
|
||||
def prepare_encode(
|
||||
self,
|
||||
state: StepRequestState,
|
||||
**kwargs: Any,
|
||||
) -> StepRequestState:
|
||||
prepared = self._prepare_request(state.prompt, state.sampling)
|
||||
state.prompt_embeds = prepared.prompt_embeds
|
||||
state.latents = prepared.latents
|
||||
state.timesteps = prepared.timesteps
|
||||
state.scheduler = copy.deepcopy(self.scheduler)
|
||||
state.scheduler.set_begin_index(0)
|
||||
state.step_index = 0
|
||||
state.extra["task_state"] = prepared.task_state
|
||||
return state
|
||||
|
||||
def denoise_step(
|
||||
self,
|
||||
input_batch: InputBatch,
|
||||
*,
|
||||
states: Sequence[StepRequestState] | None = None,
|
||||
**kwargs: Any,
|
||||
) -> torch.Tensor | None:
|
||||
validate_step_batch(input_batch, states)
|
||||
return self.transformer(**build_batched_denoise_inputs(input_batch))
|
||||
|
||||
def step_scheduler(
|
||||
self,
|
||||
state: StepRequestState,
|
||||
noise_pred: torch.Tensor,
|
||||
**kwargs: Any,
|
||||
) -> None:
|
||||
timestep = state.current_timestep
|
||||
if timestep is None:
|
||||
raise RuntimeError("step_scheduler called without a current timestep")
|
||||
state.latents = state.scheduler.step(
|
||||
noise_pred,
|
||||
timestep,
|
||||
state.latents,
|
||||
return_dict=False,
|
||||
)[0].to(state.latents.dtype)
|
||||
state.step_index += 1
|
||||
|
||||
def post_decode(
|
||||
self,
|
||||
state: StepRequestState,
|
||||
**kwargs: Any,
|
||||
) -> DiffusionOutput:
|
||||
return DiffusionOutput(output=self.decode(state.latents))
|
||||
```
|
||||
|
||||
This is a shape guide, not drop-in code: verify the target revision's
|
||||
`StepRequestState.extra`, scheduler signature, CFG handling, state fields, and
|
||||
decode contract. Follow a current native pipeline such as Qwen-Image for the
|
||||
exact revision.
|
||||
|
||||
Never use the shared `self.scheduler` to advance a request. Store scheduler,
|
||||
latents, embeddings, timesteps, masks, RNG/generator, task metadata, cache
|
||||
policy, adapter, and output assembly state on `StepRequestState` or its
|
||||
request-local extension. Pipeline fields may hold immutable weights/config and
|
||||
bounded immutable caches only.
|
||||
|
||||
### Qualification sequence
|
||||
|
||||
1. Compare request-mode and step-mode at `max_num_seqs=1` for every task using
|
||||
identical seed/schedule/shape and intermediate latents.
|
||||
2. Abort queued and in-flight requests at several step boundaries.
|
||||
3. Increase to two compatible requests; verify independent scheduler/RNG/state
|
||||
and output order.
|
||||
4. Increase to the target capacity and mix prompt lengths.
|
||||
5. Exercise mixed shape, frames, steps/schedule, task, guidance, and quality.
|
||||
Batch only compatible rows; otherwise split/reject predictably.
|
||||
6. Inject a failure in encode, denoise, scheduler, and decode independently.
|
||||
7. Run the production soak and fault matrix.
|
||||
|
||||
Candidate commands only after capability validation:
|
||||
|
||||
```bash
|
||||
# Conservative step execution and step-boundary abort.
|
||||
vllm serve '<model>' --omni \
|
||||
--step-execution \
|
||||
--max-num-seqs 1
|
||||
|
||||
# Experimental batched-step candidate after multi-request E2E.
|
||||
vllm serve '<model>' --omni \
|
||||
--step-execution \
|
||||
--max-num-seqs 4
|
||||
```
|
||||
|
||||
If an attention backend is only correct for single-request step mode, fail
|
||||
early or keep the recipe at `max_num_seqs=1`. Do not describe generic runtime
|
||||
support as model support.
|
||||
|
||||
## Abort and cleanup
|
||||
|
||||
### Required semantics
|
||||
|
||||
For queued and in-flight abort/disconnect:
|
||||
|
||||
- emit exactly one terminal aborted/error result;
|
||||
- do not run `post_decode` after abort unless a documented partial-output mode
|
||||
explicitly requires it;
|
||||
- do not increment `step_index` or apply a scheduler result after cancellation;
|
||||
- remove scheduler/state cache entries and request IDs;
|
||||
- release latents, prompt/reference tensors, cache hooks, DLO buffers, connector
|
||||
messages, uploaded/temp files, and output buffers;
|
||||
- keep other requests alive;
|
||||
- make cleanup idempotent when disconnect, timeout, and worker error race;
|
||||
- drop late results/exceptions for already-resolved or cancelled futures
|
||||
without terminating the shared result-pump thread;
|
||||
- complete the next valid request with its requested cache/offload policy.
|
||||
|
||||
Test cancellation before engine admission, while queued, after encode, during
|
||||
early/middle/final denoise steps, before decode, and while returning/persisting
|
||||
output. Include async job cancellation and client socket disconnect.
|
||||
|
||||
### Failure injection matrix
|
||||
|
||||
| Location | Injection | Assertions |
|
||||
|---|---|---|
|
||||
| Validation/upload | bad MIME, count/bytes overflow, truncated media | 400/413, no job/engine submit, no temp leak |
|
||||
| Load/start | missing fused shard, unsupported topology | startup fails, actionable error, no silent init |
|
||||
| Encode | exception/OOM/timeout | one error, request state/temp released |
|
||||
| Denoise | exception/OOM/worker exit | peers handled, hooks/buffers released, health behavior explicit |
|
||||
| Scheduler | invalid state/shape | no latent/step partial commit |
|
||||
| Decode/post | exception/codec error | no corrupt successful result, partial output removed |
|
||||
| Result/IPC | late result after cancel, large split tensor, orphan SHM | pump remains alive, one terminal result, exact ownership cleanup |
|
||||
| Connector/disagg | timeout/retry/duplicate | idempotency, backpressure, no orphan payload |
|
||||
|
||||
After each injection, check server health and issue a known-good request. If
|
||||
the process must restart, document and test the restart policy; do not claim
|
||||
transparent recovery.
|
||||
|
||||
### Large-output transport
|
||||
|
||||
Treat worker-to-engine IPC and engine-to-client transport as separate hops.
|
||||
For batched video, prove that each logical output does not accidentally retain
|
||||
or serialize the complete batch storage. Exercise the target revision's shared
|
||||
memory or artifact-handle path above and below its threshold, including
|
||||
non-contiguous views, multiple outputs, consumer failure, timeout, abort, and
|
||||
worker exit. Verify every shared segment/file is released exactly once.
|
||||
|
||||
Freeze the media contract at each hop: dtype, value range, shape/layout,
|
||||
contiguity, request ownership, encoding/color model, payload bytes, and pending
|
||||
consumers. Device-side float-to-uint8 preparation, D2H/IPC, planar/interleaved
|
||||
conversion, CPU encode/mux, and delivery off the asyncio event loop are
|
||||
different optimizations. Validate their route selection and fallbacks
|
||||
independently. Byte-identical HTTP MP4 output does not prove that a raw offline
|
||||
tensor kept its dtype or range.
|
||||
|
||||
For long-window output, validate segment ordering and audio/video continuity,
|
||||
then exercise payloads around serializer, shared-memory, HTTP, and codec size
|
||||
limits. Chunking must reassemble exactly, reject malformed/missing chunks, and
|
||||
release partial state after abort, timeout, or peer failure.
|
||||
|
||||
Qualify chunk production, transport, and public streaming as separate layers.
|
||||
A VAE callback must define owned representation, frame offsets, final marker,
|
||||
rank ownership, source/config gate, and collective-safe failure, but it does not
|
||||
prove SHM/ZMQ delivery, bounded backpressure, cancellation, codec reuse, or
|
||||
client streaming. Measure time to first owned chunk, first encoded fragment,
|
||||
steady cadence, mux finalization, and complete client artifact independently.
|
||||
|
||||
Measure remote MP4 encoding, serialization/copies, network transfer, and client
|
||||
materialization separately from denoise/decode. An inference-kernel speedup does
|
||||
not establish serving throughput if output transport dominates or leaks.
|
||||
|
||||
## Mixed-RPS soak
|
||||
|
||||
Use a realistic request mix and arrival distribution. Include small/large
|
||||
shapes, short/long prompts, all advertised tasks, cache tiers, and a small rate
|
||||
of abort/disconnect/error injections. Run phases below, near, and above measured
|
||||
saturation so queue/backpressure behavior is visible.
|
||||
|
||||
Example serving benchmark phase:
|
||||
|
||||
```bash
|
||||
python benchmarks/diffusion/diffusion_benchmark_serving.py \
|
||||
--base-url http://127.0.0.1:8091 \
|
||||
--endpoint /v1/videos \
|
||||
--model '<model>' \
|
||||
--dataset trace \
|
||||
--dataset-path /path/to/versioned-production-trace.jsonl \
|
||||
--task t2v \
|
||||
--num-prompts 500 \
|
||||
--request-rate 0.5 \
|
||||
--max-concurrency 8 \
|
||||
--warmup-requests 8 \
|
||||
--warmup-concurrency 8 \
|
||||
--output-file soak-near-saturation.json
|
||||
```
|
||||
|
||||
Use a task-appropriate trace/media dataset. Run the same phase at rates selected
|
||||
from an initial saturation curve; a literal `0.5` is illustrative only.
|
||||
|
||||
On supported GPU hosts, wrap a test or soak with the repository monitor:
|
||||
|
||||
```bash
|
||||
bash tests/dfx/stability/scripts/resource_monitor.sh run --backend gpu -- \
|
||||
pytest -s -v tests/dfx/stability/scripts/run_stability_<model>.py -m slow
|
||||
```
|
||||
|
||||
Check the target revision's monitor backends. Do not claim CPU/NPU monitoring
|
||||
from a reserved CLI choice unless collection is implemented and verified.
|
||||
|
||||
Report:
|
||||
|
||||
- offered and achieved RPS, success/error/abort rates;
|
||||
- p50/p95/p99 and queue/service/stage times, with the successful sample count
|
||||
and arrival process used for each percentile;
|
||||
- throughput per device and batch/wave utilization;
|
||||
- per-rank HBM and process-tree PSS trend/slope;
|
||||
- cache size/hit rate, temp-file/disk growth, open FDs/handles;
|
||||
- outstanding shared-memory/artifact handles and encoded/network bytes;
|
||||
- worker/engine health, restarts, OOM/timeouts, backpressure response;
|
||||
- total duration, request count/mix, raw logs/JSON/monitor artifact hashes.
|
||||
|
||||
A short burst or repeated serial curl loop is not a stability test.
|
||||
|
||||
## Four CI tracks
|
||||
|
||||
Keep the tracks independent so a failure answers a specific question. These
|
||||
four readiness tracks are not the repository's numbered CI levels. Map them
|
||||
onto the current L1-L5 taxonomy:
|
||||
|
||||
| Readiness track | Repository CI placement |
|
||||
|---|---|
|
||||
| Function | L1 unit/logic plus L2 basic online/offline E2E; extend real-model scenarios in L3/L4 |
|
||||
| Accuracy | L3 real-model gates for important cases; deeper and broader nightly coverage in L4 |
|
||||
| Performance | Important thresholds in L3 where they fit the time budget; full baselines/regression jobs in nightly L4 |
|
||||
| Reliability | Cheap rejection/recovery assertions can run earlier, but long stability, fault injection, and reliability suites belong to weekly L5 |
|
||||
|
||||
Distributed topology is a test case within the appropriate level, not a
|
||||
replacement definition for a level.
|
||||
|
||||
### 1. Function CI
|
||||
|
||||
Cover:
|
||||
|
||||
- strict unquantized and advertised quantized loading;
|
||||
- fused source-shard completeness and prefix routing;
|
||||
- official task positive boundaries and neighboring negative cases;
|
||||
- offline, sync, async normalization/parity;
|
||||
- single-device reference plus every advertised topology smoke;
|
||||
- output type/count/order and request ID isolation;
|
||||
- feature rejection paths that must fail early.
|
||||
|
||||
### 2. Accuracy CI
|
||||
|
||||
Pin official implementation/checkpoint/vLLM-Omni revisions, prompts and media
|
||||
hashes, seed, scheduler/sigma/steps, dimensions/frames/FPS, guidance, dtype,
|
||||
backend, and topology. Compare at suitable levels:
|
||||
|
||||
- component outputs or selected blocks;
|
||||
- one or more intermediate denoise latents;
|
||||
- final image/video/audio artifact metrics;
|
||||
- temporal consistency and audio/video synchronization where relevant.
|
||||
|
||||
Set tolerance per modality/hardware/backend with rationale. Keep failure
|
||||
artifacts and metadata. A single visual inspection does not define CI.
|
||||
|
||||
For quantization trajectory tooling, inspect the target revision's current
|
||||
`vllm_omni/quantization/tools/compare_diffusion_trajectory_similarity.py` and
|
||||
reuse it when compatible instead of creating an ad hoc comparator.
|
||||
|
||||
### 3. Performance CI
|
||||
|
||||
Pin workload and best deployment. Include a resident recommended row and a
|
||||
small-HBM/DLO row when both are production targets. Declare:
|
||||
|
||||
- one explicit warmup exclusion and at least three measured fixed-work
|
||||
repetitions;
|
||||
- raw result JSON and environment identity;
|
||||
- median/mean plus range for small fixed-work A/Bs; p50/p95/p99 only for an
|
||||
arrival-load run with enough samples, plus throughput/RPS, stage time,
|
||||
HBM/PSS;
|
||||
- regression metric, threshold, baseline owner, and update procedure;
|
||||
- correctness/accuracy check linked to the same configuration.
|
||||
|
||||
Use the current harness config style, for example:
|
||||
|
||||
```bash
|
||||
pytest -s -v tests/dfx/perf/scripts/run_diffusion_benchmark.py \
|
||||
--test-config-file tests/dfx/perf/tests/test_<model>_vllm_omni.json
|
||||
```
|
||||
|
||||
Verify parser options on the target revision. Do not copy a JSON schema from a
|
||||
different model without matching its endpoint, task, dataset, and deploy path.
|
||||
|
||||
### 4. Reliability CI
|
||||
|
||||
Run a bounded CI soak plus a longer scheduled soak. Cover:
|
||||
|
||||
- below/near/above-saturation arrival phases;
|
||||
- queued/in-flight abort and disconnect;
|
||||
- OOM/timeout and worker/process failure appropriate to the platform;
|
||||
- DLO/cache/connector cleanup;
|
||||
- health, fast-fail/backpressure, restart/recovery policy;
|
||||
- memory/temp/FD trend and successful known-good request after each fault.
|
||||
|
||||
Keep reliability thresholds independent from latency thresholds. A fast server
|
||||
that leaks memory or hangs after cancellation fails production readiness.
|
||||
|
||||
## PR evidence checklist
|
||||
|
||||
The PR description must include or link:
|
||||
|
||||
- official source/checkpoint revisions and API/limit matrix;
|
||||
- scoped feature compatibility matrix with the four allowed states;
|
||||
- implementation notes for FP8, DLO, request cache, attention/fusion, batching;
|
||||
- exact environment, deploy/serve, every task curl, benchmark and soak commands;
|
||||
- fixed accuracy inputs and artifacts;
|
||||
- raw before/after performance runs and memory traces;
|
||||
- output-contract manifests and stage accounting from device preparation
|
||||
through the complete client artifact;
|
||||
- abort/disconnect/fault results and cleanup evidence;
|
||||
- Function/Accuracy/Performance/Reliability CI files and ownership;
|
||||
- limitations/TODOs that remain `not tested` rather than implied support.
|
||||
|
||||
Separate the Day-0 milestone from production readiness. If the PR only closes
|
||||
some gates, title and description should say which scoped gates it closes.
|
||||
@@ -26,9 +26,6 @@ self-check.
|
||||
|
||||
- [`add-diffusion-model`](add-diffusion-model/SKILL.md): guides integration of
|
||||
a new diffusion model into `vllm-omni`
|
||||
- [`add-recipe`](add-recipe/SKILL.md): creates or updates in-repository model
|
||||
recipes with verified task/input/output contracts, concrete hardware,
|
||||
shared feature-guide links, exact commands, and qualification evidence
|
||||
- [`add-tts-model`](add-tts-model/SKILL.md): covers integration of new TTS
|
||||
models and related serving workflows
|
||||
- [`diffusion-perf-opt`](diffusion-perf-opt/SKILL.md): guides diffusion model
|
||||
@@ -40,10 +37,6 @@ self-check.
|
||||
- [`precheck-pr`](precheck-pr/SKILL.md): self-checks a branch before creating a
|
||||
PR by validating title format, dead code, simplification opportunities,
|
||||
accuracy and performance claims, and merge readiness
|
||||
- [`production-add-diffusion-model`](production-add-diffusion-model/SKILL.md):
|
||||
takes a working Day-0 diffusion integration through API/limit parity,
|
||||
feature-combination evidence, quantization/offload, performance, production
|
||||
serving, hardware recipes, and CI readiness
|
||||
- [`quantization`](quantization/SKILL.md): guides quantization method selection,
|
||||
model integration, checkpoint loading, and quality/performance validation
|
||||
for vLLM-Omni
|
||||
@@ -60,14 +53,6 @@ self-check.
|
||||
`test-weekly.yml`), and copy-paste local plus CI-like `pytest` commands; see
|
||||
`references/test-routing.md` for level-to-command mapping
|
||||
|
||||
## MiniMax H3 Creative Skills
|
||||
|
||||
The [portable H3 collection](../../.agents/skills/README.md) contains the prompt
|
||||
writing skill and eight creative workflows for product ads, 3D shorts, paper art,
|
||||
brand promos, music videos, co-op intros, and hand-drawn/live-action fusion.
|
||||
Same-named directories here link to the canonical `.agents/skills/` copies, so
|
||||
Claude Code and Codex use the same instructions and templates.
|
||||
|
||||
## Maintenance Guidelines
|
||||
|
||||
- Keep skill names short and task-oriented.
|
||||
|
||||
@@ -123,11 +123,10 @@ it changes again, report the churn and wait for a stable target.
|
||||
|
||||
For a trusted PR head, materialize the pinned head in an isolated detached
|
||||
worktree. A worktree freezes identity but is not a security sandbox. Treat fork
|
||||
heads as untrusted until the user and environment policy explicitly establish
|
||||
trust: execute their code only after that trust is recorded for this host,
|
||||
with the trusted SHA noted in the review state and secrets kept out of the
|
||||
execution scope; without recorded trust, use static SHA-addressed reads and CI
|
||||
evidence only. For a local review, freeze the committed, index, worktree,
|
||||
heads as untrusted unless the user and environment policy explicitly establish
|
||||
otherwise: execute them only in a disposable, secret-free sandbox with restricted
|
||||
filesystem, network, and resources; without one, use static SHA-addressed reads
|
||||
and CI evidence only. For a local review, freeze the committed, index, worktree,
|
||||
and NUL-safe in-scope untracked contents. Follow
|
||||
[review-execution.md](references/process/review-execution.md) for trust gates,
|
||||
state fingerprints, and byte-for-byte staleness checks.
|
||||
@@ -197,7 +196,7 @@ style preferences as simplification findings.
|
||||
|
||||
Before each validation group, verify the frozen SHA plus the tracked, index,
|
||||
untracked, and ignored-file fingerprint, or recreate a pristine snapshot. On a
|
||||
head the user explicitly trusted for this host, run an import/version preflight,
|
||||
trusted head or inside the required sandbox, run an import/version preflight,
|
||||
then the narrowest relevant tests and low-cost static checks. Bind every result
|
||||
to the head SHA, snapshot fingerprint, and environment fingerprint. Never run
|
||||
imports, tests, builds, hooks, or repo-configurable tooling from an untrusted
|
||||
|
||||
@@ -22,19 +22,6 @@ Keep exact commands and report variability rather than a cherry-picked run.
|
||||
Use the repository's benchmark when it covers the claim; otherwise use the
|
||||
smallest reproducible workload that reaches the changed production path.
|
||||
|
||||
## Isolation per claim
|
||||
|
||||
Isolate one claim per comparison. When a PR stacks several changes (a fused or
|
||||
restructured load path, a distilled step schedule, a new cache), request one
|
||||
column per claim at equal steps/workload instead of one blended base-vs-head
|
||||
delta, and name the unchanged stage or metric that pins equivalence.
|
||||
|
||||
When the changed path is runtime-switchable (feature flag or environment
|
||||
variable), run every column on the same head and toggle the switch rather than
|
||||
diffing against a base commit that cannot separate the stacked changes:
|
||||
base, head with the switch off, head with the switch on, then head with the
|
||||
switch on plus each further claim.
|
||||
|
||||
## Evidence table
|
||||
|
||||
| Dimension | Typical evidence |
|
||||
@@ -47,100 +34,6 @@ switch on plus each further claim.
|
||||
Every optimization needs correctness or quality evidence. A faster result on a
|
||||
different workload, precision, seed, or topology is not a valid comparison.
|
||||
|
||||
## E2e stage-attribution table
|
||||
|
||||
For an end-to-end latency claim, request the stage-attribution table the
|
||||
repository's perf PRs use:
|
||||
|
||||
- Rows carry the benchmark's own field names: total e2e, per-stage execution,
|
||||
postprocess, response encoding, and a request-to-output roll-up. Add a stage
|
||||
handoff/idle row when the pipeline spans stages — connector transfer and
|
||||
scheduling gaps are invisible in per-stage latency deltas.
|
||||
- Sanity rows pin comparability: inference steps, output frames/pixels/samples,
|
||||
and generated tokens held identical across columns.
|
||||
- Report P50 and P100 from the stated warmup and measured counts. One
|
||||
population per number: never mix cold-start and warm samples into one value.
|
||||
- Name the baseline and candidate commit SHAs, and close with an isolation
|
||||
statement naming the metric that stayed unchanged ("X remained effectively
|
||||
unchanged, isolating the gain to Y").
|
||||
|
||||
When the serving path caches, captures, or regenerates at runtime (CUDA-graph
|
||||
bucket growth, cache rebuilds, recompilation), medians hide the resulting tail.
|
||||
Also request the event count and the maximum observed stall.
|
||||
|
||||
Component-level tables pair with, not replace, the e2e table: per-op reference
|
||||
vs candidate timings (with the numerical-equality result per op) plus one
|
||||
complete-pipeline check under the production shapes.
|
||||
|
||||
## Proportionality
|
||||
|
||||
When a PR adds a model and explicitly labels its numbers as smoke observations
|
||||
rather than claims, do not demand the attribution machinery. Ask only for
|
||||
evidence protecting the model's stated operating contract — for a
|
||||
realtime/duplex model that is sustained cadence over a realistic session
|
||||
(enough ticks to leave warmup), with any mid-session capture/regeneration
|
||||
event and its stall reported, since a cadence that collapses mid-session is a
|
||||
functional failure, not a perf claim.
|
||||
|
||||
## Realtime/duplex world models
|
||||
|
||||
Request the operating-contract evidence as serving metrics, not claims:
|
||||
|
||||
- Sustained per-tick latency over one realistic session: median, max, and the
|
||||
argmax tick.
|
||||
- Stall events with their causes.
|
||||
- Peak memory at first versus last tick — session-owned state must not grow
|
||||
per tick.
|
||||
- Control latency from a submitted action to the tick that applies it.
|
||||
- Realtime-versus-offline parity under the same controls, using the
|
||||
repository's video-similarity helpers. Parity plotted over tick index is the
|
||||
drift-over-horizon curve and the honest measure of session lifetime;
|
||||
distribution-level suites (FVD, VBench) belong to model evaluation, not the
|
||||
serving integration.
|
||||
|
||||
Watch the contract boundaries the model documents: if a control input is
|
||||
tick-API-only, the parity pair must use a horizon both paths can run.
|
||||
|
||||
## Speech-generation (TTS) models
|
||||
|
||||
The streaming contract outranks the average — a synthesis that pauses
|
||||
mid-stream is broken for realtime use regardless of its averages. Request:
|
||||
|
||||
- The benchmark's own speech metrics over one long utterance (enough audio to
|
||||
leave warmup, tens of seconds): `audio_ttfp` (time to first packet),
|
||||
`audio_rtf` (real-time factor; above 1.0 cannot keep up), `e2el`, and
|
||||
`ttft`/`tpot` where tokens precede audio.
|
||||
- Mid-utterance stalls as the benchmark's continuity fields, not prose:
|
||||
`audio_underrun` percentiles and the `audio_continuity_ok` rate
|
||||
(`vllm_omni/benchmarks/metrics/metrics.py`), computed by
|
||||
`vllm_omni/benchmarks/audio_continuity.py` — the companion to `audio_ttfp`
|
||||
and `audio_rtf`. A synthesis that bursts chunks fast enough on average but
|
||||
underruns mid-stream fails the streaming contract even when the averages
|
||||
pass.
|
||||
- Quality as a paired comparison against the reference implementation under
|
||||
the same text, speaker prompt, and seed; report sample rate and frame
|
||||
count, and assert output completeness — truncated or dropped trailing audio
|
||||
is a known failure class, not a quality nuance.
|
||||
- For realtime or duplex TTS: interruption (barge-in) latency and the
|
||||
per-session memory trend across turns.
|
||||
|
||||
## Omni models
|
||||
|
||||
Apply the e2e stage-attribution table directly to the voice-to-voice path of
|
||||
omni (speech-to-speech and multimodal AR) models:
|
||||
|
||||
- TTFT for text output, `audio_ttfp` for audio output, and per-stage
|
||||
ITL/TPOT cadence across the perception-understanding-generation chain.
|
||||
- Cross-modal parity — the omni analogue of realtime-versus-offline parity:
|
||||
the same semantic request through text and audio inputs, compared on output
|
||||
content or timing.
|
||||
- For duplex omni sessions: turn-taking latency, interruption handling, the
|
||||
concurrent session count actually validated, and a per-turn memory trend —
|
||||
session-owned KV state must not grow per turn. When either turn produces
|
||||
audio, continuity is judged with the same `audio_underrun` and
|
||||
`audio_continuity_ok` fields as TTS, since turn-taking latency alone does
|
||||
not expose mid-utterance underruns.
|
||||
|
||||
## Reviewer verification ladder
|
||||
|
||||
1. Run base/head A/B on suitable hardware when affordable.
|
||||
|
||||
@@ -121,19 +121,6 @@ Search bounded callers and sibling implementations before reporting dead code,
|
||||
duplication, or a missed consumer. Keep new abstractions only when they have a
|
||||
distinct live caller, invariant, or compatibility need.
|
||||
|
||||
Verify absence and equality claims with whole-file evidence. A line window, a
|
||||
grep for one symbol, or a commit message proves presence, never absence: to
|
||||
assert that a file matches another version or lacks a change, diff or hash the
|
||||
full files (for example `git diff <sha1>:<path> <sha2>:<path>` or the contents
|
||||
API's size and SHA fields) before anchoring the finding. A verification
|
||||
shortcut that would itself be wrong to run is not evidence.
|
||||
|
||||
For a value written or finalized in more than one place, establish which site
|
||||
executes first on each entry path — and whether the earlier site makes the
|
||||
later ones unreachable — before naming where a fix belongs. Enumerating all
|
||||
call sites correctly still misses the bug when a path the enumeration ranked
|
||||
last actually finalizes first and clears the state the proposed fix checks.
|
||||
|
||||
## Finding bar
|
||||
|
||||
Anchor each finding to a changed `path:line`; name the trigger or call path,
|
||||
|
||||
@@ -39,13 +39,13 @@ isolated detached worktree. Run every source read, `rg` search, import, and test
|
||||
from that snapshot, not the caller's checkout. A detached worktree provides
|
||||
snapshot isolation, not a security boundary.
|
||||
|
||||
Treat a fork head as untrusted until the user and environment policy
|
||||
explicitly establish trust. Without recorded trust, never run its imports,
|
||||
tests, builds, hooks, package setup, or repo-configurable linters/plugins on
|
||||
the reviewer host. Once the user explicitly trusts this pinned commit for
|
||||
this host, record the trusted SHA with the review state and execute with
|
||||
credentials, agent sockets, and other secrets out of scope. Without recorded
|
||||
trust, limit the review to the remote diff, SHA-addressed
|
||||
Treat a fork head as untrusted unless the user and environment policy explicitly
|
||||
establish trust. Never run its imports, tests, builds, hooks, package setup, or
|
||||
repo-configurable linters/plugins on the reviewer host. Execute them only in a
|
||||
disposable sandbox or VM with no credentials or inherited secrets, no host agent
|
||||
or service sockets, minimal read-only host mounts, disabled or explicitly
|
||||
allowlisted network, resource/time limits, and destruction after the run. If
|
||||
that boundary is unavailable, limit the review to the remote diff, SHA-addressed
|
||||
reads such as `git show <head_sha>:<path>`, and existing CI evidence; report all
|
||||
executable validation as a gap.
|
||||
|
||||
|
||||
@@ -52,8 +52,8 @@ Always attach markers deliberately:
|
||||
- `tts` — speech synthesis / TTS-only models (`/v1/audio/speech`, voice clone, etc.)
|
||||
- `diffusion` — generative diffusion models (image / audio / text / video from noise)
|
||||
- **Cross-cutting area** (when relevant): `parallel`, `cache`, `example`, `benchmark`
|
||||
- **Hardware**: `cpu`, `gpu`, `cuda`, `rocm`, `npu`, …; SKU (`H100`, `L4`, …) and `cards_{n}` via `@hardware_test(...)` / `hardware_marks(...)` only — never `pytest.mark.H100` (`check-mark` rejects it)
|
||||
- Optional: `slow`. Multi-card: `num_cards=N` on the helper (filter with `cards_N` / `not cards_1`)
|
||||
- **Hardware**: `cpu`, `gpu`, `cuda`, `rocm`, `npu`, `L4`, `H100`, `distributed_cuda`, …
|
||||
- Optional: `slow`, distributed markers when multi-card is required
|
||||
|
||||
**Baseline smoke (L2 + L3):** The simplest e2e case per model — default deploy, minimal request — should usually carry **both** `@pytest.mark.core_model` and `@pytest.mark.advanced_model` on the **same** test function so `test-ready.yml` and `test-merge.yml` share one test. `send_*_request` picks validation depth from `--run-level`. References: `test_voxcpm2_tts.py::test_text_to_audio_001`, `test_qwen3_tts_customvoice.py::test_text_to_audio_001`. Heavier scenarios use **`advanced_model` only**; L4 expansion uses **`full_model`**.
|
||||
|
||||
@@ -94,8 +94,8 @@ Existing references: `tests/e2e/offline_inference/test_qwen2_5_omni.py` (L2-styl
|
||||
|
||||
| Scenario | Typical location | Fixtures / runner pattern | Baseline markers & level |
|
||||
|----------|------------------|---------------------------|---------------------------|
|
||||
| **Offline inference e2e** | `tests/e2e/offline_inference/` | **Module (default):** `omni_runner` + `offline_client`. **Function (isolation only):** `omni_runner_function` + `offline_client_function`. Diffusion/TTS may use `Omni(...).generate` directly | L2: `core_model` + **one of** `omni` / `tts` / `diffusion`; `@hardware_test(...)` when GPU/NPU is required |
|
||||
| **Online serving e2e** | `tests/e2e/online_serving/` | **Module (default):** `omni_server` + `online_client`. **Function (isolation only):** `omni_server_function` + `online_client_function`. Clients: `send_omni_request` (omni), `send_audio_speech_request` (tts), `send_diffusion_request` / `send_video_diffusion_request` / `send_images_generations_request` (diffusion) | Baseline smoke: **`core_model` + `advanced_model`**; heavier paths: `advanced_model` only; L4 expansion: `full_model` |
|
||||
| **Offline inference e2e** | `tests/e2e/offline_inference/` | **Module (default):** `omni_runner` + `omni_runner_handler`. **Function (isolation only):** `omni_runner_function` + `omni_runner_handler_function`. Diffusion/TTS may use `Omni(...).generate` directly | L2: `core_model` + **one of** `omni` / `tts` / `diffusion`; `@hardware_test(...)` when GPU/NPU is required |
|
||||
| **Online serving e2e** | `tests/e2e/online_serving/` | **Module (default):** `omni_server` + `openai_client`. **Function (isolation only):** `omni_server_function` + `openai_client_function`. Clients: `send_omni_request` (omni), `send_audio_speech_request` (tts), `send_diffusion_request` / `send_video_diffusion_request` / `send_images_generations_request` (diffusion) | Baseline smoke: **`core_model` + `advanced_model`**; heavier paths: `advanced_model` only; L4 expansion: `full_model` |
|
||||
| **Documentation / runnable examples** | `tests/examples/offline_inference/`, `tests/examples/online_serving/` | **Offline docs (preferred):** extract Python/Bash blocks from the doc README (e.g. `ReadmeSnippet.extract_readme_snippets`), `pytest.mark.parametrize` each snippet, run via `example_runner.run` with a stable `output_subfolder`. **Online docs:** copy client/request scripts into dedicated tests and keep them in sync with the doc page. | Usually **L4**: `advanced_model`, often `example` plus hardware marks matching the nightly docs-example job (see `.buildkite/cuda/test-nightly.yml`). Full conventions: [docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md) (introduced in [PR #1910](https://github.com/vllm-project/vllm-omni/pull/1910): naming, output directory layout, skip rules, avoid trimming `num_inference_steps` without a strong CI reason). |
|
||||
| **Performance / benchmark** | `tests/dfx/perf/tests/*.json` + `run_*_benchmark.py` | JSON or script-driven server + load config; assert explicit metrics / baselines | L4 Perf: JSON `mark` with `full_model` + `omni`/`tts`/`diffusion`; wire `test-nightly.yml` Perf steps |
|
||||
| **Invalid parameter / negative HTTP validation** | `tests/dfx/reliability/invalid_param_test/` | Live `omni_server` + low-level `send_*_http_request` with `err_code` / `err_message` | `pytest.mark.slow` + `omni` / `tts` / `diffusion` + `@hardware_marks` (`H100` or `L4`); CI in **`test-weekly.yml`** (not ready/merge/nightly) |
|
||||
@@ -110,9 +110,9 @@ After choosing offline/online/docs/perf, classify the **product under test** and
|
||||
|-----------|------------------------------|----------------------------|----------------------------------------|
|
||||
| **What it is** | Multimodal LLM pipeline (thinker/talker/stages; text + vision + audio I/O) | Speech synthesis / voice models | Generative diffusion (noise → image, audio, text, or video) |
|
||||
| **Examples** | Qwen2.5-Omni, Qwen3-Omni | Qwen3-TTS, VoxCPM2, Higgs-Audio, Step-Audio2 | Qwen-Image, BAGEL, Wan2.2, HunyuanVideo |
|
||||
| **Offline runner** | `omni_runner` + `offline_client`, `generate_multimodal` | `Omni(...).generate` with deploy YAML + TTS params | `Omni(...).generate` + `OmniDiffusionSamplingParams` |
|
||||
| **Online client** | `online_client.send_omni_request` (chat completions, modalities) | `online_client.send_audio_speech_request` (`/v1/audio/speech`) | `send_diffusion_request` (chat/T2I), `send_video_diffusion_request` (`/v1/videos`, X2V), or `send_images_generations_http_request` / `send_images_edits_http_request` (DALL-E routes) |
|
||||
| **Typical assertions** | Stage outputs, text/audio keywords via `OfflineOmniClient` / response handler | WAV bytes, stream chunks, speech endpoint contract | Image/video dimensions, `final_output_type`, `assert_diffusion_response` |
|
||||
| **Offline runner** | `omni_runner` + `omni_runner_handler`, `generate_multimodal` | `Omni(...).generate` with deploy YAML + TTS params | `Omni(...).generate` + `OmniDiffusionSamplingParams` |
|
||||
| **Online client** | `openai_client.send_omni_request` (chat completions, modalities) | `openai_client.send_audio_speech_request` (`/v1/audio/speech`) | `send_diffusion_request` (chat/T2I), `send_video_diffusion_request` (`/v1/videos`, X2V), or `send_images_generations_http_request` / `send_images_edits_http_request` (DALL-E routes) |
|
||||
| **Typical assertions** | Stage outputs, text/audio keywords via `OmniRunnerHandler` / response handler | WAV bytes, stream chunks, speech endpoint contract | Image/video dimensions, `final_output_type`, `assert_diffusion_response` |
|
||||
| **Deploy YAML** | Per-model deploy configs (`ci/qwen3_omni_moe.yaml`, …) | `qwen3_tts.yaml`, `voxcpm2.yaml`, … | Often default serve; parallel/offload YAML for heavy DiT |
|
||||
| **Nightly group** (`test-nightly.yml`) | **Omni Model Test** — `-m "full_model and omni"` | **TTS Model Test** — `-m "full_model and tts"` | **Diffusion X2I(&A&T)** *or* **Diffusion X2V** (see Step 3 table; same `diffusion` marker, different YAML group / file shard) |
|
||||
| **L4 pressure** | Expansion per modality/model as needed | Expansion + accuracy/perf in TTS group | X2I: merge feature combos per [#1832](https://github.com/vllm-project/vllm-omni/issues/1832); X2V: separate nightly group |
|
||||
@@ -175,9 +175,9 @@ See **L1 unit test constraints (mocking)** below for the full do/don't list.
|
||||
@pytest.mark.omni
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
def test_<scenario>(omni_runner, offline_client) -> None:
|
||||
def test_<scenario>(omni_runner, omni_runner_handler) -> None:
|
||||
request_config = {"prompts": ..., "modalities": [...]} # optional: images, videos, audios
|
||||
offline_client.send_omni_request(request_config)
|
||||
omni_runner_handler.send_omni_request(request_config)
|
||||
```
|
||||
|
||||
*Offline generative e2e — **Diffusion** (representative):*
|
||||
@@ -187,8 +187,8 @@ def test_<scenario>(omni_runner, offline_client) -> None:
|
||||
@pytest.mark.diffusion
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", test_params, indirect=True)
|
||||
def test_text_to_image_001(offline_client) -> None:
|
||||
offline_client.send_diffusion_request({"prompt": "...", "extra_body": {"num_inference_steps": 4, ...}})
|
||||
def test_text_to_image_001(omni_runner_handler) -> None:
|
||||
omni_runner_handler.send_diffusion_request({"prompt": "...", "extra_body": {"num_inference_steps": 4, ...}})
|
||||
```
|
||||
|
||||
*Offline TTS e2e — **Qwen3-TTS** (two-stage; representative):*
|
||||
@@ -198,8 +198,8 @@ def test_text_to_image_001(offline_client) -> None:
|
||||
@pytest.mark.tts
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", tts_server_params, indirect=True)
|
||||
def test_text_to_audio_001(omni_runner, offline_client) -> None:
|
||||
offline_client.send_audio_speech_request({
|
||||
def test_text_to_audio_001(omni_runner, omni_runner_handler) -> None:
|
||||
omni_runner_handler.send_audio_speech_request({
|
||||
"input": "...",
|
||||
"task_type": "Base",
|
||||
"ref_audio": REF_AUDIO_URL,
|
||||
@@ -214,8 +214,8 @@ def test_text_to_audio_001(omni_runner, offline_client) -> None:
|
||||
@pytest.mark.tts
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_runner", tts_server_params, indirect=True)
|
||||
def test_voice_clone_001(offline_client) -> None:
|
||||
offline_client.send_single_stage_tts_request({
|
||||
def test_voice_clone_001(omni_runner_handler) -> None:
|
||||
omni_runner_handler.send_single_stage_tts_request({
|
||||
"input": "...",
|
||||
"language": "en",
|
||||
"prompt_audio_path": REF_AUDIO_PATH,
|
||||
@@ -231,9 +231,9 @@ def test_voice_clone_001(offline_client) -> None:
|
||||
@pytest.mark.omni
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_server", test_params, indirect=True)
|
||||
def test_text_to_text_001(omni_server, online_client) -> None:
|
||||
def test_text_to_text_001(omni_server, openai_client) -> None:
|
||||
request_config = {"model": omni_server.model, "messages": ..., "modalities": ["text"]}
|
||||
online_client.send_omni_request(request_config)
|
||||
openai_client.send_omni_request(request_config)
|
||||
```
|
||||
|
||||
*Online serving e2e — **TTS** (representative):*
|
||||
@@ -244,9 +244,9 @@ def test_text_to_text_001(omni_server, online_client) -> None:
|
||||
@pytest.mark.tts
|
||||
@hardware_test(...)
|
||||
@pytest.mark.parametrize("omni_server", tts_server_params, indirect=True)
|
||||
def test_text_to_audio_001(omni_server, online_client) -> None:
|
||||
def test_text_to_audio_001(omni_server, openai_client) -> None:
|
||||
request_config = {"model": omni_server.model, "input": "...", "response_format": "wav", ...}
|
||||
online_client.send_audio_speech_request(request_config)
|
||||
openai_client.send_audio_speech_request(request_config)
|
||||
```
|
||||
|
||||
*Online serving e2e — **Diffusion** X2I (representative):*
|
||||
@@ -255,8 +255,8 @@ def test_text_to_audio_001(omni_server, online_client) -> None:
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.diffusion
|
||||
@pytest.mark.parametrize("omni_server", _get_default_case(MODEL), indirect=True)
|
||||
def test_text_to_image_001(omni_server, online_client) -> None:
|
||||
online_client.send_diffusion_request({...}) # chat completions + extra_body
|
||||
def test_text_to_image_001(omni_server, openai_client) -> None:
|
||||
openai_client.send_diffusion_request({...}) # chat completions + extra_body
|
||||
```
|
||||
|
||||
*Online serving e2e — **Diffusion** X2V (representative):*
|
||||
@@ -264,8 +264,8 @@ def test_text_to_image_001(omni_server, online_client) -> None:
|
||||
```python
|
||||
@pytest.mark.core_model
|
||||
@pytest.mark.diffusion
|
||||
def test_text_to_video_001(omni_server, online_client) -> None:
|
||||
online_client.send_video_diffusion_request({"model": ..., "form_data": {...}}) # /v1/videos
|
||||
def test_text_to_video_001(omni_server, openai_client) -> None:
|
||||
openai_client.send_video_diffusion_request({"model": ..., "form_data": {...}}) # /v1/videos
|
||||
```
|
||||
|
||||
*Documentation example tests:* follow the **Preferred Test Strategy** in [l4_doc_example_tests.inc.md](../../../docs/contributing/ci/test_examples/l4_doc_example_tests.inc.md): dynamic extraction for offline READMEs; explicit copied client code for online pages until extraction is justified; use the documented **naming**, **output directory** (page folder + case id), and **skipping** rules (e.g. Gradio-only scripts).
|
||||
@@ -274,7 +274,7 @@ def test_text_to_video_001(omni_server, online_client) -> None:
|
||||
|
||||
**3. Cross-cutting rules**
|
||||
|
||||
- New `.py` files need the Omni SPDX header (`Copyright contributors to the vLLM-Omni project`, not `vLLM project`). `tests/` may keep stdlib `re`/`base64`; pickle is still banned unless the file is already on the pickle allowlist. Every collected `tests/**/test_*.py` needs a CI **level** mark and a **hardware** platform mark/helper (`check-mark` runs locally; GHA skips it). Do not write `pytest.mark.H100` / `pytest.mark.L4`; use the helpers so `cards_{n}` is attached. Do not grow `allowed_files` to land a test. Policy: [docs/contributing/README.md](../../../docs/contributing/README.md#linting).
|
||||
- New `.py` files need the Omni SPDX header (`Copyright contributors to the vLLM-Omni project`, not `vLLM project`). `tests/` may keep stdlib `re`/`base64`; pickle is still banned unless the file is already on the pickle allowlist. Every collected `tests/**/test_*.py` needs a CI **level** mark and a **hardware** mark/helper (`check-test-ci-coverage` runs locally; GHA skips it). Do not grow `allowed_files` to land a test. Policy: [docs/contributing/README.md](../../../docs/contributing/README.md#linting).
|
||||
- Reuse existing fixtures for the chosen scenario; do not mix “online client” assumptions into offline `OmniRunner` tests without a clear reason.
|
||||
- Avoid external network dependency in assertions unless the scenario is explicitly “online serving” or doc examples that require a model hub (then align with CI secrets/cache).
|
||||
- Keep **one test function = one intent** (one modality combo, one endpoint contract, or one acceleration combo).
|
||||
@@ -324,21 +324,21 @@ vLLM-Omni e2e tests start a real **OmniServer** (online) or **OmniRunner** (offl
|
||||
|
||||
| Scope | Online fixtures | Offline fixtures | When to use |
|
||||
|-------|-----------------|------------------|-------------|
|
||||
| **Module** (default) | `omni_server` → `online_client` | `omni_runner` → `offline_client` | **Default for L2/L3/L4 expansion** — amortize model/server init across `test_*` in the same module. Same `OmniServerParams` / runner config can be reused by multiple tests. |
|
||||
| **Function** | `omni_server_function` → `online_client_function` | `omni_runner_function` → `offline_client_function` | **Only when required** — each `test_*` must get a **clean** server/runner (no shared engine/GPU state). Typical: `tests/dfx/reliability/`, sleep/wakeup, crash/restart, tests that mutate global server state. |
|
||||
| **Module** (default) | `omni_server` → `openai_client` | `omni_runner` → `omni_runner_handler` | **Default for L2/L3/L4 expansion** — amortize model/server init across `test_*` in the same module. Same `OmniServerParams` / runner config can be reused by multiple tests. |
|
||||
| **Function** | `omni_server_function` → `openai_client_function` | `omni_runner_function` → `omni_runner_handler_function` | **Only when required** — each `test_*` must get a **clean** server/runner (no shared engine/GPU state). Typical: `tests/dfx/reliability/`, sleep/wakeup, crash/restart, tests that mutate global server state. |
|
||||
|
||||
**Rules:**
|
||||
|
||||
1. **Default to module scope** (`omni_server` / `omni_runner`) unless the scenario **explicitly** needs a fresh instance per test function.
|
||||
2. **Indirect parametrize name must match the fixture name**: `@pytest.mark.parametrize("omni_server", ...)` with `omni_server` + `online_client`; `@pytest.mark.parametrize("omni_server_function", ...)` with `omni_server_function` + `online_client_function`. Do not mix module fixture with function client (or vice versa).
|
||||
2. **Indirect parametrize name must match the fixture name**: `@pytest.mark.parametrize("omni_server", ...)` with `omni_server` + `openai_client`; `@pytest.mark.parametrize("omni_server_function", ...)` with `omni_server_function` + `openai_client_function`. Do not mix module fixture with function client (or vice versa).
|
||||
3. Different `OmniServerParams` per `test_*` (e.g. default vs `--enable-cpu-offload`) is still OK with **module** `omni_server` — pytest parametrizes per test node; only switch to `_function` when isolation between tests matters, not merely because `server_args` differ.
|
||||
4. One `test_*` with **many** server configs (expansion matrix) → single function + `@pytest.mark.parametrize("omni_server", [...], indirect=True)` + module `omni_server` (see in-tree `test_qwen_image_expansion.py`).
|
||||
|
||||
```python
|
||||
# Default — module-scoped server (L4 expansion)
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=H100)], indirect=True)
|
||||
def test_foo_images_generations_default_1024(omni_server, online_client) -> None:
|
||||
online_client.send_images_generations_request({...})
|
||||
def test_foo_images_generations_default_1024(omni_server, openai_client) -> None:
|
||||
openai_client.send_images_generations_request({...})
|
||||
|
||||
# Function-scoped — reliability / per-test clean state only
|
||||
@pytest.mark.parametrize(
|
||||
@@ -346,9 +346,9 @@ def test_foo_images_generations_default_1024(omni_server, online_client) -> None
|
||||
[pytest.param(OmniServerParams(model=MODEL), marks=H100)],
|
||||
indirect=True,
|
||||
)
|
||||
def test_foo_sleep_wakeup_cycle(omni_server_function, online_client_function) -> None:
|
||||
online_client_function.send_omni_sleep_http_request({...})
|
||||
online_client_function.send_omni_wakeup_http_request({...})
|
||||
def test_foo_sleep_wakeup_cycle(omni_server_function, openai_client_function) -> None:
|
||||
openai_client_function.send_omni_sleep_http_request({...})
|
||||
openai_client_function.send_omni_wakeup_http_request({...})
|
||||
```
|
||||
|
||||
#### Runtime send helpers — online **and** offline (`tests/helpers/runtime.py`)
|
||||
@@ -357,19 +357,19 @@ def test_foo_sleep_wakeup_cycle(omni_server_function, online_client_function) ->
|
||||
|
||||
| Principle | Action |
|
||||
|-----------|--------|
|
||||
| **Reuse first** | Grep `runtime.py` for an existing **`send_*_request`** on `OnlineOmniClient` (online) or `OfflineOmniClient` (offline) that matches the endpoint / pipeline shape. |
|
||||
| **Reuse first** | Grep `runtime.py` for an existing **`send_*_request`** on `OpenAIClientHandler` (online) or `OmniRunnerHandler` (offline) that matches the endpoint / pipeline shape. |
|
||||
| **Extend when close** | If an existing `send_*_request` almost fits (e.g. missing one optional field), extend it in `runtime.py` — do not fork logic in the test file. |
|
||||
| **Add when missing** | No suitable helper → add **`send_<feature>_request`** (high-level: call + general `assert_*`) or **`send_<route>_<verb>_http_request`** (low-level HTTP for negative/dfx) in **`runtime.py` first**, then call it from tests. |
|
||||
| **Test module owns payload only** | In `test_*.py`: `MODEL`, deploy path, vendored media, `get_prompt()`, and inline **`request_config` dicts** only. **No** `omni.generate(...)`, raw `requests.post`, OpenAI SDK calls, or `_collect_audio()` / `_process_output()` in e2e tests. |
|
||||
|
||||
**Online** (`online_client` from `omni_server`): `OnlineOmniClient.send_*_request`.
|
||||
**Online** (`openai_client` from `omni_server`): `OpenAIClientHandler.send_*_request`.
|
||||
|
||||
**Offline** (`offline_client` from `omni_runner`): `OfflineOmniClient.send_*_request`.
|
||||
**Offline** (`omni_runner_handler` from `omni_runner`): `OmniRunnerHandler.send_*_request`.
|
||||
|
||||
| Do | Don't |
|
||||
|----|-------|
|
||||
| Online: `online_client.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_video_diffusion_request`, `send_images_generations_request`, … | `requests.post(f"{base_url}/v1/...", json=…)` or `client.chat.completions.create(...)` inside a test |
|
||||
| Offline: `offline_client.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_single_stage_tts_request`, `send_single_stage_tts_batch_request`, … | `omni_runner.omni.generate(...)` + hand-rolled tensor/WAV extraction in `test_*.py` |
|
||||
| Online: `openai_client.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_video_diffusion_request`, `send_images_generations_request`, … | `requests.post(f"{base_url}/v1/...", json=…)` or `client.chat.completions.create(...)` inside a test |
|
||||
| Offline: `omni_runner_handler.send_omni_request`, `send_diffusion_request`, `send_audio_speech_request`, `send_single_stage_tts_request`, `send_single_stage_tts_batch_request`, … | `omni_runner.omni.generate(...)` + hand-rolled tensor/WAV extraction in `test_*.py` |
|
||||
| Add missing `send_*` to **`runtime.py`** first; bundle general `assert_*` inside the send helper | A one-off `def _post_*` / `def _collect_audio` at the bottom of a test module |
|
||||
| Mirror naming/style of neighboring `send_*` (docstring, `request_config` dict, optional `run_level`, `err_code` / `err_message` for negative cases) | Different parameter shapes per test file for the same endpoint |
|
||||
|
||||
@@ -392,7 +392,7 @@ def test_foo_sleep_wakeup_cycle(omni_server_function, online_client_function) ->
|
||||
|
||||
Changing `request_config` fields (`size`, `n`, `seed`, server flags) is **not** special validation — the general `assert_*` should read those from `request_config` inside `send_*`.
|
||||
|
||||
**`send_*` ↔ `assert_*` pairing (online `OnlineOmniClient`):**
|
||||
**`send_*` ↔ `assert_*` pairing (online `OpenAIClientHandler`):**
|
||||
|
||||
| High-level `send_*` (prefer in L2+ e2e) | Assert already invoked inside `runtime.py` | Low-level HTTP-only sibling |
|
||||
|----------------------------------------|--------------------------------------------|-----------------------------|
|
||||
@@ -405,7 +405,7 @@ Changing `request_config` fields (`size`, `n`, `seed`, server flags) is **not**
|
||||
|
||||
**Rule:** Tests call high-level `send_*_request` for the general contract. **Never** call the same bundled `assert_*` again. Call an extra `assert_*` in the test **only** for special, case-specific validation.
|
||||
|
||||
**Common `OnlineOmniClient` entry points** (non-exhaustive — grep `runtime.py` before adding):
|
||||
**Common `OpenAIClientHandler` entry points** (non-exhaustive — grep `runtime.py` before adding):
|
||||
|
||||
| Area | High-level (SDK + assert) | Low-level HTTP (`*_http_request`) |
|
||||
|------|---------------------------|-----------------------------------|
|
||||
@@ -416,7 +416,7 @@ Changing `request_config` fields (`size`, `n`, `seed`, server flags) is **not**
|
||||
| TTS (online) | `send_audio_speech_request` | `send_audio_speech_http_request`, `send_audio_generate_http_request`, … |
|
||||
| Ops / meta | — | `send_health_http_request`, `send_models_http_request`, `send_omni_sleep_http_request`, … |
|
||||
|
||||
**Common `OfflineOmniClient` entry points** (offline — grep `runtime.py` before adding):
|
||||
**Common `OmniRunnerHandler` entry points** (offline — grep `runtime.py` before adding):
|
||||
|
||||
| Area | High-level (`send_*_request` + assert) | Notes |
|
||||
|------|----------------------------------------|-------|
|
||||
@@ -426,7 +426,7 @@ Changing `request_config` fields (`size`, `n`, `seed`, server flags) is **not**
|
||||
| Single-stage TTS (Coqui XTTS, MOSS-TTS-Nano, …) | `send_single_stage_tts_request`, `send_single_stage_tts_batch_request` | `prompt` + `additional_information` + `omni.generate` — **do not** duplicate in tests |
|
||||
| New model family | **Add** `send_<family>_request` here first | Then call from `tests/e2e/offline_inference/test_*.py` |
|
||||
|
||||
L1 tests under `tests/entrypoints/` may use **FastAPI `TestClient`** or direct handler calls with mocks; they do **not** need `OnlineOmniClient` / `OfflineOmniClient`. **L2+ online and offline e2e** must use `runtime.py` `send_*` helpers.
|
||||
L1 tests under `tests/entrypoints/` may use **FastAPI `TestClient`** or direct handler calls with mocks; they do **not** need `OpenAIClientHandler` / `OmniRunnerHandler`. **L2+ online and offline e2e** must use `runtime.py` `send_*` helpers.
|
||||
|
||||
#### Invalid parameter validation (`tests/dfx/reliability/invalid_param_test/`)
|
||||
|
||||
@@ -452,7 +452,7 @@ When the user asks for **invalid parameter validation**, **invalid request bodie
|
||||
| **Module markers** | `pytestmark = [pytest.mark.slow, pytest.mark.<omni or tts or diffusion>]` |
|
||||
| **Server fixture** | `_PARAMS` / `_QWEN3_TTS_SPEECH`-style list of `pytest.param(OmniServerParams(...), id="...", marks=hardware_marks(...))`; `@pytest.mark.parametrize("omni_server", _PARAMS, indirect=True)` |
|
||||
| **Hardware** | `hardware_marks(res={"cuda": "H100"})` for heavy diffusion/omni/video; `hardware_marks(res={"cuda": "L4"})` for smaller TTS models (must match weekly `-m "slow and L4"` step) |
|
||||
| **HTTP client** | **Low-level** `online_client.send_*_http_request({..., "err_code": 400, "err_message": (...)} )` — **not** `send_*_request` (success path) |
|
||||
| **HTTP client** | **Low-level** `openai_client.send_*_http_request({..., "err_code": 400, "err_message": (...)} )` — **not** `send_*_request` (success path) |
|
||||
| **Case shape** | Prefer **one `test_*` per route family** + `@pytest.mark.parametrize("body_spec, err_message", [...])` with stable `id=` per case; or dedicated `test_<route>_malformed_json` when not parametrized |
|
||||
| **Body helpers** | `_minimal_<endpoint>_json(omni_server)` / `_minimal_*_form_data()`; merge overrides with `body.update(body_spec)` |
|
||||
| **Known gaps** | `pytest.mark.skip(reason="…#3649")` as `_SKIP_ISSUE_3649` when server validation is not yet strict (mirror neighboring cases) |
|
||||
@@ -468,10 +468,10 @@ When the user asks for **invalid parameter validation**, **invalid request bodie
|
||||
| **Invalid parameters Test · H100** | `pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and H100"` | Diffusion / omni / video invalid-param tests with `H100` hardware mark |
|
||||
| **Invalid parameters Test · L4** | `pytest -s -v tests/dfx/reliability/invalid_param_test/ -m "slow and L4"` | TTS / lighter models marked `L4` |
|
||||
|
||||
- **Trigger:** `build.env("WEEKLY") == "1"` (all weekly jobs) or PR label `weekly-test` (Reliability / Perf, then filtered by `source_file_dependencies`).
|
||||
- **Trigger:** `build.env("WEEKLY") == "1"` or PR label `weekly-test`.
|
||||
- **Default:** extending an existing `invalid_param_test` script needs **no YAML edit** — the weekly steps already sweep the whole directory.
|
||||
- **Edit YAML only** when adding a **new hardware queue**, a **new top-level script** that must run in isolation, or a **model-specific weekly shard** (mirror neighboring reliability steps).
|
||||
- Weekly Reliability / Perf jobs declare `source_file_dependencies`. Filtering applies only on the `weekly-test` label; `WEEKLY=1` on `main` keeps every job in those groups. CPU coverage (`WEEKLY=1` only) and the E2E Tests group (`NON_CRITICAL=1` only) do **not** use this field.
|
||||
- Weekly steps **do not** use `source_file_dependencies`.
|
||||
|
||||
**Example — append to `test_invalid_image_generation.py`:**
|
||||
|
||||
@@ -485,13 +485,13 @@ When the user asks for **invalid parameter validation**, **invalid request bodie
|
||||
@pytest.mark.parametrize("omni_server", _PARAMS, indirect=True)
|
||||
def test_images_generations_invalid_requests(
|
||||
omni_server: OmniServer,
|
||||
online_client: OnlineOmniClient,
|
||||
openai_client: OpenAIClientHandler,
|
||||
body_spec: dict[str, object],
|
||||
err_message: str | tuple[str, ...],
|
||||
) -> None:
|
||||
body = _minimal_images_gen_json(omni_server)
|
||||
body.update(body_spec)
|
||||
online_client.send_images_generations_http_request(
|
||||
openai_client.send_images_generations_http_request(
|
||||
{"json": body, "timeout": 300, "err_code": 400, "err_message": err_message}
|
||||
)
|
||||
```
|
||||
@@ -542,18 +542,18 @@ def send_images_generations_request(self, request_config: dict[str, Any], ...) -
|
||||
|
||||
# tests/e2e/.../test_foo_expansion.py — one case per test_* (no case_id branching)
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True)
|
||||
def test_foo_images_generations_default_1024(omni_server, online_client) -> None:
|
||||
online_client.send_images_generations_request({"json": body, "timeout": 300})
|
||||
def test_foo_images_generations_default_1024(omni_server, openai_client) -> None:
|
||||
openai_client.send_images_generations_request({"json": body, "timeout": 300})
|
||||
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True)
|
||||
def test_foo_images_generations_sizes_256_512_1024(omni_server, online_client) -> None:
|
||||
def test_foo_images_generations_sizes_256_512_1024(omni_server, openai_client) -> None:
|
||||
for size in ("256x256", "512x512", "1024x1024"):
|
||||
online_client.send_images_generations_request({"json": {**body, "size": size}, "timeout": 300})
|
||||
openai_client.send_images_generations_request({"json": {**body, "size": size}, "timeout": 300})
|
||||
|
||||
# chat diffusion — separate test; send_diffusion_request bundles assert_diffusion_response
|
||||
@pytest.mark.parametrize("omni_server", [pytest.param(OmniServerParams(model=MODEL), marks=SINGLE_L4)], indirect=True)
|
||||
def test_foo_chat_completions_t2i_fallback(omni_server, online_client) -> None:
|
||||
online_client.send_diffusion_request({...})
|
||||
def test_foo_chat_completions_t2i_fallback(omni_server, openai_client) -> None:
|
||||
openai_client.send_diffusion_request({...})
|
||||
```
|
||||
|
||||
L1 tests under `tests/entrypoints/` may keep **minimal** asserts next to the handler under test when validating a single branch; still prefer `assertions.py` when the same JSON/media check appears in more than one file or level (e.g. L1 protocol test + L4 e2e).
|
||||
@@ -644,7 +644,7 @@ If the test is not already collected by an existing pipeline command (for exampl
|
||||
| **L2** | `test-ready.yml` with `core_model` + `source_file_dependencies` | merge / nightly pipelines |
|
||||
| **Invalid param** | `test-weekly.yml` — **Invalid parameters Test** group (`-m "slow and H100"` / `-m "slow and L4"`). Usually **no YAML edit** when appending cases to existing `invalid_param_test` scripts | `test-ready.yml`, `test-merge.yml`, `test-nightly.yml` |
|
||||
|
||||
`test-nightly.yml` shards use **explicit pytest file paths** in `commands` plus `source_file_dependencies`; PR trigger is `nightly-test` (path filter), or `NIGHTLY=1` on `main` (all jobs). Only suggest merge/ready YAML when the requested level is L3/L2 (or the user explicitly asks for multi-level CI).
|
||||
`test-nightly.yml` shards use **explicit pytest file paths** in `commands` (and PR labels like `diffusion-x2iat-test`); they generally **do not** use `source_file_dependencies`. Only suggest merge/ready YAML when the requested level is L3/L2 (or the user explicitly asks for multi-level CI).
|
||||
|
||||
**`test-nightly.yml` top-level groups** (each uses `full_model` + type marker in pytest `-m`):
|
||||
|
||||
@@ -653,7 +653,7 @@ If the test is not already collected by an existing pipeline command (for exampl
|
||||
| **Omni Model Test** | `omni` | Function / doc / accuracy / **perf** for Qwen-Omni family |
|
||||
| **TTS Model Test** | `tts` | `-m "full_model and L4 and tts"` (and H100 variants when enabled); **Perf Test** uses `run_benchmark.py` + `test_tts.json` |
|
||||
| **Diffusion X2I(&A&T) Model Test** | `diffusion` | Image/audio/text diffusion; explicit file shards for **Function**; **Perf Test · <Model>** uses `run_diffusion_benchmark.py` + JSON config |
|
||||
| **Diffusion X2V Model Test** | `diffusion` | Video-only (Wan, HunyuanVideo, …); separate **Perf** steps when in-tree |
|
||||
| **Diffusion X2V Model Test** | `diffusion` | Video-only (Wan, HunyuanVideo, …); PR label `diffusion-x2v-test`; separate **Perf** steps when in-tree |
|
||||
|
||||
Guidelines when editing YAML:
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ Every model-centric e2e test must declare **exactly one** type marker:
|
||||
| **Diffusion X2I(&A&T) Model Test** | x2**i** / x2**a** / x2**t** — image, audio, text (non-video) | Qwen-Image*, BAGEL, FLUX, SD3, Z-Image, LongCat, DreamZero |
|
||||
| **Diffusion X2V Model Test** | x2**v** — video only | Wan2.2, HunyuanVideo 1.5, Wan VACE |
|
||||
|
||||
PR label for L4 on a branch: `nightly-test` (jobs then filtered by `source_file_dependencies`). Scheduled `main` uses `NIGHTLY=1` and runs every nightly job. L5 Reliability / Perf use `weekly-test` with the same path filter; CPU coverage and the weekly E2E group are env-only (`WEEKLY=1` / `NON_CRITICAL=1`) and are not source-filtered.
|
||||
PR labels for selective nightly runs: `diffusion-x2iat-test`, `diffusion-x2v-test` (plus `nightly-test` / `NIGHTLY=1`).
|
||||
|
||||
## Level and Marker Mapping
|
||||
|
||||
@@ -79,7 +79,7 @@ E2E (L2+) should not rely on mocks unless documenting a rare exception; prefer r
|
||||
- `full_model` — L4 nightly (`test-nightly.yml`); some expansion tests still carry both `advanced_model` and `full_model` during migration
|
||||
2. **Model type** (pick one): `omni`, `tts`, or `diffusion`
|
||||
3. **Cross-cutting** when relevant: `parallel`, `cache`, `example`, `benchmark`
|
||||
4. **Hardware**: `cpu`, `cuda`, `rocm`, `npu`, `L4`, `H100`, `cards_1`, …
|
||||
4. **Hardware**: `cpu`, `cuda`, `rocm`, `npu`, `L4`, `H100`, `distributed_cuda`, …
|
||||
5. Multi-card: `@hardware_test(...)` in `tests/helpers/mark.py`
|
||||
|
||||
## Command Templates
|
||||
@@ -238,13 +238,13 @@ pytest -s -v dfx/reliability/invalid_param_test/ -m "slow and L4"
|
||||
pytest -s -v dfx/reliability/invalid_param_test/test_invalid_image_generation.py::test_images_generations_invalid_requests -m "slow and H100"
|
||||
```
|
||||
|
||||
**Trigger:** `WEEKLY=1` (all jobs) or PR label `weekly-test` (Reliability / Perf, filtered by `source_file_dependencies`). Appending cases to existing scripts usually needs **no YAML edit** (directory sweep).
|
||||
**Trigger:** `WEEKLY=1` or PR label `weekly-test`. Appending cases to existing scripts usually needs **no YAML edit** (directory sweep). No `source_file_dependencies` on weekly steps.
|
||||
|
||||
### Platform-targeted examples
|
||||
|
||||
```bash
|
||||
cd tests
|
||||
pytest -s -v -m "core_model and L4 and not cards_1" --run-level=core_model
|
||||
pytest -s -v -m "core_model and distributed_cuda and L4" --run-level=core_model
|
||||
```
|
||||
|
||||
### Concrete e2e paths (common in-tree)
|
||||
@@ -279,7 +279,7 @@ When adding or modifying tests, do not stop at “where the file lives” — al
|
||||
4. **API client + assert placement**: **General** validation → implement in `assertions.py`, call **inside** `send_*_request` in `runtime.py`; tests call **`send_*_request` only**. **Special** case validation → `assert_*` in `assertions.py`, called **in the test** after `send_*_request`. Low-level `send_*_http_request` is for negative/dfx tests (`err_code`), not ordinary L2+ success e2e.
|
||||
5. **Shared assertions**: logic in **`tests/helpers/assertions.py`**; general checks inside `send_*_request`; special checks only in the test. No `_assert_*` in `test_*.py`.
|
||||
6. **One case → one `test_*`**: function name reflects what is validated; no `if case_id == ...` mega-test merging multiple scenarios.
|
||||
7. **Fixture scope**: default **`omni_server` + `online_client`** / **`omni_runner` + `offline_client`** (module). Use **`omni_server_function` + `online_client_function`** / **`omni_runner_function` + `offline_client_function`** only when each `test_*` must spawn a fresh instance. Parametrize name must match fixture (`omni_server` vs `omni_server_function`).
|
||||
7. **Fixture scope**: default **`omni_server` + `openai_client`** / **`omni_runner` + `omni_runner_handler`** (module). Use **`omni_server_function` + `openai_client_function`** / **`omni_runner_function` + `omni_runner_handler_function`** only when each `test_*` must spawn a fresh instance. Parametrize name must match fixture (`omni_server` vs `omni_server_function`).
|
||||
8. **Type marker**: `omni`, `tts`, or `diffusion` on every model e2e module.
|
||||
9. **Diffusion L4 Function**: wire `*_expansion.py` into **X2I(&A&T)** or **X2V** **Function Test** in **`test-nightly.yml` only** — do not add `test-merge.yml` unless the user also requested L3.
|
||||
10. **Diffusion L4 Perf** (only when requested): add `tests/dfx/perf/tests/test_<slug>_vllm_omni.json` + **Perf Test · <Model>** step (artifact upload); not part of “L4 functional cases” by default.
|
||||
|
||||
@@ -67,11 +67,6 @@
|
||||
# File-level overrides within entrypoints/
|
||||
/vllm_omni/entrypoints/openai/errors.py @alex-jw-brooks @NickCao
|
||||
/vllm_omni/entrypoints/cli/benchmark/ @alex-jw-brooks @Bounty-hunter
|
||||
# Shared wire codec: protocol/realtime (OpenAI Realtime GA) and protocol/duplex
|
||||
# (vLLM-Omni's extension of it) — RFC #6592 P0a. Layer-neutral by design — the
|
||||
# engine depends on it — so it is owned by the serving-boundary document rather
|
||||
# than by any one consumer. — docs/design/module/entrypoints.md
|
||||
/vllm_omni/protocol/ @alex-jw-brooks @linyueqian @NickCao
|
||||
|
||||
# Error contracts — docs/design/module/error_contracts.md
|
||||
/vllm_omni/errors.py @alex-jw-brooks @NickCao
|
||||
|
||||
@@ -12,7 +12,4 @@ PLEASE FILL IN THE PR DESCRIPTION HERE.
|
||||
## Test Result
|
||||
|
||||
**BEFORE SUBMITTING:** read [CONTRIBUTING.md](https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md) and run the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) with the code agent for a self-check against project conventions.
|
||||
|
||||
> [!NOTE]
|
||||
> To avoid agent spam, vLLM-Omni limits contributors without write access to **5 open PRs**. If this blocks well-intentioned critical work, please contact a committer to request bypass list access.
|
||||
(anything written below this line will be removed by GitHub Actions)
|
||||
@@ -56,35 +56,6 @@ jobs:
|
||||
run: |
|
||||
bash scripts/build_wheel.sh --python python
|
||||
|
||||
- name: Validate runtime package data from source archive
|
||||
run: |
|
||||
set -euo pipefail
|
||||
archive_dir="$(mktemp -d)"
|
||||
install_dir="$(mktemp -d)"
|
||||
vcs_wheels=(dist/*.whl)
|
||||
if [[ ${#vcs_wheels[@]} -ne 1 || ! -f "${vcs_wheels[0]}" ]]; then
|
||||
echo "Expected exactly one VCS-built wheel" >&2
|
||||
exit 1
|
||||
fi
|
||||
git archive HEAD | tar -x -C "${archive_dir}"
|
||||
(
|
||||
cd "${archive_dir}"
|
||||
VLLM_OMNI_TARGET_DEVICE=cuda \
|
||||
VLLM_OMNI_VERSION_OVERRIDE=0.0.0 \
|
||||
python -m build --wheel --outdir archive-dist
|
||||
)
|
||||
wheels=("${archive_dir}"/archive-dist/*.whl)
|
||||
if [[ ${#wheels[@]} -ne 1 || ! -f "${wheels[0]}" ]]; then
|
||||
echo "Expected exactly one archive-built wheel" >&2
|
||||
exit 1
|
||||
fi
|
||||
python -m pip install --no-deps --target "${install_dir}" "${wheels[0]}"
|
||||
python tools/check_wheel_package_data.py \
|
||||
--source-root "${archive_dir}" \
|
||||
--wheel "${wheels[0]}" \
|
||||
--vcs-wheel "${vcs_wheels[0]}" \
|
||||
--installed-root "${install_dir}"
|
||||
|
||||
- name: Upload Wheel Artifact
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
|
||||
@@ -27,4 +27,4 @@ jobs:
|
||||
# Don't run historical-debt hooks against the whole tree in CI yet.
|
||||
# They still run locally on changed files.
|
||||
env:
|
||||
SKIP: check-mark,markdownlint-cli2,shellcheck,check-spdx-header,mypy-3.10
|
||||
SKIP: check-test-ci-coverage,markdownlint-cli2,shellcheck,check-spdx-header,mypy-3.10
|
||||
@@ -243,15 +243,9 @@ datasets/
|
||||
*.csv
|
||||
*.json
|
||||
!recipes/cosmos3/negative_prompt.json
|
||||
!benchmarks/mammoth_moda2/reference_results.json
|
||||
!tests/dfx/perf/tests/*.json
|
||||
!apps/ComfyUI-vLLM-Omni/example_workflows/*.json
|
||||
# Omni-DuplexEval CI config JSON (e.g. accuracy/omni_duplex_eval)
|
||||
!tests/e2e/accuracy/omni_duplex_eval/*.json
|
||||
*.jsonl
|
||||
# Shared Hunyuan IT2I input for accuracy tests and vllm bench.
|
||||
!tests/assets/hunyuan_image3/it2i.jsonl
|
||||
!tests/assets/livekit/client_vad_replay.jsonl
|
||||
*.parquet
|
||||
|
||||
# Output files
|
||||
|
||||
@@ -105,8 +105,8 @@ repos:
|
||||
<<: *mypy_common
|
||||
stages: [manual]
|
||||
|
||||
- id: check-mark
|
||||
name: Ensure test files have CI marks (no direct SKU)
|
||||
- id: check-test-ci-coverage
|
||||
name: Ensure test files have pytestmarks
|
||||
entry: python tools/pre_commit/check_test_marks.py
|
||||
language: python
|
||||
types: [python]
|
||||
|
||||
@@ -2,8 +2,6 @@
|
||||
|
||||
You may find information about contributing to vLLM-Omni on [Contributing](https://vllm-omni.readthedocs.io/en/latest/contributing/).
|
||||
|
||||
AI-assisted contributions must follow the [AI Usage Policy](docs/contributing/README.md#ai-usage-policy), including disclosure, human review, and validation requirements.
|
||||
|
||||
Local `pre-commit` gates (SPDX, forbidden imports, `torch.cuda`, shellcheck,
|
||||
mypy, test marks, TTS adapter ratchet, markdownlint, Buildkite schema, and
|
||||
how to extend allowlists) are documented in the
|
||||
@@ -11,5 +9,3 @@ how to extend allowlists) are documented in the
|
||||
section of that guide.
|
||||
|
||||
Before submitting a PR, run the [precheck-pr skill](https://github.com/vllm-project/vllm-omni/blob/main/.claude/skills/precheck-pr/SKILL.md) with the code agent for a self-check against project conventions.
|
||||
|
||||
vLLM-Omni limits contributors without write access to **5 open PRs**; see the [Pull Request Limit](docs/contributing/README.md#pull-request-limit) section of the contributing guide.
|
||||
@@ -16,8 +16,6 @@ Easy, fast, and cheap omni-modality model serving for everyone
|
||||
|
||||
*Latest News* 🔥
|
||||
|
||||
- [2026/09] We released [0.30.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.30.0), rebased onto vLLM 0.30.0, featuring a unified full-duplex serving framework around engine-owned sessions for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md) and AURA, native cross-stage KV and multimodal payload transfer (Mooncake AR-to-DiT handoff and NIXL connectors), interactive world-model serving with [LingBot World](recipes/Robbyant/LingBot-World-2.0.md), and realtime [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) Turbo inference on NVIDIA Blackwell and Ascend 950.
|
||||
- [2026/08] We released [0.28.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.28.0), featuring production-ready [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) serving on GPU and NPU, a unified AR/DiT paged KV cache runtime, and enhanced realtime full-duplex serving for the [MiniCPM-o series](recipes/OpenBMB/MiniCPM-o-4_5.md).
|
||||
- [2026/08] [VeRL-Omni](https://github.com/verl-project/verl-omni) `v0.2.0` is released: faster diffusion RL powered by vLLM-Omni (request-level/step-wise batching with FA3), rebuilt Qwen3-Omni multimodal training (DPO & GSPO), plus LTX-2.3, Qwen-Image-Edit support and more. See the [release notes](https://github.com/verl-project/verl-omni/releases/tag/v0.2.0).
|
||||
- [2026/08] We released [0.26.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.26.0) - aligned with the vLLM 0.26 release line, featuring [MiniMax H3](recipes/MiniMaxAI/MiniMax-H3.md) joint video/audio generation, an experimental full-duplex realtime runtime for [MiniCPM-o 4.5](recipes/OpenBMB/MiniCPM-o-4_5.md), distributed layerwise diffusion offload, and broader model, hardware, streaming, TTS, and quantization support.
|
||||
- [2026/07] We released [0.24.0](https://github.com/vllm-project/vllm-omni/releases/tag/v0.24.0) - aligned with the vLLM 0.24 release line, expanding production-ready coverage across TTS, speech, diffusion, image/video generation, and robot-policy serving, with major Omni stage runtime refactoring, diffusion request-level batching, async output materialization, quantization/cache/memory improvements, and broad CUDA/ROCm/XPU/NPU support.
|
||||
@@ -54,14 +52,14 @@ vLLM-Omni is flexible and easy to use with:
|
||||
- Tensor, pipeline, data and expert parallelism support for distributed inference
|
||||
- Streaming outputs
|
||||
- OpenAI-compatible API server
|
||||
- [Full-duplex realtime serving](docs/serving/realtime_duplex_api.md) with streaming audio input and output
|
||||
- Full-duplex realtime serving with streaming audio input and output (experimental)
|
||||
|
||||
vLLM-Omni seamlessly supports most popular open-source models on HuggingFace, including:
|
||||
|
||||
- **Omni-modality models** (e.g. Qwen3-Omni, MiniCPM-o 4.5, Cosmos3, HunyuanImage, BAGEL)
|
||||
- **TTS models** (e.g. Qwen3-TTS, Tencent AuK, Breeze-TTS-2, CosyVoice3)
|
||||
- **Diffusion models** — image, video, and audio generation (e.g. MiniMax H3, LingBot World, MAGI-2, LTX-2.5, Wan2.2)
|
||||
- **Robot-policy and action models** (e.g. π0.5, GR00T-N1.7, DreamZero-DROID, InternVLA-A1)
|
||||
- **TTS models** (e.g. Qwen3-TTS, VoxCPM2, Ming-Omni-TTS, CosyVoice3)
|
||||
- **Diffusion models** — image, video, and audio generation (e.g. MiniMax H3, Qwen-Image, Wan2.2, FLUX)
|
||||
- **Robot-policy and action models** (e.g. GR00T-N1.7, DreamZero-DROID, InternVLA-A1, Cosmos3 action policy)
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -69,7 +67,6 @@ Visit our [documentation](https://vllm-omni.readthedocs.io/en/latest/) to learn
|
||||
|
||||
- [Installation](https://vllm-omni.readthedocs.io/en/latest/getting_started/installation/)
|
||||
- [Quickstart](https://vllm-omni.readthedocs.io/en/latest/getting_started/quickstart/)
|
||||
- [Nightly CUDA Docker images](https://hub.docker.com/r/vllm/vllm-omni/tags?name=nightly) built from the latest `main`
|
||||
- [List of Supported Models](https://vllm-omni.readthedocs.io/en/latest/models/supported_models/)
|
||||
- [Deployment Recipes](https://recipes.vllm.ai) for vLLM-Omni model serving
|
||||
|
||||
|
||||
@@ -13,6 +13,7 @@ It can send model inference requests to either a locally running vLLM-Omni servi
|
||||
> [!TIP]
|
||||
> If you run both ComfyUI and vLLM-Omni on the same device, you can create separate virtual environments and use different Python versions for them.
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
Copy this folder to the `custom_nodes` subfolder of your ComfyUI installation. Your directory should look like `ComfyUI/custom_nodes/ComfyUI-vLLM-Omni`.
|
||||
@@ -20,10 +21,9 @@ Copy this folder to the `custom_nodes` subfolder of your ComfyUI installation. Y
|
||||
If you are running ComfyUI during copying, you should restart ComfyUI to load this extension.
|
||||
|
||||
> [!TIP]
|
||||
> You can use utility websites such as <https://download-directory.github.io/> to download a subdirectory of a repo. Also checkout community discussions (e.g., <https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository>) for more info.
|
||||
> You can use utility websites such as https://download-directory.github.io/ to download a subdirectory of a repo. Also checkout community discussions (e.g., https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repository) for more info.
|
||||
|
||||
On the device and virtual environment you run ComfyUI, launch ComfyUI with
|
||||
|
||||
```bash
|
||||
cd ComfyUI
|
||||
|
||||
@@ -35,7 +35,6 @@ python main.py --cpu
|
||||
```
|
||||
|
||||
On the device and virtual environment you run vLLM-Omni, start a model service with
|
||||
|
||||
```bash
|
||||
vllm serve The_Model_ID_to_Serve --omni --port 8000
|
||||
```
|
||||
@@ -49,31 +48,14 @@ This extension offers the following nodes based on the output modalities (at **C
|
||||
|
||||
- **Generate Image** for text-to-image and image-to-image tasks
|
||||
- **Generate Video** for text-to-video, first-frame/image-to-video, and reference-conditioned video
|
||||
- **Latent Mask Editing** for MiniMax-H3 latent-mask editing (source media plus video/audio noise masks)
|
||||
- **FastH3 Deployment** for routing text-to-video requests to a MiniMax-H3 server with FastH3 fused at startup
|
||||
- **Multimodality Understanding** for multimodality-to-text and multimodality-to-audio tasks
|
||||
- **TTS** and **TTS Voice Clone** for TTS tasks
|
||||
- **Generate Music** for lyrics-and-description-to-music generation
|
||||
|
||||
This extension also offers example workflows (at **ComfyUI sidebar -> Templates -> vLLM-Omni**)
|
||||
|
||||
> [!NOTE]
|
||||
> The node UI and feature designs are intended to match vLLM-Omni online serving interfaces. It cannot offer more than what the interfaces support.
|
||||
|
||||
Every node carries the vLLM-Omni mark in its title bar and is tinted by what it outputs, so a graph is readable at a glance:
|
||||
|
||||
| Colour | Nodes | What they produce |
|
||||
| --- | --- | --- |
|
||||
| Blue | Generate Image, Generate Video, Multimodality Understanding, TTS, TTS Voice Clone | A generated image, video, audio, or text. These are the only nodes that reach a server. |
|
||||
| Amber | AR / Diffusion / Multi-Stage Sampling Params | Sampling parameters that apply to any model |
|
||||
| Purple | Qwen TTS Params, Wan Video Params, MiniMax-H3 Video Params | Parameters that only one model family accepts |
|
||||
| Red | LoRA, FastH3 Deployment | Which weights the server is expected to have loaded |
|
||||
| Teal | Video References, Latent Mask Editing | Reference media |
|
||||
|
||||
Recolouring a node by hand (right click -> Colors) overrides its tint, and the choice is kept.
|
||||
|
||||
**Generate Video** takes a clip length in seconds (`duration`), not a frame count. Frames stay the wire unit and are derived with the node's `fps`, so the length is always measured against the rate that is actually served; models that accept only certain frame counts still round to their own lattice server-side. Graphs saved before this widget existed stored `num_frames` in its place and are converted on load, using the fps recorded alongside it -- the browser console names every node it rewrites.
|
||||
|
||||
To build a simple workflow yourself,
|
||||
|
||||
- Drag a generation node onto the canvas.
|
||||
@@ -135,146 +117,11 @@ You can configure per-stage sampling parameters for multi-stage models.
|
||||
</p>
|
||||
|
||||
> [!TIP]
|
||||
> Connect **frame** for backward-compatible first-frame / image-to-video behavior. For MiniMax-H3 FL2VA,
|
||||
> connect **first_frame**, **last_frame**, or both to condition the start frame, end frame, or both.
|
||||
> Connect a **frame** image for first-frame / image-to-video (e.g., Wan I2V, MiniMax-H3 FL2VA).
|
||||
>
|
||||
> For reference-conditioned generation (MiniMax-H3 Ref2VA), connect a **Video References** node instead.
|
||||
>
|
||||
> Do not combine `frame` with `first_frame` or `last_frame`, and do not combine any frame input with
|
||||
> `references`. Task routing is automatic from which inputs you connect.
|
||||
|
||||
For MiniMax-H3 Ref2VA, **Video References** accepts up to 9 images (`image_1`–`image_9`),
|
||||
3 videos (`video_1`–`video_3`), and 3 audio clips (`audio_1`–`audio_3`), with at most
|
||||
12 connected inputs in total. Any mixture containing at least one image or video
|
||||
is supported; empty and audio-only references are rejected. For example, you can
|
||||
combine 6 images, 3 videos, and 3 audio clips in one request.
|
||||
|
||||
Within each media type, references follow slot-number order, skipping unconnected
|
||||
slots. For example, connecting `image_2` and `image_9` sends `image_2` as the first
|
||||
image and `image_9` as the second. Each image slot uses the first image in its batch.
|
||||
Existing connections to `image_1`, `image_2`, `audio_1`, `audio_2`, `video_1`, and
|
||||
`video_2` remain valid in saved workflows.
|
||||
|
||||
#### MiniMax-H3 Reference to Video
|
||||
|
||||
Load [MiniMax-H3 Reference to Video](example_workflows/vLLM-Omni%20MiniMax-H3%20Reference%20to%20Video.json)
|
||||
from **Templates → ComfyUI-vLLM-Omni**, or drag the JSON onto the canvas.
|
||||
It adapts the [official ComfyUI R2V workflow](https://github.com/Comfy-Org/workflow_templates/blob/main/templates/video_minimax_h3_r2v.json)
|
||||
to remote vLLM-Omni execution. Model loading, sampling, and VAE decoding run on
|
||||
the server; ComfyUI loads references and saves the returned video.
|
||||
|
||||
Use an extension version containing the audio-preserving output fix from
|
||||
[#7456](https://github.com/vllm-project/vllm-omni/pull/7456). Older versions can
|
||||
drop the server's audio while decoding the response.
|
||||
|
||||
1. Start a Ref2VA-capable service using the [MiniMax-H3 recipe](../../recipes/MiniMaxAI/MiniMax-H3.md).
|
||||
Set **Generate Video** to its `/v1` URL and served model name.
|
||||
2. Select your own image in **Load Image**. The template's filenames are placeholders
|
||||
relative to ComfyUI's input directory; no sample assets or model weights are bundled.
|
||||
3. Connect **Load Video** and **Load Audio** when needed. Duplicate the loaders to
|
||||
fill more reference slots, or disconnect the image when using video-only input.
|
||||
Keep the references connected to **Generate Video** and leave `frame` disconnected.
|
||||
4. Match prompt tags such as `<Picture 1>`, `<Video 1>`, and `<Audio 1>` to the
|
||||
connected references, counting each media type separately and skipping empty slots.
|
||||
5. Run the workflow. **Save Video** writes an MP4 under `output/video/` and preserves
|
||||
the generated audio when the audio-output prerequisite is installed.
|
||||
|
||||
The default is **1344×768, 24 FPS, 124 frames** (about 5.17 seconds), 50 sampling
|
||||
points, video flow shift 12, audio flow shift 3, and seed 42. The duration widget
|
||||
is set to 5.167 seconds, which converts to 124 frames at 24 FPS. Other H3 canvas presets
|
||||
are 1024×768, 768×768, 768×1024, and 768×1344. Keep frame counts at `17k+5` within
|
||||
the 4–15 second output range; examples are 107, 124, 209, and 345 frames.
|
||||
Reference video/audio clips must each be 2–15 seconds, with at most 15 seconds of
|
||||
reference video and at most 15 seconds of standalone reference audio per request.
|
||||
|
||||
For **Ref2VA Turbo**, use the same template. Configure a Ref2VA-only server with
|
||||
`--lora-backend peft --lora-path "$TURBO_LORA" --task-type ref2va`, following the
|
||||
recipe's hardware setup. For `minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors`,
|
||||
set **LoRA** `local_path` to that file's path on the server and connect it to
|
||||
**Generate Video**. Use `num_inference_steps=5`, `flow_shift=12`,
|
||||
`audio_flow_shift=3`, and LoRA `scale=1`. The API counts sigma points, so five
|
||||
points produce four denoiser evaluations. Use the Diffusers artifact, not its
|
||||
`_comfyui_` export. To return to base mode, disconnect LoRA and restore 50 steps.
|
||||
The 8-step v1.0 Ref2VA adapter instead requires nine sampling points and video
|
||||
flow shift 6. FastH3 is a separate startup-fused T2VA deployment.
|
||||
|
||||
Validate the template's wiring and defaults locally with:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/e2e/features/comfyui/test_h3_reference_workflow.py -q
|
||||
```
|
||||
|
||||
For a real-model check, run the workflow against H3 and inspect the saved file:
|
||||
|
||||
```bash
|
||||
ffprobe -v error -show_entries stream=codec_type,width,height,r_frame_rate,sample_rate,channels \
|
||||
-show_entries format=duration -of json "$OUTPUT_MP4"
|
||||
```
|
||||
|
||||
Check for 24 FPS video, stereo audio, the requested canvas, and aligned duration;
|
||||
also play the result to assess reference conditioning and audio/video synchronization.
|
||||
Record the server commit, model/adapter, input assets, prompt, settings, and output
|
||||
alongside the result. Local schema or mocked-server checks do not establish H3
|
||||
generation quality or replace this real-model validation.
|
||||
|
||||
### MiniMax H3 Text to Video
|
||||
|
||||
Import [the H3 text-to-video template](example_workflows/MiniMax_H3_Text_to_Video.json)
|
||||
for remote video generation with audio. It includes connected base presets and
|
||||
optional Turbo sampling, H3 parameters, and Remote LoRA nodes. See the
|
||||
[workflow guide](docs/minimax-h3-t2v.md) for server setup, artifact-specific Turbo
|
||||
settings, and saved-video/audio validation. Set Remote LoRA’s `local_path` to
|
||||
the downloaded Turbo artifact on your server before enabling Turbo.
|
||||
|
||||
#### H3 video upscale (WF-07)
|
||||
|
||||
The **vLLM-Omni MiniMax H3 Video Upscale** template generates video remotely, upscales it with SeedVR2, and saves the original and upscaled videos with the generated audio and FPS. See [workflow setup](docs/wf07-h3-upscale.md).
|
||||
|
||||
#### FastH3 text-to-video
|
||||
|
||||
FastH3 is fused into MiniMax-H3 when the vLLM-Omni server starts; it is not a request-switchable LoRA. Download one adapter variant and start a non-offloaded FL2VA server. For the Dense / Data-Free profile:
|
||||
|
||||
```bash
|
||||
export FASTH3_DIR=/path/to/fasth3
|
||||
hf download FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA \
|
||||
dense-datafree/adapter_model.safetensors --local-dir "${FASTH3_DIR}"
|
||||
|
||||
vllm serve /path/to/MiniMax-H3 \
|
||||
--omni \
|
||||
--task-type fl2va \
|
||||
--lora-path "${FASTH3_DIR}/dense-datafree/adapter_model.safetensors" \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
For the VSA / Data-Free profile, download `vsa-datafree/adapter_model.safetensors`, use that file as `--lora-path`, and add:
|
||||
|
||||
```bash
|
||||
--diffusion-attention-backend FASTVIDEO_VSA \
|
||||
--fastvideo-vsa-topk 64
|
||||
```
|
||||
|
||||
The VSA profile also requires a compatible `fastvideo-kernel` installation. See the [MiniMax-H3 FastH3 recipe](../../recipes/MiniMaxAI/MiniMax-H3.md#fasth3-adapter) for the full serving contract, supported parallel layouts, and kernel requirements.
|
||||
|
||||
Open the **vLLM-Omni FastH3 Text to Video** template, then:
|
||||
|
||||
- Set the server URL and served model name on **FastH3 Deployment**.
|
||||
- Connect its output to **Generate Video → fast_h3**. When connected, the deployment node's URL and model take precedence over the corresponding Generate Video widgets.
|
||||
- Keep `frame`, `references`, **LoRA**, and **MiniMax-H3 Video Params** disconnected. FastH3 Preview v1 supports T2VA only, is already fused, and owns both flow shifts.
|
||||
- A connected **Diffusion Sampling Params** node may set seed and other ordinary sampling options. The integration always enforces four denoising steps and 24 FPS for FastH3.
|
||||
|
||||
The node records which server the workflow targets; it does not start one, nor switch adapters or attention backends on a running server.
|
||||
|
||||
#### Latent-mask editing (MiniMax-H3)
|
||||
|
||||
The [WF-05 template](example_workflows/vLLM-Omni%20MiniMax-H3%20Latent%20Mask%20Editing.json) contains inpainting, object removal, continuation, and extension examples in one graph. See the [workflow guide](docs/wf05-h3-latent-editing.md) for inputs, mask settings, dependencies, and preview limitations.
|
||||
|
||||
Connect a **Latent Mask Editing** node to **Generate Video → latent_edit** to edit a source clip instead of generating from scratch. It uploads the source media and serializes the video/audio noise masks the MiniMax H3 API accepts:
|
||||
|
||||
- `source_video` / `source_audio` — the media to edit.
|
||||
- `video_mask` — a ComfyUI mask image; `0` preserves a region, `1` regenerates it, fractional values blend. A 2D mask `[H, W]` is applied to every frame; a 3D mask `[T, H, W]` is treated as a temporal mask (one slice per frame) for continuation or extension. The client area-downsamples it by 16 to keep the upload small, and the server resizes it to the video latent grid.
|
||||
- `audio_mask` — a scalar in `[0, 1]`; `0` keeps the source audio, `1` regenerates it, fractional values blend.
|
||||
|
||||
The node requires at least one mask; the server then enforces the cross-field rules (a source requires a matching mask, and a non-trivial mask requires its source). This node only forwards inputs to the server; the served model must declare latent-mask editing support (MiniMax-H3).
|
||||
> Do not use `frame` and `references` together. Task routing is automatic from which inputs you connect.
|
||||
|
||||
### TTS (e.g., Qwen TTS series)
|
||||
|
||||
@@ -290,17 +137,6 @@ The node requires at least one mask; the server then enforces the cross-field ru
|
||||
> [!TIP]
|
||||
> There is a dedicated node for VoiceClone tasks with reference audio input. Other simple text-to-speech tasks should use the regular TTS node.
|
||||
|
||||
### Music generation (e.g., MiniMax Music 3)
|
||||
|
||||
(Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Music Generation**)
|
||||
|
||||
<p align="center">
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/apps/ComfyUI-vLLM-Omni/docs/images/comfyui-music-generation.jpg">
|
||||
<img alt="vLLM-Omni Music Generation" src="https://raw.githubusercontent.com/vllm-project/vllm-omni/refs/heads/main/apps/ComfyUI-vLLM-Omni/docs/images/comfyui-music-generation.jpg" width=55%>
|
||||
</picture>
|
||||
</p>
|
||||
|
||||
### Chaining multiple model services
|
||||
|
||||
(Also available at **ComfyUI sidebar->Template->vLLM-Omni->vLLM-Omni Chaining Services**)
|
||||
@@ -316,8 +152,6 @@ The node requires at least one mask; the server then enforces the cross-field ru
|
||||
|
||||
Follow the [development convention and rules of vLLM-Omni](https://docs.vllm.ai/projects/vllm-omni/en/latest/contributing/).
|
||||
|
||||
Node tints and the title-bar mark are applied in `web/main.js`, keyed off each node's declared output types rather than a list of node names. A new node that returns an existing type is themed with no front-end change; a new output type needs one entry in `FAMILY_BY_OUTPUT` there.
|
||||
|
||||
## Limitation and Non-Goals
|
||||
|
||||
- Single server mode only. No automatic load balancing or failover.
|
||||
@@ -344,11 +178,11 @@ Whenever you find an issue or problem, please
|
||||
|
||||
Features
|
||||
|
||||
- <https://github.com/dougbtv/comfyui-vllm-omni/> The official reference implementation for ComfyUI integration with vLLM-Omni's DALL-E compatible image generation API.
|
||||
- <https://github.com/Comfy-Org/ComfyUI/tree/master/comfy_extras> ComfyUI's built-in node implementations.
|
||||
- https://github.com/dougbtv/comfyui-vllm-omni/ The official reference implementation for ComfyUI integration with vLLM-Omni's DALL-E compatible image generation API.
|
||||
- https://github.com/Comfy-Org/ComfyUI/tree/master/comfy_extras ComfyUI's built-in node implementations.
|
||||
|
||||
UI/UX design references
|
||||
|
||||
- <https://github.com/sgl-project/sglang/pull/15271> SGLang Diffusion's official ComfyUI integration for image and video generation.
|
||||
- <https://github.com/SXQBW/ComfyUI-Qwen-Omni> A third party ComfyUI integration for Qwen Omni series.
|
||||
- <https://github.com/flybirdxx/ComfyUI-Qwen-TTS> <https://github.com/DarioFT/ComfyUI-Qwen3-TTS> Third party ComfyUI integrations for Qwen TTS series.
|
||||
- https://github.com/sgl-project/sglang/pull/15271 SGLang Diffusion's official ComfyUI integration for image and video generation.
|
||||
- https://github.com/SXQBW/ComfyUI-Qwen-Omni A third party ComfyUI integration for Qwen Omni series.
|
||||
- https://github.com/flybirdxx/ComfyUI-Qwen-TTS https://github.com/DarioFT/ComfyUI-Qwen3-TTS Third party ComfyUI integrations for Qwen TTS series.
|
||||
@@ -1,8 +1,4 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
# ruff: noqa: N999
|
||||
|
||||
"""Top-level package for comfyui_vllm_omni.""" # This is not a Python library intended to be imported.
|
||||
"""Top-level package for comfyui_vllm_omni.""" # noqa: N999 # This is not a python library intended to be imported
|
||||
|
||||
__all__ = [
|
||||
"NODE_CLASS_MAPPINGS",
|
||||
@@ -17,13 +13,9 @@ __version__ = "0.0.1"
|
||||
from .comfyui_vllm_omni.nodes import (
|
||||
VLLMOmniARSampling,
|
||||
VLLMOmniDiffusionSampling,
|
||||
VLLMOmniFastH3Deployment,
|
||||
VLLMOmniGenerateImage,
|
||||
VLLMOmniGenerateMusic,
|
||||
VLLMOmniGenerateVideo,
|
||||
VLLMOmniLatentMaskEditing,
|
||||
VLLMOmniMiniMaxH3Params,
|
||||
VLLMOmniMiniMaxH3TemporalMask,
|
||||
VLLMOmniQwenTTSParams,
|
||||
VLLMOmniRemoteLoRA,
|
||||
VLLMOmniSamplingParamsList,
|
||||
@@ -41,17 +33,13 @@ NODE_CLASS_MAPPINGS = {
|
||||
"VLLMOmniGenerateVideo": VLLMOmniGenerateVideo,
|
||||
"VLLMOmniUnderstanding": VLLMOmniUnderstanding,
|
||||
"VLLMOmniTTS": VLLMOmniTTS,
|
||||
"VLLMOmniGenerateMusic": VLLMOmniGenerateMusic,
|
||||
"VLLMOmniVoiceClone": VLLMOmniVoiceClone,
|
||||
"VLLMOmniVideoReferences": VLLMOmniVideoReferences,
|
||||
"VLLMOmniLatentMaskEditing": VLLMOmniLatentMaskEditing,
|
||||
"VLLMOmniMiniMaxH3TemporalMask": VLLMOmniMiniMaxH3TemporalMask,
|
||||
# === Params ===
|
||||
"VLLMOmniARSampling": VLLMOmniARSampling,
|
||||
"VLLMOmniDiffusionSampling": VLLMOmniDiffusionSampling,
|
||||
"VLLMOmniSamplingParamsList": VLLMOmniSamplingParamsList,
|
||||
"VLLMOmniRemoteLoRA": VLLMOmniRemoteLoRA,
|
||||
"VLLMOmniFastH3Deployment": VLLMOmniFastH3Deployment,
|
||||
"VLLMOmniQwenTTSParams": VLLMOmniQwenTTSParams,
|
||||
"VLLMOmniWanParams": VLLMOmniWanParams,
|
||||
"VLLMOmniMiniMaxH3Params": VLLMOmniMiniMaxH3Params,
|
||||
@@ -64,17 +52,13 @@ NODE_DISPLAY_NAME_MAPPINGS = {
|
||||
"VLLMOmniGenerateVideo": "Generate Video",
|
||||
"VLLMOmniUnderstanding": "Multimodality Understanding",
|
||||
"VLLMOmniTTS": "TTS (Text to Speech)",
|
||||
"VLLMOmniGenerateMusic": "Generate Music",
|
||||
"VLLMOmniVoiceClone": "TTS Voice Cloning",
|
||||
"VLLMOmniVideoReferences": "Video References",
|
||||
"VLLMOmniLatentMaskEditing": "Latent Mask Editing",
|
||||
"VLLMOmniMiniMaxH3TemporalMask": "MiniMax-H3 Temporal Mask",
|
||||
# === Params ===
|
||||
"VLLMOmniARSampling": "AR Sampling Params",
|
||||
"VLLMOmniDiffusionSampling": "Diffusion Sampling Params",
|
||||
"VLLMOmniSamplingParamsList": "Multi-Stage Sampling Params List",
|
||||
"VLLMOmniRemoteLoRA": "LoRA",
|
||||
"VLLMOmniFastH3Deployment": "FastH3 Deployment",
|
||||
"VLLMOmniQwenTTSParams": "Qwen TTS Params",
|
||||
"VLLMOmniWanParams": "Wan Video Params",
|
||||
"VLLMOmniMiniMaxH3Params": "MiniMax-H3 Video Params",
|
||||
|
||||
@@ -1,25 +1,15 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
import math
|
||||
from typing import Literal
|
||||
|
||||
import torch
|
||||
from comfy_api.input import AudioInput, VideoInput
|
||||
|
||||
from .utils.api_client import VLLMOmniClient
|
||||
from .utils.latent_mask import _align_frame_count
|
||||
from .utils.logger import get_logger
|
||||
from .utils.models import lookup_model_spec
|
||||
from .utils.types import (
|
||||
MAX_REFERENCE_AUDIOS,
|
||||
MAX_REFERENCE_IMAGES,
|
||||
MAX_REFERENCE_VIDEOS,
|
||||
AudioFormat,
|
||||
AutoregressionSamplingParams,
|
||||
DiffusionSamplingParams,
|
||||
FastH3Deployment,
|
||||
LatentMaskEditing,
|
||||
MiniMaxH3ModelSpecificParams,
|
||||
QwenTTSModelSpecificParams,
|
||||
VideoReferences,
|
||||
@@ -32,23 +22,6 @@ from .utils.validators import (
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
FASTH3_INFERENCE_STEPS = 4
|
||||
FASTH3_FPS = 24
|
||||
# A FastH3 deployment may expose H3 under any --served-model-name, so the payload
|
||||
# spec has to be named outright rather than recovered from that alias.
|
||||
FASTH3_SPEC_MODEL = "MiniMax-H3"
|
||||
|
||||
|
||||
def _resolve_fast_h3_deployment(deployment: dict) -> tuple[str, str]:
|
||||
if not isinstance(deployment, dict):
|
||||
raise ValueError("FastH3 deployment must be provided by a FastH3 Deployment node.")
|
||||
|
||||
url = str(deployment.get("url") or "").strip().rstrip("/")
|
||||
model = str(deployment.get("model") or "").strip()
|
||||
if not url or not model:
|
||||
raise ValueError("FastH3 deployment requires both URL and model.")
|
||||
return url, model
|
||||
|
||||
|
||||
class _VLLMOmniGenerateBase:
|
||||
"""Base class for vLLM-Omni generation nodes with shared functionality."""
|
||||
@@ -189,31 +162,14 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
"width": ("INT", {"default": 832, "min": 1}),
|
||||
"height": ("INT", {"default": 480, "min": 1}),
|
||||
"fps": ("INT", {"default": 16, "min": 1}),
|
||||
"duration": (
|
||||
"FLOAT",
|
||||
{
|
||||
"default": 4.0,
|
||||
"min": 0.1,
|
||||
"step": 0.1,
|
||||
"round": 0.001,
|
||||
"tooltip": (
|
||||
"Clip length in seconds, converted to frames with the fps above. "
|
||||
"Models that only accept certain frame counts (e.g. MiniMax-H3) round to "
|
||||
"their own lattice, so the served clip can be slightly longer than requested."
|
||||
),
|
||||
},
|
||||
),
|
||||
"num_frames": ("INT", {"default": 41, "min": 1}),
|
||||
},
|
||||
"optional": {
|
||||
"frame": ("IMAGE",),
|
||||
"first_frame": ("IMAGE",),
|
||||
"last_frame": ("IMAGE",),
|
||||
"references": ("VIDEO_REFERENCES",),
|
||||
"sampling_params": ("SAMPLING_PARAMS",),
|
||||
"lora": ("REMOTE_LORA",),
|
||||
"model_params": ("VIDEO_PARAMS",),
|
||||
"fast_h3": ("FASTH3_DEPLOYMENT",),
|
||||
"latent_edit": ("LATENT_MASK_EDITING",),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -222,30 +178,12 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
FUNCTION = "generate"
|
||||
|
||||
@classmethod
|
||||
def VALIDATE_INPUTS(
|
||||
cls,
|
||||
url,
|
||||
model,
|
||||
frame=None,
|
||||
first_frame=None,
|
||||
last_frame=None,
|
||||
references=None,
|
||||
fast_h3=None,
|
||||
**_kwargs,
|
||||
) -> str | Literal[True]:
|
||||
def VALIDATE_INPUTS(cls, url, model, frame=None, references=None, **_kwargs) -> str | Literal[True]:
|
||||
base = super().VALIDATE_INPUTS(url, model)
|
||||
if base is not True:
|
||||
return base
|
||||
if frame is not None and references is not None:
|
||||
return "Provide only one of frame or references, not both."
|
||||
if frame is not None and (first_frame is not None or last_frame is not None):
|
||||
return "Provide either frame or first_frame/last_frame, not both."
|
||||
if references is not None and (first_frame is not None or last_frame is not None):
|
||||
return "Provide either first_frame/last_frame or references, not both."
|
||||
if fast_h3 is not None and any(value is not None for value in (frame, first_frame, last_frame, references)):
|
||||
return (
|
||||
"FastH3 Preview supports T2VA only; disconnect frame, first_frame, last_frame, and references inputs."
|
||||
)
|
||||
return True
|
||||
|
||||
async def generate(
|
||||
@@ -256,72 +194,20 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
width: int,
|
||||
height: int,
|
||||
fps: int,
|
||||
duration: float,
|
||||
num_frames: int,
|
||||
negative_prompt: str | None = None,
|
||||
frame: torch.Tensor | None = None,
|
||||
first_frame: torch.Tensor | None = None,
|
||||
last_frame: torch.Tensor | None = None,
|
||||
references: dict | None = None,
|
||||
sampling_params: dict | list[dict] | None = None,
|
||||
model_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
fast_h3: dict | None = None,
|
||||
latent_edit: dict | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
logger.debug("Got sampling params: %s", sampling_params)
|
||||
logger.debug("Got model params: %s", model_params)
|
||||
|
||||
# Which spec builds the payload. Only a FastH3 deployment separates this
|
||||
# from the served name; every other path keeps them equal.
|
||||
spec_model = model
|
||||
|
||||
if fast_h3 is not None:
|
||||
if any(value is not None for value in (frame, first_frame, last_frame, references)):
|
||||
raise ValueError(
|
||||
"FastH3 Preview supports T2VA only; disconnect frame, first_frame, "
|
||||
"last_frame, and references inputs."
|
||||
)
|
||||
if lora is not None:
|
||||
raise ValueError(
|
||||
"FastH3 is already fused into the selected server; disconnect the request-level LoRA input."
|
||||
)
|
||||
|
||||
url, model = _resolve_fast_h3_deployment(fast_h3)
|
||||
logger.info("Using FastH3 deployment at %s", url)
|
||||
fps = FASTH3_FPS
|
||||
# The served name is whatever the operator passed to --served-model-name.
|
||||
# Left alone, lookup_model_spec would miss H3 for an alias such as
|
||||
# "fasth3", drop the params builder, and send a t2va request carrying no
|
||||
# aspect_ratio -- which the server refuses.
|
||||
spec_model = FASTH3_SPEC_MODEL
|
||||
|
||||
if sampling_params is None:
|
||||
sampling_params = DiffusionSamplingParams()
|
||||
elif isinstance(sampling_params, list):
|
||||
if len(sampling_params) != 1:
|
||||
raise ValueError("FastH3 expects a single diffusion sampling params group.")
|
||||
sampling_params = sampling_params[0].__class__(sampling_params[0])
|
||||
else:
|
||||
sampling_params = sampling_params.__class__(sampling_params)
|
||||
sampling_params["num_inference_steps"] = FASTH3_INFERENCE_STEPS
|
||||
|
||||
# FastH3 owns both modality shifts. Sending the ordinary H3 values
|
||||
# from a connected H3 Params node would turn a deployment choice
|
||||
# into a request-level override, which the server intentionally
|
||||
# rejects when it differs from the fused adapter contract.
|
||||
if model_params is not None:
|
||||
model_params = model_params.__class__(model_params)
|
||||
model_params.pop("flow_shift", None)
|
||||
model_params.pop("audio_flow_shift", None)
|
||||
|
||||
# Frames stay the wire unit. Convert after the FastH3 branch above, so the
|
||||
# duration is measured against the fps the server will actually apply.
|
||||
num_frames = max(1, round(duration * fps))
|
||||
|
||||
validate_model_and_sampling_params_types(spec_model, sampling_params)
|
||||
validate_model_and_sampling_params_types(model, sampling_params)
|
||||
|
||||
# Currently, all video generation models are single-stage diffusion models
|
||||
if isinstance(sampling_params, list):
|
||||
@@ -341,11 +227,8 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
client = VLLMOmniClient(url)
|
||||
output = await client.generate_video(
|
||||
model=model,
|
||||
spec_model=spec_model,
|
||||
prompt=prompt,
|
||||
frame=frame, # frame present => fl2va / Wan I2V
|
||||
first_frame=first_frame,
|
||||
last_frame=last_frame,
|
||||
references=references,
|
||||
width=width,
|
||||
height=height,
|
||||
@@ -355,7 +238,6 @@ class VLLMOmniGenerateVideo(_VLLMOmniGenerateBase):
|
||||
sampling_params=sampling_params,
|
||||
lora=lora,
|
||||
model_params=model_params,
|
||||
latent_edit=latent_edit,
|
||||
)
|
||||
return (output,)
|
||||
|
||||
@@ -536,66 +418,6 @@ class VLLMOmniTTS(_VLLMOmniGenerateBase):
|
||||
return (audio,)
|
||||
|
||||
|
||||
class VLLMOmniGenerateMusic(_VLLMOmniGenerateBase):
|
||||
"""Generate a song from lyrics and a musical description with MiniMax Music 3."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"url": ("STRING", {"default": "http://localhost:8000/v1"}),
|
||||
"model": ("STRING", {"default": "MiniMaxAI/MiniMax-Music3"}),
|
||||
"instructions": (
|
||||
"STRING",
|
||||
{
|
||||
"multiline": True,
|
||||
"display_name": "caption",
|
||||
"tooltip": "Music caption: describe genre, instruments, tempo and mood.",
|
||||
},
|
||||
),
|
||||
"lyrics": ("STRING", {"multiline": True}),
|
||||
"max_duration_seconds": (
|
||||
"FLOAT",
|
||||
{
|
||||
"default": 300.0,
|
||||
"min": 1,
|
||||
"max": 360,
|
||||
"step": 0.01,
|
||||
"tooltip": "Upper limit; rounded down to whole 25 Hz audio frames. May end earlier.",
|
||||
},
|
||||
),
|
||||
"seed": ("INT", {"default": 0, "min": 0, "max": 2**53 - 1, "control_after_generate": True}),
|
||||
"response_format": (["wav", "mp3", "flac", "opus"],),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("AUDIO",)
|
||||
RETURN_NAMES = ("audio",)
|
||||
FUNCTION = "generate"
|
||||
|
||||
async def generate(
|
||||
self,
|
||||
url: str,
|
||||
model: str,
|
||||
lyrics: str,
|
||||
instructions: str,
|
||||
response_format: AudioFormat,
|
||||
max_duration_seconds: float,
|
||||
seed: int = 0,
|
||||
) -> tuple[AudioInput]:
|
||||
audio = await VLLMOmniClient(url.rstrip("/")).generate_speech(
|
||||
model=model,
|
||||
input=lyrics,
|
||||
instructions=instructions,
|
||||
voice="default",
|
||||
speed=1.0,
|
||||
response_format=response_format,
|
||||
max_new_tokens=int(max_duration_seconds * 25),
|
||||
seed=seed,
|
||||
)
|
||||
return (audio,)
|
||||
|
||||
|
||||
class VLLMOmniVoiceClone(_VLLMOmniGenerateBase):
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -873,52 +695,6 @@ class VLLMOmniRemoteLoRA:
|
||||
return (lora,)
|
||||
|
||||
|
||||
class VLLMOmniFastH3Deployment:
|
||||
"""Select a vLLM-Omni service that fused FastH3 at startup."""
|
||||
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"url": (
|
||||
"STRING",
|
||||
{
|
||||
"default": "http://localhost:8000/v1",
|
||||
"tooltip": "URL of a vLLM-Omni server started with a FastH3 --lora-path.",
|
||||
},
|
||||
),
|
||||
"model": (
|
||||
"STRING",
|
||||
{
|
||||
"default": "MiniMaxAI/MiniMax-H3",
|
||||
"tooltip": "Model name exposed by the FastH3 deployment.",
|
||||
},
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("FASTH3_DEPLOYMENT",)
|
||||
RETURN_NAMES = ("deployment",)
|
||||
FUNCTION = "get_deployment"
|
||||
CATEGORY = "vLLM-Omni"
|
||||
DESCRIPTION = (
|
||||
"Selects a server with FastH3 fused at startup. The connected Generate Video node uses T2VA, "
|
||||
"four inference steps, and 24 FPS without sending a request-level LoRA."
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def VALIDATE_INPUTS(cls, url, model) -> str | Literal[True]:
|
||||
try:
|
||||
_resolve_fast_h3_deployment({"url": url, "model": model})
|
||||
except ValueError as exc:
|
||||
return str(exc)
|
||||
return True
|
||||
|
||||
def get_deployment(self, url: str, model: str):
|
||||
url, model = _resolve_fast_h3_deployment({"url": url, "model": model})
|
||||
return (FastH3Deployment({"url": url, "model": model}),)
|
||||
|
||||
|
||||
class VLLMOmniQwenTTSParams:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
@@ -1014,10 +790,6 @@ class VLLMOmniVideoReferences:
|
||||
"audio_2": ("AUDIO",),
|
||||
"video_1": ("VIDEO",),
|
||||
"video_2": ("VIDEO",),
|
||||
# Append ports to preserve connections in saved workflows.
|
||||
**{f"image_{i}": ("IMAGE",) for i in range(3, MAX_REFERENCE_IMAGES + 1)},
|
||||
**{f"audio_{i}": ("AUDIO",) for i in range(3, MAX_REFERENCE_AUDIOS + 1)},
|
||||
**{f"video_{i}": ("VIDEO",) for i in range(3, MAX_REFERENCE_VIDEOS + 1)},
|
||||
},
|
||||
}
|
||||
|
||||
@@ -1034,112 +806,21 @@ class VLLMOmniVideoReferences:
|
||||
audio_2: AudioInput | None = None,
|
||||
video_1: VideoInput | None = None,
|
||||
video_2: VideoInput | None = None,
|
||||
image_3: torch.Tensor | None = None,
|
||||
image_4: torch.Tensor | None = None,
|
||||
image_5: torch.Tensor | None = None,
|
||||
image_6: torch.Tensor | None = None,
|
||||
image_7: torch.Tensor | None = None,
|
||||
image_8: torch.Tensor | None = None,
|
||||
image_9: torch.Tensor | None = None,
|
||||
audio_3: AudioInput | None = None,
|
||||
video_3: VideoInput | None = None,
|
||||
**kwargs,
|
||||
):
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
refs = VideoReferences()
|
||||
for kind, values in (
|
||||
("image", (image_1, image_2, image_3, image_4, image_5, image_6, image_7, image_8, image_9)),
|
||||
("video", (video_1, video_2, video_3)),
|
||||
("audio", (audio_1, audio_2, audio_3)),
|
||||
):
|
||||
for index, value in enumerate(values, start=1):
|
||||
if value is not None:
|
||||
refs[f"{kind}_{index}"] = value
|
||||
if image_1 is not None:
|
||||
refs["image_1"] = image_1
|
||||
if image_2 is not None:
|
||||
refs["image_2"] = image_2
|
||||
if audio_1 is not None:
|
||||
refs["audio_1"] = audio_1
|
||||
if audio_2 is not None:
|
||||
refs["audio_2"] = audio_2
|
||||
if video_1 is not None:
|
||||
refs["video_1"] = video_1
|
||||
if video_2 is not None:
|
||||
refs["video_2"] = video_2
|
||||
return (refs,)
|
||||
|
||||
|
||||
class VLLMOmniLatentMaskEditing:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {},
|
||||
"optional": {
|
||||
"source_video": ("VIDEO",),
|
||||
"source_audio": ("AUDIO",),
|
||||
"video_mask": ("MASK",),
|
||||
"audio_mask": ("FLOAT", {"default": -1.0, "min": -1.0, "max": 1.0, "step": 0.01}),
|
||||
},
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("LATENT_MASK_EDITING",)
|
||||
RETURN_NAMES = ("latent_edit",)
|
||||
FUNCTION = "get_latent_edit"
|
||||
CATEGORY = "vLLM-Omni"
|
||||
|
||||
def get_latent_edit(
|
||||
self,
|
||||
source_video: VideoInput | None = None,
|
||||
source_audio: AudioInput | None = None,
|
||||
video_mask: torch.Tensor | None = None,
|
||||
audio_mask: float = -1.0,
|
||||
**kwargs,
|
||||
):
|
||||
if kwargs:
|
||||
logger.info("Uncaught kwargs: %s", kwargs)
|
||||
edit = LatentMaskEditing()
|
||||
if source_video is not None:
|
||||
edit["source_video"] = source_video
|
||||
if source_audio is not None:
|
||||
edit["source_audio"] = source_audio
|
||||
if video_mask is not None:
|
||||
edit["video_mask"] = video_mask
|
||||
if audio_mask >= 0.0:
|
||||
edit["audio_mask"] = audio_mask
|
||||
return (edit,)
|
||||
|
||||
|
||||
class VLLMOmniMiniMaxH3TemporalMask:
|
||||
@classmethod
|
||||
def INPUT_TYPES(cls):
|
||||
return {
|
||||
"required": {
|
||||
"images": ("IMAGE",),
|
||||
"source_fps": ("FLOAT", {"default": 24.0, "min": 0.01}),
|
||||
"duration": ("FLOAT", {"default": 5.0, "min": 0.01, "max": 15.0}),
|
||||
"mode": (["continuation", "extension"],),
|
||||
"preserve_fraction": ("FLOAT", {"default": 0.5, "min": 0.0, "max": 1.0}),
|
||||
}
|
||||
}
|
||||
|
||||
RETURN_TYPES = ("MASK", "FLOAT", "IMAGE", "MASK")
|
||||
RETURN_NAMES = ("mask", "preview_fps", "preview_images", "preview_mask")
|
||||
FUNCTION = "build"
|
||||
CATEGORY = "vLLM-Omni"
|
||||
|
||||
def build(self, images, source_fps, duration, mode, preserve_fraction=0.5):
|
||||
if images.shape[0] <= 0 or not math.isfinite(source_fps) or source_fps <= 0:
|
||||
raise ValueError("Source must contain frames and have a positive finite FPS.")
|
||||
if not math.isfinite(duration) or duration <= 0 or not 0 <= preserve_fraction <= 1:
|
||||
raise ValueError("Invalid duration or preserve_fraction.")
|
||||
frames = _align_frame_count(max(1, round(duration * 24)))
|
||||
source_seconds = images.shape[0] / source_fps
|
||||
if mode == "extension":
|
||||
if frames / 24 <= source_seconds:
|
||||
raise ValueError("Extension output must be longer than the source; increase duration.")
|
||||
boundary = source_seconds
|
||||
elif mode == "continuation":
|
||||
boundary = min(source_seconds, frames / 24) * preserve_fraction
|
||||
else:
|
||||
raise ValueError(f"Unknown temporal mask mode: {mode}")
|
||||
available = min(frames, math.floor(boundary * 24 + 1e-8))
|
||||
# Snap the preserved prefix to whole VAE clips (5 + 17n frames) so every
|
||||
# latent it maps to is fully preserved.
|
||||
prefix = 0 if available < 5 else 5 + 17 * ((available - 5) // 17)
|
||||
# One slice per output frame; the server pools frames to the latent grid.
|
||||
mask = torch.ones(frames, 1, 1)
|
||||
mask[:prefix] = 0
|
||||
indices = torch.arange(frames, device=images.device)
|
||||
source_indices = (indices * (source_fps / 24)).floor().long().clamp(max=images.shape[0] - 1)
|
||||
preview_images = images.index_select(0, source_indices)
|
||||
return mask, 24.0, preview_images, mask
|
||||
@@ -1,6 +1,3 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM-Omni project
|
||||
|
||||
"""
|
||||
An high-level API client adapter that forwards ComfyUI inputs to vLLM-Omni's REST API,
|
||||
and transforms the API responses back to ComfyUI formats.
|
||||
@@ -20,7 +17,6 @@ from comfy_api.input import AudioInput, VideoInput
|
||||
|
||||
from .format import (
|
||||
audio_to_base64,
|
||||
audio_to_bytes,
|
||||
base64_to_audio,
|
||||
base64_to_image_tensor,
|
||||
bytes_to_audio,
|
||||
@@ -30,16 +26,9 @@ from .format import (
|
||||
video_to_base64,
|
||||
video_to_bytes,
|
||||
)
|
||||
from .latent_mask import scalar_mask_to_json, video_mask_to_json
|
||||
from .logger import get_logger, pretty_printer
|
||||
from .models import lookup_model_spec
|
||||
from .types import (
|
||||
MAX_REFERENCE_AUDIOS,
|
||||
MAX_REFERENCE_IMAGES,
|
||||
MAX_REFERENCE_VIDEOS,
|
||||
MAX_TOTAL_REFERENCES,
|
||||
AudioFormat,
|
||||
)
|
||||
from .types import AudioFormat
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -278,34 +267,14 @@ class VLLMOmniClient:
|
||||
fps: int,
|
||||
negative_prompt: str | None = None,
|
||||
frame: torch.Tensor | None = None,
|
||||
first_frame: torch.Tensor | None = None,
|
||||
last_frame: torch.Tensor | None = None,
|
||||
references: dict | None = None,
|
||||
sampling_params: dict | None = None,
|
||||
model_params: dict | None = None,
|
||||
lora: dict | None = None,
|
||||
latent_edit: dict | None = None,
|
||||
spec_model: str | None = None,
|
||||
**extra_params,
|
||||
) -> VideoInput:
|
||||
"""Post a video job and return the decoded result.
|
||||
|
||||
``spec_model`` names the model whose payload spec builds the request, for
|
||||
deployments that serve a known model under a different ``model`` alias. It
|
||||
never reaches the wire; defaults to ``model``.
|
||||
"""
|
||||
if frame is not None and references is not None:
|
||||
raise ValueError("Provide only one of frame or references, not both.")
|
||||
if frame is not None and (first_frame is not None or last_frame is not None):
|
||||
raise ValueError("Provide either frame or first_frame/last_frame, not both.")
|
||||
if references is not None and (first_frame is not None or last_frame is not None):
|
||||
raise ValueError("Provide either first_frame/last_frame or references, not both.")
|
||||
|
||||
spec, matched_pattern = lookup_model_spec(spec_model or model)
|
||||
if (first_frame is not None or last_frame is not None) and (
|
||||
matched_pattern is None or "MiniMax-H3" not in matched_pattern
|
||||
):
|
||||
raise ValueError("first_frame and last_frame are supported only for MiniMax-H3; use frame for this model.")
|
||||
|
||||
# === regular payload fields ===
|
||||
form = aiohttp.FormData()
|
||||
@@ -325,62 +294,43 @@ class VLLMOmniClient:
|
||||
|
||||
# === multimodal inputs (first-last-frames, references, etc.) ===
|
||||
input_reference_image: torch.Tensor | None = None
|
||||
keyframe_images: list[tuple[str, torch.Tensor]] = []
|
||||
audio_reference: AudioInput | None = None
|
||||
reference_videos: list[VideoInput] = []
|
||||
video_task: str | None = None
|
||||
|
||||
if frame is not None:
|
||||
input_reference_image = frame
|
||||
video_task = "fl2va"
|
||||
elif first_frame is not None or last_frame is not None:
|
||||
frame_indices: list[int] = []
|
||||
if first_frame is not None:
|
||||
keyframe_images.append(("first_frame.png", first_frame))
|
||||
frame_indices.append(0)
|
||||
if last_frame is not None:
|
||||
keyframe_images.append(("last_frame.png", last_frame))
|
||||
frame_indices.append(-1)
|
||||
if len(keyframe_images) == 1:
|
||||
input_reference_image = keyframe_images[0][1]
|
||||
extra_params["frame_indices"] = frame_indices
|
||||
video_task = "fl2va"
|
||||
elif references is not None:
|
||||
reference_formats = (
|
||||
("image", MAX_REFERENCE_IMAGES, "png", "image/png", image_tensor_to_png_bytes),
|
||||
("video", MAX_REFERENCE_VIDEOS, "mp4", "video/mp4", video_to_bytes),
|
||||
("audio", MAX_REFERENCE_AUDIOS, "mp3", "audio/mpeg", audio_to_bytes),
|
||||
)
|
||||
supported_inputs = {f"{kind}_{i}" for kind, limit, *_ in reference_formats for i in range(1, limit + 1)}
|
||||
connected = {name: value for name, value in references.items() if value is not None}
|
||||
unsupported = connected.keys() - supported_inputs
|
||||
if unsupported:
|
||||
raise ValueError(f"Unsupported reference input(s): {', '.join(sorted(unsupported))}.")
|
||||
if not any(name.startswith(("image_", "video_")) for name in connected):
|
||||
images = [references[k] for k in ("image_1", "image_2") if k in references and references[k] is not None]
|
||||
audios = [references[k] for k in ("audio_1", "audio_2") if k in references and references[k] is not None]
|
||||
videos = [references[k] for k in ("video_1", "video_2") if k in references and references[k] is not None]
|
||||
|
||||
if not images and not audios and not videos:
|
||||
raise ValueError("references is empty; connect at least one image/audio/video.")
|
||||
|
||||
if videos:
|
||||
if images or audios:
|
||||
raise ValueError(
|
||||
"Video references cannot be combined with image or audio references. "
|
||||
"Connect only video_1/video_2 for multi-video Ref2VA."
|
||||
)
|
||||
reference_videos = videos
|
||||
video_task = "ref2va"
|
||||
elif len(images) == 1 and len(audios) == 1:
|
||||
input_reference_image = images[0]
|
||||
audio_reference = audios[0]
|
||||
video_task = "ref2va"
|
||||
else:
|
||||
raise ValueError(
|
||||
"references requires at least one image or video; audio-only inputs are not supported."
|
||||
"Invalid references combination. Supported modes: "
|
||||
"(1) one or more videos only, or (2) exactly one image and one audio."
|
||||
)
|
||||
if len(connected) > MAX_TOTAL_REFERENCES:
|
||||
raise ValueError(
|
||||
f"references supports at most {MAX_TOTAL_REFERENCES} inputs in total "
|
||||
f"(up to {MAX_REFERENCE_IMAGES} images, {MAX_REFERENCE_VIDEOS} videos, "
|
||||
f"and {MAX_REFERENCE_AUDIOS} audios)."
|
||||
)
|
||||
for kind, limit, extension, content_type, encode in reference_formats:
|
||||
for index in range(1, limit + 1):
|
||||
name = f"{kind}_{index}"
|
||||
if name in connected:
|
||||
filename = f"{name}.{extension}"
|
||||
form.add_field(
|
||||
"input_references",
|
||||
encode(connected[name], filename),
|
||||
filename=filename,
|
||||
content_type=content_type,
|
||||
)
|
||||
video_task = "ref2va"
|
||||
else:
|
||||
video_task = "t2va"
|
||||
|
||||
if input_reference_image is not None:
|
||||
image_filename = keyframe_images[0][0] if keyframe_images else "image.png"
|
||||
image_filename = "image.png" # Required for multipart form
|
||||
form.add_field(
|
||||
"input_reference",
|
||||
image_tensor_to_png_bytes(input_reference_image, image_filename),
|
||||
@@ -388,69 +338,32 @@ class VLLMOmniClient:
|
||||
content_type="image/png",
|
||||
)
|
||||
|
||||
if len(keyframe_images) == 2:
|
||||
for image_filename, image in keyframe_images:
|
||||
form.add_field(
|
||||
"input_references",
|
||||
image_tensor_to_png_bytes(image, image_filename),
|
||||
filename=image_filename,
|
||||
content_type="image/png",
|
||||
)
|
||||
if audio_reference is not None:
|
||||
form.add_field(
|
||||
"audio_reference",
|
||||
json.dumps({"audio_url": audio_to_base64(audio_reference)}, ensure_ascii=False),
|
||||
)
|
||||
|
||||
# === latent-mask editing (MiniMax H3) ===
|
||||
if latent_edit is not None:
|
||||
source_video = latent_edit.get("source_video")
|
||||
source_audio = latent_edit.get("source_audio")
|
||||
video_mask = latent_edit.get("video_mask")
|
||||
audio_mask = latent_edit.get("audio_mask")
|
||||
|
||||
if video_mask is None and audio_mask is None:
|
||||
raise ValueError("Latent-mask editing requires at least one mask.")
|
||||
|
||||
if source_video is not None:
|
||||
form.add_field(
|
||||
"source_video",
|
||||
video_to_bytes(source_video),
|
||||
filename="source.mp4",
|
||||
content_type="video/mp4",
|
||||
)
|
||||
if source_audio is not None:
|
||||
# The filename extension selects the audio codec (mp3), so it is
|
||||
# meaningful even though aiohttp uses the explicit filename below.
|
||||
form.add_field(
|
||||
"source_audio",
|
||||
audio_to_bytes(source_audio, "source_audio.mp3"),
|
||||
filename="source_audio.mp3",
|
||||
content_type="audio/mpeg",
|
||||
)
|
||||
if video_mask is not None:
|
||||
mask_json = video_mask_to_json(video_mask)
|
||||
form.add_field(
|
||||
"video_noise_mask",
|
||||
mask_json.encode("utf-8"),
|
||||
filename="video-mask.json",
|
||||
content_type="application/json",
|
||||
)
|
||||
if audio_mask is not None:
|
||||
form.add_field(
|
||||
"audio_noise_mask",
|
||||
scalar_mask_to_json(audio_mask).encode("utf-8"),
|
||||
filename="audio-mask.json",
|
||||
content_type="application/json",
|
||||
)
|
||||
for idx, video in enumerate(reference_videos, start=1):
|
||||
video_filename = f"reference_{idx}.mp4"
|
||||
form.add_field(
|
||||
"input_references",
|
||||
video_to_bytes(video, video_filename),
|
||||
filename=video_filename,
|
||||
content_type="video/mp4",
|
||||
)
|
||||
|
||||
# === model specific params. Either use a specialized builder, or add flattened fields as-is ===
|
||||
if model_params is not None:
|
||||
model_params = dict(model_params)
|
||||
model_params.pop("type", None)
|
||||
|
||||
spec, _ = lookup_model_spec(model)
|
||||
params_builder = spec.get("params_builder") if spec else None
|
||||
if params_builder is not None:
|
||||
form_fields = params_builder(
|
||||
model_params or {},
|
||||
extra_params={**extra_params, "task": video_task},
|
||||
width=width,
|
||||
height=height,
|
||||
)
|
||||
for k, v in form_fields.items():
|
||||
form.add_field(k, v if isinstance(v, str) else str(v))
|
||||
|
||||
Loaded 100 of 3022 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user